There are good restaurants and bad restaurants. Good cars and bad cars. Good movies and bad movies. And, when it comes to statistics about religion, there are good sources and bad sources. This is easy to miss because statistics seem so official and trustworthy. Actually, a lot of statistics about American religion—especially Christianity—are wrong and unhelpful.
Unfortunately, evaluating a social statistic thoroughly requires time and expertise. A reasonable shortcut, however, is to evaluate the source of a religion statistic—the person or organization who presents the statistic as accurate. Generally speaking, trustworthy researchers produce trustworthy statistics.
Here are four guidelines for evaluating researchers:
- Expertise and experience. Is the researcher formally trained in social statistics? Do they have a master’s degree or PhD in a social science? Think of it this way. Would you want a physician who is self-taught or one who went to medical school? While it’s possible for someone to teach themselves expert-level social science methods, I haven’t met many who have. Empirical analyses can go wrong in many, many ways, and it takes training and practice to get them right. Myself, I had seven years of graduate school followed by a two-year post-doctoral training position. Since then, I have evaluated hundreds of studies and conducted dozens of my own. And I still learn new things all the time.
- Financial incentives. Does the researcher have financial incentive to spin their findings? Some researchers use their writing, speaking engagements, and consulting to pay the bills. This gives the incentive to present gripping, exciting, and counter-intuitive findings. After all, who wants to pay for a boring story—even if it is accurate? Other researchers don’t need to sell their research. We tenured professors, for example, get steady paychecks regardless of how many people read our work. Certainly, this is a privilege (albeit one that I had to work 15 years for), and it gives us the opportunity to (hopefully) present what we think is most accurate and important. Well-funded research organizations, likewise, have less financial pressure.
- Ideological bias. Can a researcher step away from his/her ideological biases? We researchers all have our own worldviews—ideas about what is true and good and what is false and bad. It comes with being human. This matters because data don’t speak for themselves. Rather, they are interpreted. When we read someone’s research, we are getting some combination of how the world really is and how the researcher thinks the world should be. And researchers vary in their desire and ability to emphasize the former instead of the latter. Some researchers only present findings that support their worldview. They hypothesize, analyze, and, gasp, their findings somehow always fit with what they started out believing. For example, there is a professor at a well-known liberal arts college whose research consistently casts Christianity and other religions in a negative light. When I see his name on a study, I already know what its conclusion will be. In this sense, his work isn’t that informative. I am a professing Christian. Certainly, my ideology guides which questions I find interesting. However, once I arrive at a question, I try to let the data speak for themselves. Many times they have contradicted my expectations. I assume that what’s best for Christianity is being accurate. If American Christianity is doing badly at something, let’s identify it and work to fix it. If it’s doing well, let’s celebrate it. Either way, we want to know what’s really happening.
- Uses really good data. There are an abundance of data sets that measure religion. But you’d be surprised at how often researchers do not use the best data for their question. Sometimes, they use their own, weaker data out of financial incentive. Other times, they only use data that produces results that fit with their worldview.
These guidelines are not perfect predictors of trustworthiness. Some researchers meet all of them and still produce work that isn’t stellar. Others miss one or two but still do good work. Nonetheless, on average, the more of these guidelines met by a religion researcher, the more trustworthy their research.