(Part 5 in a series)
In writing a critique of UnChristian, and let me preface it by saying that I am mildly uncomfortable doing so. Not because of the book itself, but because I get no particular joy out of pointing out the shortcomings and limitations of other peoples’ research. I certainly have plenty in my own research to worry about, so why should I bother with other peoples’ research?
A somewhat unique situation has arisen in American Christianity in which much of the data we get about the Church comes from organizations that, to my knowledge, don’t go through any peer-review process. With academic journals, in contrast, when I submit a paper for publication, they send it to several reviewers. These anonymous reviewers, fellow experts in the field, read the manuscript and suggest improvements as well as make recommendations for publication. (They always find a lot to fix with my research.) Certainly the anonymous review system has its own problems, but any academic journal article has been vetted by at least several researchers.
I don’t think that this is the case with research such as published by Barna, Reveal, Lifeway, and various other popular Christian data sources. Instead, there’s an element of “trust us” in accepting their material. I have no doubt that these organizations seek to be as honest and rigorous as possible with their data, but that doesn’t mean that they (or anyone) always get it right. This is confounded by what seems to be a general reluctance to publish the specific methods of their research. When reading material from these sources, I rarely find the survey instrument itself or basic methodological information such as response rates. On top of this, these organizations rarely share their data with others, which makes their analyses difficult to replicate.
The end result is almost a blind faith in the quality of the analyses being presented. Certainly these researchers are devout Christians dedicated to advancing the Kingdom (of that there is no doubt), and their work is often cited by experts, so shouldn’t we just accept what they say at face value? It feels like bad form to question or critique what they are doing—as if we’re hindering someone else’s ministry.
Instead of faith, I would like to promote a model of informed consumption of research about Christianity. Some of it is outstanding, some of it is off the mark, and there’s a lot in between. As readers of it, we should critically evaluate it. Rather than having faith in all data about Christianity, we should first separate the wheat from the chaff.
Ultimately, a more critical engagement of Christian statistics should identify those findings from which the church has the most to learn, that can best guide us in advancing the Kingdom.
Next: Compared to whom?