The creation of a useful, but inaccurate, statistic

(Part 1 in a series)

As a follow-up to yesterday's post, I am exploring the history of Christian-related statistic that might best be described as useful but inaccurate. I do this not to pick on this one statistic (though it is pretty bad), but to illustrate the social processes involved in creating, selecting, and presenting information about Christians to Christians.

In a December, 2002 report, The Barna Research Group details a study they conducted in which they asked 270 non-Christian respondents about their opinions of various groups in society. Specifically, they asked “Is your impression of people in this group generally favorable, generally unfavorable, or somewhere in-between?” They asked this question about eleven different groups in society, including born-again Christians, ministers, and Evangelicals.

As shown in Figure 1, military officers commanded the most favorable impressions while prostitutes had the least. Among religious groups, ministers and born-again Christians scored high (44% and 33% favorable impressions) while Evangelicals scored somewhat low (22%). In describing these findings, Barna concluded that “Surprisingly Few Adults Outside of Christianity Have Positive Views of Christians.”

Respondents’ Impression
Social group Favorable In-Between Unfavorable Don’t Know
Military officers 56% 32% 6% 6%
Ministers 44% 40% 9% 7%
Born again Christians 32% 41% 17% 10%
Democrats 32% 47% 12% 9%
Real estate agents 30% 51% 11% 8%
Movie & TV performers 25% 54% 14% 7%
Lawyers 24% 53% 18% 5%
Republicans 23% 47% 22% 8%
Lesbians 23% 38% 30% 11%
Evangelicals 22% 33% 23% 22%
Prostitutes 5% 29% 55% 11%

In looking at these data, we can make several observations about how Barna measured and presented these statistics.

A striking finding is the different attitudes expressed toward born-again Christians and Evangelicals. Despite the two terms being virtually synonymous, the respondents reacted rather differently to each. Why would someone think well of born-again Christians but poorly of Evangelicals? My guess is that respondents were confused by the question wording. The question asks about “evangelicals” instead of “evangelical Christians,” and so I wonder if perhaps respondents misinterpreted the evangelical term to mean evangelists or some specific denomination they’ve never heard of. As shown in the table, more respondents (22%) did not know who this group was than any other group.

In presenting these data, Barna emphasizes the low favorability scores for Evangelicals. He writes that non-Christians are “dismissive” of evangelicals, and that “one reason why evangelical churches across the nation are not growing is due to the image that non-Christian adults have of evangelical individuals.” One could make the opposite case just as strongly—while only 22% had favorable impressions of Evangelicals, only 23% had unfavorable impressions. These data could be spun either way, the cup being half-empty or half-full, and Barna chooses half-empty.

The sample size is rather small for this type of survey. With only 270 respondents, there is relatively little statistical power. Later in the report, Barna acknowledges this in writing that “the maximum margin of sampling error … is ±6.2 percentage points at the 95% confidence level.” That means that the estimates given above are probably accurate within 6 percentage points. So, our best guess about the favorability score for Evangelicals is between 16% and 28%--quite a wide range. Looking at the data, we can probably assume that the true population scores for military officers and ministers are high, and for prostitutes it is low. However, the remaining groups, from born-again Christians to Evangelicals are probably statistically indistinguishable. If so, that means these data are not sufficiently “powerful” to tell us if Evangelicals are more or less favored than most these other groups.

Now, understanding issues of statistical significance is not something that the average reader should be expected to know, but Barna and associates should. As such, their distinction between statistically-indistinguishable groups seems inappropriate.

Part 2 in the series.