(Part 5 in a series)
This post is a bit out of place, because I'm using it to expand a point made in the first post of this series. I'm addressing a statistic that, except for prostitutes, Christians have the worst reputation in American society.
I wrote here about various problems with that statistic. Here I want to elaborate on one of those points. Namely, Barna's study found few significant differences in reputation between Evangelicals and other groups. There are two ways we can think about this.
1) The first has to do with the small sample size. Suppose I told you that I surveyed 100 people and 52 of them are going to vote for Obama, 48 for McCain. Therefore, I predict that Obama will win the election. My guess is that you wouldn't find my prediction very compelling because that's a fairly small difference with such a small sample. Well, it's the same point here. Barna's data was based on 270 people, and the study emphasized distinctions of only a percentage point or few. This works out to be only several people. To show how this works, here are Barna's data translated into numbers, rather than percentages. For simplicity, I show only the first column of data.
Respondents’ Favorable Impression
151 respondents viewed military officers favorably
120 viewed ministers favorably
86 viewed born-again Christians favorably
86 viewed democrats favorably
81 viewed real estate agents favorably
68 viewed movie & TV performers favorably
65 viewed lawyers favorably
62 viewed republicans favorably
62 viewed lesbians favorably
59 viewed evangelicals favorably
14 viewed prostitutes favorably
So, the difference between Evangelicals placing next to last and placing in the middle is... 9 people. I'm know that that is worth a headline.
[In statistical terms, Barna reports a confidence interval of 6 percentage points, which suggests that most the groups are not "significantly" different from Evangelicals.]
2) Barna's presentation uses the wrong denominator. Each of the 11 groups queried about had some percentage that were not familiar with them and as such had no opinion. For example, 22% were unfamiliar with the Evangelical group (probably due to awkward question wording). It doesn't make for these people to have an opinion about Evangelicals. As such, if we want to document how many people have a favorable opinion of a group, it should be out of how many people have heard of that group. For example, 22% of respondents had a favorable opinion of Evangelicals out of the 78% who had opinions. This works out to be 31%.
Here are Barna's groups, organized by this probably-more-appropriate computation:
60% respondents viewed military officers favorably
47% viewed ministers favorably
36% viewed born-again Christians favorably
35% viewed democrats favorably
33% viewed real estate agents favorably
28% viewed evangelicals favorably
27% viewed movie & TV performers favorably
26% viewed lesbians favorably
25% viewed lawyers favorably
25% viewed republicans favorably
6% viewed prostitutes favorably
Suddenly, Evangelicals are in the middle of the pack, and it becomes much less of a story. My guess is that if Barna had presented the data this way, that few, if any, news outlets, books, or conference producers would have picked it up. Why? It wouldn't be "newsworthy."
What's my point? Frankly, this statistic by itself doesn't really matter that much, and so the fact that many people appear to have gotten it wrong isn't all that important.
What is more interesting, however, is the way this statistic illustrates how data about Evangelical Christians gets used and transformed, often by Evangelicals themselves. Even a cursory look at this study sees the resulting conclusion as problematic, and yet, because of its provocative nature, the conclusion has lived on.
Part 6 in the series
This post is a bit out of place, because I'm using it to expand a point made in the first post of this series. I'm addressing a statistic that, except for prostitutes, Christians have the worst reputation in American society.
I wrote here about various problems with that statistic. Here I want to elaborate on one of those points. Namely, Barna's study found few significant differences in reputation between Evangelicals and other groups. There are two ways we can think about this.
1) The first has to do with the small sample size. Suppose I told you that I surveyed 100 people and 52 of them are going to vote for Obama, 48 for McCain. Therefore, I predict that Obama will win the election. My guess is that you wouldn't find my prediction very compelling because that's a fairly small difference with such a small sample. Well, it's the same point here. Barna's data was based on 270 people, and the study emphasized distinctions of only a percentage point or few. This works out to be only several people. To show how this works, here are Barna's data translated into numbers, rather than percentages. For simplicity, I show only the first column of data.
Respondents’ Favorable Impression
151 respondents viewed military officers favorably
120 viewed ministers favorably
86 viewed born-again Christians favorably
86 viewed democrats favorably
81 viewed real estate agents favorably
68 viewed movie & TV performers favorably
65 viewed lawyers favorably
62 viewed republicans favorably
62 viewed lesbians favorably
59 viewed evangelicals favorably
14 viewed prostitutes favorably
So, the difference between Evangelicals placing next to last and placing in the middle is... 9 people. I'm know that that is worth a headline.
[In statistical terms, Barna reports a confidence interval of 6 percentage points, which suggests that most the groups are not "significantly" different from Evangelicals.]
2) Barna's presentation uses the wrong denominator. Each of the 11 groups queried about had some percentage that were not familiar with them and as such had no opinion. For example, 22% were unfamiliar with the Evangelical group (probably due to awkward question wording). It doesn't make for these people to have an opinion about Evangelicals. As such, if we want to document how many people have a favorable opinion of a group, it should be out of how many people have heard of that group. For example, 22% of respondents had a favorable opinion of Evangelicals out of the 78% who had opinions. This works out to be 31%.
Here are Barna's groups, organized by this probably-more-appropriate computation:
60% respondents viewed military officers favorably
47% viewed ministers favorably
36% viewed born-again Christians favorably
35% viewed democrats favorably
33% viewed real estate agents favorably
28% viewed evangelicals favorably
27% viewed movie & TV performers favorably
26% viewed lesbians favorably
25% viewed lawyers favorably
25% viewed republicans favorably
6% viewed prostitutes favorably
Suddenly, Evangelicals are in the middle of the pack, and it becomes much less of a story. My guess is that if Barna had presented the data this way, that few, if any, news outlets, books, or conference producers would have picked it up. Why? It wouldn't be "newsworthy."
What's my point? Frankly, this statistic by itself doesn't really matter that much, and so the fact that many people appear to have gotten it wrong isn't all that important.
What is more interesting, however, is the way this statistic illustrates how data about Evangelical Christians gets used and transformed, often by Evangelicals themselves. Even a cursory look at this study sees the resulting conclusion as problematic, and yet, because of its provocative nature, the conclusion has lived on.
Part 6 in the series
5 comments:
Thanks, Brad, for being willing to point out aspects of data analysis that most people don't think about, and that can possibly lead to faulty conclusions about the data. Thanks also for giving me some great examples that I can use in my Experimental Psychology class when we talk about how careful you must be when interpreting data. That is one of the main points I try to teach my students throughout the semester in that class.
John Williams
Hey John,
Let me know if you figure out how to teach that--being careful with data analysis!
It's tough to break the habit of trusting all numbers.
One thing that's important in interpreting statistics is too look for more data rather than resting your case on just one number (unless it's really all that there is). The iPOLL database maintained by the Roper Center for Public Opinion Research contains a number of similar questions. This is from a 1977 Gallup Poll:
"Now, I'd like to get your opinion about various religious faiths and denominations. You notice that the boxes on this card go from the HIGHEST POSITION OF PLUS 5--for something you have a very favorable opinion of --all the way down to the lowest position of minus 5--for something you have a very unfavorable opinion of. How far up the scale or how far down the scale would you rate the following? Evangelicals" The results were 49% positive, 9% negative, and 43% don't know.
That's the only question about "Evangelicals", but starting in the 1990s, there have been some about "Evangelical Christians". One from 1994:
"(Now I'd like your opinion of some groups and organizations.) Would you say your overall opinion of... Evangelical Christians... is very favorable, mostly favorable, mostly unfavorable, or very unfavorable? " The results were 43% favorable, 32% unfavorable, 14% can't rate, and 11% never heard of them.
Then from 2008:
"Now thinking about some specific religious groups...Is your overall opinion of...Evangelical Christians very favorable, mostly favorable, mostly unfavorable, or very unfavorable? " The results are 60% favorable, 19% unfavorable, 16% can't rate, and 5% never heard of.
What these data suggest to me is that (a) Americans have a pretty positive opinion of Evangelicals (b) and opinions have become MORE positive since the mid-1990s. The first point doesn't surprise me, but the second does--I would have guessed little or no change.
These numbers are all from samples of the general public, so they're not comparable to the Barna statistics, which referred to people they classified as "non-Christian." But I believe the original data are available, so it would be possible to look at trends among non-religious people. Some energetic young sociologist should look into that.
Hey Brad... great post. You made that abundantly clear, even for a pastor like me :-) Thanks!
David W... great comment. Thanks.
Loving this series!
Thanks David! You're a wizard at finding data. I'll actually post these data next week, because they address the issue so clearly.
Ben, glad that you liked it. Look at what you started in finding that stat.
Post a Comment