(Part 5 in a series)
Given that Follow Me is a follow-up of Reveal, it’s worth asking how it’s different. While Follow Me has the same look as Reveal, with similar graphics and layout, it has a somewhat different feel. Reveal was parsimonious in presentation and told a memorable story. Follow Me, in contrast, meanders somewhat, and it feels less focused. The book presents two breakthrough discoveries (Chapter 5): Christians can do better, and reading the Bible prompts spiritual growth. Now, I’m not an expert in church matters, but it’s not clear to me that these represent breakthroughs—I’ve certainly heard them in countless sermons. Still, it’s good to be reminded of their truth.
As I understand the chronology of the Reveal ministry, they collected data several years before publishing Reveal and then they published Follow Me a year or so after Reveal—may just months after they finished collected the additional data used in Follow Me. I wonder if the book would have benefited from additional time to assimulate and make sense of the data. I found myself wondering if the publication of the book was somewhat rushed. Not only did it not have the tight focus of their previous book, but it had some duplicated sections. For example pages 47 and 77 contain similar discussions of whether spiritual growth is linear. Pages 19 and 26 present the same figure.
It’s worth thinking about how much value is added by Follow Me’s increased sample size, and as far as I understand their sampling procedure, it’s an open question as to how much the substantial increase in sample size actually increases the validity of the findings. Here’s why: The value of sample is its representativeness of a larger population, not in it size per se (at least after a certain point). For example, if we want to use a public opinion poll to predict a presidential election, we’d much rather have a national, random sample of only 100 voters than we would a sample of 10,000 voters from the same conservative or liberal region of the country.
The original Reveal sample of about 5,000 respondents and the Follow-Me sample of about 80,000 respondents are both plenty big to detect statistical differences between groups (called statistical power). Both Reveal and Follow-Me use convenience samples, meaning the researchers took easily-accessed churches rather than drawing a random sample of all churches in the country. According to Reveal (Reveal, p. 93), its seven churches were selected to be geographically and culturally diverse. Follow Me chose 200 churches out of 1,700 that applied at Willow Creek’s annual Leadership Conference. Probably the 200-church sample better represents American Christians than the 7 church sample, but we can’t know for sure with both being convenience samples. At the very least, Follow Me studies a certain kind of church—those that have leaders who attend the Leadership Conference and apply to be in surveys. Perhaps this type of church is not representative of American churches.
The real value of having 200, rather than 7, churches is not in the increased sample size but rather it allows for across-church comparisons. In describing its sample, Follow Me describes its churches as varying along a number of dimensions. They are from different regions of the country, they are of varying sizes, they come from different denominations, and they have different worship styles. Why not, then, see how these different types of churches vary in the processes described in Follow Me? For example, does the spiritual continuum work differently for Baptists versus Lutherans? Does satisfaction with the church services matter more for people in mega-churches than small churches? These types of findings would allow the folks at Willow Creek to test if their conclusions hold for different types of churches. Put differently, Reveal concluded that church programs were not “one-size-fits-all.” It’s likewise a possibility that explanations for spiritual formation do not apply equally well for all types of churches, that they too are not “one-size-fits-all.”
Next: Cross-sectional data
Given that Follow Me is a follow-up of Reveal, it’s worth asking how it’s different. While Follow Me has the same look as Reveal, with similar graphics and layout, it has a somewhat different feel. Reveal was parsimonious in presentation and told a memorable story. Follow Me, in contrast, meanders somewhat, and it feels less focused. The book presents two breakthrough discoveries (Chapter 5): Christians can do better, and reading the Bible prompts spiritual growth. Now, I’m not an expert in church matters, but it’s not clear to me that these represent breakthroughs—I’ve certainly heard them in countless sermons. Still, it’s good to be reminded of their truth.
As I understand the chronology of the Reveal ministry, they collected data several years before publishing Reveal and then they published Follow Me a year or so after Reveal—may just months after they finished collected the additional data used in Follow Me. I wonder if the book would have benefited from additional time to assimulate and make sense of the data. I found myself wondering if the publication of the book was somewhat rushed. Not only did it not have the tight focus of their previous book, but it had some duplicated sections. For example pages 47 and 77 contain similar discussions of whether spiritual growth is linear. Pages 19 and 26 present the same figure.
It’s worth thinking about how much value is added by Follow Me’s increased sample size, and as far as I understand their sampling procedure, it’s an open question as to how much the substantial increase in sample size actually increases the validity of the findings. Here’s why: The value of sample is its representativeness of a larger population, not in it size per se (at least after a certain point). For example, if we want to use a public opinion poll to predict a presidential election, we’d much rather have a national, random sample of only 100 voters than we would a sample of 10,000 voters from the same conservative or liberal region of the country.
The original Reveal sample of about 5,000 respondents and the Follow-Me sample of about 80,000 respondents are both plenty big to detect statistical differences between groups (called statistical power). Both Reveal and Follow-Me use convenience samples, meaning the researchers took easily-accessed churches rather than drawing a random sample of all churches in the country. According to Reveal (Reveal, p. 93), its seven churches were selected to be geographically and culturally diverse. Follow Me chose 200 churches out of 1,700 that applied at Willow Creek’s annual Leadership Conference. Probably the 200-church sample better represents American Christians than the 7 church sample, but we can’t know for sure with both being convenience samples. At the very least, Follow Me studies a certain kind of church—those that have leaders who attend the Leadership Conference and apply to be in surveys. Perhaps this type of church is not representative of American churches.
The real value of having 200, rather than 7, churches is not in the increased sample size but rather it allows for across-church comparisons. In describing its sample, Follow Me describes its churches as varying along a number of dimensions. They are from different regions of the country, they are of varying sizes, they come from different denominations, and they have different worship styles. Why not, then, see how these different types of churches vary in the processes described in Follow Me? For example, does the spiritual continuum work differently for Baptists versus Lutherans? Does satisfaction with the church services matter more for people in mega-churches than small churches? These types of findings would allow the folks at Willow Creek to test if their conclusions hold for different types of churches. Put differently, Reveal concluded that church programs were not “one-size-fits-all.” It’s likewise a possibility that explanations for spiritual formation do not apply equally well for all types of churches, that they too are not “one-size-fits-all.”
Next: Cross-sectional data
No comments:
Post a Comment