Summary of this series
(Post 11 in an 11-part series)
I would summarize this series on Willow Creek’s Reveal study with four points.
1) Well done. Collecting survey data is a powerful but virtually unused tool in the world of Christian churches. By conducting Reveal, Willow Creek has modeled to other churches the usefulness of church surveys, and given Willow’s influence in American Christendom, I would hope that this message takes hold.
2) Weaknesses. As with any survey, Reveal has its weaknesses. The cross-sectional design of its sample (i.e., a one-time snapshot) limits the conclusions that can be drawn from the data. Also, the authors use of “maximizing predictability,” a technique apparently popular in brand marketing studies, doesn’t fit well with studies of human behavior. Reveal constitutes a good pilot study that should prepare the way for more definitive studies in the future.
3) Overinterpretation of the data. The conclusion draw by the study’s authors, and loudly echoed by critics of Willow Creek, is that the Willow model is flawed. The data presented here are sufficiently ambiguous to make such strong claims. Given the weaknesses of the study design and analytic strategy, it’s possible that the results indicate strong support for the Willow Creek model.
4) Future studies. Simply repeating the Reveal study with hundreds more churches potentially adds very little knowledge. Much better would be a smaller, longitudinal study of, say, a thousand respondents. If many churches are studied, measure characteristics of the churches as a whole as well as of individuals.
I hope that this series has been helpful to the authors and readers of Reveal. In academics, critical attention is a form of flattery indicating that the discussed work —it indicates that one thinks a work is worth consideration. I believe that Reveal will be looked back upon as an important step in the American Christian Church discovering the value of empirical data.
To start the series
(Post 11 in an 11-part series)
I would summarize this series on Willow Creek’s Reveal study with four points.
1) Well done. Collecting survey data is a powerful but virtually unused tool in the world of Christian churches. By conducting Reveal, Willow Creek has modeled to other churches the usefulness of church surveys, and given Willow’s influence in American Christendom, I would hope that this message takes hold.
2) Weaknesses. As with any survey, Reveal has its weaknesses. The cross-sectional design of its sample (i.e., a one-time snapshot) limits the conclusions that can be drawn from the data. Also, the authors use of “maximizing predictability,” a technique apparently popular in brand marketing studies, doesn’t fit well with studies of human behavior. Reveal constitutes a good pilot study that should prepare the way for more definitive studies in the future.
3) Overinterpretation of the data. The conclusion draw by the study’s authors, and loudly echoed by critics of Willow Creek, is that the Willow model is flawed. The data presented here are sufficiently ambiguous to make such strong claims. Given the weaknesses of the study design and analytic strategy, it’s possible that the results indicate strong support for the Willow Creek model.
4) Future studies. Simply repeating the Reveal study with hundreds more churches potentially adds very little knowledge. Much better would be a smaller, longitudinal study of, say, a thousand respondents. If many churches are studied, measure characteristics of the churches as a whole as well as of individuals.
I hope that this series has been helpful to the authors and readers of Reveal. In academics, critical attention is a form of flattery indicating that the discussed work —it indicates that one thinks a work is worth consideration. I believe that Reveal will be looked back upon as an important step in the American Christian Church discovering the value of empirical data.
To start the series
17 comments:
This has been a great series, Brad. I have enjoyed watching your analysis unfold post by post. Thanks for sharing your expertise on this for a wider audience.
This has been a great series. Thanks Brad for doing this. Hopefully, the folks at Willow will look at your analysis and implement some of your ideas for future study. They certainly have the resources to do so!
I'm not sure I agree with you that doing the same study with 500 other churches won't help much. You posted before that we really have no idea if Willow's numbers 25% or so "stalled" are good or bad. Won't doing this study across 500 churches give some context for comparing Willow's stalled to the other 500 churches?
Then again I'm not a sociologist, and don't know much about sociology in general. But hey, I'm beginning to learn a bit, thanks to you!
Excellent series! Thanks so much.
Hey Brad
A couple of thoughts to throw out at you:
(1)(Agreeing with Vince's comment) It seems to me that one of the benefits of the 500 church phase will be to provide comparison data for Willow. My understanding is that they will not be reading the results as 50,000 people but as 500 churches. They have purposely chosen among several categories of churches (seeker, purpose-driven, emerging, mainline, etc) and will be looking for trends across type. This seems to me to be a helpful next step. I also agree with you that a longitudinal study would also be very helpful as a next step after that.
(2) One of the things I have always respected and appreciated about Willow is that they aren't interested in being better than any other church--they are interested in being the most effective church they can possibly be... and then helping other churches be as effective as they can possibly be. From this perspective, the comparison data will only be helpful to them in terms of identifying best practices, etc. If it urns out that a rate of 25% stalled Christians is 10x better than any other church, my guess is that Hybels, et al, will say "so what, we can still do better... and then help other churches do better too!"
(3) I hope that some folks from Willow will read your posts because I think it will be helpful in their next stage. Again, one of the things I really love about Willow, is their generosity to other churches and their near-obsession with being a learning and growing organization.
This has been a great series... thanks!
BD
One last thought Brad (at least for now, lol):
One of the reasons I am excited to have St. Paul's participate in the next round of the survey is so that we can have some comparison data too.
It will be interesting to see the way we line up against other churches--especially controlling for regional differences, age of church, style, etc.
If it were just about our own folk, I'm sure that we could develop a better survey and I hope we keep doing the survey we have developed. But I think having the ability to compare will also be very helpful.
BD
You're welcome Ray and Michael... At times I was wondering if I was overinvesting, so your comments are helpful.
Vince & Ben, yeah, you're right that it would be interesting to know the range of these data (e.g., how many are stalled) across churches.
It's still difficult to interpret,though, because in a good church the stalled might stay and work through it.
In a bad church, they might just leave and look for answers elsewhere.
Still, the range across churches would be interesting to know.
Thanks so much for your coments. I have bought the book and have read most of it. I have also collected your comments and complied them to be read again and think about the study. I hope that was okay. But it was very helpful to have what you wrote.
Absolutely Kent. I'm pleased and a bit surprised that anyway has read the whole series...
Thanks Brad. I like Willow Creek's ministry model but feel they have really bungled this survey. I too am thankful for them doing some quantitative research but am appalled at the questions and how they have been analyzed.
For an example of a more a more sociologically sophisticated study see the U.S. Congregations Study which surveyed 300,000 congregations in 2001.
U.S. Congregations Survey
U.S. Congregational Life Survey, 2001, Random Attenders
Or see the:
National Congregations Study
Andy Rowell
Th.D. Student
Duke Divinity School
Blog: Church Leadership Conversations
Hello Andy,
Thank you for your comment, and I'm glad that you liked the series.
I've heard of the national cong. survey, but I've not looked at it closely, so I appreciate the links.
Hi Brad. Great work. I enjoyed reading it. I hope you send your analysis to the folks at Willow and not just wait for them to find it here. You have raised many important considerations for future survey work by the Willow team. I know they will benefit from your insights.
Mark
Well done, Brad. It was nice to read a more balanced review of this important book.
Very helpful to those of us who lead churches and are constantly seeking to understand what we observe. We have done various surveys and recently a series of evenings listening to where people are at spiritually and their perceptions of the church. This was very timely for me.
Thanks heaps for making this available.
Brad, great series with solid insights and methodological critique. You've reshaped the conversation about the Willow Creek study and provided a valuable social sciences perspective. Thanks.
Chuck Warnock
http://smallchurchpastor.outreachmagazine.com
I missed this series when it was first posted, but have recently read "Reveal". I find it surprising, and I have to take it seriously since it nailed accurately the "segment" of which I am a part.
The authors' conclusions may be right on. Church leadership has for years complained that people who attend "events" and programs are hesitant to volunteer in others. That isn't surprising if our volunteer base is built on spiritual maturity that begins somewhere other than "programs".
There are other puzzles concerning what really makes Christians active or inactive in the church that may be partially answered by pursuing the investigation that "Reveal" may incite in the hearts of church leaders.
The book is worth reading, and taking seriously.
Brad, just came upon this series today via googling REVEAL. My reason: I am considering doing a survey of my church with respect to their felt connectdness. The occasion: too many inactive members, some who continue to leave our church through the "back door" (while others continue to come through the front). I am lead pastor of a congregation in MB, Canada. Was intrigued by your insights into methodology about doing such surveys. You have put me on the alert for sure. Much appreciated! Any spontaneous idea on how to quantify something like "felt connectedness", ie. finding out why people are leaving, why they don't "feel at home" in our church any longer?
Am also considering a research/survey project for my D.Min. dissertation covering our 39regional churches, also on a similar but slightly more focussed theme: connectedness of a church's membership with their pastors (taking off more on the hospitality them prevalent in New Testament times but sorely lacking in many pastors' ministries today. Any thoughts or pointed questions you might have to guide me?
Thanks again for this insightful blog.
Walter Fast
Post a Comment