Church Surveys: Willow Creek's Reveal Study, VII

Is the Willow Creek flawed?
(Post 7 in an 11-post series)

The big story to come out of Willow Creek's Reveal Study is that the seeker-sensitive ministry made famous by Willow is supposedly flawed and ineffective. This is based on two findings, the first of which I’ll post on today:

The first finding is that levels of participation in church activities didn't predict measures of spiritual growth. As shown in this graph, the relationship between church activity and spiritual growth appears modest at best.

These findings seem to have hit the leadership of Willow really hard. From the Christianity Today blog,

"Having spent thirty years creating and promoting a multi-million dollar organization driven by programs and measuring participation, and convincing other church leaders to do the same, you can see why Hybels called this research “the wake up call” of his life.
Hybels confesses:

'We made a mistake. What we should have done when people crossed the line of faith and become Christians, we should have started telling people and teaching people that they have to take responsibility to become ‘self feeders.’ We should have gotten people, taught people, how to read their bible between service, how to do the spiritual practices much more aggressively on their own.'"

Critics of Willow have seized upon this admission to term the Willow as deeply flawed.

Is the Willow Creek actually flawed? Maybe, maybe not, but we can't tell from these data for several reasons.

1) How steep a line? As per the chart printed above, both spiritual behaviors increase somewhat with church activities while spiritual attitudes don’t really. The fact that increased church activity correlates with moderate growth is a good sign for Willow. Maybe it’s not as steep a line as they expected, but it’s still going up. They found a much steeper line using a spiritual continuum to predict spiritual growth, but as I posted previously, this may not be a particularly interesting finding. It could be that this is as big effect as can be realistically expected, which leads me to the second point.

2) Compared to what? To judge the effectiveness of any program, you need to compare it to something. For example, if we give a pill to people with some disease, and 25% of the pass away, does the pill work? The answer: We don’t know, and we have to ask what would have happened if they didn’t take the pill. There are several useful comparisons for a study such as Reveal. A) What would have happened to the people if they had not attended church activities? B) What would have happened if they had attended church activities elsewhere? C) What would have happened if they had attended a different level of church activities? The Reveal study ignores the first two comparisons and uses cross-sectional data to approximate the third one. This raises another issue…

3) Contamination. People talk to each other, they learn from each other. In a religious community such as Willow Creek, what is taught to one group of people affects others as well. As such, the impact of programs on high-activity people will probably be spread to low-activity people as well. This effect on low-activity people could change their behavior to make it more similar to the high-activity people, giving the false impression that the activities don’t actually work.

In reality, the ideas being tested here operate at the church-wide level, i.e., is Willow Creek effective as a church? To fully test this question, in turn, requires church level data. For example, find 100 churches, 50 that subscribe to the Willow Creek and 50 that don’t. How are they different? How are they the same? (Ideally you would randomly assign which churches get the Willow Creek but this gets difficult).

4) Activity and maturity. Another interpretation of these data goes as follows. Maybe the individuals who have recently started attending church, especially if they are new Christians, are also the most active in its activities. With time they grow spiritually, but they also settle down into a routine involving fewer activities. This would mask the correlation between church activity and spiritual growth, even if church activities were effective.

It appears that the Reveal Study is over-interpreting the data. The big changes apparently in place at Willow as the result of Reveal may in fact not be warranted. In short, they may be fixing something that isn’t broken.

Here’s an analogy. If someone feels a lump under their arm, they would be rightly concerned about the possibility of cancer. What they should not do, however, is immediately start full-scale cancer treatment such as chemotherapy. What they should do is get a better test, such as a biopsy, to find out if the lump is cancerous and then decide what to do—the lump may be something else.

The findings of the Reveal Study perhaps suggest problems with the Willow , but they may not. They are inherently limited by the cross-sectional design and other features of the study. A more careful look using a more powerful survey design is warranted before an institution as successful as Willow makes serious changes.

Next: Overinterpreting the data?

To start the series

To read the final summary