Church Surveys: Willow Creek's Reveal Study, X

Suggestions for future Reveal studies.

(Post 10 in an 11-part series)

The Reveal Study, conducted by Willow Creek Church, used what sociologists call a cross-sectional survey to collect data at one point in time. Think of it as a snap-shot of life at Willow during the time of the survey.

A cross-sectional survey was a good choice to get the Reveal program going, for it is an inexpensive method to collect data, and it’s a good way to test measures and start exploring a phenomenon.

Cross-sectional surveys, however, have some inherent weaknesses. They don’t do a good job in describing how processes work over time—think of trying to document a child’s growth by taking a picture at only one point in time. This matters because some of Reveal’s hypotheses are longitudinal—over time—in nature. One example is the idea that as Christians should act differently as they become more mature in their faith.

Another weakness with cross-sectional studies regards causality. When we want to say that “A” causes “B”, we usually assume that “A” precedes “B” in time (short of having a time machine). As such, we would like to measure “A” at an earlier time than “B” to lessen the chance of reverse causation—“B” causing “A.” (There’s more to establishing causality, but it’s not worth going into here).

It’s in this context that we might review Reveal’s plans for the future. As described in the book, the study’s authors plan to give the current Reveal study in 500 other churches. By doing this they could easily get up to 50,000 respondents (figuring 100 per church). The value of doing this is that they can replicate the Reveal study in other contexts, other than just Willow (and the other six churches surveyed originally). This should broaden the generalizability of the findings (something sociologists call external validity).

Nonetheless, this next phase of Reveal isn’t necessarily a big step forward for two reasons. 1) It is still saddled with the limitations of cross-sectional studies, and 2) the extra sample size by itself doesn’t make much of a difference. The value of having bigger samples is that they give more certainty in the accuracy of the resulting statistics (called statistical power). However, for a study like this, a sample of several thousand is more than enough, and the additional tens of thousands of respondents won’t really make much a difference. It doesn’t make things worse, but it doesn’t much increase the validity of the data.

What then would I recommend for Reveal? Three things.

1) Consider conducting a longitudinal study. The best way to test Reveal’s ideas would be to conduct a longitudinal study by observing some people over time. For example, recruit 1,000 people at Willow Creek. Some who are there for their first Sunday, some who have been there for years, and some in between. Collect information about them, such as is already collected in Reveal. Then wait six months to a year and collect the same information about them. Maybe wait another time period and do it again. At the end, you’ll have data to test how people change over time in response to being at Willow Creek. You can test what stages people go through, and how they change with time in the church. These would be fascinating data, and I don’t know of any church that has collected them.

2) Collect church-level measures. As I discuss above, the additional tens of thousands of respondents in the next phase of Reveal really don’t buy much in terms of statistical power. What would be useful, however, would be to collect data about the 500 churches themselves. Record data such as:
- how big is the church
- what size town is it in
- how many pastors do they have
- do they have a “seeker-sensitive” approach to ministry
- what is their budget
- what percentage of their leadership are women or racial minorities
- anything other church characteristic that might affect the individuals’ experiences at a church

With sufficient church-level measures, the Reveal authors could then conduct what is called a multi-level analysis. They could examine which individual characteristics make a difference versus which church characteristics.

The classic example of multi-level analysis comes from educational studies. If a child succeeds in education, it could be due to who they are as students—smart, hard working, high goals, etc. Success could also be due to the school they go to—low teacher/student ratio, highly educated and paid teachers, fellow students who are going to college, etc. Or, it could be both the child and the school.

Multi-level education studies pick a series of schools, measuring school characteristics, and then pick individual students within those schools, measuring individual characteristics. They then analyze these multi-level data using various statistical techniques (such as “hierarchical linear modeling”).

The result: A reasonably clear statement about whether it’s the person, the organization, or both that affect individual outcomes.

3) Draw from social science. The analytic approach used in Reveal comes from business marketing studies, and this poses some problems when applied to people, for people are a more complex unit of study than most business products.

Consider studying perfume sales at a department store versus people’s behavior. A bottle of perfume won’t up and leave by itself if it doesn’t like where it is placed. People will change due to situations if they are unhappy. A bottle of perfume won’t be different if it is surrounded by other bottles of perfume. People are different depending on who’s around them. The actual contents of a perfume bottle don’t change depending on what people think of it. People do change based on other peoples’ opinions.

As described in Reveal, the driving analytic strategy in marketing/ business studies is “maximizing predictability.” This means identifying which predictor variables best predict the outcome variable (it’s also called “maximizing explained variance”). In sociology this is generally in sociology this is considered a not a good idea when it comes to people, for it capitalizes on chance, potentially amplifies measurement and sampling error, and produces less interesting results.

There are plenty of social scientists who apply the latest statistical and sampling methods in applying religious behaviors, and Willow Creek should get some of these people are board. To see the difference between a social science approach and the marketing approach of Reveal, pick up any book by Christian Smith, a sociologist at Notre Dame. He’s about the best in the business when it comes to surveys about religion, and his work produces simple, clear, believable findings.

These three suggestions would substantially increase the power and importance of the Reveal study, helping it to further achieve its laudable goals of helping the church grow.

Next: Final summary

To start the series

To read the final summary