Wednesday, November 28, 2007

Church Surveys: Willow Creek's Reveal Study, X

Suggestions for future Reveal studies.
(Post 10 in an 11-part series)

The Reveal Study, conducted by Willow Creek Church, used what sociologists call a cross-sectional survey to collect data at one point in time. Think of it as a snap-shot of life at Willow during the time of the survey.

A cross-sectional survey was a good choice to get the Reveal program going, for it is an inexpensive method to collect data, and it’s a good way to test measures and start exploring a phenomenon.

Cross-sectional surveys, however, have some inherent weaknesses. They don’t do a good job in describing how processes work over time—think of trying to document a child’s growth by taking a picture at only one point in time. This matters because some of Reveal’s hypotheses are longitudinal—over time—in nature. One example is the idea that as Christians should act differently as they become more mature in their faith.

Another weakness with cross-sectional studies regards causality. When we want to say that “A” causes “B”, we usually assume that “A” precedes “B” in time (short of having a time machine). As such, we would like to measure “A” at an earlier time than “B” to lessen the chance of reverse causation—“B” causing “A.” (There’s more to establishing causality, but it’s not worth going into here).

It’s in this context that we might review Reveal’s plans for the future. As described in the book, the study’s authors plan to give the current Reveal study in 500 other churches. By doing this they could easily get up to 50,000 respondents (figuring 100 per church). The value of doing this is that they can replicate the Reveal study in other contexts, other than just Willow (and the other six churches surveyed originally). This should broaden the generalizability of the findings (something sociologists call external validity).

Nonetheless, this next phase of Reveal isn’t necessarily a big step forward for two reasons. 1) It is still saddled with the limitations of cross-sectional studies, and 2) the extra sample size by itself doesn’t make much of a difference. The value of having bigger samples is that they give more certainty in the accuracy of the resulting statistics (called statistical power). However, for a study like this, a sample of several thousand is more than enough, and the additional tens of thousands of respondents won’t really make much a difference. It doesn’t make things worse, but it doesn’t much increase the validity of the data.

What then would I recommend for Reveal? Three things.

1) Consider conducting a longitudinal study. The best way to test Reveal’s ideas would be to conduct a longitudinal study by observing some people over time. For example, recruit 1,000 people at Willow Creek. Some who are there for their first Sunday, some who have been there for years, and some in between. Collect information about them, such as is already collected in Reveal. Then wait six months to a year and collect the same information about them. Maybe wait another time period and do it again. At the end, you’ll have data to test how people change over time in response to being at Willow Creek. You can test what stages people go through, and how they change with time in the church. These would be fascinating data, and I don’t know of any church that has collected them.

2) Collect church-level measures. As I discuss above, the additional tens of thousands of respondents in the next phase of Reveal really don’t buy much in terms of statistical power. What would be useful, however, would be to collect data about the 500 churches themselves. Record data such as:
- how big is the church
- what size town is it in
- how many pastors do they have
- do they have a “seeker-sensitive” approach to ministry
- what is their budget
- what percentage of their leadership are women or racial minorities
- anything other church characteristic that might affect the individuals’ experiences at a church

With sufficient church-level measures, the Reveal authors could then conduct what is called a multi-level analysis. They could examine which individual characteristics make a difference versus which church characteristics.

The classic example of multi-level analysis comes from educational studies. If a child succeeds in education, it could be due to who they are as students—smart, hard working, high goals, etc. Success could also be due to the school they go to—low teacher/student ratio, highly educated and paid teachers, fellow students who are going to college, etc. Or, it could be both the child and the school.

Multi-level education studies pick a series of schools, measuring school characteristics, and then pick individual students within those schools, measuring individual characteristics. They then analyze these multi-level data using various statistical techniques (such as “hierarchical linear modeling”).

The result: A reasonably clear statement about whether it’s the person, the organization, or both that affect individual outcomes.

3) Draw from social science. The analytic approach used in Reveal comes from business marketing studies, and this poses some problems when applied to people, for people are a more complex unit of study than most business products.

Consider studying perfume sales at a department store versus people’s behavior. A bottle of perfume won’t up and leave by itself if it doesn’t like where it is placed. People will change due to situations if they are unhappy. A bottle of perfume won’t be different if it is surrounded by other bottles of perfume. People are different depending on who’s around them. The actual contents of a perfume bottle don’t change depending on what people think of it. People do change based on other peoples’ opinions.

As described in Reveal, the driving analytic strategy in marketing/ business studies is “maximizing predictability.” This means identifying which predictor variables best predict the outcome variable (it’s also called “maximizing explained variance”). In sociology this is generally in sociology this is considered a not a good idea when it comes to people, for it capitalizes on chance, potentially amplifies measurement and sampling error, and produces less interesting results.

There are plenty of social scientists who apply the latest statistical and sampling methods in applying religious behaviors, and Willow Creek should get some of these people are board. To see the difference between a social science approach and the marketing approach of Reveal, pick up any book by Christian Smith, a sociologist at Notre Dame. He’s about the best in the business when it comes to surveys about religion, and his work produces simple, clear, believable findings.

These three suggestions would substantially increase the power and importance of the Reveal study, helping it to further achieve its laudable goals of helping the church grow.

9 comments:

Glen Davis said...

Thank you for this fascinating series.

Have you ever taken a look at Christian Schwarz's research on healthy churches? It goes under the title Natural Church Development. I'd be very curious to learn what you think of it. The basic idea is that he's surveyed a couple of million believers in a couple of thousand churches over a lot of years and has developed some surveys and techniques that allow him to make observations about the health of a church and predictions about the growth of that church.

I can't find any great links to it right now, but there's a summary at http://www.ntevangelism.org/NTEAssets/4-5%20How%20Did%20Natural%20Church%20Development%20Get%20Started.pdf

Brad Wright said...

Thank you Glen... this looks really interesting. I wonder why his work isn't even better known?

peter said...

As someone in the UK at a church that has just done the Natural Church Development survey, I'd be very interested to see what you think about it Brad. For our church, with a combined congregation of about 800, 100 people were surveyed (how they were chosen was the first issue - it wasn't entirely random). When the results were presented, the greatest strength the survey identified was not one I, or others I talked to at the results day, could see as a strength in the church - but as something which needed a lot of work.

As the analysis of the data was hidden it was impossible to tell if there were flaws in the method at which the results were arrived at.

I don't think our running of the survey was great, but I'd like to know if there were also statistical problems. I think it could have worked a lot better and been more helpful to the church.

Brad Wright said...

Hello Peter,

Hm-m-m-m, sounds like an interesting survey. I hadn't heard of it before you and Glen mentioned it, but I'll probably take a look at it (after the holidays!).

Brad

Benjamin said...

Natural Church Development is a great ministry/concept that has been around a fairly long time. My understanding (and I could be wrong) is that a lot of its foundations come out of the church growth movement, which comes out of the missiological work of Donald McGavran (http://en.wikipedia.org/wiki/Donald_McGavran).

Interestingly, McGavran's work was pretty influential on people like Hybels and Warren... his ideas come from research in India, primarily.

Purpose-Driven (church, not life) is in many respects a re-working of NTD stuff.

Brad, you would like McGavran's work I think. His book "Understanding Church Growth" is a classic!

One of the things I love about him is that he is a data-driven, research-based missiologist. He pretty much said, let's study what works, figure out the variables and factors, and try and reproduce it.

BD

Pastor Jeff said...

I don't understand your point about perfume bottles that can't leave. My understanding of their brand management background is that it is not about perfume bottles, but about customers that buy the perfume or don't and customers that stay loyal to a brand or not. In other words, they studied consumers (complex people) not simple bottles of perfume.

Brad Wright said...

Yes, they studied people buying perfume, but the perfume doesn't change in reaction to the people buying it.

In contrast, church members will respond differently to church leaders. There are people on both sides of the interaction.

Peter said...

Thank you for a thoughtful review.

My suggestions for future study are for the Reveal folk to look more closely as the psychology of motivation. As you say, their basis is marketing -- not the best motivational starting point.

For example, Carol Dweck (Stanford) is an education psychologist who has done a lot of work on "fixed" and "growth" mindsets. Many (most?) of those who pursue spiritual growth probably have a growth mindset and those who don't probably have a "fixed" mindset (that is, they are content with their status quo sense of self, and they respond to assessments of inadequacy (by themselves or others, ie. "I need to strengthen my spiritual practice") by dispensing with the challenge rather than "improving themselves"). The Reveal studies is based on a theological premise that all Christians are oriented to grow in faith. But the Reveal studies themselves (and psychological work like Dweck's) would suggest otherwise.

The Reveal people would be much further ahead learning how to help people with a fixed mindset develop a growth one.

This is but one example of motivation psychology that would greatly improved [1] their theoretical understanding, and [2] redirect their efforts in congregational research.

Peter C.

Brad Wright said...

Interesting ideas, Peter. I hadn't even thought about the psychological aspect of it.

Thanks,

Brad