Tuesday, November 20, 2007

Church Surveys: Willow Creek's Reveal Study, VI

The problem of maximizing predictiveness
(Post 6 in an 11-part series)

The big message in the Reveal Study is that there is a spiritual continuum that predicts spiritual growth. As discussed in previous posts, the spiritual continuum is a four-stage scale of self-defined relationship with Christ, and spiritual growth is various behaviors and attitudes such as tithing, reading the Bible, praying, worshiping.

To make it easier to understand, I drew a picture. As shown below, the Reveal Study assumes the causal #1. The more you define yourself as close to Christ, the more you do Christian things.

The problem with this causal is that several other, equally viable ones exist. Maybe doing Christian things, like reading the Bible, make you feel closer to Christ (i.e., 2). Or, maybe one's self-defined relationship with Christ and one's spiritual practices are mutually interdependent (Model 3).

The that I find most compelling is #4. Various factors "X" affect both self-perceived relationship with Christ as well as spiritual practices. Obviously, from an applied perspective, churches should be looking for those "X"s that increase both.

It's not clear to me that separating "relationship with Christ (i.e., spiritual continuum) and spiritual growth gives us any particular insight, for I think they should be clumped together as outcome measures.

***** Warning: Somewhat technical language below. You can stop reading here and still get the main point of this post *****

The authors support the validity of their continuum -> growth by emphasizing its high predictive power. "Our research experts told us this was one of the most highly predictive s that they had seen (p. 36, emphasis theirs)." Predictive here means that levels of the "spiritual growth" systematically vary by levels of the spiritual continuum. As shown in various tables, the Reveal Study documents that people with higher scores on the continuum also score higher on measures of spiritual growth.

It's not that this statement is inaccurate, rather it's uninteresting.

If you have two measures of about the same thing, well of course they will be highly correlated and thus predictive of each other. Here are some everyday examples of this:

* What's a strong predictor of whether people smoke cigarettes today? If they smoked cigarettes yesterday.

* What's a strong predictor of running a mile fast? Being able to run two miles fast.

* What's a strong predictor of having an in-depth knowledge of the book of John? Knowing the books of Luke well.

Suppose that a church hired a consultant to help it increase the size of its congregation. The consultant came back saying that they the best predictor of church attendance in any given church is the number of cars in the parking lot during services, and so the church should focus on increasing them. My guess is that the church wouldn't necessarily find this help--it's accurate, but not very interesting.

They key in research is to examine the empirical connectedness (i.e., correlation) between variables that are somewhat different, that have an interesting and meaningful connection, if it exists. It's not clear to me that the Reveal study has done this in their emphasis on a spiritual continuum.

Next: Is the Willow model flawed?

To start the series

To read the final summary

4 comments:

Mat said...

great post Brad! To me, this is the central point of your helpful critique -- What are we measuring and are they distinct enough from each other to be helpful predictors of spiritual growth (however that is defined)? It seems to me that Reveal hinges heavily on subjective feeling of being close to Christ as a mark of maturity, leaving Mother Teresa out in the cold. That said, I'm grateful for the journey Willow is on and how they are publicly involving others to repent and learn in such ways.

Brad Wright said...

Good thoughts Mat, especially about Mother Teresa. She would not have done well on the spiritual continuum...

I also agree that Willow is doing well tackling issues with data and being honest about it.

kent said...

Thank for your extended review of the book. I have copied all onto one sheet and want to read it again after I finish the book.

You da man!

john said...

I'm enjoying your REVEAL review but I hope you meant you were looking for causation, not correlation, in analyzing multiple data arrays. Remember, European researchers found a high correlation between the frequency of storks and the level of birthrates. The missing explanatory variables were the more houses, the more chimneys, the more nesting opportunities for storks, the more storks and the more houses, the more people, the more births.