Interpreting data is an art and a science. I spend a considerable amount of time deciding on the degree of confidence I should have in published research concerning religion. It is hard because no study is perfect. If you don't want to believe something, you can always find something wrong with the methodology. In this election year, with the heavy emphasis today on surveys, I thought it might be worth going over some general guidelines for data interpretation. The rules for deciding whether the study you are reading has reliable data are simple.
The study needs to have a good response rate to be able to generalize to the population, that is, to be sure the results apply to all members of the group being studied. That's an easy rule, but no one tells you what a good response rate is. When I was in graduate school, a 60 percent response rate was considered about the lowest acceptable rate. Today, because of increased resistance to filling our questionnaires, response rates of less than 50 percent are sometimes seen as respectable. I personally think 50 percent is the minimal acceptable rate. In all cases the author had best explain why he or she thinks those who responded are similar to those who did not respond. Otherwise you will not know if those who take the trouble to answer a questionnaire are similar to those who do not return questionnaires.
You need to make sure the population from which the sample was drawn includes the people whose opinions you want to study. If the report concerns whether your congregation should have a Saturday evening service, you need to make sure the survey was not handed out only at a Sunday service unless the Saturday service is aimed at those who regularly attend on Sunday. If it is aimed at inactive members or members not regularly attending on Sunday, the survey needs to have included all such members. You have to be careful not to totally disregard results from a study you don't think is representative of the population. In the 1970s, studies done in California showed that youth were becoming alienated from denominations. It was easy to disregard those studies because they were done in California and "everyone knows they're a little weird out there." The response should have been, "a nationwide study is needed to see if the results are the same."
The sample has to be large enough so that the error rate is small. This is usually not a problem with published results. If a study has returns from 500 persons (and if this is a good response rate), there is a 95 percent chance that the results will not vary by more than 5 percent from the reported results.
Survey questions have to be clear and understood in the same manner by everyone. This can be a major problem. What is evangelism? What is social action? There are many different activities that fall under both of those headings, and some respondents would not agree on what activities should be included under each expression. Stewardship is another example. Stewardship is more than raising money, and I suspect most members know that, but if you asked a question about stewardship with no explanation of the term, I suspect members would think about raising money.
The rules for deciding the accuracy or adequacy of the interpretation are also simple. First, comparisons with past behavior cannot be made unless data from the past are available. I happen to think that the financial burden for recent seminary graduates may be greater today than in the past, but I have not seen any studies on the subject. I have seen studies showing the burden today, but nothing on what the burden was in the past. It is possible that indebtedness may not have been as large in actual dollars, but it is also possible that the burden was still heavy.
Correlations (smoking and cancer go together) do not demonstrate cause and effect, but they do show relationships. This is a hard one--for example, the tobacco industry used the "correlation is not causation" argument for many years to attack studies linking smoking and cancer. Correlations should not be ignored (or denied) just because they do not constitute proof.
Ecological fallacy is another mistake to watch for--this occurs when a researcher makes comments about individuals based on aggregate data. For example, if we know a presbytery voted against Proposition B, we cannot say an individual pastor is opposed to Proposition B. Since the presbytery is more than pastors, we cannot even say the majority are opposed. Elders also voted. You need a lot of information before you can go from the aggregate to the individual.
In some ways, the opposite of the ecological fallacy is over-generalizing-- making judgments about the whole based on one or several individuals. I am ending with this one because I find it the easiest to fall into. This occurs, for example, when we generalize from personal experience (so it must be "true"). Some complain when the Gallup organization generalizes to all Americans based on around 1,000 responses. Our opposition may really be because we do not agree with the results. We often make generalizations based on a sample of one, ourselves.
So who can you believe? Be open to believing everyone, keep the above comments in mind, and look for confirming evidence in other studies before believing anyone.
Email the author: Keith Wulff
Research Services