Reply to Alston

We agree with much of Jon Alston’s critique of “Toward a Social Science of Contemporary Mormondom.” Certainly the potentially fruitful paradigms he recommends to supplement our metaphors of boomtown and underdeveloped nation are appropriate. Alston’s most useful observation, we think, is that social scientists of Mormondom have been distressingly ethnocentric in their research; only rarely have they contrasted Mormons with members of other faiths. Alston suggests that this narrow focus be supplanted by comparisons across space and time—between modern Mormons and the Mormondom of yesteryear, between Mormondom and splinter groups of Mormon origin, and between Mormons and non-Mormons.

Alston suggests that there is no harm—perhaps even benefit—in continuing to encourage impressionistic, intuitive, and pseudoscientific writing about Mormons. He is far more optimistic than we about the likelihood that sometime, somewhere there will arise devoted empiricists to assess the accumulated results of studies of unrepresentative samples and tell us which findings are valid and which are errors stemming from faulty procedures, improper research design, or uncontrolled overgeneralization.

Indeed, our careful review of the published research of the past fifteen years, along with a less exhaustive assessment of research done in the 1950s and 1960s, suggests the opposite conclusion: as long as scientists are rewarded for studies of “convenient” or “accidental” samples, most of us will not be motivated to do the more difficult and often more costly work necessary to secure defensible probability samples or to study Mormons who live beyond the environs of the universities where we and our students are located. Thus the present status of the social science of Mormondom clearly illustrates the “principle of least effort”: if our convenient samples generate findings that are treated as legitimate contributions to the field, there is less motivation to design and administer data collection programs that meet more rigorous standards.

There may also be a kind of Gresham’s Law operating: bad research spawns more bad research, and in the process the few studies based on representative samples are overbalanced by the sheer volume of “evidence” from inadequate samples or unsystematic small-scale observation.

Many times in the literature review we encountered a “fact” about Mormons duly certified by a handful of citations: on the face of it, the generalization was well-supported. However, careful checking of the alleged support—an examination of each study cited—generally revealed that the “scientific support” reflected such flawed research design that the evidence offered provided shaky support at best and, at worst, was positively misleading.

Perhaps the most telling argument against Alston’s optimistic hope that someone will come along and separate all that misleading research chaff from the valuable kernals of fact is the absence of any such winnowing process in the research literature in the past twenty-five years. As we noted in the paper, there are some hopeful signs. For a few topics there has been some highly credible, generalizable work. But the accumulation of “soft” and pseudoscientific literature continues to outpace the production of defensible empirical work. Excepting the discipline of history, most of the social science literature on Mormondom is an untrustworthy guide to the characteristics of Mormon people and the social processes that affect them.

Alston concludes that our suggestion for curtailing “exploratory” work—that we “stop surveying and start shoveling”—is as likely to leave us shoveling offal as rich ore. That may be so. If it is, it reflects the quality of three decades of often uncontrolled and sometimes irresponsible surveying. We are certain that some of the surveyors are better than others at pointing us toward pay dirt. Unfortunately, the available empirical work—the appropriate, well-directed shoveling necessary to assess a surveyor’s credentials and his “success rate” is insufficient to allow us to decide which surveyors to dismiss and which to put on long-term contract.

Therefore, the conclusion that it is time to do some serious “digging,” and thereby learn enough that we may dismiss the least efficient surveyors, remains in our view a most important task facing the social scientists of Mormondom. We suspect that the rate of progress would be much improved if, rather than continuing to encourage impressionistic journalism and the study of idiosyncratic Mormon populations, there were indeed a swing toward “rank empiricism.” Then, when a brilliant theorist or social critic does come along, he or she will have a body of solidly grounded research to build on. We believe that the chances for genuine progress in our disciplines are much enhanced if our best thinkers can work from good data bases.

There are swings and cycles in the evolution of science. For several decades the social science of Mormondom has been heavily skewed to the impressionistic-intuitive, exploratory side of things. A period of overemphasis on empiricism—on social bookkeeping and low-level theorizing along with high-level attention to methodological rigor—is long overdue.

We therefore reaffirm our call for a partial moratorium on the study of Mormon college students—to be relaxed only when defensible, as in the case of studies of college samples that can be generalized to some wider population. And we repeat our call for a period of “overemphasis” on careful enumeration and careful description of the various segments of contemporary Mormondom and of the major processes that seem to determine their characteristics. It would be well for us to have a firm fix on what, in fact, is so before we proceed to the “why” questions or to recommending changes. In any case, we need to do enough shoveling to know which surveyors are most likely to guide us to pay dirt. Multiplying exploratory studies without an appropriate testing of the accuracy and legitimacy of the surveyor/explorer seems a misguided policy at this stage.