Psychology

Rejoinder

Should I Pack My Umbrella? Clinical Versus Statistical Prediction

of Mental Health Decisions

Stefanía Ægisdóttir Paul M. Spengler Michael J. White

Ball State University

In this rejoinder, the authors respond to the insightful commentary of Strohmer and Arm, Chwalisz, and Hilton, Harris, and Rice about the meta-analysis on statistical versus clinical prediction techniques for mental health judgments. The authors address issues including the availability of statistical prediction techniques for real-life psychology applications, the development of these prediction techniques for future applications, and the training of counseling and other psychologists in using statistical prediction techniques. Many of these issues are couched in the historical debate about clinical versus statistical prediction.

We were pleased to read the reactions to our article by such eminent scientist practitioners as Strohmer and Arm (2006 [this issue]), Chwalisz (2006 [this issue]), and Hilton, Harris, and Rice (2006 [this issue]). These scholars provide an important appraisal of our work, pose questions, sug- gest and extend implications for counseling psychology and mental health practice and training, and offer guidance for future research. The reactants provide similar yet slightly different vantage points on the utility (and status) of statistical prediction for clinical decision making. We agree with the majority of their points. In this regard, we recognize that our rejoin- der is confounded by a shared bias that statistical prediction techniques have an important (yet underdeveloped) place in mental health decision making.

It may be relevant to note here that we perceive the majority of the issues the reactants raise to be similar to those that have been debated ever since Paul Meehl’s (1954) book Clinical Versus Statistical Prediction. When we decided more than 10 years ago to conduct this meta-analytic “labor of love,” our hope was that the field of psychology would be propelled forward in its scientific evaluation of the utility of statistical prediction techniques in clinical practice. We believe that the reactants’ input and commentary significantly add to achieving this goal.

THE COUNSELING PSYCHOLOGIST, Vol. 34 No. 3, May 2006 410-419 DOI: 10.1177/0011000006286696 © 2006 by the Society of Counseling Psychology

410

•

TCP286696.qxd 3/29/2006 1:06 PM Page 410

Ægisdóttir et al. / UMBRELLAS AND STATISTICAL PREDICTION 411

UMBRELLAS AND STATISTICAL PREDICTION

Before we address the reactants’ issues, consider the act of packing an umbrella. It is a strategic, yet almost effortless, act. All that is required to decide to bring one’s umbrella is to check the forecasted probability of rain. It is such a commonplace act that it is easy to forget that the forecast, which prompted taking the umbrella, is based on a sophisticated statistical math- ematical model that considers extensive meteorological data. Although some might consider it nonsensical to compare the complexities of weather forecasting to the even more complicated decisions that counseling psy- chology practitioners face, we argue that it has particular relevance. Namely, if counseling (and other) psychologists will, without question, use statisti- cal prediction techniques to predict the weather, why would they not also readily use statistical techniques for the significant enterprise of predicting human behavior?

SELECTION CRITERIA FOR STUDIES INCLUDED IN THE META-ANALYSIS

Chwalisz (2006) admonished us for our “decision to include studies with homosexuality as the target of prediction” (p. 392). We respond that we fully agree with her comment about the inappropriateness of “diagnosing” or pre- dicting a person’s sexual identity. Chwalisz contends that the professional prediction of homosexuality reflects an archaic and discriminatory practice. Her belief that including extant research that predicted homosexuality (e.g., Lindsey, 1965) legitimates such predictions; however, it leaves us in a quandary, and we would like to address this comment.

Meta-analyses, by nature, examine what other researchers have chosen to examine. Although we had rigorous criteria about how selected research was conducted, we did not impose a second layer of criteria regarding the current legitimacy of the predicted outcomes. Studies were included, for example, that examined outdated prediction tasks (e.g., the Minnesota Multiphasic Personality Inventory vs. the Minnesota Multiphasic Personality Inventory–2) and those that relied on obsolete criterion (e.g., older ver- sions of Diagnostic and Statistical Manual of Mental Disorders) to evalu- ate accuracy. Our purpose was to statistically synthesize and report what has been done in terms of comparative research on clinical versus statistical without making judgment about the content’s appropriateness. We by no means intend to legitimize discriminatory behavior, just report on the results of extant research.

TCP286696.qxd 3/29/2006 1:06 PM Page 411

412 THE COUNSELING PSYCHOLOGIST / May 2006

THE ROLE OF STATISTICAL PREDICTION IN CLINICAL PREDICTION

We found that the reactants view the role of statistical prediction in relation to clinical judgment in similar yet different ways. Chwalisz (2006) portrays statistical prediction as a useful “adjunct” to clinical decision making. Strohmer and Arm (2006) argue for a mixed clinical-statistical model where clinical prediction is used at the stage of discovery and statistical prediction is used at the stage of hypothesis testing or justification. By contrast, Hilton et al. (2006) argue for the primary utility of statistical prediction versus clinical prediction and caution clinicians from watering down statistical prediction with clinical techniques. They note, for example, that “the evi- dence suggests that combining assessments does not increase the accuracy of prediction and can reduce it” (p. 403). This comment is interesting and may have relevance to the other two perspectives.

We believe that Hilton et al. (2006) capture the essence of the place for actuarial prediction when such formulas exist (more on this below) in the following statement: “Because some of the best indicators require clinical skill to measure, accurately appraising violence risk is likely to remain a task for the clinician, but the place for the clinical judgment is within rather than outside actuarial tools [italics added]” (p. 402). We take this comment to mean that the clinician remains an essential part of decision making—that is, forming judgments that can be quantified and imbedded in statistical for- mulas when such formulas are available. After acknowledging that some of the best predictors are counterintuitive and nonclinical (e.g., age) when pre- dicting such serious criteria as violence, Hilton et al. aptly note that the clinician’s role is one of information gatherer and, in cases where clinical diagnosis is a relevant predictor, a data synthesizer.

Given the fact, however, that a judgment must be first made about when to test a specific hypothesis (e.g., future violence), there may be a need for something more than a “pure” statistical model, which Hilton et al. (2006) advocate. That is, the trained clinician must recognize the need to generate a hypothesis to test (e.g., risk of future violence), and then the clinician must make decisions about the methods used to conduct such a test (e.g., use a proven statistical formula). In this regard, Strohmer and Arm’s (2006) process model for clinical decision making intrigued us. What we read was that they embedded a pure statistical model within a model for clinical deci- sion making. That is, Hilton et al.’s (2006) model for statistical prediction would be embedded within a process where statistical formulas might be used at the hypothesis-testing stage (see also Spengler, Strohmer, Dixon, & Shivy, 1995). We believe that Strohmer and Arm have provided a means by which clinicians may more intuitively and nondefensively perceive an appropriate place for statistical prediction in their practice activities.

TCP286696.qxd 3/29/2006 1:06 PM Page 412

Ægisdóttir et al. / UMBRELLAS AND STATISTICAL PREDICTION 413

Strohmer and Arm’s (2006) critique about our description of clinical judgment as much more than an intuitive process is also well taken. In fact, this is the same argument Holt (1970) made. Although we agree, the defin- ition we used is more traditional when comparing and contrasting clinical (intuitive) versus statistical (rational or objective) methods of prediction. Perhaps herein lies an age-old problem: that the two techniques are pre- sented as opposites of a bipolar continuum when, in fact, they can be com- plimentary and additive to one another. In this respect, we agree with all three of the reactions that statistical prediction should be used. We contend that statistical prediction models should be used as much as possible and in concert with clinical decision making.

AVAILABILITY OF STATISTICAL PREDICTION MODELS IN CLINICAL PRACTICE

Strohmer and Arm (2006) point out that there are not many examples of statistical prediction techniques for counseling psychologists in the 5 years of Journal of Counseling Psychology articles they searched. Specific statis- tical prediction rules, in fact, have infrequently been described and recom- mended for mental health decisions (Garb, 2000). One example involves forensic decisions where statistical tools are common. Indeed, the past decade has seen the development of a great number of statistical and semi- statistical tools to aid forensic decisions (see Hilton et al., 2006). Several reasons may explain the greater advancement of statistical prediction aids for forensic decisions compared with other mental health decisions. One is that false-negative forensic predictions are often too costly to be ignored (e.g., future violence against others). Despite being less than perfect, behavioral criteria such as arrest records offer a reasonably valid indication of criminal recidivism and of the prediction’s accuracy. Compare this with the difficulty in predicting whether a premarital couple will have a satisfy- ing marriage. Once a person enters the criminal justice system, follow-up or longitudinal data about him or her are more accessible, less costly, and more routine than in the typical clinical setting. The presence of such foren- sic data facilitates the development and availability of accurate prediction models. We believe that more programs like the one Hilton et al. outline will add significantly to counseling practice.

GENERALIZABILITY TO REAL-LIFE SETTINGS

Chwalisz (2006) and Strohmer and Arm (2006) raise important questions about the generalizability of the results, given the small overall advantage of statistical over clinical prediction. Chwalisz questions if statistical models

TCP286696.qxd 3/29/2006 1:06 PM Page 413

414 THE COUNSELING PSYCHOLOGIST / May 2006

perform in the same way outside the research context and how one would construct studies to address this issue. Strohmer and Arm question if clini- cians in the studies might have been limited by not having the opportunity to use a model-building process. Both concerns are legitimate. In fact, similar concerns have been posed before. Holt (1958, 1970), for instance, criticized earlier findings about the advantage of statistical over clinical prediction on the bases of unfair comparison between the two methods. Holt contended that in many studies, conditions for the clinicians were artificial and not representative of real-life clinical situations (e.g., lack of relevant clinical information). He further criticized such comparisons because of the lack of representativeness of the prediction tasks that were studied. These issues were only partially addressed in our meta-analysis and should be addressed by future researchers. We analyzed the differential effect size for various prediction tasks and looked at the amount of information provided to the clinicians versus the formulas (cf. Ægisdóttir et al., 2006, pp. 361-362 Table 3). We found that the differential accuracy of clinical and statistical prediction varied by prediction tasks. The amount of information provided to the clinicians (i.e., no restrictions placed on them), however, did not influence the differential accuracy of statistical and clinical prediction. Thus, we agree with Chwalisz and Strohmer and Arm that despite years of research, we need a more thorough scrutiny of clinical versus statistical pre- diction for real-life clinical situations where the clinicians are free to par- ticipate in client process model building.

DEVELOPING STATISTICAL PREDICTION FORMULAS FOR CLINICAL PRACTICE

In addition to the reactants’ ideas, several other criteria could benefit from developing statistical prediction models. We discuss these in a framework we believe useful for clinical practice and for future research by identifying three judgment categories. First, as Meehl (1954) suggested, statistical formulas could aid many routine clinical decisions. Examples are (a) decisions made in acute psychiatry about whether a client should be placed in a locked or an unlocked floor where the criterion is dangerousness to self or others, (b) short-term or long-term therapy where the criterion might be based on rate of improvement (e.g., see Lambert, Hansen, & Finch, 2001), or (c) graduate- school admissions (yes or no) where the criteria could be indexes of graduate- school success (e.g., grade point average, length of completion). The second judgment category with promise for statistical prediction formulas might include those with “high-stakes” outcomes and where accuracy is at a pre- mium, such as (a) custody evaluations where the criteria might be follow-up measures of child functioning and adaptation, (b) suicide risk where the

TCP286696.qxd 3/29/2006 1:06 PM Page 414

criteria might be suicide attempts and suicide completion, and (c) child-abuse risk where substantiated abuse would be the criterion. Finally, judgments that could be studied more are those where counseling psychologists strive to help clients make optimal choices, such as (a) vocational choice where job satisfaction is the criterion, (b) marital and relationship matching where relationship satisfaction is the criterion, and (c) controlled drinking versus abstinence where optimal interpersonal and intrapersonal function- ing is the criterion.

POTENTIAL PITFALLS USING STATISTICAL PREDICTION RULES IN COUNSELING

All three reactants discuss how statistical prediction models could be used in counseling practice, were they available. They also express caution about this use. Chwalisz (2006) embraces the idea of using statistical pre- diction models in evidence-based practice. She warns, however, that statis- tical procedures may miss important data uncovered by the observant clinician. To emphasize her point, Chwalisz cites an example of a World War II veteran presenting with atypical symptoms of delirium and dementia. On further assessment, it was discovered that the client had lead shrapnel embedded in his shoulder since serving in the war. Following the shrapnel’s removal, his symptoms disappeared. Chwalisz correctly concludes that a statistical prediction model would have misclassified this client. Meehl (1954) discussed an example just like this one that has since been referred to as the “broken-leg problem” (e.g., Bishop & Trout, 2002). It goes like this: A statistical prediction model accurately predicts movie attendance. Whereas this model would accurately predict most attendance at movies, it would not do so for Person X, who has a broken leg. By not relying on this idiosyncratic information, the formula would make an inaccurate predic- tion. In contrast, a clinician would take advantage of this information and arrive at the correct decision because he or she can modify the prediction by observing the broken leg. Broken-leg problems will inevitably arise in clinical practice, yet it has been argued that clinicians find more broken-leg problems than there really are (Bishop & Trout, 2002)!

Similar to Chwalisz (2006), Strohmer and Arm (2006) ask how the aver- age clinician would incorporate statistical prediction methods into his or her clinical work. They recommend that clinicians rely on outcomes from statistical prediction models in addition to other data in making decisions about their clients. Drawing from this array of data, clinicians should develop, test, and maintain a tentative working client model. Strohmer and Arm’s recommendations make sense. Counseling psychologists and other mental health practitioners should use disconfirmatory and confirmatory hypotheses

Ægisdóttir et al. / UMBRELLAS AND STATISTICAL PREDICTION 415

TCP286696.qxd 3/29/2006 1:06 PM Page 415

testing strategies and engage in a client model-building process (cf. Spengler et al., 1995). One cannot help wonder, though, on the basis of empirical evidence to date, if this suggestion may also be akin to the broken- leg problem. For instance, in the five studies reviewed in our meta-analysis where clinicians were provided access to the statistical formulas and were allowed to override them, the clinicians did worse than the statistical formula (d+ = –.14). They would have done better had they relied on only the formula. Perhaps clinicians have a tendency to add greater weight to data that appear relevant to the criterion outcome (e.g., clinical impression, theoretical notions, idiosyncratic information) than to information without obvious relevance but with an empirical relationship to the outcome. Clinicians may need more training on how to “trust” what appears to be counterintuitive use of predic- tors when testing clinical hypotheses.

Hilton et al. (2006), drawing from work done on violence risk assess- ment, also discuss the use of statistical prediction rules. Their position, as it applies to forensic decisions, could be in contrast to Strohmer and Arm’s (2006) and Chwalisz’s (2006) points of view. Whereas Strohmer and Arm and Chwalisz seem to lean toward merging clinical and statistical prediction methods, Hilton et al. advocate relying solely on statistical prediction mod- els, at least for violence risk assessment. According to Hilton et al., forensic clinicians can rely on three ways to render judgment about violence risk: sta- tistical prediction models (e.g., Violence Risk Appraisal Guide [VRAG]; Quinsey et al., 1998), unaided clinical judgment, and structured professional judgment (SPJ) schemes. SPJ schemes represent a merger between statistical and clinical methods (cf. Hilton et al., 2006). After reviewing the relevant literature for the past 10 years, they conclude that despite the availability of statistical prediction rules to aid violence risk assessment (that in many cases provide the most accurate prediction), many forensic clinicians preferred the moderate position (e.g., SPJ). In this method, the forensic clinician has the freedom of adjusting his or her judgment on the basis of idiosyncratic data as well as relying on predictors found to be empirically related to the outcome. Hilton et al. conclude that SPJ schemes were merely unaided clinical pre- diction dressed in new clothing and recommend using statistical models to render more accurate prediction.

MORE ON TRAINING

In our meta-analytic report, we offer suggestions for training counseling psychologists in using statistical prediction. In addition to our suggestions, Chwalisz (2006) and Strohmer and Arm (2006) emphasize the importance of the scientist-practitioner training model as well as training in effectiveness

416 THE COUNSELING PSYCHOLOGIST / May 2006

TCP286696.qxd 3/29/2006 1:06 PM Page 416

and outcome research while recognizing the challenges involved in each. They discuss the apparent lack of transfer from past research to practice (e.g., Strohmer & Arm, 2006) and perhaps the mistaken idea many trainees have about counseling psychology as art rather than as science (see also Hilton et al., 2006). We agree with these concerns. In fact, these concerns may signal a need for a new emphasis in counseling psychology training.

Despite many scholars’ hesitation about whether research on clinical versus statistical prediction (e.g., Ægisdóttir et al., 2006; Dawes, Faust, & Meehl, 1989; Grove, Zald, Lebow, Snitz, & Nelson, 2000; Meehl, 1954) will inform psychotherapy practice (e.g., Dawes et al., 1989; Strohmer & Arm, 2006), we are optimistic that the field may be ready. Hilton et al. (2006) offer words of encouragement. They state that in forensic assess- ment, progress has been made in transferring knowledge from actuarial research to training professionals. Following the development of statistical assessment models—VRAG (Quinsey et al., 2006) and the Ontario Domestic Assault Risk Assessment (Hilton et al., 2004)—hundreds of professionals have been trained in their use.

CONCLUSION

Predicting the weather sometimes has relevance for safety. For most routine purposes, however, we look to weather reports for less dramatic reasons—whether we should take an umbrella. With all the effort placed on forming mathematical models of the weather, why would counseling and other psychologists resist their use in predicting human behavior? Statistical prediction is routinely used in other aspects of our society, some of them purely for entertainment purposes. For example, when watching baseball, it is not uncommon to hear the commentator refer to a detailed analysis of statistics used by managers to make decisions about pitchers, batting order, and so on. More serious decisions are routinely based on statistical formulas, such as insurance companies’ rates, medical decisions, NASA’s decision to launch a spacecraft, and stock-market investments. Failure to rely on these mathematical models would be perceived as irresponsible and cavalier. We must ask, Why should it be any different when predicting human behavior?

In our optimism that statistical prediction models will be further devel- oped and used in counseling practice, we cannot help but acknowledge lessons learned from the field of violence risk assessment. As previously noted, the field has taken advantage of the evidence accumulated during the past 60 to 70 years on the relative accuracy of statistical over clinical prediction. In this respect, we cite Hilton et al. (2006), who state,

Ægisdóttir et al. / UMBRELLAS AND STATISTICAL PREDICTION 417

TCP286696.qxd 3/29/2006 1:06 PM Page 417

Ægisdóttir et al. (2006) have performed an extremely valuable service in examining and communicating the size, nature, and limits of the superiority of actuarial methods, but recent experience in the field of violence risk assessment suggests that their work will be insufficient to produce measur- able improvement in practice. Our position has been described as extreme . . . yet the apparent middle ground has, in our opinion of the empirical evidence, proved to be a stronghold for unaided clinical judgment. (p. 406)

Our meta-analysis is hardly the definitive or last analysis of this topic. But it, like most meta-analyses, allows one to take a broad view of individ- ual findings, which have been accumulating for some time now (e.g., Dawes et al., 1989; Grove & Meehl, 1996; Grove et al., 2000; Meehl, 1954, 1986). These findings argue that statistical decision making is almost always as good as clinical decision making and usually better. What we need now are strategies to apply these conclusions in counseling and clini- cal practice. We urge counseling psychologists to take up the challenge our reactants pose in their informed and considered comments to our study so that at a future time, some decisions relevant to counseling psychology will be as easy as deciding whether to pack an umbrella.

REFERENCES

Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., et al. (2006). The meta-analysis of clinical judgment project: Fifty-six years of accumu- lated research on clinical versus statistical prediction. The Counseling Psychologist, 34, 341-382.

Bishop, M. A., & Trout, J. D. (2002). 50 years of successful predictive modeling should be enough: Lessons for philosophy of science. Philosophy of Science, 69, S197-S208.

Chwalisz, K. (2006). Statistical versus clinical prediction: From assessment to psychotherapy process and outcome. The Counseling Psychologist, 34, 391-399.

Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.

Garb, H. N. (2000). Computers will become increasingly important for psychological assess- ment: Not that there’s anything wrong with that! Psychological Assessment, 12, 31-39.

Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impres- sionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2, 293-323.

Grove, W. M., Zald, D. H., Lebow, S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19-30.

Hilton, N. Z., Harris, G. T., & Rice, M. E. (2006). Sixty-six years of research on the clinical versus actuarial prediction of violence. The Counseling Psychologist, 34, 400-409.

Hilton, N. Z., Harris, G. T., Rice, M. E., Lines, K. J., Lang, C., & Cormier, C. A. (2004). A brief actuarial assessment for the prediction of wife assault recidivism: The Ontario Domestic Assault Risk Assessment. Psychological Assessment, 16, 267-275.

418 THE COUNSELING PSYCHOLOGIST / May 2006

TCP286696.qxd 3/29/2006 1:06 PM Page 418

Holt, R. R. (1958). Clinical and statistical prediction: A reformulation and some new data. The Journal of Abnormal and Social Psychology, 56, 1-12.

Holt, R. R. (1970). Yet another look at clinical and statistical prediction: Or, is clinical psy- chology worthwhile? American Psychologist, 25, 337-349.

Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: Using patient outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69, 159-172.

Lindsey, G. R. (1965). Seer versus sign. Journal of Experimental Research in Personality, 1, 17-26.

Meehl, P. E. (1954). Clinical vs. statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis: University of Minnesota Press.

Meehl, P. E. (1986). Causes and effects of my disturbing little book. Journal of Personality Assessment, 50, 370-375.

Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (1998). Violent offenders: Appraising and managing risk. Washington, DC: American Psychological Association.

Spengler, P. M., Strohmer, D. M., Dixon, D. N., & Shivy, V. A. (1995). A scientist-practitioner model of psychological assessment: Implications for training, practice, and research. The Counseling Psychologist, 23, 506-534.

Strohmer, D. C., & Arm, J. R. (2006). The more things change, the more they stay the same: Reaction to Ægisdóttir et al. The Counseling Psychologist, 34, 383-390.

Ægisdóttir et al. / UMBRELLAS AND STATISTICAL PREDICTION 419

TCP286696.qxd 3/29/2006 1:06 PM Page 419