Projective

W A R H O L C R E D I T H E R E

w w w . s c i a m . c o m SCIENTIFIC AMERICAN 41

PICTURE?PICTURE? What’s Wrong with This

PSYCHOLOGISTS OFTEN USE THE FAMOUS RORSCHACH INKBLOT TEST AND RELATED

TOOLS TO ASSESS PERSONALITY AND MENTAL ILLNESS. BUT RESEARCH SHOWS

THAT INSTRUMENTS ARE FREQUENTLY INEFFECTIVE FOR THOSE PURPOSES

by Scott O. Lilienfeld, James M. Wood and Howard N. Garb

PHOTOGRAPHS BY JELLE WAGAANER

But how correct would they be? The answer is important because psychologists frequently apply such “projective” in- struments (presenting people with ambiguous images, words or objects) as components of mental assessments, and because the outcomes can profoundly affect the lives of the respondents. The tools often serve, for instance, as aids in diagnosing men- tal illness, in predicting whether convicts are likely to become violent after being paroled, in assessing the mental stability of parents engaged in custody battles, and in discerning whether children have been sexually molested.

We recently reviewed a large body of research into how well projective methods work, concentrating on three of the most extensively used and best-studied instruments. Overall our find- ings are unsettling.

Butterflies or Bison? T H E F A M O U S R O R S C H A C H inkblot test—which asks people to describe what they see in a series of 10 inkblots—is by far the most popular of the projective methods, given to hundreds of thousands, or perhaps millions, of people every year. The com- ments that follow refer to the modern, rehabilitated version, not to the original construction, introduced in the 1920s by Swiss psychiatrist Hermann Rorschach.

The initial tool came under severe attack in the 1950s and 1960s, in part because it lacked standardized procedures and a set of norms (averaged results from the general population).

Standardization is needed because seemingly trivial differences in the way an instrument is administered can affect a person’s responses. Norms provide a reference point for determining when someone’s responses fall outside an acceptable range.

In the 1970s John E. Exner, Jr., then at Long Island Uni- versity, ostensibly corrected those problems in the early Rorschach test by introducing what he called the Comprehen- sive System. This set of instructions established detailed rules for delivering the inkblot exam and for interpreting the re- sponses, and it provided norms for children and adults.

In spite of the Comprehensive System’s current popularity, the Rorschach generally falls short on two additional, and critical, cri- teria: scoring reliability and validity. A tool possessing scoring re- liability yields similar results regardless of who tabulates and in- terprets the responses. A valid technique measures what it aims to measure: its results are consistent with those produced by other trustworthy instruments or are able to predict behavior, or both.

To understand the Rorschach’s scoring reliability problems, it helps to know something about how reactions to the inkblots are interpreted. First, a psychologist rates the collected reactions on more than 100 characteristics, or variables. The evaluator records, for instance, whether the person looked at whole blots or just parts, notes whether the detected images were unusual or typical of most test takers, and indicates the aspects of the inky swirls (such as form or color) that most determined what the respondent saw.

42 SCIENTIFIC AMERICAN M A Y 2 0 0 1

What if you were asked to describe images you saw in an inkblot or to invent a story for an ambiguous illustration— say, of a middle-aged man looking away from a woman who was grabbing his arm? To comply, you would draw on your own emotions, experiences, memories and imagination. You would, in short, project yourself into the images. Once you did that, many practicing psychologists would assert, trained evaluators could mine your musings to reach conclusions about your personality traits, unconscious needs and overall mental health.

Then he or she compiles the findings into a psychological profile of the indi- vidual. As part of that interpretive process, psychologists might conclude that focusing on minor details (such as stray splotches) in the blots, instead of on whole images, signals obsessiveness in a patient and that seeing things in the white spaces within the larger blots, instead of in the inked areas, reveals a negative, contrary streak.

For the scoring of any variable to be considered highly reliable, two different assessors should be very likely to produce similar ratings when examining any giv- en person’s responses. Recent studies demonstrate, however, that strong agree- ment is achieved for only about half the characteristics examined by those who score Rorschach responses; evaluators might well come up with quite different ratings for the other variables.

Equally troubling, analyses of the Rorschach’s validity indicate that it is poorly equipped to identify most psychi- atric conditions—with the notable excep- tions of schizophrenia and other distur- bances marked by disordered thoughts, such as bipolar disorder (manic-depres- sion). Despite claims by some Rorschach proponents, the method does not consis- tently detect depression, anxiety disorders or psychopathic personality (a condition characterized by dishonesty, callousness and lack of guilt).

Moreover, although psychologists frequently administer the Rorschach to assess propensities toward violence, im- pulsiveness and criminal behavior, most research suggests it is not valid for these purposes either. Similarly, no compelling evidence supports its use for detecting sexual abuse in children.

Other problems have surfaced as well. Some evidence suggests that the Rorschach norms meant to distinguish mental health from mental illness are un-

representative of the U.S. population and mistakenly make many adults and children seem maladjusted. For in- stance, in a 1999 study of 123 adult volunteers at a California blood bank, one in six had scores supposedly in- dicative of schizophrenia.

The inkblot results may be even more misleading for minorities. Sever- al investigations have shown that scores for African-Americans, Native Americans, Native Alaskans, Hispan- ics, and Central and South Americans differ markedly from the norms. To- gether the collected research raises se- rious doubts about the use of the Rorschach in the psychotherapy office and in the courtroom.

Doubts about TAT A N O T H E R P R O J E C T I V E T O O L—the Thematic Apperception Test (TAT)— may be as problematic as the Rorschach. This method asks respon- dents to formulate a story based on ambiguous scenes in drawings on cards. Among the 31 cards available to psychologists are ones depicting a boy contemplating a violin, a distraught woman clutching an open door, and the man and woman who were men- tioned at the start of this article. One card, the epitome of ambiguity, is to- tally blank.

The TAT has been called “a clini- cian’s delight and a statistician’s night- mare,” in part because its administra- tion usually is not standardized: different clinicians present different numbers and selections of cards to re- spondents. Also, most clinicians inter- pret the stories intuitively instead of following a well-tested scoring proce- dure. Indeed, a recent survey of nearly 100 North American psychologists practicing in juvenile and family courts found that only 3 percent relied on a

“It looks like two dinosaurs with huge heads and tiny bodies. They’re moving away from each other, looking over their shoulders. The black blob in the middle reminds me of a spaceship.”

Once deemed an “x-ray of the mind,” the Rorschach inkblot test remains the most famous—and infamous—projective psychological technique. An examiner hands 10 symmetrical inkblots, one at a time in a set order, to a respondent, who says what each blot resembles. A few blots include colored shapes, but most are black and gray—like artist Andy Warhol’s rendering above (the actual blots cannot be published).

Responses to the inkblots purportedly reveal aspects of a person’s personality and mental health. Advocates believe, for instance, that references to moving animals—such as the dinosaurs mentioned above—often indicate impulsiveness, whereas allusions to a blot’s “blackness”—as in a spaceship—often indicate depression.

Swiss psychiatrist Hermann Rorschach probably got the idea of showing inkblots from a European parlor game. The test debuted in 1921 and reached high status by 1945. But a critical backlash began taking shape in the 1950s, as researchers found that psychologists often interpreted the same responses differently and that particular responses did not correlate well with specific mental illnesses or personality traits.

Today the methodological response to those weaknesses—the Comprehensive System (CS)—is used widely to score and interpret Rorschach responses. But it has been criticized on similar grounds. Moreover, several recent findings indicate that the Comprehensive System incorrectly labels many normal respondents as pathological.PH

O TO

C R

E D

IT H

E R

E

SCOTT O. LILIENFELD, JAMES M. WOOD and HOWARD N. GARB all conduct research on psychological assessment tools and recently collaborated on an extensive review of re- search into projective instruments that was published by the American Psychological So- ciety (see “More to Explore,” on page 47). Lilienfeld and Wood are associate professors in the departments of psychology at Emory University and the University of Texas at El Paso, respectively. Garb is a clinical psychologist at the Pittsburgh Veterans Administra- tion Health Care system and the University of Pittsburgh and author of the book Studying the Clinician: Judgment Research and Psychological Assessment.

TH E

A U

TH O

R S

w w w . s c i a m . c o m SCIENTIFIC AMERICAN 43

RORSCHACH TEST

Wasted Ink?

44 SCIENTIFIC AMERICAN M A Y 2 0 0 1

standardized TAT scoring system. Un- fortunately, some evidence suggests that clinicians who interpret the TAT intu- itively are likely to overdiagnose psycho- logical disturbance.

Many standardized scoring systems are available for the TAT, but some of the more popular ones display weak “test-retest” reliability: they tend to yield inconsistent scores from one picture- viewing session to the next. Their validi-

ty is frequently questionable as well; studies that find positive results are often contradicted by other investigations. For example, several scoring systems have proved unable to differentiate normal in- dividuals from those who are psychotic or depressed.

A few standardized scoring systems for the TAT do appear to do a good job of discerning certain aspects of person- ality—notably the need to achieve and a

person’s perceptions of others (a proper- ty called “object relations”). But many times individuals who display a high need to achieve do not score well on mea- sures of actual achievement, so the abili- ty of that variable to predict behavior may be limited. These scoring systems currently lack norms and so are not yet ready for use outside of research settings, but they merit further investigation.

The Thematic Apperception Test (TAT), created by Harvard University psychiatrist Henry Murray and his student Christiana Morgan in the 1930s, is among the most commonly used projective measures. Examiners present individuals with a subset (typically five to 12) of 31 cards displaying pictures of ambiguous situations, mostly featuring people. Respondents then construct a story about each picture, describing the events that are occurring, what led up to them, what the characters are thinking and feeling, and what will happen later. Many variations of the TAT are in use, such as the Children’s Apperception Test, featuring animals interacting in ambiguous situations, and the Blacky Test, featuring the adventures of a black dog and its family.

Psychologists have several ways of interpreting responses to the TAT. One promising approach—developed by Boston University psychologist Drew Westen—relies on a specific scoring system to assess people’s perceptions of others (“object relations”). According to that approach, if someone wove a story about an older woman plotting against a younger person in response to the image visible in the photograph at the right, the story would imply that the respondent tends to see malevolence in others—but only if similar themes turned up in stories told about other cards.

Surveys show, however, that most practitioners do not use systematic scoring systems to interpret TAT stories, relying instead on their intuitions. Unfortunately, research indicates that such “impressionistic” interpretations of the TAT are of doubtful validity and may make the TAT a projective exercise for both examiner and examinee.

THEMATIC APPERCEPTION TEST

Picture Imperfect

P H

O TO

C R

E D

IT H

E R

E

Faults in the Figures I N C O N T R A S T T O T H E R O R S C H A C H and the TAT, which elicit reactions to exist- ing images, a third projective approach instructs the people being evaluated to draw the pictures. A number of these in- struments, such as the frequently applied Draw-a-Person Test, have people depict a human being; others have them draw houses or trees as well. Clinicians com- monly interpret the sketches by relating specific “signs”—such as features of the body or clothing—to facets of personali- ty or to particular psychological disor- ders. They might associate large eyes with paranoia, long ties with sexual ag- gression, missing facial features with de- pression, and so on.

As is true of the other methods, the re- search on drawing instruments gives rea- son for serious concern. In some studies, raters agree well on scoring, yet in others the agreement is poor. What is worse, no strong evidence supports the validity of the sign approach to interpretation; in other words, clinicians apparently have no grounds for linking specific signs to particular personality traits or psychiatric diagnoses. Nor is there consistent evi- dence thats signs purportedly linked to child sexual abuse (such as tongues or genitalia) actually reveal a history of mo- lestation. The only positive result found repeatedly is that, as a group, people who draw human figures poorly have some- what elevated rates of psychological dis- orders. On the other hand, studies show that clinicians are likely to attribute men- tal illness to many normal individuals who lack artistic ability.

Certain proponents argue that sign approaches can be valid in the hands of seasoned experts. Yet one group of re- searchers reported that experts who ad- ministered the Draw-a-Person Test were less accurate than graduate students at distinguishing psychological normality from abnormality.

w w w . s c i a m . c o m SCIENTIFIC AMERICAN 45

Hand Test Subjects say what hands pictured in various positions might be doing. This method is used to assess aggression, anxiety and other personality traits, but it has not been well studied.

Handwriting Analysis Graphology Interpreters rely on specific “signs” in a person’s handwriting to assess personality characteristics. Though useless, the method is still used to screen prospective employees.

Lüscher Color Test People rank colored cards in order of preference to reveal personality traits. Most studies find the technique to lack merit.

Play with Anatomically Correct Dolls Research finds that sexually abused children often play with the dolls’ genitalia; however, that behavior is not diagnostic, because many nonabused children do the same thing.

Rosenzweig Picture Frustration Study After one cartoon character makes a provocative remark to another, a viewer decides how the second character should respond. This instrument, featured in the movie A Clockwork Orange, successfully predicts aggression in children.

Sentence Completion Test Test takers finish a sentence such as, “If only I could . . .” Most versions are poorly studied, but one developed by Jane Loevinger of Washington University is valid for measuring aspects of ego development, such as morality and empathy.

Szondi Test From photographs of patients with various psychiatric disorders, viewers select the ones they like most and least. This technique assumes that the selections reveal something about the choosers’ needs, but research has discredited it.

OTHER PROJECTIVE TOOLS:

What’s the Score?

Even when projective methods assess what they claim to measure, they RARELY ADD MUCH to information

that can be obtained in other, more practical ways

Psychologists have dozens of projective methods to choose from beyond the Rorschach Test, the TAT and figure drawings. As the sampling below indicates, some stand up well to the scrutiny of research, but many do not.

46 SCIENTIFIC AMERICAN M A Y 2 0 0 1

P H

O TO

C R

E D

IT H

E R

E

A few global scoring systems, which are not based on signs, might be useful. In- stead of assuming a one-to-one correspon- dence between a feature of a drawing and a personality trait, psychologists who ap- ply such methods combine many aspects of the pictures to come up with a general impression of a person’s adjustment.

In a study of 52 children, a global scoring approach helped to distinguish normal individuals from those with mood or anxiety disorders. In another

report, global interpretation of the Draw-a-Person Test correctly differenti- ated 54 normal children and adolescents from those who were aggressive or ex- tremely disobedient. The global ap- proach may work better than the sign approach because the act of aggregating information can cancel out “noise” from variables that provide misleading or in- complete information.

What to Do?

OUR LITERATURE REVIEW, then, indicates that, as usually administered, the Rorschach, TAT and human figure draw- ings are useful in only very limited cir- cumstances. The same is true for many other projective techniques, some of which are described in the box on the pre- ceding page.

We have also found that even when the methods assess what they claim to measure, they tend to lack what psychol- ogists call “incremental validity”: they

Psychologists have many projective drawing instruments at their disposal, but the Draw-a-Person Test is among the most popular—especially for assessing children and adolescents. A clinician asks the child to draw someone of the same sex and then someone of the opposite sex in any way that he or she wishes. (A variation involves asking the child to draw a person, house and tree.) Those who employ the test believe that the drawings reveal meaningful information about the child’s personality or mental health.

In a sketch of a man, for example, small feet would supposedly indicate insecurity or instability—a small head, inadequacy. Large hands or teeth would be considered signs of aggression; short arms, a sign of shyness. And feminine features—such as eyelashes or darkly colored lips—would allegedly suggest sex-role confusion.

Yet research consistently shows that such “signs” bear virtually no relation to personality or mental illness. Scientists have denounced these sign interpretations as “phrenology for the 20th century,” recalling the 19th-century pseudoscience of inferring people’s personalities from the pattern of bumps on their skulls.

Still, the sign approach remains widely used. Some psychologists even claim they can identify sexual abuse from certain key signs. For instance, in the child’s drawing at the right, alleged signs of abuse include a person older than the child, a partially unclothed body, a hand near the genitals, a hand hidden in a pocket, a large nose and a moustache. In reality, the connection between these signs and sexual abuse remains dubious, at best.

HUMAN FIGURE DRAWINGS

Misleading Signs

rarely add much to information that can be obtained in other, more practical ways, such as by conducting interviews or administering objective personality tests. (Objective tests seek answers to rel- atively clear-cut questions, such as, “I fre- quently have thoughts of hurting my- self—true or false?”) This shortcoming of projective tools makes the costs in mon- ey and time hard to justify.

Some mental health professionals dis- agree with our conclusions. They argue that projective tools have a long history of constructive use and, when administered and interpreted properly, can cut through the veneer of respondents’ self-reports to provide a picture of the deepest recesses of the mind. Critics have also asserted that we have emphasized negative find- ings to the exclusion of positive ones.

Yet we remain confident in our con- clusions. In fact, as negative as our over- all findings are, they may paint an over- ly rosy picture of projective techniques because of the so-called file drawer effect. As is well known, scientific journals are more likely to publish reports demon- strating that some procedure works than reports finding failure. Consequently, re- searchers often quietly file away their negative data, which may never again see the light of day.

We find it troubling that psychologists commonly administer projective instru- ments in situations for which their value has not been well established by multiple studies; too many people can suffer if er- roneous diagnostic judgments influence therapy plans, custody rulings or criminal court decisions. Based on our findings, we strongly urge psychologists to curtail their use of most projective techniques and, when they do select such instruments, to limit themselves to scoring and interpret- ing the small number of variables that have been proved trustworthy.

Our results also offer a broader lesson for practicing clinicians, psychology stu- dents and the public at large: even seasoned professionals can be fooled by their intu- itions and their faith in tools that lack strong evidence of effectiveness. When a substantial body of research demonstrates that old intuitions are wrong, it is time to adopt new ways of thinking.

w w w . s c i a m . c o m SCIENTIFIC AMERICAN 47

MORE TO EXPLORE The Rorschach: A Comprehensive System, Vol. 1: Basic Foundations. Third edition. John E. Exner. John Wiley & Sons, 1993.

The Comprehensive System for the Rorschach: A Critical Examination. James M. Wood, M. Teresa Nezworski and William J. Stejskal in Psychological Science, Vol. 7, No. Tk, pages 3–10; DateTK, 1996.

Studying the Clinician: Judgment Research and Psychological Assessment. Howard N. Garb. American Psy- chological Association, 1998.

Evocative Images: The Thematic Apperception Test and the Art of Projection. Edited by Lon Gieser and Morris I. Stein. American Psychological Association, 1999.

Projective Measures of Personality and Psychopathology: How Well Do They Work? Scott O. Lilienfeld in Skep- tical Inquirer, Vol. 23, No. TK, pages 32–39; DateTK 1999.

The Scientific Status of Projective Techniques. Scott O. Lilienfeld, James M. Wood and Howard N. Garb in Psy- chological Science in the Public Interest, Vol. 1, No. TK, pages 27–66; DateTk, 2000. Available at www.psycho- logicalscience.org/newsresearch/publications/journals/pspi1_2.html

In 1995 a survey of 412 randomly selected clinical psychologists in the American Psychological Association asked how often they used various projective and nonprojective assessment tools, including those listed below, in their practices. Projective instruments present people with ambiguous pictures, words or things; nonprojective measures are less open-ended. The percentages who use the projective methods might have declined slightly since 1995, but these techniques remain widely used.

HOW OF TEN THE TO0LS ARE USED

Popularity Poll

PROJECTIVE TECHNIQUES

USE AT LEAST OCCASIONALLY

USE ALWAYS OR FREQUENTLY

USE AT LEAST OCCASIONALLY

USE ALWAYS OR FREQUENTLY

Rorschach 43% 82%

Human Figure Drawings 39% 80%

Thematic Apperception Test (TAT) 34% 82%

Sentence Completion Tests 34% 84%

CAT (Children’s version of the TAT) 6% 47%

NONPROJECTIVE TECHNIQUES*

Weshler Adult Intelligence Scale (WAIS) 59% 93%

Minnesota Multiphasic 58% 85% Personality inventory-2 (MMPI-2)

Weschler Intelligence 69% 42% Scale for Children (WISC)

Beck Depression Inventory 71% 21%

* Those listed are the most commonly used nonprojective tests for assessing adult IQ (WAIS), personality (MMPI-2), childhood IQ (WISC) and depression (Beck Depression Inventory).

SOURCE: “Contemporary Practice of Psychological Assessment by Clinical Psychologists,” by C.E. Watkins et al. in Professional Pscyhology: Research and Practice, Vo. 26, No. 1, pages 54–60; 1995