week 3 discusion
Journal of Vocational Rehabilitation 32 (2010) 191–200 DOI:10.3233/JVR-2010-0509 IOS Press
191
Use of psychological tests within vocational rehabilitation
Oscar A. Donoso a,∗, Brigida Hernandez b and Elizabeth V. Horin c a DePaul University, Chicago, IL, USA bYAI Network, New York, NY, USA cEdward Hines, Jr. VA Hospital, Hines, IL, USA
Revised/Accepted: January 2010
Abstract. This study examined the testing practices of 150 professionals who conduct psychological assessments for the vocational rehabilitation (VR) system. A convenience sample, participants represented five large and culturally diverse states (i.e. California, Florida, Illinois, New York, and Texas). They responded to a semi-structured, online survey that was created by the authors to examine testing practices. Results indicated that (a) the Wechsler Adult Intelligence Scale and Minnesota Multiphasic Personality Inventory ranked highly in terms of use; (b) the overall test ranking, for the most part, did not vary based on participants’ demographic and professional characteristics; (c) neuropsychological instruments were commonly endorsed; and (d) projectives were not commonly used. Although findings corroborate prior research with regards to continued use of “popular” instruments, they also suggest increased use of psychometrically sound tests.
Keywords: Psychological assessment, vocational rehabilitation, test usage
1. Introduction
For decades, the testing practices of psychologists have been examined in order to determine the rank- ing of commonly used tests [5, 17, 18]. This line of research has grown as primary theoretical orientations have shifted over the years [19] and managed care has played a more significant role in service deliv- ery [21, 24]. To date, professionals from a number of sub-disciplines have been surveyed including forensic psychology, neuropsychology, health psychology, and school psychology [1, 15, 23, 25]. In a review of the lit- erature, Piotrowski [22] noted that test rankings tended to vary based on client characteristics, test settings, and assessors’ professional affiliations. The overall purpose of this study was to augment this body of research by examining the testing practices of professionals who conduct assessments for the vocational rehabilitation
∗Address for correspondence: Oscar A. Donoso, MA, DePaul University, 2219 N. Kenmore Ave Rm. 420, Chicago, IL 60614, USA. Tel.: +1 949 400 6854; E-mail: odonoso@depaul.edu.
(VR) system, a group that has been largely absent from this research.
1.1. Psychological test usage
Lubin and colleagues [18] examined test usage pat- terns in the United States from 1935 through 1982, with particular attention paid to projective and intelligence tests. Since 1969, five projective tests (i.e. Rorschach Inkblot Test, Thematic Apperception Test, Sentence Completion Test, House-Tree-Person Test, and Draw- a-Person Test) ranked in the top ten. The popularity of these measures was attributed in part to perceived improvements in scoring and interpretation as evident with Exner’s Rorschach Comprehensive System [8]. By the 1980s, the Wechsler intelligence scales for both adults and children started to rank highly, while the Stanford-Binet Intelligence Scales dropped from the top ten list. Overall, the authors concluded that there were minimal substantive changes in test rankings from 1969 to 1982.
1052-2263/10/$27.50 © 2010 – IOS Press and the authors. All rights reserved
192 O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation
In a more recent study, Watkins et al. [32] surveyed the assessment practices of 412 clinical psychologists. They found that the Wechsler Adult Intelligence Scale – Revised, Minnesota Multiphasic Personality Inventory – 2nd Edition, and a number of projective tests (Sen- tence Completion Test, Thematic Apperception Test, and Rorschach Inkblot Test) ranked highly. Moreover, participants reported that the following factors, listed in descending order of importance, influenced test selec- tion: (1) referral question, (2) prior testing experiences, (3) test reliability and validity, (4) graduate training, (5) agency requirements, and (6) test availability.
In a study of clinical psychologists and neuropsy- chologists, Camara et al. [5] found that both groups reported frequent use of the Wechsler intelligence scales, Minnesota Multiphasic Personality Inventory (1st and 2nd Editions), Wide Range Achievement Test (Revised and 3rd Edition), and Wechsler Memory Scale (Revised). Among the clinical psychologists, projective tests were still popular in spite of concerns with their psychometric properties [13, 33]. Neuropsychologists endorsed less use of projective tests and greater use of neuropsychological tests.
To date, test usage studies have covered a wide range of settings and disciplines, including the juvenile justice system [2], health psychology [23], foren- sic psychology [1, 16], neuropsychology [4, 25, 28], school psychology [15], and the Society for Personal- ity Assessment [20]. When Piotrowski [22] compared these studies, he noted that rankings varied based on clients tested (e.g., adults, juveniles), test settings (e.g., inpatient, career center, college counseling center), and assessors’ professional affiliations. Even though vari- ability was noted, the Wechsler scales and Minnesota Multiphasic Personality Inventory remained popular. As a group, individuals testing for the VR system are for the most part absent from this body of research. This discipline is of particular importance, given that many VR consumers undergo psychological testing to determine eligibility and service delivery [11].
1.2. Vocational rehabilitation and psychological testing
The employment rate for working-age adults with disabilities (37.7%) is quite low when compared to the rate of those without disabilities (79.9%) [26]. The VR system was established to help address the employment struggles experienced by the disability community. To be eligible for VR services, an individual must have a physical or mental impairment that impedes employ-
ment and is deemed likely to benefit from VR services to achieve an employment outcome [30]. VR can pro- vide up to 57 different services including psychological and vocational assessment; restoration of physical or mental functioning; academic, business, or vocational training; personal or vocational adjustment training; employment counseling; and job placement and referral [11, 14].
Psychological testing plays a critical role in the VR process. In a longitudinal study of 8,500 current and former VR consumers, approximately 35% of the sample received cognitive/psychological assessment services for the purpose of eligibility determination [11]. Furthermore, an additional 39% had pre-existing cognitive/psychological evaluations retrieved for this purpose. Of note, 92% of individuals who received a psychological evaluation were subsequently accepted into the VR system. Given that a high percentage of VR consumers undergo psychological testing for the purpose of eligibility determination and service provi- sion, it is important to gage which tests are being used for these important decision-making processes. For instance, concerns have been raised with the psycho- metric properties of projectives [13, 15, 33] and appro- priateness of certain standardized tests with culturally diverse groups [6, 7, 12]. Furthermore, Piotrowski [22] noted shifts in test usage patterns based on clientele, setting, and affiliation. Thus, when considering the literature, to what extent do the testing patterns of pro- fessionals conducting psychological assessments for the VR system differ from other settings? What is their pattern of projective test usage? How do demographic and professional characteristics impact testing patterns?
1.3. Purposes of the study
The purposes of this study were fivefold. First, this study assessed the practices of professionals conduct- ing psychological assessments for the VR system in order to determine the overall test ranking. Second, the impact of participants’ demographic (e.g., age, sex) and professional characteristics (e.g., years testing for the VR system, theoretical orientation, and freedom in test selection) on the overall test ranking was considered. Third, the frequency of use for each test ranking in the overall top 20 was examined. Fourth, to underscore the diversity of tests available to practitioners, rank- ings were determined for eight testing domains (i.e. achievement, adaptive behavior, cognitive/intelligence, neuropsychological, objective personality, projective personality, symptom checklists, and vocational apti-
O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation 193
tude and interests). Finally, the overall test ranking of this study was compared generally with the rank- ings from the Camara et al. [5] study, which was selected because it was recently conducted and included a sample of professionals (clinical psychologists and neuropsychologists) who likely reflect individuals test- ing for the VR system [3].
2. Method
2.1. Participants
To be eligible for this study, participants had to conduct psychological assessments for the VR sys- tem during the past 12 months (from the point of data collection) and practiced in one of five target states: California, Florida, Illinois, New York, or Texas. These states were selected because they are the largest by pop- ulation estimates, represent various geographic regions of the country, and have major cities with great cultural diversity [27, 29]. One hundred and ninety-five partic- ipants completed the survey online and three over the phone with the assistance of a research assistant, for a total of 198 participants. Of the 198 surveys, 150 (76%) were completed sufficiently, with remaining surveys being discarded because of numerous missing items or respondents not having met eligibility criteria (e.g., did not perform testing during the 12-month period).
2.2. Instrument
The Psychological Assessment Survey (PAS) was developed by the authors, following a review of the psychological assessment and vocational rehabilita- tion literature. The PAS is a semi-structured, 78-item survey that gathers information on demographic and professional characteristics, testing practices includ- ing reasons for referral, standardized tests used, and frequency of test usage during the 12-month span. Specifically, participants were provided with a list of tests commonly cited in the literature (without specification of edition or version) for 8 domains: (1) achievement, (2) adaptive behavior, (3) cogni- tive/intelligence, (4) neuropsychological, (5) objective personality, (6) projective personality, (7) symptom checklists, and (8) vocational aptitude and interests. In addition, for each domain, participants were able to write in up to three tests that were not listed. For each test endorsed, participants also indicated the frequency of use by selecting one of 24 range options. Each range
had a 5-point interval (i.e. 0, 1–5, 6–10, 11–15, etc.), with the exception of 101–150, 151–200, and 201 or more. Lastly, participants indicated how much over- all “freedom” they had to select tests administered to VR consumers using a 4-point Likert-type scale (i.e. no freedom, a little freedom, some freedom, and a lot of freedom). The PAS was pilot-tested with three psy- chologists who met criteria to participate in this study. They were asked to comment on the clarity of items and thoroughness of the survey; they provided minimal feedback for survey improvement.
2.3. Procedures
A number of recruitment strategies were used to reach out to potential participants. First, information about the study was posted on the American Psycho- logical Association’s (APA) listserv for the Division of Rehabilitation Psychology. Second, contact infor- mation was obtained for 855 “vendors” contracted to perform psychological testing for the VR system. Each vendor was emailed and/or mailed a recruitment flyer (with 2 reminders); when available, telephone reminders were used as well. Third, information about the study was posted on each target state’s psycho- logical association’s website for one month. Lastly, snowball sampling was employed where upon comple- tion of the survey participants were asked to inform colleagues about the study. Since recruitment informa- tion was posted on a listserv and numerous websites and snowball sampling was used, it is not possible to calculate a response rate.
With all recruitment methods, participants had the option of completing the survey online or over the phone with a research assistant, who concurrently entered the data into the online database. The survey took approxi- mately 30 to 45 min to complete. Initially, participants were compensated for their time and effort with a $25 gift card. Midway through data collection, this com- pensation was increased to a $50 gift card in order to increase participation.
3. Results
3.1. Description of the participants
Table 1 displays the demographic characteristics of the sample. In sum, 65% were male and the mean age of the total sample was 50 years, with a range of 27 to 85. In terms of racial/ethnic background, the sam- ple was predominantly Caucasian/White (81%). With
194 O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation
Table 1 Demographic characteristics of the sample
Variable n %
Sex Male 97 65.0 Female 53 35.0
Age <40 years 38 25.3 40–49 years 34 22.7 50–59 years 40 26.7 >60 years 38 25.3
Race/Ethnicity Caucasian/White 121 80.7 Latino/Hispanic 10 6.7 Multiraciala 7 4.7 African American/Black 4 2.7 Middle Eastern 2 1.3 Asian American/Pacific Islander 1 0.7 Other race/Ethnicity 5 3.3
aParticipants selecting more than one racial/ethnic category were grouped in the multiracial category.
regards to professional characteristics (see Table 2), 95% had a doctorate degree, with the majority indi- cating a background in clinical psychology (84%). In terms of theoretical orientation, 54% identified
as cognitive-behavioral, 20% eclectic/integrative, 18% psychodynamic, and 8% other (e.g., family systems, interpersonal). On average, participants had conducted 71 batteries over the 12-month period, and had 15 years of testing experience within the VR sys- tem. Ninety-three percent conducted testing to rule out cognitive difficulties and 94% indicated assessing clients with cognitive/developmental disabilities. Fur- thermore, 67% reported they had “a lot” or “some” freedom in selecting tests, while 29% indicated they had “little” or “no” freedom. Geographically, the largest percentage of participants were from Texas (39%), fol- lowed by Florida (25%), California (13%), New York (11%), and Illinois (10%).
3.2. PAS results
The results from the PAS are organized under five sections: (1) overall test ranking, (2) participants’ demographic and professional variables and overall test ranking, (3) weighted score test ranking, (4) rankings by testing domain, and (5) comparison of overall test ranking with the Camara et al. [5] study.
Table 2 Professional characteristics of the sample
Variable n % Variable n %
Academic degree Clients’ type of disability Ph.D. Clinical psychology 91 60.7 Cognitive/Developmental disability 141 94.0 Psy.D. Clinical psychology 34 23.7 Psychiatric/Emotional disability 133 88.7 Other doctoral degree 17 11.3 Medical disability/Chronic illness 118 78.7 MA/MS 8 5.3 Physical disability 104 69.3
Theoretical orientation Hearing 68 45.0 Cognitive-Behavioral 81 54.0 Visual disability 66 44.0 Eclectic/Integrative 30 20.0 Other disability 9 6.0 Psychodynamic 27 18.0 Reason for referral Other 12 8.0 Cognitive difficulty 139 92.7
State testing for VR system Depressive disorders 115 76.7 Texas 58 38.7 Anxiety disorders 102 68.0 Florida 37 24.7 Personality disorders 89 59.3 California 19 12.7 Psychotic symptoms 82 54.7 New York 17 11.2 Other 25 16.7 Illinois 16 10.7 Substance abuse 13 8.7 Multiple states 3 2.0 Number of test batteries during the
Freedom in test selection last 12 months A lot of freedom 73 48.7 <10 34 22.7 Some freedom 28 18.7 10–25 39 26.0 A little freedom 27 18.0 26–50 28 18.7 No freedom 17 11.3 51–100 24 16.0 Missing 5 3.3 >101 25 16.6
Years testing for VR system <10 60 40.0 10–19 42 28.0 20–29 25 16.7 30–39 18 12.0 >40 5 3.3
O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation 195
3.2.1. Overall test ranking Participants endorsed using 154 different tests when
assessing VR consumers during a 12-month period. Table 3 displays the top 20 instruments that partici- pants endorsed using at least once, with the Wechsler Adult Intelligence Scale (93%, n = 139) ranking first, followed by the Wide Range Achievement Test (70%, n = 105), Mental Status Exam (69%, n = 104), and Minnesota Multiphasic Personality Inventory (68%, n = 102). Other tests that were used by at least half of the sample included the Wechsler Memory Scale (60%, n = 90), Beck Depression Inventory (58%, n = 87), and Trail Making Test (50.7%, n = 76). The Rorschach Inkblot Test and Thematic Apperception Test ranked 16th and 18th, respectively.
3.2.2. Participants’ demographic and professional variables and overall test ranking
With the exception of the Bender Visual-Motor Gestalt Test and Trail Making Test, when the overall test ranking was examined by participants’ demographic (i.e. age, sex) and professional characteristics (i.e. years testing for the VR system, theoretical orientation, and freedom in test selection), significant group differ- ences were not found. For the Bender Visual-Motor Gestalt Test, Chi-square analyses indicated signifi-
cant differences based on participants’ age, χ2 (3, n = 150) = 12.69, p = 0.05, and years testing for the VR system, χ2 (3, n = 150) = 19.53, p < 0.001. Upon closer examination of the data, it was apparent that participants who were 39 years of age and younger and had been testing for the VR system between 1 and 5 years used the Bender Visual-Motor Gestalt Test less frequently (with the instrument ranking 20th for both groups) than those 40 years of age and older and with more testing experience. Similarly, for the Trail Making Test, sig- nificant differences were evident by participants’ age, �2 (3, n = 150) = 7.94, p = 0.04, and years testing for the VR system, �2 (3, n = 150) = 7.83, p = 0.05. Once again, those who were in the youngest age category and with least number of years conducting assessments for the VR system reported using the Trail Making Test less often (with the instrument ranking 10th and 8th, respec- tively) when compared to their older counterparts.
3.2.3. Weighted score test ranking To determine the frequency to which each test rank-
ing in the top 20 was used, a weighted score similar to the one used by Piotrowski and Lubin [23] was calcu- lated. First, participants’ responses with regard to how frequently they were using tests were grouped into one of six ranges with a corresponding numerical weight
Table 3 Overall test ranking and weighted score test ranking
Test Overall n % 0 1–25 26–50 51–75 76–100 101+ Weighted Weighted score test ranking score test ranking
Wechsler Adult Intelligence Scale 1 139 92.7 11 71 21 7 15 25 319 1 Wide Range Achievement Test 2 105 70.0 45 57 12 10 7 19 234 3 Mental Status Exam 3 104 69.3 46 56 13 5 11 19 236 2
Minnesota Multiphasic Personality Inventory
4 102 68.0 48 77 10 1 8 6 162 4
Wechsler Memory Scale 5 90 60.0 60 69 11 3 6 1 129 6 Beck Depression Inventory 6 87 58.0 63 64 14 2 3 4 130 5 Trail Making Test (any version) 7 76 50.7 74 48 18 7 2 1 118 7 Bender Visual-Motor Gestalt Test 8 61 40.7 89 37 14 3 2 5 107 8 Incomplete Sentences (any version) 9 60 40.0 90 45 7 3 4 1 89 9 Beck Anxiety Inventory 10 54 36.0 96 41 6 1 2 4 84 10 Wechsler Individual Achievement Test 10 54 36.0 96 39 8 6 1 0 77 12 Woodcock-Johnson Tests of
Achievement 12 53 35.3 97 37 9 2 2 3 84 10
Projective Drawings (any version) 13 52 34.7 98 40 6 2 2 2 76 13
Woodcock-Johnson Tests of Cognitive Abilities
14 50 33.3 100 36 8 2 2 2 76 13
Vineland Adaptive Behavior Scales 15 47 31.3 103 36 6 2 3 0 66 16 Rorschach Inkblot Test 16 44 29.3 106 34 3 3 1 3 68 15 Millon Clinical Multiaxial Inventory 17 43 28.7 107 35 4 2 1 1 58 17 Thematic Apperception Test 18 42 28.0 108 36 3 1 1 1 54 18 Symptom Checklist-90 19 32 21.3 118 25 2 1 3 1 49 19 Wisconsin Card Sorting Test 19 32 21.3 118 25 5 1 1 0 42 21 Personality Assessment Inventory 23 26 17.3 124 16 4 0 3 2 46 20
196 O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation
(i.e. 0 = 0; 1–25 = 1; 26–50 = 2; 51–75 = 3; 76–100 = 4; and 101+ = 5). Then, each numerical weight was mul- tiplied by the number of participants who fell within the range to determine the weighted score. Table 3 displays the weighted score for each test, along with the overall test ranking when the weighted scores were considered. In general, the weighted score ranking was consistent with the overall test ranking, as the Wechsler Adult Intelligence Scale ranked first and was followed by the Mental Status Exam, Wide Range Achievement Test, Minnesota Multiphasic Personality Test, and Beck Depression Inventory.
3.2.4. Rankings by testing domain Rankings were also determined for eight specific
testing domains (see Table 4). The Wechsler Adult Intelligence Scale (92.7%), Wide Range Achievement Test (70.0%), Mental Status Exam (69.3%), Minnesota
Multiphasic Personality Inventory (68.0%), Beck Depression Inventory (58.0%), Incomplete Sentences Blank (40.0%), Vineland Adaptive Behavior Scales (31.3%), and Career Assessment Inventory (15.3%) ranked highest within the (1) cognitive/intelligence, (2) achievement, (3) neuropsychological, (4) objec- tive personality, (5) symptom checklist, (6) projective personality, (7) adaptive behavior, and (8) vocational aptitude and interests domains, respectively.
3.2.5. Comparison of rankings with Camara et al. [5] study
The overall rankings of this study were compared with rankings from the Camara et al. [5] sample of both clinical psychologists and neuropsychologists (see Table 5). Six tests appeared in the top ten ranking of the current sample and Camara’s clinical sample: Wechsler Adult Intelligence Scale, Wide Range Achievement
Table 4 Overall test rankings by test domain
Test domain and test Ranking n % Test domain and test Ranking n %
Achievement Neuropsychological Wide Range Achievement Test 1 105 70 Mental Status Exam 1 104 69.3 Wechsler Individual Achievement Test 2 54 36 Wechsler Memory Scale 2 90 60 Woodcock-Johnson Tests of Achievement 3 53 35.3 Trail Making Test (any version) 3 76 50.7 Kaufman Test of Educational Achievement 4 11 7.3 Bender Visual-Motor Gestalt Test 4 61 40.7 Stanford Achievement Test 5 5 3.3 Wisconsin Card Sorting Test 5 32 21.3
Adaptive behavior Beery-Buktenica Developmental Test of 6 20 13.3 Vineland Adaptive Behavior Scales 1 47 31.3 Visual Motor Integration Adaptive Behavior Assessment System 2 26 17.3 Dementia Rating Scale 7 19 12.7
Cognitive Purdue Pegboard 8 16 10.7 Wechsler Adult Intelligence Scale 1 139 92.7 Halstead Reitan Battery 9 9 6.1 Woodcock Johnson Tests of Cognitive Abilities 2 50 33.3 Repeatable Battery for the Assessment of Raven’s Progressive Matrices 3 29 19.3 Neuropsychological Status 9 9 6.1 Kaufman Brief Intelligence Test 4 14 9.3 Crawford Small Parts 11 4 2.7 Escala de Inteligencia Wechsler para Adultos 4 14 9.3 Objective personality Stanford-Binet Intelligence Scales 6 13 8.7 Minnesota Multiphasic Personality Inventory 1 102 68 Wechsler Abbreviated Scale of Intelligence 6 13 8.7 Million Clinical Multiaxial Inventory 2 43 28.7 Test of Nonverbal Intelligence 8 11 7.4 Sixteen Personality Factor 3 28 18.7 Kaufman Adult Intelligence Test 9 8 5.3 Personality Assessment Inventory 4 26 17.3 Batería Woodcock-Muñoz 10 6 4 California Personality Inventory 5 5 3.3
Symptom checklists Projective Beck Depression Inventory 1 87 58 Incomplete Sentence (any version) 1 60 40 Beck Anxiety Inventory 2 54 36 Projective Drawings (any version) 2 52 34.7 Symptom Checklist-90 3 32 21.3 Rorschach Inkblot Test 3 44 29.3
Thematic Apperception Test 4 42 28 Vocational aptitude and interest
Career Assessment Inventory 1 23 15.3 Self Directed Search 2 16 10.7 Strong Interest Inventory 3 14 9.3 Strong Campbell Inventory 4 9 6 Differential Aptitude Test 5 6 4 Geist Picture Inventory 6 4 2.7 Non-reading Aptitude Test Battery 6 4 2.7 U.S. Employment Services 8 3 2 Armed Services Vocational Aptitude 9 1 0.7 Minnesota Importance Questionnaire 9 1 0.7
O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation 197
Table 5 Overall test ranking of current sample and the Camara et al. (2000) samples
Test Camara et al. Camara et al. PASa Clinical psychologistsb Neuropsychologistsc
Rank n % Rank n % Rank n %
Wechsler Adult Intelligence Scale 1 139 92.7 1 151 93.7 2 331 76.9 Wide Range Achievement Test 2 105 70.0 7 86 53.4 9 203 47.2 Mental Status Exam 3 104 69.3 – – – – – – Minnesota Multiphasic Personality Inventory 4 102 68.0 2 138 85.7 1 359 83.4 Wechsler Memory Scale 5 90 60.0 9 58 36.0 3 257 59.7 Beck Depression Inventory 6 87 58.0 10 53 32.9 11 200 46.5 Trail Making Test (any version) 7 76 50.7 12 52 32.2 4 246 57.2 Bender Visual-Motor Gestalt Test 8 61 40.7 5 112 69.5 25 96 22.3 Incomplete Sentences (any version) 9 60 40.0 14d 45 27.9 51d 41 0.09 Beck Anxiety Inventory 10 54 36.0 – – – – – – Wechsler Individual Achievement Test 10 54 36.0 – – – – – – Woodcock-Johnson Tests of Achievement 12 53 35.3 – – – – – – Projective Drawings (any version) 13 52 34.7 8e 60 37.2 31e 78 18.1 Woodcock-Johnson Tests of Cognitive Abilities 14 50 33.3 – – – – – – Vineland Adaptive Behavior Scales 15 47 31.3 18 37 22.9 44 51 11.8 Rorschach Inkblot Test 16 44 29.3 4 124 77.0 18 153 35.5 Millon Clinical Multiaxial Inventory 17 43 28.7 10 53 32.9 24 100 23.2 Thematic Apperception Test 18 42 28.0 6 107 66.4 26 91 21.1 Symptom Checklist-90 19 32 21.3 – – – – – – Wisconsin Card Sorting Test 19 32 21.3 33 19 11.8 12 196 45.5
Note. Dashes indicate these tests were not in the top 20 and therefore not reported by Camara et al. (2000). The data presented from the Camara study appear in “Psychological test usage: Implications in professional psychology”, by W.J. Camara, J.S. Nathan and A.E. Puente, 2000, Professional Psychology: Research and Practice 31, p. 148. Copyright 2000 by the American Psychological Association. Adapted with permission. an = 150. bn = 161. cn = 430. dRotters Incomplete Sentences Blank. eHouse-Tree-Person Projective Technique.
Test, Minnesota Multiphasic Personality Inventory, Wechsler Memory Scale, Beck Depression Inventory, and Bender Visual-Motor Gestalt Test. Five tests ranked in the top ten of the current sample and Camara’s neuropsychologist sample: Wechsler Adult Intelligence Scale, Wide Range Achievement Test, Minnesota Mul- tiphasic Personality Inventory, Wechsler Memory Scale and Trail Making Test. With regard to areas of differ- ence, the Mental Status Exam, Beck Anxiety Inventory, and Wechsler Individual Achievement Test did not rank within the top 20 of either Camara’s samples. Lastly, incomplete sentences tests were the only projective test to rank in the top ten for the current sample; in contrast, several projective tests (i.e. Rorschach Inkblot Test, Thematic Appreciation Test, and projective drawings) ranked in the top ten of Camara’s clinical psychologist sample, although this particular trend was not evident with the neuropsychologist sample.
4. Discussion
For the last four decades, researchers have been examining the testing practices of psychologists [5, 18, 32]. Throughout that time period, a few tests (e.g.,
Wechsler intelligence scales and Minnesota Multipha- sic Personality Inventory) have ranked highly in terms of use. There have also been some changes in test popu- larity over time as theoretical orientations have shifted and managed care has had implications for reimburse- ment. Moreover, the field of psychology has grown and become more specialized, particularly when one con- siders the influence of neuropsychology [25] and the diversity of clinical populations and settings. For the most part, professionals testing for the VR system have been absent from this body of work. Thus, the overall findings of this study are noteworthy in that the testing practices of this group were examined to determine: (1) the overall test ranking, (2) influence of demo- graphic and professional characteristics on the overall test ranking, (3) frequency of use for each test ranking in the overall top 20, (4) rankings by testing domain, and (5) similarities and differences with rankings from the Camara et al. [5] study.
First, this study found that professionals conducting VR assessments used a variety of instruments across a number of testing domains and, consistent with prior research, the Wechsler Adult Intelligence Scale and Minnesota Multiphasic Personality Inventory reigned supreme [9, 18, 21, 32]. Also, continuing to rank highly
198 O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation
were the Wide Range Achievement Test, Wechsler Memory Scale, Bender Visual-Motor Gestalt Test, and incomplete sentences tests. To some extent, psychol- ogists have deemed the utility of these instruments to be particularly strong, with perhaps a few considering them standards of a test battery. However, it is worth noting that adherence to particular tests (as in the case of projectives) may do more harm than good, given that some “popular” instruments have been criticized for their weak theoretical foundation and/or psychometric properties [13, 15, 33].
Second, for the most part, the overall test rank- ing did not vary significantly based on participants’ demographic and professional characteristics including age, sex, years testing for the VR system, theoreti- cal orientation, and freedom in test selection. To the authors’ knowledge, the impact of these characteristics has not been extensively examined by prior research. Third, the stability of the overall test ranking was also evident when frequency of test usage was examined using weighted scores. The Wechsler Adult Intelligence Scale ranked highest and only slight differences were observed when the overall test ranking and weighted score test rankings were compared.
Fourth, recognizing the advances that psycho- logical testing has made over the decades, test rankings were also examined by eight domains (i.e. achievement, adaptive behavior, cognitive/intelligence, neuropsychological, objective personality, projective personality, symptom checklists, and vocational apti- tude and interests). Noteworthy, participants used numerous neuropsychological tests, with five tests appearing in the top 20: Mental Status Exam, Wechsler Memory Scale, Trail Making Test, Bender Visual- Motor Gestalt Test, and Wisconsin Card Sorting Test. In addition, a number of achievement tests (i.e. Wide Range Achievement Test, Wechsler Individual Achievement Test, and Woodcock-Johnson Tests of Achievement) were frequently endorsed.
Fifth, the overall test ranking of this study was compared generally with rankings from the Camara et al. [5] study, which was selected because it was recently conducted and included professionals who likely reflected professionals testing for the VR sys- tem [3]. Overall, the test stability noted earlier was apparent across both studies (with the Wechsler Adult Intelligence Scale, Minnesota Multiphasic Personality Inventory, Wide Range Achievement Test, and Wech- sler Memory Scale ranking in the top ten for all three samples). Moreover, for this study, less than one-third of participants endorsed use of the Rorschach Inkblot
Test and Thematic Apperception Test, which ranked 16th and 18th, respectively. Over the last 40 years, these two instruments have ranked consistently in the top ten [18] and this trend was evident with Camara’s clinical sample. However, Camara’s neuropsychologi- cal sample did not endorse frequent use of these two measures, as the Rorschach Inkblot Test and Thematic Apperception Test ranked 18th and 26th, respectively. Possible reasons for the decreased use of projectives include the need for specialized training, lengthiness of administration, ambiguous scoring and interpretation [5], concerns with clinical utility [10, 13], and reluc- tance of third-party payers to reimburse time-intensive assessments [24]. Further, although it is unknown how many participants of the current study identified as neu- ropsychologists, they did report administering a wide variety of neuropsychological tests (n = 55).
4.1. Limitations
Although a number of important findings emerged from this study, there are limitations to consider. First, participants represented a convenience sample, and consequently results reported herein are not necessarily reflective of those who declined participation. Second, the sample was drawn from populous states and may not represent demographic and fiscal resources of other states. Third, participants’ responses may not have been entirely accurate given that estimates of test usage were likely used, the survey did not specify test edition or version, and there may have been test overlap across the testing domains. Lastly, data on participants’ level of neuropsychological training was not gathered. Such information would have aided the understanding of results.
4.2. Implications of findings
With regards to future research, there are a number of areas worth examining. It is clear that throughout the decades professionals have been drawn to particu- lar tests [18, 32], and it would be prudent to ask why this pattern continues (particularly when one consid- ers the number of tests now available and continued concerns with projectives). After reviewing 30 years of research related to psychological assessments, Watkins [31] presented three reasons for the continued stability and frequent use of certain instruments: (1) assessors’ belief that they provide the best information to respond to the assessment question at hand, (2) longevity of instruments and their frequent mention in research jour-
O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation 199
nals, and (3) assessors’ graduate training. Watkins [31] contended that popular tests may be caught in a rein- forcement cycle, whereby graduate training programs emphasize particular tests which in turn are selected by both practitioners and researchers.
In a survey of APA members who identified as clin- ical psychologists, Watkins et al. [32] asked them to rate how important six specified variables were in test selection. Findings indicated that the capacity to answer specific referral questions was most important followed by prior experiences with an instrument, the psycho- metric properties of an instrument, graduate training, organizational requirements, and test availability. How- ever, are other factors playing a role? It would be valuable for future research to examine with greater depth the testing practices of professionals with a focus on why particular tests are chosen. To what extent are critical decisions tied to time constraints, grad- uate training, testing experiences, lack of resources, or reimbursement. A qualitative study may be helpful in providing some answers. Furthermore, it would be important to examine whether graduate training pro- grams are indeed caught in a reinforcement cycle [31]. With the plethora of tests available, are assessment courses relying on the gold standards and not expos- ing students to other tests? If so, why might that be the case? A study that extends this line of research to the perspectives of training directors and assessment instructors would be quite informative.
5. Conclusion
Clearly, psychological assessments play a critical role for individuals seeking entry into the VR system and subsequent services. It is incumbent on profession- als to select the best instruments to address referral questions. Selected tests should also be psychomet- rically sound and appropriate to use with culturally diverse groups, including people with disabilities. With that stated, it is important to acknowledge that test selec- tion is just one part of the assessment process. Even more important is the role of the assessor who has the charge of interpreting and writing the results and recommendations.
Acknowledgements
We extend our gratitude to Andrea Saul and Carrie Kaufman for their assistance during the course of this
project as well as our participants for their willing- ness to share their experiences. This work was funded by the National Institute on Disability and Rehabilita- tion Research (NIDRR), U.S. Department of Education, through a grant to the Center for Capacity Build- ing on Minorities with Disabilities Research (Grant # H133A040007). The opinions expressed herein are those of the authors and not necessarily those of NIDRR or the Center. This study was conducted by the authors while they were affiliated with DePaul University, and we wish to thank the YAI Network and the Edward Hines, Jr. VA Hospital for their support in completing this work.
References
[1] R. Archer, J. Buffington-Vollum, R. Stredny and R. Handel, A survey of psychological test use patterns among forensic psychologists, Journal of Personality Assessment 87 (2006), 84–94.
[2] S. Bailey, T. Doreleijers and P. Tarbuck, Recent developments in mental health screening and assessment in juvenile justice systems, Child and Adolescent Psychiatric Clinics of North America 15 (2006), 391–406.
[3] M.T. Barisa and M.W. Barisa, Neuropsychological evaluation applied to vocational rehabilitation, NeuroRehabilitation 16 (2001), 289–293.
[4] M. Butler, P. Retzlaff and R.Vanderploeg, Neuropsychological test usage, Professional Psychology: Research and Practice 22 (1991), 510–512.
[5] W.J. Camara, J.S. Nathan and A.E. Puente, Psychological test usage: Implications in professional psychology, Professional Psychology: Research and Practice 31 (2000), 141–154.
[6] R. Dana, Multicultural Assessment Perspectives for Profes- sional Psychology, Allyn & Bacon, Needham Heights, MA, 1993.
[7] R.H. Dana, Multicultural Assessment: Principles, Applica- tions, and Examples, Lawrence Erlbaum Associates, Mahwah, NJ, 2005.
[8] J.E. Exner, The Rorschach: A comprehensive system. 4th edi- tion, Basic Foundations and Principles of Interpretation, John Wiley & Sons, Hoboken, NJ, 2003.
[9] D. Frauenhoffer, M.J. Ross, J. Gfeller, H.R. Searight and C. Piotrowski, Psychological test usage among licensed men- tal health practitioners: A multidisciplinary survey, Journal of Psychological Practice 4 (1998), 28–33.
[10] H. Garb, J. Wood, S. Lilienfeld and M. Nezworski, Roots of the Rorschach controversy, Clinical Psychology Review 25 (2005), 97–118.
[11] B.J. Hayward and H. Schmidt-Davis, Longitudinal Study of Vocational Rehabilitation Service Program: Final Report 2: VR Services and Outcomes, Rehabilitation Services Admin- istration, U.S. Department of Education, RTI International, 2003.
[12] B. Hernandez, E.V. Horin, O.A. Donoso and A. Saul, Psycho- logical testing and multicultural populations, in: F. Balcazar, Y. Suarez-Balcazar, T. Taylor-Ritzler and C. Keys, eds., Race, Culture and Disability: Rehabilitation Science and Practice, Jones and Bartlett, Boston, 2010, pp. 55–79.
200 O.A. Donoso et al. / Use of psychological tests within vocational rehabilitation
[13] J. Hunsley and J. Bailey, The clinical utility of the Rorschach: Unfulfilled promises and an uncertain future, Psychological Assessment 11 (1991), 266–277.
[14] H.S. Kaye, Vocational rehabilitation in the United States, in: Disability Statistics Abstract 20, Disability Statistics Center, San Francisco, CA, March 1998.
[15] M. Kennedy, D. Faust, W. Willis and C. Piotrowski, Social- emotional assessment practices in school psychology, Journal of Psychoeducational Assessment 12 (1994), 228–240.
[16] P. Lees-Haley, H. Smith, C. Williams and J. Dunn, Forensic neuropsychological test usage: An empirical survey, Archives of Clinical Neuropsychology 11 (1996), 45–51.
[17] C. Louttit and C. Browne, The use of psychometric instruments in psychological clinics, Journal of Consulting Psychology 11 (1947), 49–54.
[18] B. Lubin, R.M. Larsen and J.D. Matarazzo, Patterns of psycho- logical test usage in the United States: 1935–1982, American Psychologist 39 (1984), 451–454.
[19] J.C. Norcross, C.P. Karpiak and S.O. Santoro, Clinical psy- chologists across the years: The division of clinical psychology from 1960–2003, Journal of Clinical Psychology 61 (2005), 1467–1483.
[20] C. Piotrowski, Clinical assessment: Attitudes of the society for personality assessment membership, The Southern Psycholo- gist 2 (1985), 80–83.
[21] C. Piotrowski, Assessment practices in the era of managed care: Current status and future directions, Journal of Clinical Psychology 55 (1999), 787–796.
[22] C. Piotrowski, Forensic psychological testing as a function of affiliation and organizational setting, Organization Develop- ment Journal 25 (2007), 94–98.
[23] C. Piotrowski and B. Lubin, Assessment practices of health psychologists: Survey of APA Division 38 clinicians, Pro-
fessional Psychology: Research and Practice 21 (1990), 99–106.
[24] C. Piotrowski, R. Belter and J. Keller, The impact of ‘managed care’ on the practice of psychological testing: Preliminary find- ings, Journal of Personality Assessment 70 (1998), 441–447.
[25] L.A. Rabin, W.B. Barr and L.A. Burton, Assessment prac- tices of clinical neuropsychologists in the United States and Canada: A survey of INS, NAN, and APA Division 40 mem- bers, Archives of Clinical Neuropsychology 20 (2005), 33–65.
[26] Rehabilitation Research and Training Center on Disability Demographics and Statistics, in: 2006 Disability Status Report, Cornell University, New York, 2007.
[27] C.P. Ryan, Work isn’t what it used to be: Implications, rec- ommendations, and strategies for vocational rehabilitation, Journal of Vocational Rehabilitation 61 (1995), 8–15.
[28] K. Sullivan and S. Bowden, Which tests do neuropsychologists use? Journal of Clinical Psychology 53 (1997), 657–661.
[29] U.S. Census Bureau, 2000 U.S. Census with race and ethnicity with Hispanic origin, U.S. Department of Commerce, Bureau of the Census, Washington, DC, 2000.
[30] U.S. Department of Education, Vocational rehabilita- tion state grants. Retrieved on April 12, 2009 from http://www.ed.gov/programs/rsabvrs/gteprsabvrs.pdf, 2009.
[31] C.E. Jr. Watkins, What have surveys taught us about the teaching and practice of psychological assessment? Journal of Personality Assessment 56 (1991), 426–437.
[32] C.E. Watkins, V.L. Campbell, R. Nieberding and R. Hallmark, Contemporary practice of psychological assessment by clinical psychologists, Professional Psychology: Research & Practice 24 (1995), 54–60.
[33] J.M. Wood, M.T. Nezworski and W.J. Stejskal, The Compre- hensive System for the Rorschach: A critical examination, Psychological Science 7 (1996), 3–10.
Copyright of Journal of Vocational Rehabilitation is the property of IOS Press and its content may not be copied
or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.
However, users may print, download, or email articles for individual use.