Psychology Statistics data report project

DepressionAnxietyStressOsmanetal.2012.pdf

Home >Psychology homework help >Psychology Statistics data report project

The Depression Anxiety Stress Scales—21 (DASS-21): Further Examination of Dimensions, Scale Reliability, and Correlates

Augustine Osman,1 Jane L. Wong,2 Courtney L. Bagge,3 Stacey Freedenthal,4 Peter M. Gutierrez,5 and Gregorio Lozano6

1 The University of Texas at San Antonio 2 Armstrong Atlantic State University 3 University of Mississippi Medical Center 4 University of Denver 5 Denver VA Medical Center, MIRECC 6 The University of Texas at San Antonio

Objectives: We conducted two studies to examine the dimensions, internal consistency reliability estimates, and potential correlates of the Depression Anxiety Stress Scales—21 (DASS-21; Lovibond & Lovibond, 1995). Method: Participants in Study 1 included 887 undergraduate students (363 men and 524 women, aged 18 to 35 years; mean [M] age = 19.46, standard deviation [SD] = 2.17) recruited from two public universities to assess the specificity of the individual DASS-21 items and to evaluate estimates of internal consistency reliability. Participants in a follow-up study (Study 2) included 410 students (168 men and 242 women, aged 18 to 47 years; M age = 19.65, SD = 2.88) recruited from the same universities to further assess factorial validity and to evaluate potential correlates of the original DASS-21 total and scale scores. Results: Item bifactor and confirmatory factor analyses revealed that a general factor accounted for the greatest proportion of common variance in the DASS-21 item scores (Study 1). In Study 2, the fit statistics showed good fit for the bifactor model. In addition, the DASS-21 total scale score correlated more highly with scores on a measure of mixed depression and anxiety than with scores on the proposed specific scales of depression or anxiety. Coefficient omega estimates for the DASS-21 scale scores were good. Conclusions: Further investigations of the bifactor structure and psychometric properties of the DASS-21, specifically its incremental and discriminant validity, using known clinical groups are needed. C© 2012 Wiley Periodicals, Inc. J. Clin. Psychol. 68:1322–1338, 2012.

Keywords: depression anxiety stress; self-report inventory; bifactor IRT models; nonclinical

Theorists have posited that a common factor may underline depression and anxiety (cf. Clark & Watson, 1991), accounting for both observations of high comorbidity between the mood and anxiety disorders (e.g., Sanderson, Di Nardo, Rapee, & Barlow, 1990; Seligman & Ollendick, 1998) and the lack of specificity of measures designed to assess specific disorders (e.g., depression; Clark & Watson, 1990). Clinicians have also suggested that a new diagnosis be created which combines elements of both disorders into a single mixed anxiety-depressive disorder. Lovibond and Lovibond (1995) developed the original Depression Anxiety Stress Scales-42 (DASS-42) to maximize discrimination between self-reported anxiety and depression while assessing the full range of these disorders’ core symptoms. Antony, Bieling, Cox, Enns, and Swinson (1998) subsequently confirmed that both the original DASS-42 and a shorter version, the DASS-21, distinguish “well between features of depression, physical arousal, and psychological tension and agitation” (p. 176) in clinical and nonclinical groups.

The DASS is not intended to be used as a diagnostic instrument as the three subscales as- sess dimensional components of the anxiety and depressive disorders (Psychology Foundation of Australia, 2011). Additionally, it is not only a distress measure, but rather a measure of

Please address correspondence to: Augustine Osman, The University of Texas at San Antonio, Department of Psychology, One UTSA Circle, San Antonio, Texas 78249-0652. E-mail: [email protected]

Depression Anxiety Stress 1323

the shared causes across anxiety, depression, and stress characterized as a distinct syndrome. Because the measure is not diagnosis-specific, it is appropriate for use in clinical populations broadly speaking. This makes it a measure appropriate in a wide range of clinical and research settings where measures of the interplay of different forms of emotional distress are needed. However, it is probably not appropriate as a measure of individuals’ state distress as the time- frame assessed is more than the current moment (Psychology Foundation of Australia). It may perform similarly to other existing measures such as the Hospital Anxiety and Depression Scale (HADS; Zigmond & Snaith, 1983), as the two yielded similar results in a study of for- mer intensive care unit patients (Sukantarat, Williamson, & Brett, 2007). However, Sukantarat et al. cited the DASS’ ability to assess the additional dimension of stress not covered by the HADS and superior internal consistency reliability as two advantages of the DASS-21 over the HADS.

The DASS-21 has been used in a range of settings across age groups. For example, Page, Hooke, and Morrison (2007) evaluated the psychometric properties of the DASS-21 in three samples of mostly adult psychiatric patients with mood disorders, comprising over 1,400 patients between the ages of 14 and 83. Their primary findings support the strong internal consistency of the total and scale scores, the factor structure as determined by Crawford and Henry (i.e., allowing three items to cross-load and correlated error terms; 2003), and sensitivity to change resulting from treatment. The authors also report a significant ceiling effect for the depression subscale, less of a ceiling effect (although still notable) for the anxiety subscale, and no ceiling effect for the stress scale. They were unable to eliminate the ceiling effect by adjusting the response options, and they suggest that the revision of the items to better capture the most severe symptoms of depression is in order.

The DASS-21 was found to be an appropriate measure to assess improvement as a result of treatment for patients admitted to a private psychiatric hospital in Australia (Ng et al., 2007). In this sample of 388, majority female (61%), adult (mean age = 52) patients, the measure was found to be easy to administer, low cost, and effective at detecting change in a sample of patients diagnosed primarily with depressive and anxiety disorders. Sukantarat et al. (2007) evaluated the DASS for use with 51 adult patients (mean age = 57 years) who survived intensive care unit admissions. Internal consistency reliability for the DASS was acceptable at both 3-month and 9- month follow-up and scale scores correlated at acceptable levels with another validated measure of depression and anxiety. Finally, Allen and Annells (2009) concluded after conducting a critical review of the literature that one advantage of using the DASS-21 to assess elderly patients is that the lack of items on somatic complaints allows for more accurate assessment in this age group.

The current studies build on previous investigations with the DASS-21 by examining the factor structure, scale reliability, and potential correlates of the scores in two large undergraduate student samples. The DASS items are scored on a 4-point scale ranging from 0 (did not apply to me at all) to 3 (applied to me very much, or most of the time). Higher scores indicate more frequent symptomatology. Seven items comprise each of three scales: Depression (example item: “I couldn’t seem to experience any positive feeling at all”), Anxiety (e.g., “I experienced breathing difficulty”), and Stress (e.g., “I found it hard to wind down”).

Factor Structure

Previous studies investigating the factor structure of the DASS-21 have identified a two-factor solution, a three-factor solution, or a second-order factor defined by three lower first-order factor solutions. However, several modifications were made to each of these models to at- tain good fit estimates. As an example, in studies confirming support for a two-factor solu- tion, researchers grouped together the DASS-21 Stress and Anxiety scale items into a general anxiety-stress factor to help improve the fit of the model (Daza, Novy, & Stanley, 2002; Duffy, Cunningham, & Moore, 2005; Tully, Zajac, & Venning, 2009). In studies involving a three- factor solution, researchers have allowed for (a) multiple correlated errors within domain-specific scales or (b) cross loading of items to attain best-fitting models (e.g., see Antony et al., 1998;

1324 Journal of Clinical Psychology, December 2012

Brown, Chorpita, Korotitsch, & Barlow, 1997; Clara, Cox, & Enns, 2001; Henry & Crawford, 2005).1

To date, only one study has attempted to address empirically the unique contributions of the individual DASS-21 items to the depression, anxiety, and stress constructs. Using a confirma- tory factor-analytic approach, Henry and Crawford (2005) reported support for a four-factor (i.e., quadripartite) model in samples of nonclinical adults in the United Kingdom. Specifically, unlike previous studies with the DASS-21 items, the investigators constrained each item to load on a general distress dimension, and also on only one of the three domain-specific dimen- sions. However, the researchers allowed for several correlated errors between items within the domain-specific factors (i.e., the Anxiety and Stress scale items) to help establish good fit of the quadripartite model and did not discuss how to derive scores for the DASS-21 scales that included correlated errors (see Smith, McCarthy, & Anderson, 2000). Importantly, by relying on model fit statistics to evaluate the adequacy of the study models, the authors did not evalu- ate empirically the extent to which each individual DASS-21 item is related more strongly to a proposed domain-specific dimension (i.e., specificity) than to a general distress (nonspecificity) dimension (see Cook, Kallen, & Amtmann, 2009; Haynes, Richard, & Kubany, 1995).

Consequently, one of the goals of our Study 1 was to examine the relationships between each individual DASS-21 item and a domain-specific dimension or scale. Specifically, we wanted to determine the specificity and nonspecificity of the individual DASS-21 items (e.g., see Reise, Morizot, & Hays, 2007; Simms, Grös, Watson, & O’Hara, 2008). As an example, if the DASS- 21 items are linked strongly with both the general and domain-specific dimensions, the items should be conceptualized as multidimensional (i.e., nonspecific to the underlying dimension), rather than specific, in nature. Accordingly, we conducted several item-level bifactor analyses to address the current study goals. First, however, we conducted an exploratory Schmid-Leiman (1957) transformation analysis to examine the (a) dimensions and (b) amount of common variance accounted for by the DASS-21 scale scores. This approach is exploratory because the structural dimensions of the DASS-21 items have not been examined in a large sample (e.g., N ≥ 500) of U.S. nonclinical college-age samples.

Scale Reliability

Beyond investigations of factor structure or dimensionality, studies have reported good estimates of internal consistency reliability for the original scale scores (range = .82 to .97) of the DASS-21 in clinical and nonclinical samples (e.g., Henry & Crawford, 2005; Lovibond & Lovibond, 1995). Because the coefficient-α estimation procedure has been observed to underestimate or overesti- mate internal consistency reliability for multidimensional instruments (e.g., between-item), we also employed an alternate internal consistency reliability estimation method, the McDonald coefficient-omega (coefficient-ω; McDonald, 1999), to calculate internal consistency reliability for scores on the DASS-21 scales (see Raykov, 2004; Sijtsma, 2009; Zumbo, Gadermann, & Zeisser, 2007).

Confirmatory Analyses and Concurrent Validity

In Study 2, we used confirmatory factor analysis to evaluate fit estimates of the bifactor model against two competing solutions that have been reported in the extant literature for the DASS-21 (e.g., Henry & Crawford, 2005; Tully et al., 2009). Specifically, unlike Study 1, the goal of this study was to extend our understanding of the structure and nature of the relationships among the proposed dimensions of the DASS-21. Accordingly, we hypothesized that the bifactor model (i.e., a model comprising a general distress dimension plus three domain-specific dimensions) would attain a better fit to the sample data than any of the alternative models (in particular, the original oblique three-factor solution).

1Although some researchers tend to remove the DASS-21 Stress scale items from validation studies, we did not modify the DASS-21 items given that the goal of the current study was not to revise the instrument (see Clara et al., 2001; Smith & McCarthy, 1995).

Depression Anxiety Stress 1325

In Study 2, we also examined potential correlates of the DASS-21. The concurrent validity of the DASS-21 scale scores has been supported by moderate to high correlations (rs = .40 to .65) with related measures of depression and anxiety (e.g., Antony et al., 1998; Brown et al., 1997; Crawford & Henry, 2003). Also, previous studies have demonstrated some evidence of discrim- inant validity by showing that between-group specific correlations (e.g., DASS-21 depression with a number of self-report measures of anxiety) were significantly lower than within-group specific correlations (e.g., DASS-21 depression with other related measures of depression) among clinical and nonclinical samples (e.g., Antony et al., 1998; Henry & Crawford, 2005; Gloster et al., 2008). In this study, we further examined estimates of concurrent validity for scores on the DASS-21.

Study 1

First, we conducted an exploratory factor analysis to evaluate the amount of common variance in the DASS-21 item scores. Second, because our focus was on evaluating the characteristics of the individual DASS-21 items, we conducted an item bifactor analysis. Third, we conducted estimates of internal consistency reliability for the original DASS-21 total and subscale scores. We also computed descriptive statistics (mean and standard deviation) for the DASS-21 total and scales. Only participants with complete data were included in the analyses.2

Method

Participants

The sample included undergraduate students from large midwestern and southwestern public universities. Because participants from the separate universities did not differ significantly on relevant demographic variables, including age, gender, and marital status (all ps > .05), we combined the data for all the subsequent analyses. The sample included 524 women (mean [M] age = 19.38, standard deviation [SD] = 2.32 years) and 363 men (M age = 19.59, SD = 1.93 years); they did not differ significantly in age, t (885) = 1.42, p = .16. Of the sample, 652 (73.5%) self-identified as Caucasian/White, 96 (10.8%) Hispanic American, 80 (9.0%) African American, 53 (6.0%) Asian American, and 6 (0.7%) as “other ethnic racial groups.” The majority of the participants were single, never married (n = 785, 88.5%), 19 (2.1%) were married, 60 (6.8%) were engaged, seven (0.8%) were separated, four (0.5%) were divorced, and 12 (1.3%) reported “live-in” partner. The majority were freshmen (n = 705, 79.5%), 100 (11.3%) were sophomores, 52 (5.9%) were juniors, and 30 (3.3%) were seniors.

Measures and Procedure

Consistent with approvals from the institutional review boards, all the study participants volun- tarily completed written informed consent forms, a brief demographic questionnaire (assessing gender, age, ethnicity, year in college, and marital status), and the Depression Anxiety Stress Scales—21. senior research undergraduates, who were trained and supervised by the first au- thor, administered all questionnaire packets. All participants received partial course credit for completing the questionnaire packet.

Results

Exploratory Factor (Schmid-Leiman) Analysis

To examine (a) the dimensions of the DASS-21 and (b) the amount of common variance ac- counted for by the specific and general distress factors, we conducted an exploratory item

2To address concerns regarding missing data, responses to all individual questionnaire items were reviewed for completeness at the time of data collection.

1326 Journal of Clinical Psychology, December 2012

Table 1 Exploratory Factor Analysis and Schmid-Leiman Analysis of the DASS-21 Items

Oblique three-factor solution Schmid-Leiman bifactor

Item F1 F2 F3 GF F1 F2 F3

Factor 1 (F1; Depression) 3. no positive feeling .74 −.03 .05 .58 .49 −.02 .03 5. no initiatives .32 .07 .24 .50 .21 .05 .13 10. had nothing .76 −.03 .01 .56 .51 −.02 .01 13. down-hearted .58 −.10 .15 .49 .38 −.07 .08 16. unable about things .60 −.03 .10 .52 .40 −.02 .06 17. wasn’t worth much .82 .02 −.12 .53 .54 .01 −.07 21. life is meaningless .79 .03 −.18 .47 .52 .02 −.10 Factor 2 (F2; Anxiety) 2. dryness of mouth −.09 .57 −.04 .32 −.06 .39 −.02 4. breathing problem −.06 .80 −.11 .45 −.04 .54 −.06 7. trembling .04 .57 .09 .52 .02 .39 .05 9. worried .11 .35 .27 .57 .08 .24 .15 15. close to panic .15 .49 .23 .66 .10 .33 .13 19. action of my heart −.01 .71 −.12 .41 −.01 .49 −.07 20. felt scared .06 .45 .23 .57 .04 .31 .13

Factor 3 (F3; Stress) 1. hard to wind down −.16 −.03 .83 .55 −.11 −.02 .46 6. over-react .11. −.10 .69 .59 .08 −.07 .38 8. nervous energy −.10 .05 .80 .63 −.07 .04 .44 11. getting agitated .06 −.03 .71 .62 .04 −.02 .39 12. difficult to relax −.08 −.03 .85 .63 −.05 −.02 .47 14. intolerant .07 .10 .53 .57 .04 .07 .29 18. rather touchy .16 −.11 .66 .59 .11 −.07 .36

Note. DASS-21 = Depression Anxiety Stress Scale – 21; GF = general distress factor. Factor loadings ≥ .40 are set in bold.

bifactor analysis. The analysis was also designed to help provide additional information for de- riving scores on the DASS-21. As in most exploratory factor analyses with the Schmid-Leiman transformation, each item was allowed to load on a general distress factor and on one or more of the domain-specific factors. A higher order structure would be indicated to the extent that the general distress factor accounts for at least 20% of the composite variance (see Reckase, 1979).

Parallel analysis (95th percentile of random eigenvalues) of the matrix indicated that a three- factor solution could be extracted. Results of the principal axis factoring (PAF) analysis with promax rotation, followed by the Schmid-Leiman transformation are presented in Table 1. The first five eigenvalues from the preliminary analyses were as follows: 8.06, 1.85, 1.67, 1.11, and 0.91. Regarding the three-factor oblique solution (columns 2–4), most of the items had good loadings (i.e., values ≥ .40) on one of the domain-specific factors. However, the factor inter- correlations were high: stress versus depression (.63), stress versus anxiety (.61), and depression versus anxiety (.55).

Regarding results of the Schmid-Leiman transformation (columns 5–8), we found that the general distress factor accounted for the greatest proportion of the common variance among the DASS-21 item scores, 61.9%. The DASS-21 Depression factor (14.7%), the DASS-21 Stress factor (12.3%), and the DASS-21 Anxiety factor (11.1%) accounted for smaller amounts of the variance among the DASS-21 items. Using item-factor loadings ≥ .40 as salient, 20 of the 21 (95.2%) DASS-21 items loaded more substantially on the general distress factor than on the domain-specific factors that were identified in the exploratory factor analysis (EFA). Further analyses showed that the domain-specific DASS-21 Stress factor correlated with the general distress factor at .84. The domain-specific DASS-21 Depression factor correlated with

Depression Anxiety Stress 1327

the general distress factor at .75, and the domain-specific DASS-21 Anxiety factor correlated with the general distress factor at .73. The obtained (hierarchical) coefficient-ωH for the general distress factor of .87 was also high, suggesting that 87% of the variance for the composite DASS-21 score was due to the variance on the general distress factor.

Confirmatory Item Bifactor Analysis: Analysis of Item Responses

We used the item response theory (IRT) for Patient-Reported Outcomes (IRTPRO) 2.1 statistical program (Cai, du Toit, & Thissen, 2012) to conduct the analysis.3 As an IRT modeling program, IRTPRO 2.1 also allows for the evaluation of several characteristics of an item such as (a) the degree to which an individual item is specific or related to a specified construct (i.e., a slope parameter) and (b) the intercept of the item (i.e., c parameter). In the current study, however, we addressed the specific aims by focusing on (a) the magnitudes of the item-factor/dimension loadings and (b) the slope parameters (a-parameters). We considered items with moderate to high standardized loadings (i.e., values ≥ .40) and high slopes (values ≥ 1.50) to be strongly linked with (i.e., most discriminating) the specified dimension (see also, Kim & Pilknois, 1999; Reise & Waller, 1990).

In the analysis involving the item-bifactor modeling, we constrained each DASS-21 item to load onto a general distress factor and on only a domain-specific factor (see Table 2). The three domain-specific factors or dimensions were orthogonal to each other and to the general distress factor. As an example, for the domain-specific DASS-21 Depression dimension items, we constrained items 3, 5, 10, 13, 16, 17, and 21 to load on the DASS-21 Depression dimension (specificity) and the general distress factor shared by all the DASS-21 items, including the domain-specific DASS-21 Anxiety and DASS-21 Stress scale items (nonspecificity).

We used the Bock and Aitkin (BAEM; 1981) algorithm in IRTPRO 2.0 to conduct the analysis. Convergence criterion was attained with the maximum number of E-steps set at 500, and the quadrature (Q = 21) points were between -6.0 and 6.0. The result of the item bifactor analysis is presented in Table 2. Of note is that for multidimensional instruments such as the DASS-21, the threshold (b) parameters do not yield substantive interpretations. Thus, these parameters were not discussed further. As shown in the table, items 3, 10, 17, and 21 had the highest slopes and standardized loadings on the domain-specific factor, and are therefore most associated with the DASS-21 Depression dimension. However, these items were also strongly associated with the general distress (mixed) factor. Item 5 was least associated with the domain-specific DASS-21 Depression dimension; this item also had the lowest loading on this dimension. Taken together, item 21 provided the most information about the DASS-21 Depression dimension.

Examination of the slope parameters for the DASS-21 Anxiety dimension showed that items 4 and 19 were most strongly associated with the domain-specific Anxiety dimension. Items 7, 9, 15, and 20 were more strongly associated with the general distress dimension than with the domain-specific anxiety dimension. In particular, items 9 and 20 were least associated with the domain-specific DASS-21 Anxiety dimension; these items also had the lowest loadings on the domain-specific dimension. Item 4 provided the most important information about the domain-specific anxiety dimension. For the domain-specific DASS-21 Stress dimension, we found that items 1 and 12 were most strongly associated with this dimension. Each DASS- 21 Stress dimension item, however, was associated strongly with the general distress (mixed) factor. Items 6, 11, 14, and 18 were least associated with the domain-specific DASS-21 Stress dimension. In general, most of the Anxiety and Stress subscale items were strongly associated with the general distress dimension.

3We obtained similar results when we used the Metropolis Robbins-Monro (MH-RM) algorithm in IRTPRO. This is a new computer program that is designed to estimate unidimensional and multidimensional item response models (and related statistics). The program manual provides comprehensive information regarding several item parameter estimation methods (e.g., the Bock-Aitkin, 1981 for bifactor modeling), test statistics (e.g., RMSEA), and scale score computation strategies.

1328 Journal of Clinical Psychology, December 2012

Table 2 Bifactor (Confirmatory) Analyses of the DASS-21 Items (N = 887)

Standardized factor loading (slope parameter estimates)

General Depression Anxiety Stress Item Abbreviation factor (a1) factor (a2) factor (a3) factor (a4)

Factor 1 (Depression) 3. no positive feeling .70 (2.39) .51 (1.75) 5. no initiatives .64 (1.47) .19 (0.42) 10. had nothing .68 (2.44) .57 (2.05) 13. down-hearted .63 (1.70) .46 (1.23) 16. unable about things .63 (1.70) .45 (1.20) 17. wasn’t worth much .63 (2.16) .60 (2.06) 21. life is meaningless .65 (2.51) .62 (2.38)

Factor 2 (Anxiety) 2. dryness of mouth .32 (0.72) .56 (1.23) 4. breathing problem .51 (1.66) .69 (2.26) 7. trembling .63 (1.74) .47 (1.28) 9. worried .73 (1.87) .19 (0.50) 15. close to panic .83 (3.04) .30 (1.09) 19. action of my heart .46 (1.24) .62 (1.68) 20. felt scared .76 (2.20) .30 (0.87)

Factor 3 (Stress) 1. hard to wind down .59 (2.71) .71 (3.26) 6. over-react .80 (2.28) .12 (0.34) 8. nervous energy .75 (2.50) .43 (1.44) 11. getting agitated .81 (2.45) .17 (0.52) 12. difficult to relax .68 (3.45) .66 (3.37) 14. intolerant .77 (2.04) .08 (0.21) 18. rather touchy .81 (2.36) .07 (0.19)

Note. DASS-21 = Depression Anxiety Stress Scales. Estimates ≥ .40 are set in bold. Slope (a1-a4) parameter estimates of the latent dimensions are in parenthesis.

Scale Reliability Analysis and Descriptive Statistics (Means and Standard Deviations)

We interpreted reliability estimates of .70 to .79 as moderate, and estimates of .80 or greater as high for use of the scale scores in research settings (see Clark & Watson, 1995; Cicchetti, 1994). For the current study sample, the traditional coefficient-α estimates for the DASS-21 scale scores were as follows: DASS-21 Depression (α = .85; 95% confidence interval [CI], .83- .87, average inter-item correlation [AIC], .47); DASS-21 Anxiety (α = .81; 95% CI, .79-.84, AIC, .40); and DASS-21 Stress (α = .88; 95% CI, .87-.89, AIC, .52). The McDonald’s coefficient-ω for the DASS-21 Depression scale score was .86 (M = 4.18, SD = 3.60). The coefficient-ω for the DASS-21 Anxiety scale score was .82 (M = 2.93, SD = 3.38), and the coefficient-ω for the DASS-21 Stress scale score was .88 (M = 5.29, SD = 4.57). As expected, each scale reliability estimate was ≥ .80.

Study 2

First, we used confirmatory factor analysis to evaluate fit estimates of the bifactor (target) model against alternative models to the sample data. Second, we used both the coefficient-ω and the traditional Cronbach (coefficient-α) reliability analytic procedures to re-examine internal consistency reliability estimates of the DASS-21 scale scores in an independent sample. Third, based on empirical support obtained for the best-fitting model, we examined potential correlates for the DASS-21.

Depression Anxiety Stress 1329

Method

Participants and Procedure

Following the analyses in Study 1, we collected data from an independent sample of 410 un- dergraduate students in Midwestern and Southwestern state universities. The combined sample, with no statistically significant differences in demographic variables, comprised 242 women (M age = 19.95, SD = 3.15; range = 18–47 years) and 168 men (M age = 19.45, SD = 2.66; range = 18–39 years). The mean age of the sample was 19.65 years (SD = 2.88; range = 18–47); 259 (63.2%) self-reported as Caucasian/White, 90 (21.9%) as Hispanic/Latino American, 32 (7.8%) African American, 24 (5.9%) Asian American, and 5 (1.2%) indicated “other” ethnicities. The total sample comprised 305 (74.4%) freshmen, 52 (12.7%) sophomores, 33 (8.0%) juniors, and 20 (4.9%) seniors. After obtaining informed consent, participants completed the questionnaire packets in small groups for partial course credit.

Confirmatory factor analytic (CFA) sample. All the study participants (N = 410) provided data for inclusion in the descriptive statistics (mean and standard deviation) and CFA.

Concurrent validation subsample4. We included concurrent validation self-report ques- tionnaires in 223 (149 women and 74 men, aged 18–25 years) of the 410 (approximately 54%) study packets (see the Measures section). These measures have good psychometric properties for assessing dimensions of constructs that are related to those evaluated with the DASS-21:

� Anxiety: Beck Anxiety Inventory total score (Beck & Steer, 1990), the Mood and Anxiety Symptom Questionnaire-90 (MASQ-90; Watson, Clark et al., 1995), and Anxious Arousal scale score.

� Depression: Beck Depression Inventory-II total score (BDI-II; Beck, Steer, & Brown, 1996) and the MASQ-90 Anhedonic Depression scale score.

� Perceived stress: The Perceived Stress Scale total score (PSS; Cohen, Kamarck, & Mermel- stein, 1983).

Measures

Besides a brief background questionnaire and the DASS-21, all participants completed the Mood and Anxiety Symptom Questionnaire-90 (MASQ-90; Watson, Clark et al., 1995). The subsample (n = 223) completed the following concurrent validation self-report measures.

The Mood and Anxiety Symptom Questionnaire-90 (MASQ-90; Watson, Clark et al., 1995). We used the MASQ-90 as a concurrent validity measure of anxiety, depression, and general distress (mixed depression and anxiety symptoms). Participants rated the instrument items from 1 (not at all) to 5 (extremely). An example MASQ-90 item is “felt discouraged.” The MASQ-90 has been shown to have adequate estimates of reliability and concurrent validity (see Keogh & Reidy, 2000; Watson, Clark et al., 1995; Watson, Weber et al., 1995). As an example, in the concurrent validation subsample (n = 223), estimates of internal consistency for scores on the anhedonic depression (low positive affect; coefficient-α for 14 items = .94, 95% CI, .93-.95; AIC, .53), anxiety (anxious arousal, coefficient-α for 17 items = .87, 95% CI, .81-.90; AIC, .28), general distress depression (coefficient-α for 12 items = .91, 95% CI, .89-.93; AIC, .46), and general distress anxiety (coefficient-α for 11 items = .81, 95% CI, .76-.85; AIC, .28) were good.

4To ensure that participants in Study 2 did not participate in the Study 1 data collection process, we included a checklist assessing familiarity with studies conducted in our laboratory during the Study 1 data collection phase. The measures were also included randomly in each packet to control for order effects.

1330 Journal of Clinical Psychology, December 2012

The Perceived Stress Scale (PSS; Cohen, Kamarck, & Mermelstein, 1983)5. We used scores on the PSS, a 10-item self-report instrument, to assess perceptions of the individual to stress (negative affectivity) “in the past month.” Each PSS item is scored on a 5-point scale ranging from 0 (never) to 4 (very often). Four of the items are reverse-scored and the ratings summed on all the items to obtain a total scale score. Example PSS items are, “felt like you were on top of things” and “felt nervous and stressed.” Satisfactory estimates of internal consistency and concurrent validity have been established for scores on the PSS (see Hughes, 2007; Kohn & Gurevich, 1993). Cronbach coefficient-α estimate of the PSS total scale score for the current subsample (n = 223) was satisfactory at .87 (95% CI, .84-.89; AIC, .40). It is important to note that scores on stress-related measures have been useful in assessing negative affectivity (see Lucas, Michalopoulou, Falzarano, Menon, & Cunningham, 2008; Shewchuk, Elliott, MacNair- Semands, & Harkins, 1999).

The Beck Depression Inventory-II (BDI-II; Beck, Steer, & Brown, 1996). We used the BDI-II total scale score to assess severity of depressive symptoms. The BDI-II is a 21-item self-report measure; it includes a cluster of items such as “I am disappointed in myself” and “I dislike myself.” Scores on the 21 items are summed to derive a depressive symptom severity score. Adequate estimates of factor structure, internal consistency, and concurrent validity of the BDI-II total scale scores have been reported for undergraduate students (Dozois, Dobson, & Ahnberg, 1998; Osman, Kopper, Barrios, Gutierrez, & Bagge, 2004; Storch, Roberti, & Roth, 2004). For the subset of the study sample (n = 223), the Cronbach coefficient-α estimate was .90 (95% CI, .87-.92; AIC, .30).

The Beck Anxiety Inventory (BAI; Beck & Steer, 1990). We used the BAI total scale score to evaluate anxiety symptom severity in the study sample. The BAI is a 21-item self-report instrument that is composed of specific anxiety symptoms such as “feeling hot,” “scared,” and “faint.” Each BAI item is rated on a 4-point scale, ranging from 0 (not at all) to 3 (severely). The BAI total scale score has shown good estimates of internal consistency, test-retest reliability, and concurrent validity in clinical and nonclinical samples (Leyfer, Ruberg, & Woodruff-Borden, 2006; Osman et al., 2002). The BAI total score showed a good estimate of internal consistency reliability (n = 223), coefficient-α = .87 (95% CI, .84-.90; AIC, .24).

Results

CFA

We conducted a series of CFAs to determine whether the bifactor (target) model of emotions retained in Study 1 represents a good fit for the current sample data (N = 410). First, we evaluated the fit of a one-factor model by constraining all 21 items to load on a single factor (see illustrative Figure 1a). Second, we examined the fit of the original DASS-21 three-factor oblique model to represent the DASS model of emotions (see Figure 1b). Specifically, the Depression, Anxiety, and Stress items were constrained to load only on their respective original factors; the three first-order factors were allowed to be correlated. Because a good-fitting first- order correlated three-factor model would yield similar fit indices as a second-order model (see Figure 1c), we presented the same set of indices for both of these models (see Brown, 2006).6

Third, we conducted a bifactor analysis in which each of the 21 items was constrained to load on

5The PSS was included in the current study because of the large body of unpublished descriptive (mean and standard deviation) and psychometric data for nonclinical samples in our laboratory for this instrument. The obtained scores included in this report are similar to the unpublished PSS scales. 6We note that an alternate second-order model (equivalent to a bifactor model) could be derived by allowing the individual items to load directly from the second-order factor to each item on the general factor. However, a researcher would need to specify at least four orthogonal first-order factors for the second-order factor model prior to conducting the chi-square difference test of the potential models (see Chen et al., 2006 for detailed discussion of this model).

Depression Anxiety Stress 1331

Figure 1.a,b. CFA models for the DASS-21.

a general factor of negative (mixed) emotions and on a domain-specific factor (see Figure 1d). The general and domain-specific factors in the bifactor model were specified to be uncorrelated. The variance of the factor in each CFA model was set to 1.0 (see Flora & Curran, 2004; Floyd & Widaman, 1995).

To evaluate the goodness of fit of the proposed models, we examined several goodness-of-fit indices. Specifically, values of .90 or higher for the robust comparative fit index (CFI) and the Tucker Lewis index (TLI) were adopted as good fit to the data. Values of .06 or less for the robust root mean square error of approximation (RMSEA) and robust standardized root mean-square residual (SRMR) were adopted as indicative of general model fit to the data. Also, the model with the smallest Akaike information Criterion (AIC) was considered the “best-fitting” for the sample data (see Bentler & Bonett, 1980; Browne & Cudeck, 1993; Marsh, Hau, & Wen, 2004; Satorra & Bentler, 1994; Tucker & Lewis, 1973).

Preliminary analyses showed that responses on the DASS-21 items were not normal, Mardia’s normalized estimate = 72.27, p < .001. Therefore, we used the maximum likelihood mean- adjusted (MLM) robust estimator in the Mplus 6.11 program (Muthén & Muthén, 1998–2011) to conduct all the analyses.7 In the analysis involving the one-factor model, we found that this

7Because of questions regarding the performance of the Weighted least squares mean and variance (WLSMV) estimator when comparing complex models (e.g., bifactor and second-order), we used an alternative robust estimator that performs as well as the WLSMV estimator (e.g., Rast, Zimprich, Boxtel, & Jolles, 2009; Sass, 2011). As an example, Chen et al. (2006) observed that some fit statistics (e.g., RMSEA) obtained for complex models such as bifactor actually tend to become worse when using the WLSMV estimator. Furthermore, we note that, as demonstrated by Page et al. (2007), scoring the DASS-21 items as a four-factor or five-factor Likert-type scale does not substantially alter fit estimates for models in clinical and nonclinical samples.

1332 Journal of Clinical Psychology, December 2012

Figure 1.c,d. CFA models for the DASS-21.

model showed a poor fit to the sample data: Robust χ2 (189, N = 410) = 593.10, scaling = 1.44, CFI = .830, TLI = .812, SRMR = .067, AIC, 17,373.32 and RMSEA = .072 (90% CI, .066-.079). Consistent with the observation by Brown (2006), fit estimates for the oblique three- factor model and the second-order model were similar: Robust χ2 (186, N = 410) = 317.86, scaling = 1.44, CFI = .945, TLI = .938, SRMR = .047, AIC, 16,981.67 and RMSEA = .042 (90% CI, .034-.049). Although the oblique three-factor model attained good fit to the sample data, the links among the related factors were high: The Depression factor was associated with the Anxiety factor at .66, and with the Stress factor at .77. The Anxiety factor was associated with the Stress factor at .75.

Further analysis involving the bifactor model showed that this model also attained good fit to the sample data: Robust χ2 (168, N = 410) = 264.28, scaling = 1.43, CFI = .960, TLI = .949, SRMR = .040, AIC, 16,938.07, and RMSEA = .037 (90% CI, .028-.046). Although the second-order model can be conceptualized to be nested within the bifactor model, statistical comparison of these models is recommended only when the second-order model is defined by four or more first-order factors (see Bludworth, Tracey, & Glidden-Tracey, 2010; Brown, 2006; Chen, West, & Sousa, 2006). Additionally because of the small sample size, we used the AIC index to compare the fit of the non-nested oblique three-factor and bifactor models. As expected, the AIC for the bifactor model (16,938.07) was lower than the obtained AIC for the original oblique three-factor model (16,981.67), providing strong support for the adequacy of the bifactor model for the sample data.

Depression Anxiety Stress 1333

Table 3 Descriptive Statistics, Concurrent Estimates, and Coefficient Alpha Estimatesa

Measures DASS-21 total scoreb M SD Coefficient-α

Specific scales MASQ-Anhedonic Depression .65** 58.60 14.25 .94 MASQ-Anxious Arousal .50** 23.15 6.90 .87 Perceived Stress Scale .73** 17.52 7.41 .87

Nonspecific scales MASQ-General Distress Depression .68** 23.45 8.67 .91 MASQ-General Distress Anxiety .64** 18.42 5.64 .81 MASQ-Mixed Depression-Anxiety .73** 34.69 11.52

Depression severity Beck Depression Inventory-II .80** 11.01 8.11 .90

Anxiety severity Beck Anxiety Inventory .69** 8.57 7.06 .87

Note. DASS-21 = Depression Anxiety Stress Scales; M = mean; SD = standard deviation; MASQ = Mood and Anxiety Symptom Questionnaire. aN = 223. bZero-order correlations between the DASS-21 total scale score and the concurrent measures. **p < .001.

Descriptive Statistics, Reliability Analyses, and Scale Intercorrelations

The correlations among the DASS-21 scale scores ranged from .53 to .67. For the total sample (N = 410), the highest mean score was on the DASS-21 Stress scale (M = 5.56, SD = 4.52), followed by the DASS-Depression scale (M = 3.87, SD = 3.98), and the DASS-21 Anxiety scale (M = 3.18, SD = 3.38). The DASS-21 total scale had a mean score of 12.61 (SD = 10.23). The coefficient omega (ω) estimates for the DASS-21 total and scales in the full sample were good, .88 for the DASS-21 Depression, .83 for the DASS-21 Anxiety, and .85 for the DASS-21 Stress scale scores. For the DASS-21 total scale score, the obtained estimate was high, .89.

Correlates of the DASS-21 total Scale Score (n = 223) We conducted bivariate analyses to further examine potential correlates for the DASS-21 total scale score. Given the strong empirical evidence for deriving a total score for the DASS-21, we included only the DASS-21 total scale score in the analyses. In particular, as a unidimensional self-report instrument, we examined the extent to which the DASS-21 total scale score is related to the MASQ-90 total and the specific anhedonic depression and anxious arousal scales scores. In particular, we conducted significance tests to evaluate empirically the extent of some of these relationships (see Meng, Rosenthal, & Rubin, 1992).

As shown in Table 3, the highest correlates (i.e., rs ≥ .70) for the DASS-21 total scale score were the BDI-II (depression symptom severity, r = .80, p < .001), the MASQ-Mixed anxiety and depression (r = .73, p < .001), and the perceived stress scale (r = .73, p < .001) scores. In addition, the DASS-21 total scale score correlated highly (rs = .60 to .69) with the validation measures of anxiety symptom severity, general distress depression, general distress anxiety, and anhedonic depression. A moderate and statistically significant correlation was observed between the DASS-21 total scale score and the MASQ-90 anxious arousal scale score (r = .50, p < .05).

Regarding specificity of the correlates, the DASS-21 total scale score was more strongly linked with the MASQ-90 mixed depression-anxiety score than with the specific anhedonic depression measure score (z = 2.08, p < .04) or with the specific anxious arousal measure score (z = 5.44, p < .001). Furthermore, the DASS-21 total scale score was more strongly associated with the BDI-II scale score than with the BAI scale score (z = 3.10, p < .001).

1334 Journal of Clinical Psychology, December 2012

Discussion

The DASS-21 was constructed to assess the multiple dimensions of depression, anxiety, and stress (Lovibond & Lovibond, 1995). We conducted two studies to evaluate the specificity of individual items to each dimension, estimate scale reliability, and identify potential correlates of the DASS-21. In both studies, we found strong support for deriving a DASS-21 total scale score rather than independent scores for the hypothesized dimensions. First, taken together, a greater percentage of the items within each dimension had stronger associations with the general distress dimension than with the domain-specific dimension (e.g., depression). Second, in the exploratory bifactor analyses, the general distress factor accounted for a greater percentage of the variance (approximately 62%) in the sample data than did any of the domain-specific factors. Third, the confirmatory factor analytic procedures supported the bifactor model in finding that the majority of the DASS-21 items did not contribute uniquely to the proposed domain-specific dimensions. In fact, four or more of the items were not identified as specific to any group dimension. Taken together, these findings are consistent with Henry and Crawford’s (2005) analyses of DASS-21 data from a large British nonclinical sample, and with findings of structural indeterminacy in other multidimensional symptom report instruments (Leue & Beauducel, 2011; Vassend & Skrondal, 1999). Our findings suggest that the representativeness of the DASS-21 dimensions remains questionable (see Haynes et al., 1995).

Overall, as in the study by Henry and Crawford (2005), we used data from large samples of college-age students to examine the psychometric properties of the DASS-21. Given that clinical and nonclinical samples may differ in the specificity of responses to measures of emotion, the bifactor analytic procedure adopted in the current study should be extended to different clinical groups to assess the differential functioning of the DASS-21 items. For example, in their analyses of the Symptom Checklist (SCL)-90-R, Vassend and Skrondal (1999) found that the factorial structure of the instrument was different across levels of negative affectivity. Consequently, it would be important to explore the dimensionality of the DASS-21 in inpatient and outpatient psychiatric samples with varied diagnoses and levels of severity.

Review of the results from our current factor analyses, along with those reported in previous studies (e.g., Clara et al., 2001; Crawford & Henry, 2003; Henry & Crawford, 2005; Tully et al., 2009), suggests that four potential items might more clearly delineate each dimension in future revisions of the DASS-21. Specifically, items 3, 10, 13, and 17 should be considered potential Depression-specific items; items 2, 4, 7, and 19 should be considered potential Anxious Arousal- specific items; and items 1, 11, 12, and 18 should be considered potential Negative Affectivity- specific items. We note, however, that additional research is needed to replicate the current findings, and to generate relevant and representative sets of items for the DASS dimensions.

Building on Henry and Crawford’s (2005) investigation, we used both the traditional and contemporary scale reliability analytic procedures to evaluate estimates of internal consistency reliability for the DASS-21 total and scale scores. Estimates of internal consistency for the DASS- 21 total and scale scores were observed to be in the moderate to high ranges. In particular, the high estimates of internal consistency reliability reported frequently in the extant literature (i.e., values ≥ .90) for the DASS-21 total scale, compared with that of the proposed specific scale scores, raised additional questions about the independence of the individual DASS-21 scale scores (see Cicchetti, 1994; Clark & Watson, 1991; Gutierrez & Osman, 2008).

In Study 2, we examined further evidence for potential correlates of the DASS-21 total scale score in a series of correlation analyses. Consistent with the findings of the bifactor analyses, the DASS-21 total scale score correlated more highly with scores on a measure of mixed depression and anxiety than with scores on the specific anhedonic depression or anxious arousal scales of the MASQ-90. Antony et al. (1998) reported similar patterns of correlates for the DASS-21 scale scores in a clinical sample.

Conclusions and Directions for Future Research

Our investigations of the DASS-21 with large samples of nonclinical U.S. college students found strong evidence for the presence of a general distress factor that accounted for a majority of

Depression Anxiety Stress 1335

the variance in both the DASS-21 items and total score. Clinically, these findings are consistent with the comorbidity of mood and anxiety disorders that is frequently observed in clinical populations (e.g., Antony et al., 1998; Clark & Watson, 1991). As noted previously, it would be important to investigate the items’ and scales’ ability to discriminate between known groups of clinical patients at varying levels of symptom severity.

The current studies were the first to explore the structure and psychometric properties of the DASS-21 in large samples of nonclinical U.S. college students. The use of a bifactor-analytic procedure provided us with the opportunity to evaluate the unique contributions of each item to a general and a domain-specific factor. We were also able to identify 12 potential items that could be included in a future revision of the DASS-21. Several other issues should be addressed in future studies with the DASS-21. First, our college samples were not demographically representative of the general U.S. college-age population. Consequently, it is unclear how the structure of the DASS-21 would be represented when using data from more diverse populations (see Daza et al., 2002; Norton, 2007). Second, we relied solely on self-reported data in all the analyses; thus, the findings could be accounted for by method variance (see Gutierrez & Osman, 2008; Lonigan, Carey, & Finch, 1994). Future studies should obtain data from multiple sources, including direct observation and semistructured interviews, when investigating the properties of the DASS-21 or its revision.

Third, the DASS-21 is a theoretically relevant measure of negative emotions that include mixed symptoms of depression, anxiety, and stress “over the past week.” However, to optimize its clinical utility, instructions for completing the DASS-21 items might be revised to include extended (e.g., “over the past 2 weeks”) timeframes (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision; American Psychiatric Association, 2000). Similarly, both of our studies were cross-sectional in nature. It would be important to investigate the stability of the DASS-21 dimensions over time. Overall, findings from the current and previous studies suggest that additional investigations are needed to provide stronger support for the psychometric properties of the DASS-21 and its clinical utility compared with other established instruments.

References

Allen, J., & Annells, M. (2009). A literature review of the application of the Geriatric Depression Scale, Depression Anxiety Stress Scales and Post-traumatic Stress Disorder Checklist to community nursing cohorts. Journal of Clinical Nursing, 18, 949–959.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric properties of the 42-item and 21-item versions of the depression anxiety stress scales in clinical groups and a community sample. Psychological Assessment, 10, 176–181.

Beck, A. T., & Steer, R. A. (1990). Manual for the Beck Anxiety Inventory. San Antonio, TX: Psychological Corporation.

Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the revised Beck Depression Inventory-II. San Antonio, TX: Psychological Corporation.

Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606.

Bludworth, J. L., Tracey, T. J. G., & Glidden-Tracey, C. (2010). The bilevel structure of the Outcome Questionnaire-45. Psychological Assessment, 22, 350–355.

Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459.

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

Brown, T. A., Chorpita, B. F., Korotitsch, W., & Barlow, D. (1997). Psychometric properties of the depression anxiety stress scales (DASS) in clinical samples. Behaviour Research Therapy, 35, 79–89.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage.

1336 Journal of Clinical Psychology, December 2012

Cai, L., du Toit, S. H. C., & Thissen, D. (2012). IRTPRO 2.1: Flexible, multidimensional, multiple categorical IRT modeling [Computer software]. Chicago, IL: Scientific Software International.

Chen, F. F., West, S. G., & Sousa, K. H. (2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189–225.

Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.

Clara, I. P., Cox, B., & Enns, M. W. (2001). Confirmatory factor analysis of the Depression-Anxiety Scales in depressed and anxious patients. Journal of Psychopathology and Behavioral Assessment, 23, 61–67.

Clark, L. A., & Watson, D. (1990). Theoretical and empirical issues in differentiating depression from anxiety. In Becker, J. & Kleinman, A. (Eds.), Advances in mood disorders, Vol. I. Psychosocial aspects of depression. Hillsdale, NJ: Erlbaum.

Clark, L. A., & Watson, D. (1991). Tripartite model of anxiety and depression: Psychometric evidence and taxonomic implications. Journal of Abnormal Psychology, 100, 316–336.

Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309–319.

Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 385–396.

Cook, K. F., Kallen, M. A., & Amtmann, D. (2009). Having a fit: Impact of number of items and distribution of data on traditional criteria for assessing IRT’s unidimensionality assumption. Quality of Life Research, 18, 447–460.

Crawford, J. R., & Henry, J. D. (2003). The Depression Anxiety Stress Scales (DASS): Normative data and latent structure in a large non-clinical sample. British Journal of Clinical Psychology, 42, 111–131.

Daza, P., Novy, D. M., & Stanley, M. A. (2002). The Depression Anxiety Stress Scale-21: Spanish translation and validation with a Hispanic sample. Journal of Psychopathology and Behavioral Assessment, 24, 195– 205.

Dozois, D. J. A., Dobson, K. S., & Ahnberg, J. L. (1998). A psychometric evaluation of the Beck Depression Inventory-II. Psychological Assessment, 10, 83–89.

Duffy, C. J., Cunningham, E. G., & Moore, S. M. (2005). Brief Report: The factor structure of mood states in an early adolescent sample. Journal of Adolescence, 28, 677–680.

Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466–491.

Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299.

Gloster, A. T., Rohades, H. M., Novy, D., Klotsche, J., Senior, A., Kunik, M. . . . Stanley, M. A. (2008). Psychometric properties of the Depression Anxiety and Stress Scale-21 in older primary care patients. Journal of Affective Disorders, 110, 248–259.

Gutierrez, P. M., & Osman, A. (2008). Adolescent suicide: An integrated approach to the assessment of risk and protective factors. DeKalb: Northern Illinois University Press.

Haynes, S. N., Richard, D., & Kubany, E. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7, 238–247.

Henry, J. D., & Crawford, J. R. (2005). The short-form version of the Depression Anxiety Stress Scales (DASS-21): Construct validity and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 44, 227–239.

Hughes, B. M. (2007). Social support at university scale, a brief index. Psychological Reports, 100, 76–82.

Keogh, E., & Reidy, J. (2000). Exploring the factor structure of the Mood and Anxiety Symptom Question- naire (MASQ). Journal of Personality Assessment, 74, 106–125.

Kim, Y., & Pilknois, P. (1999). Selecting the most informative items in the IIP scales for personality disorders: An application of item response theory. Journal of Personality Disorders, 13, 157–174.

Kohn, P., & Gurevich, M. (1993). On the adequacy of the indirect method of measuring the primary appraisal of hassles-based stress. Personality and Individual Differences, 14, 679–684.

Leue, A., & Beauducel, A. (2011). The PANAS structure revised: On the validity of a bifactor model in community and forensic samples. Psychological Assessment, 23, 215–225.

Leyfer, O. T., Ruberg, J. L., & Woodruff-Borden, J. (2006). Examination of the utility of the Beck Anxiety Inventory and its factors as a screener for anxiety disorders. Journal of Anxiety Disorders, 20, 444– 458.

Depression Anxiety Stress 1337

Lonigan, C. J., Carey, M. P., & Finch, A. J. (1994). Anxiety and depression in children and adolescents: Negative affectivity and utility of self-reports. Journal of Consulting and Clinical Psychology, 62, 1000– 1008.

Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33, 335–343.

Lucas, T., Michalopoulou, G., Falzarano, P., Menon, S., & Cunningham, W. (2008). Healthcare provider cultural competency: Development and initial validation of a patient report measure. Health Psychology, 27, 185–193.

Marsh, H. W., Hau, K-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11, 320–341.

McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

Meng, X., Rosenthal, R., & Rubin, D. B. (1992). Comparing correlated correlations. Psychological Bulletin, 111, 172–175.

Mineka, S., Watson, D., & Clark, L. A. (1998). Comorbidity of anxiety and unipolar mood disorder. Annual Review of Psychology, 49, 377–412.

Muthén, L. K., & Muthén, B. O. (1998–2011). Mplus 6.1 [Computer software]. Los Angeles, CA: Author.

Ng, F., Trauer, T., Dodd, S., Callaly, T., Campbell, S., & Berk, M. (2007). The validity of the 21-item version of the Depression Anxiety Stress Scales as a routine clinical outcome measure. Acta Neuropsychiatrica, 19, 304–310.

Norton, P. (2007). Depression Anxiety and Stress Scales (DASS-21): Psychometric analysis across four racial groups. Anxiety, Stress & Coping: An International Journal, 20, 253–265.

Osman, A., Hoffman, J., Barrios, F. X., Kopper, B. A., Breitenstein, J. L., & Hahn, S. K. (2002). Factor structure, reliability, and validity of the Beck Anxiety Inventory in adolescent psychiatric inpatients. Journal of Clinical Psychology, 58, 443–456.

Osman, A., Kopper, B. A., Barrios, F. X., Gutierrez, P. M., & Bagge, C. L. (2004). Reliability and validity of the Beck Depression Inventory—II with adolescent psychiatric inpatients. Psychological Assessment, 16, 120–132.

Page, A. C., Hooke, G. R., & Morrison, D. L. (2007). Psychometric properties of the Depression Anxiety Stress Scales (DASS) in depressed clinical samples. British Journal of Clinical Psychology, 46, 283– 297.

Psychology Foundation of Australia. (2011). Depression Anxiety Stress Scale – DASS. Retrieved from http://www2.psy.unsw.edu.au/groups/dass//

Rast, P., Zimprich, D., Boxtel, M. V., & Jolles, J. (2009). Factor structure and measurement invariance of the Cognitive Failures Questionnaire across the adult life span. Assessment, 16, 145–158.

Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy, 35, 299–331.

Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4, 207–230.

Reise, S. P., Morizot, J., & Hays, R. D. (2007). The role of bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16, 19–31.

Reise, S. P., & Waller, N. G. (1990). Fitting the two-parameter model to personality data. Applied Psycho- logical Measurement, 14, 45–58.

Sanderson, W. C., Di Nardo, P. A., Rapee, R. M., & Barlow, D. H. (1990). Syndrome comorbidity in patients diagnosed with a DSM-III-R anxiety disorder. Journal of Abnormal Psychology, 99, 308–312.

Sass, D. A. (2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29, 347–363.

Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. VonEye & C. C. Clogg (Eds.), Latent variable analysis: Applications for developmental research (pp. 399–519). Thousand Oaks, CA: Sage.

Schmid, J., & Leiman, J. M. (1957). The development of hierarchical factor solutions. Psychometrika, 22, 53–61.

Seligman, L. D., & Ollendick, T. H. (1998). Comorbidity of anxiety and depression in children and adoles- cents: An integrative review. Clinical Child and Family Psychology Review, 1, 125–144.

1338 Journal of Clinical Psychology, December 2012

Shewchuk, R. M., Elliott, T. R., MacNair-Semands, R. R., & Harkins, S. (1999). Trait influences on stress appraisal and coping: An examination of alternative frameworks. Journal of Applied Social Psychology, 29, 685–704.

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychome- trika, 74, 107–120.

Simms, L. J., Grös, D. F., Watson, D., & O’Hara, M. (2008). Parsing the general and specific components of depression and anxiety with bifactor modeling. Depression and Anxiety, 25, E34–E46.

Smith, G. T., & McCarthy, D. M. (1995). Methodological considerations in the refinement of clinical assessment instruments. Psychological Assessment, 7, 300–308.

Smith, G. T., McCarthy, D. M., & Anderson, K. G. (2000). On the sins of short-form development. Psychological Assessment, 12, 102–111.

Storch, E. A., Roberti, J. W., & Roth, D. A. (2004). Factor structure, concurrent validity, and internal consistency of the Beck Depression Inventory-Second Edition in a sample of college students. Depression and Anxiety, 19, 187–189.

Sukantarat, K. T., Williamson, R. C. N., & Brett, S. J. (2007). Psychological assessment of ICU survivors: A comparison between the Hospital Anxiety and Depression scale and the Depression, Anxiety and Stress Scale. Anaesthesia, 62, 239–243.

Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psy- chometrika, 38, 1–10.

Tully, P. J., Zajac, I. T., & Venning, A. J. (2009). The structure of anxiety and depression in a normative sample of younger and older Australian adolescents. Journal of Abnormal Child Psychology, 37, 717–726.

Vassend, O., & Skrondal, A. (1999). The problem of structural indeterminacy in multidimensional symptom report instruments. The case of SCL-90-R. Behaviour Research & Therapy, 37, 685–701.

Watson, D., Clark, L. A., Weber, K., Assenheimer, J. S., Strauss, M. E., & McCormick, R. A. (1995). Testing a tripartite model: II. Exploring the symptom structure of anxiety and depression in student, adult, and patient samples. Journal of Abnormal Psychology, 104, 15–25.

Watson, D., Weber, K., Assenheimer, J. S., Clark, L. A., Strauss, M. E., & McCormick, R. A. (1995). Testing a tripartite model. I. Evaluating the convergent and discriminant validity of anxiety and depression symptom scales. Journal of Abnormal Psychology, 104, 3–14.

Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficient alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6, 21–29.

be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written

permission. However, users may print, download, or email articles for individual use.