Ethical and Professional Issues in Psychology Testing

Title:

Database:

A comparison of three self-report measures of intolerance of uncertainty: An examination of structure and incremental explanatory power in a community sample. By: Fergus, Thomas A., Psychological Assessment, 10403590, 20131201, Vol. 25, Issue 4

APA PsycArticles

A Comparison of Three Self-Report Measures of Intolerance of Uncertainty: An Examination of Structure and Incremental Explanatory Power in a Community Sample

By: Thomas A. Fergus Baylor University; Acknowledgement:

Intolerance of uncertainty (IU) is an individual difference variable that has garnered increased interest from clinical researchers (Birrell, Meares, Wilkinson, & Freeston, 2011; Carleton, 2012). A seminal operational definition of IU within the clinical literature came from Freeston, Rhéaume, Letarte, Dugas, and Ladouceur (1994), who opined that IU represented “cognitive, emotional, and behavioral reactions to uncertainty in everyday life situations” (p. 792). A number of refinements to Freeston et al.’s definition of IU have been put forth in the literature (see Birrell et al., 2011; Carleton, 2012), with Dugas and Robichaud (2007) recently defining IU as “a dispositional characteristic that results from a set of negative beliefs about uncertainty and its implications” (p. 24). Carleton sought to identify a common theme across extant definitions of IU

!Listen"

UAGC Library

and noted that IU fundamentally represents a dispositional fear of the unknown. In taxometric studies, IU has been identified as being nontaxonic (Carleton, Weeks, et al., 2012; Olatunji, Broman-Fulks, Bergman, Green, & Zlomke, 2010), indicating that IU is continuous and should be assessed using the full range of available scores.

IU was once thought to be specific to generalized anxiety disorder (GAD), worry in particular (see Koerner & Dugas, 2006). However, the existing literature currently supports the possibility that IU spans across emotional disorders (i.e., IU appears to be transdiagnostic; Carleton, Mulvogue, et al., 2012). To understand how IU relates to various clinical phenomena, it is of course necessary to have valid assessment tools. To date, the most commonly used measure of IU has been the 27-item Intolerance of Uncertainty Scale (IUS) developed by Freeston et al. (1994). One limitation of the IUS is that a number of divergent factor structures have been found for the 27 IUS items (Berenbaum, Bredemeier, & Thompson, 2008; Buhr & Dugas, 2002; Freeston et al., 1994; Norton, 2005). Norton (2005) noted that the factor structure of the IUS items might be improved through item removal. Carleton, Norton, and Asmundson (2007) later developed a 12-item short form of the IUS, labeled the IUS–12. Carleton et al. found that scores on the IUS–12 correlated strongly with scores on the full-length IUS (r = .96), and scores on both versions of the IUS evidenced statistically equivalent correlations with key criteria. Carleton et al. identified a two-factor structure for the IUS–12 items, with McEvoy and Mahoney (2011) advocating for labeling these two factors Prospective IU and Inhibitory IU.

Sexton and Dugas (2009) critiqued Carleton et al.’s (2007) approach in developing the IUS–12, noting that content was not a chief consideration when selecting the short-form items and thus the IUS–12 might not capture the IU construct in a manner analogous to the full-length IUS. Addressing limitations of prior research examining the factor structure of the full-length IUS items (e.g., relatively small samples), Sexton and Dugas identified a replicable two-factor full- length IUS solution. The two factors were labeled Uncertainty Has Negative Behavioral and Self- Referent Implications and Uncertainty Is Unfair and Spoils Everything. Despite Sexton and Dugas’s (2009) concern as to how the IUS–12 was developed, their full-length IUS factors strongly paralleled Carleton et al.’s (2007) IUS–12 factors. In fact, all of the items from Carleton et al.’s Prospective IU factor loaded on Sexton and Dugas’s Uncertainty Is Unfair and Spoils Everything factor, and all of the items from Carleton et al.’s Inhibitory IU factor loaded on Sexton and Dugas’s Uncertainty Has Negative Behavioral and Self-Referent Implications.

Studies have been conducted to directly compare Sexton and Dugas’s (2009) full-length and Carleton et al.’s (2007) short-form versions of the IUS. In one study, Khawaja and Yu (2010) found that scores on the corresponding IUS full-length and short-form scales evidenced strong

correlations (rs ranging from .90 to .97), and scores on these sets of scales showed a similar pattern of correlations with theoretically relevant criteria. Despite some results indicating that test scores on one of Sexton and Dugas’s full-length IUS scale evidenced more favorable discriminative validity relative to test scores on the corresponding scale of the IUS-12, Khawaja and Yu concluded that “clinicians and researchers may choose either version without any serious limitations” (p. 105). In another study, McEvoy and Mahoney (2011) found that Carleton et al.’s IUS–12 two-factor solution, but not Sexton and Dugas’s full-length IUS two-factor solution, demonstrated an adequate fit to their data using confirmatory factor analyses (CFAs). In reviewing available factor analytic studies, Birrell et al. (2011) concluded that the 12 items of the IUS–12 have most consistently loaded on the same factors across studies, and Carleton et al.’s two-factor solution was supported by Birrell et al. (2011) as their proposed factor structure of the IUS items. Given its brevity and strong convergence with scores on the full-length IUS, the IUS–12 appears to be viable substitute for the full-length IUS and is the focus of the present research.

Carleton (2012) stated that the “IUS–12 has been designed specifically to research the core aspects of IU across different populations and different disorders” (p. 941). At the same time, additional self-report measures that purportedly assess IU exist in the clinical literature, and interrelations among the IUS–12 and these other measures have yet to be thoroughly examined. One additional measure of IU in the clinical literature is the Obsessive Beliefs Questionnaire (OBQ), which was developed by Steketee, Frost, and the Obsessive Compulsive Cognitions Working Group (OCCWG; 2001) to assess IU and related dysfunctional beliefs. According to the OCCWG (2001), IU represents “beliefs about the necessity for being certain, that one has poor capacity to cope with unpredictable change, and that it is difficult to function adequately in ambiguous situations” (p. 1004). Within the OBQ, IU items have historically loaded on the same factor as items assessing perfectionism (Steketee, Frost, & OCCWG, 2005 [OCCWG, 2005]). This pattern of factor loadings led researchers to label this scale Perfectionism/Certainty (P/C). The OCCWG (2005) defined P/C as reflecting “high, absolute standards of completion, rigidity, concern over mistakes, and feelings of uncertainty” (p. 1532). Moulding et al. (2011) recently developed a 20-item short form of the OBQ, labeled the OBQ–20, to provide researchers with an economical assessment of the targeted dysfunctional beliefs. Pursuant to the present research, Moulding et al. (2011) found that the OBQ–20–P/C items were best represented by a single factor and that scores of the OBQ–20–P/C strongly converged with scores of a full-length version of the OBQ–P/C (rs of .95 and .96). Moulding et al. (2011) also found that scores on the OBQ–20–P/C and full-length OBQ–P/C shared nearly identical correlations with criteria. Given its brevity and strong convergence with the full-length OBQ–P/C, the OBQ–20–P/C was used in the present research. To date, researchers have found that scores on the IUS and the OBQ–P/C moderately intercorrelate (rs of .55 and .57; Fergus & Wu, 2011). Such intercorrelations indicate

that these two measures are not redundant and likely assess IU in a somewhat distinct manner (Gentes & Ruscio, 2011).

The Intolerance of Uncertainty Index (IUI) is a self-report measure that was recently developed by Gosselin et al. (2008) to address purported limitations of the IUS. Gosselin et al. asserted that the items of the IUS tap general reactions to uncertainty rather than tolerance or acceptance of uncertainty. The IUI consists of two separate parts, but as noted by Gosselin et al., only the items of Part A of the IUI (i.e., IUI–A) assess IU in a manner consistent with how this construct has historically been defined in the literature (e.g., Dugas, Gosselin, & Ladouceur, 2001). Therefore, the IUI–A is the only part of the measure considered in this article. The content of the IUI–A items converges with Gosselin et al.’s (2008) stated attempt to assess IU as a tolerance/acceptance of uncertainty (e.g., Not knowing what will happen in advance is often unacceptable for me). Scores on the IUS and IUI–A strongly intercorrelate (r = .68; Gosselin et al., 2008), although the magnitude of this correlation does not suggest redundancy between the two measures. Carleton, Gosselin, and Asmundson (2010) validated an English version of the IUI–A, finding that the IUI–A items were best represented by a single factor. Carleton et al. (2010) called for future research to compare relations between the IUI–A and other measures of IU in an attempt to explicate the potential differential utility of these available options for assessing IU.

Indeed, despite IU receiving increased attention from researchers, little systematic research has been completed to directly compare existing self-report measures of IU. The lack of data speaking to interrelations among available measures of IU represents a gap in the existing literature, as it is currently unknown whether the measures all can be conceptualized as assessing the same construct. The different definitions used by researchers when creating the items of the IUS-12, OBQ–P/C, and IUI–A, coupled with the relatively modest interrelations among scores of these measures, suggests that it is possible that these measures assess distinct aspects of IU. The structure of the IUS–12, OBQ–P/C, and IUI–A items was examined in the present research to study this possibility. The first aim of the present research was to examine whether items of these IU measures were factorially distinct. If the items of the IUS–12, OBQ–P/C, and IUI–A were factorially distinct, the next aim of the present research was to investigate whether the distinct content assessed by these items was nonetheless representative of the same higher order construct (i.e., IU). Such a pattern of findings would support the notion that each of the three targeted measures assesses a distinct aspect of IU.

A final aim of the present research was to further investigate the distinctiveness of the IUS–12, OBQ–P/C, and IUI–A by examining unique relations between the three targeted measures and

symptom measures of emotional disorders. Although typically used to support the validity of test scores, the concept of incremental validity can also have important conceptual implications (Hunsley, 2003). In the context of the present research, finding a unique relation between each IU measure and symptom measure after accounting for variance shared across IU measures would suggest that Carleton’s (2012) notion that the IUS–12 assess the core aspects of IU across different disorders might be broadened to consider the possibility that the IUS–12, OBQ– P/C, and IUI–A each assess core aspects of IU that span across different disorders. Finding relatively robust relations between scores on certain IU measures and symptom measures might also shed light onto operational definitions of IU that are particularly relevant to symptoms of specific emotional disorders. Depression, generalized anxiety, and obsessive–compulsive symptoms were targeted in the present research, as these three symptom types have garnered particular interest from IU researchers (Gentes & Ruscio, 2011).

A higher order CFA approach, following Brown (2006), was used in the present research to examine the structure of the items of the IUS–12, OBQ–P/C, and IUI–A. This data analytic approach allowed for an examination of the initial two study aims outlined earlier. As recommended by Brown (2006), an adequate first-order model was initially fit to the data before examination for the presence of a higher order factor was performed. When testing the adequacy of first-order models, it was predicted that a four-factor model—in which the items of IUS–12–Prospective, IUS–12–Inhibitory, OBQ–P/C, and IUI–A loaded on separate, yet intercorrelated, factors—would provide an adequate fit to the data and a better fit to the data relative to alternative first-order factor structures. This prediction was based on the reviewed factor analytic studies of the targeted measures, the differences in operational definitions used by researchers when creating the items of each measure, and the relatively modest magnitude of interrelations among the scores of the measures. It was next predicted that a higher order factor would account for the intercorrelations among the four first-order factors. Despite the expectation that items of the IUS–12–Prospective, IUS–12–Inhibitory, OBQ–P/C, and IUI–A would be distinct, the presence of a higher order factor would support the notion that these measures are all representative of the same overarching construct (i.e., IU). A higher order factor was predicted to account for the latent factor correlations, as each of the targeted measures was developed to assess IU.

Structural equation modeling (SEM) was then used to examine the incremental explanatory power of each measure in the concurrent prediction of depression, generalized anxiety, and obsessive–compulsive symptoms. Using SEM provides advantages compared with using scaled scores, including the ability to account for scale unreliability (Brown, 2006). Because of such advantages, using SEM in the present research was expected to provide clearer estimates of

interrelations among the IU measures and symptom types than what would be expected using scaled scores. These tests of incremental explanatory power were considered exploratory.

Method

Participants The sample consisted of 624 adults recruited through the Internet. The mean age of the sample was 33.1 years (SD = 11.4; range from 18 to 71). Respondents primarily self-identified as female (56.1%), having received at least a 2-year college degree (55.9%), working at least part-time (68.4%), and being currently unmarried (66.9%). In terms of racial/ethnic identification, 79.2% self-identified as White, 6.6% self-identified as African American, 6.4% self-identified as Asian, 3.7% self-identified as Hispanic, 3.5% self-identified as bi- or multiracial, and 0.6% self-identified as “other” race/ethnicity.

Measures Intolerance of Uncertainty Scale–12-item version (IUS–12; Carleton et al., 2007)

As introduced, the IUS–12 is a 12-item short form of the full-length 27-item IUS (Freeston et al., 1994; English translation: Buhr & Dugas, 2002). The IUS–12 consists of seven items that assess Prospective IU (e.g., Unforeseen events upset me greatly) and five items that assess Inhibitory IU (e.g., When I am uncertain, I can’t function very well). The IUS–12 items are rated on a 5-point scale (ranging from 1 to 5). In previous research, the test scores on the Prospective IU and Inhibitory IU scales demonstrated adequate internal consistency (Cronbach’s αs of .85 and .88; Carleton et al., 2007; McEvoy & Mahoney, 2011). Further, Khawaja and Yu (2010) found that test scores on the IUS–12 evidenced satisfactory test–retest reliability (2-week r = .77). Test scores on the IUS–12 scales (total: M = 35.22, SD = 8.83; Prospective IU: M = 22.41, SD = 5.06; Inhibitory IU: M = 12.81, SD = 4.56) demonstrated good internal consistency in the present research (Cronbach’s αs ranging from .86 to .91). The average interitem r for test scores on the Prospective IU scale was .46 (ranged from .35 to .59), and the average interitem r for test scores on the Inhibitory IU scale was .62 (ranged from .54 to .75) in the present research.

Obsessive Beliefs Questionnaire–20-item version (OBQ–20; Moulding et al., 2011)

As introduced, the OBQ-20 is a 20-item short form of the 44-item version of the OBQ (Steketee, Frost, & OCCWG, 2005). The OBQ–20–PC—the OBQ–20 scale of interest in the present research—consists of five items that assess for perfectionism and certainty (e.g., For me, things

are not right if they are not perfect). The OBQ–20 items are rated on a scale ranging from 1 to 7. In previous research, test scores on the OBQ–20–P/C have demonstrated adequate internal consistency given the brevity of the scale (Cronbach’s αs of .78 and .81; Moulding et al., 2011). Test scores on full-length versions of this scale also have evidenced satisfactory test–retest reliability (2-month to 3-month rs ranging from .66 to .77; Steketee, Frost, & OCCWG, 2003). Test scores on the OBQ–20–P/C (M = 18.50, SD = 7.16) evidenced adequate internal consistency in the present research (α = .83). The average interitem r for test scores on the OBQ–20–P/C was .49 (ranged from .35 to .68) in the present research.

Intolerance of Uncertainty Index–Part A (IUI–A; Gosselin et al., 2008; English translation: Carleton et al., 2010)

As introduced, the IUI–A is a 15-item measure that assesses difficulties tolerating or accepting uncertainty (e.g., I have difficulty tolerating life’s uncertainties). IUI–A items are rated using a scale ranging from 1 to 5. Previous research has shown that test scores on the IUI–A demonstrated good internal consistency (Cronbach’s αs of .94 and .96; Carleton et al., 2010; Gosselin et al., 2008), and Gosselin et al. (2008) found that test scores on the IUI–A evidenced satisfactory test–retest reliability (5-week r = .76). Test scores on the IUI–A (M = 40.99, SD = 14.10) evidenced good internal consistency in the present research (α = .96). The average interitem r for test scores on the IUI–A was .63 (ranged from .40 to .82) in the present research.

Depression, Anxiety, and Stress Scales–21-item version (DASS–21; Lovibond & Lovibond, 1995)

The DASS–21 is a 21-item short-form version of the original 42-item version of the DASS (Lovibond & Lovibond, 1995) that assesses depression, anxiety, and stress symptoms the respondent has experienced over the previous week. The Depression scale of the DASS-21— the DASS-21 scale of interest in the present research—consists of seven items (e.g., I couldn’t seem to experience any positive feeling at all) that are rated using a scale ranging from 0 to 3. Antony, Bieling, Cox, Enns, and Swinson (1998) found that the depression items of the DASS– 21 significantly loaded on a single factor that was distinct from the anxiety and stress factors of the measure. Antony et al. (1998) further found that test scores on the Depression scale of the DASS–21 demonstrated good internal consistency (α = .94) and correlated strongly with scores on other symptom measures of depression (r = .79). Prior research has shown that test scores on the full-length Depression scale evidenced satisfactory test–retest reliability (2-week r = .71; Brown, Chorpita, Korotitsch, & Barlow, 1997). Test scores on the Depression scale of the DASS–21 (M = 5.93, SD = 5.59) evidenced good internal consistency in the present research (α

= .93). The average interitem r for test scores on the Depression scale of the DASS–21 was .64 (ranged from .51 to .74) in the present research.

Generalized Anxiety Disorder–7 (GAD–7; Spitzer, Kroenke, Williams, & Lowe, 2006)

The GAD–7 is seven-item measure that assesses generalized anxiety symptoms the respondent has experienced over the previous 2 weeks (e.g., Worrying too much about different things). Spitzer et al. (2006) found that the GAD–7 items all significantly loaded on a single factor and that test scores on the GAD–7 evidenced good internal consistency (α = .92) and satisfactory test–retest reliability (1-week r = .83). Scores on the GAD–7 correlate strongly (r = .64) with scores on another measure of generalized anxiety symptoms (Kertz, Bigda-Peyton, & Bjorgvinsson, in press). Test scores on the GAD–7 (M = 6.94, SD = 5.46) evidenced good internal consistency in the present research (α = .91). The average interitem r for test scores on the GAD–7 was .59 (ranged from .45 to .84) in the present research.

Dimensional Obsessive–Compulsive Scale (DOCS; Abramowitz et al., 2010)

The DOCS is a 20-item measure that assesses the severity of obsessive–compulsive symptoms using a scale ranging from 0 to 4. The four symptom dimensions assessed by the DOCS are contamination, responsibility for harm, unacceptable thoughts, and symmetry. Each DOCS scale assesses for the time spent, avoidance, distress, interference, and attempts of control surrounding the respective symptom dimension. Abramowitz et al. (2010) found that a four- factor solution best represented the items of the DOCS and that test scores on the DOCS scales evidenced good internal consistency (αs ranging from .83 to .96) and satisfactory test–retest reliability (5-week rs ranging from .55 to .66). Abramowitz et al. (2010) also found that scores on the DOCS scales correlated moderately to strongly (rs ranging from .39 to .88) with scores of other symptom measures that assess corresponding symptom dimensions. Test scores on each DOCS scale (Contamination: M = 2.79, SD = 3.34; Responsibility: M = 3.40, SD = 3.89; Unacceptable Thoughts: M = 3.51, SD = 4.11; Symmetry: M = 3.14, SD = 3.95) demonstrated good internal consistency in the present research (αs ranging from .86 to .93). The average interitem r for test scores on each of the DOCS scales in the present research was as follows: Contamination .57 (ranged from .54 to .61), Responsibility .69 (ranged from .63 to .77), Unacceptable Thoughts .68 (ranged from .60 to .75), and Symmetry .73 (ranged from .68 to .76).

Procedure Participants were recruited using Amazon’s Mechanical Turk (MTurk), an Internet-based

platform that allows respondents to complete jobs (e.g., survey completion) for monetary compensation. Respondents completing surveys through MTurk have been found to produce high-quality data (Buhrmester, Kwang, & Gosling, 2011). The use of an Internet sample was supported by the equivalence of IU measures across paper-and-pencil versus Internet administration (Coles, Cook, & Blake, 2007). The present research was approved by the local institutional review board. Recruitment was limited to MTurk workers over 18 years old and located in the United States. Participants were required to provide electronic consent, and there was no penalty for withdrawing from the study. Upon completion of the study, participants were debriefed and paid in full. Compensation was $1, an amount consistent with the compensation given to MTurk workers completing prior studies of similar length (Buhrmester et al., 2011).

Results

Structure As described earlier, a higher order CFA approach was used to examine the structure among the items of the IUS–12, OBQ–P/C, and the IUI–A. Tests for multivariate skewness and kurtosis were significant for some of the item scores of these measures, suggesting the presence of multivariate nonnormality. Multivariate nonnormality can negatively impact results obtained when maximum likelihood (ML) estimation is used. Robust ML estimation (Satorra & Bentler, 1994) was therefore used for all reported analyses, as this estimation procedure provides parameter estimates with standard errors that are robust to nonnormality (Brown, 2006). All models were tested by inputting covariance and asymptotic covariance matrices into LISREL Version 8.80 software (Jöreskog & Sörbom, 2007). Four commonly recommended (Brown, 2006; Hu & Bentler, 1999; Kline, 2011) goodness-of-fit statistics were used to evaluate all tested models. These fit statistics were the comparative fit index (CFI), nonnormed fit index (NNFI), root-mean- square error of approximation (RMSEA), and standard root-mean-square residual (SRMR). Hu and Bentler’s (1999) guidelines were used to evaluate fit: CFI and NNFI should be close to .95, RMSEA should be close to .06, and SRMR should be close to .08. Further, the upper limit of the 90% RMSEA confidence interval (CI) should not exceed .10 (Kline, 2011). Nested models were compared using the Satorra–Bentler scaled difference chi-square test (i.e., SDCS test; following Bryant & Satorra, 2012).

As recommended by Brown (2006), the adequacy of a first-order measurement model was initially examined. The metric of latent factors was set by fixing one of the unstandardized item factor loadings to 1.0 (Brown, 2006). Fit statistics for all of the tested measurement models are presented in Table 1. A correlated four-factor model was tested first. This model consisted of the items of the IUS–12–Prospective, IUS–12–Inhibitory, OBQ–P/C, and IUI–A loading on one of the

four latent factors, with no secondary loadings. Based on the specified guidelines, the correlated four-factor model provided an adequate fit to the data. Despite the adequacy of the correlated four-factor first-order model, alternative first-order models were considered. For example, it is possible that the IUS–12 items are not best represented by distinct factors in the context of the items of the OBQ–P/C and IUI–A. To examine this possibility, a correlated three-factor model was tested next. This model was identical to the correlated four-factor model, except all of the IUS–12 items were collapsed onto a single latent factor. With the exception of the RMSEA, this three-factor model provided an adequate fit to the data. However, the fit statistics were more favorable for the four-factor model. Further, the SDCS test indicated that the three-factor model provided a significant decrement in model fit relative to the four-factor model: χD (3) = 329.69, p < .01.

Goodness-of-Fit Statistics for Tested Models

For a more parsimonious model, a one-factor first-order model was tested next. Within this model, the items of the IUS–12, OBQ–P/C, and IUI–A were all collapsed onto a single latent construct. With the exception of the RMSEA and 90% RMSEA CI, the one-factor model provided an adequate fit to the data. However, all of the fit statistics were more favorable for the correlated four-factor model. Further, the SDCS test indicated that the one-factor model provided a significantly poorer fit to the data than did the four-factor model: χD (6) = 668.66, p < .01. Despite the decrement in model fit surrounding the one factor, the pattern of factor loadings on this factor was used to guide the consideration of any remaining tenable first-order models. The completely standardized factor loadings from the one-factor model are presented in Table 2. Generally speaking, the IUS–12 items and the IUI–A items loaded strongly on the factor, whereas the OBQ–P/C items evidenced more modest factor loadings. Therefore, it is possible that a correlated two-factor model, in which the IUS–12 items and the IUI–A items load on one factor and the OBQ–P/C items load a second factor, might provide an adequate fit to the data. The adequacy of this model was tested next. With the exception of the RMSEA and 90%

2

2

RMSEA CI, the correlated two-factor model provided an adequate fit to the data. However, the fit statistics were all more favorable for the correlated four-factor model. Further, the SDCS test indicated that the two-factor model provided a significantly poorer fit to the data than did the four-factor model: χD (5) = 571.83, p < .01.

Completely Standardized Factor Loadings From Select Confirmatory Factor Analyses

Based on the specified guidelines for model comparisons, the correlated four-factor model provided the best model fit to these data. All factor loadings within the four-factor model were significant (p < .01), and the completely standardized loadings are presented in Table 2. All of the correlations among the latent factors were significant (p < .01) and are presented in Table 3. Given the superiority of the four-factor first-order model, a second-order CFA model was tested next. Following Brown (2006), this second-order model removed the first-order latent factor intercorrelations and added direct effects from a higher order factor to each of the first-order factors (IUS–Prospective, IUS–Inhibitory, OBQ–P/C, and IUI–A). The items of each of these scales were retained as indicators of each first-order factor in the second-order model. Evidence of a lack of significant decrement in model fit between the first-order model and second-order

2

model is indicative of the presence of a higher order factor (Brown, 2006). As presented in Table 1, the second-order model provided an adequate fit to the data. All of the fit statistics met or exceeded the specified guidelines and were comparable to the correlated four-factor first-order model. Further, the SDCS test indicated that the second-order model did not provide a significant decrement in model fit relative to the four-factor first-order model: χD (2) = 0.56, ns. The second-order factor loadings of all of the first-order factors were significant (p < .01), and the completely standardized loadings on the second-order factor were as follows: IUS–12– Prospective = .90; IUS–12–Inhibitory = .89; OBQ–P/C = .68; and IUI–A = .92.

Latent Factor Correlations

Incremental Explanatory Power As described previously, unique relations between each IU measure and the symptom measures were examined using SEM. More specifically, three latent structural regression models were fit to the data. The three models only differed based on the endogenous construct in each model. Each model consisted of four exogenous constructs being allowed to predict one endogenous construct. The exogenous constructs were allowed to co-vary in each model. The exogenous constructs were each of the IU measures, and the endogenous construct was one of the symptom measures. The indicators for the exogenous constructs were the items of the respective IU measure. The indicators for the depression and generalized anxiety constructs were the items of the respective symptom measures, whereas the indicators for the obsessive– compulsive construct were the separate four scales of the respective symptom measure. The metric of each latent factor was set by fixing one of the unstandardized factor loadings to 1.0 (Brown, 2006). Partial path coefficients were used to investigate unique relations between the IU and symptom factors.

2

Latent correlations among the constructs in the structural regression models are presented in Table 3. Results from the structural regression models indicated the presence of a statistical suppression effect, such that the IUS–12-Prospective construct tended to evidence significant negative standardized partial path coefficients predicting the symptom constructs—depression = −.34, p < .01; generalized anxiety = −.14, ns; obsessive–compulsive = −.42, p < .01—despite IUS–12–Prospective sharing a significant positive latent correlation with each symptom construct. Supplemental analyses were completed to examine this suppression effect. Structural regression models in which IUS–12–Prospective and IUS–12–Inhibitory were the only two exogenous constructs in the model indicated that the overlap among these two constructs appeared to be affecting the partial path coefficients between the constructs and criteria. Specifically, in the models with either depression or obsessive–compulsive symptoms as the endogenous constructs, the IUS–12–Prospective construct shared nonsignificant unique relations with these symptom constructs after the variance shared with the IUS–12–Inhibitory construct was controlled. Moreover, the IUS–12–Inhibitory construct shared positive unique relations that were larger in magnitude than the latent correlations with these symptom constructs after the variance shared with the IUS–12–Prospective construct was controlled. The standardized partial path coefficients from these supplemental structural regression models, in which IUS–12–Prospective and IUS–12–Inhibitory were the only two exogenous constructs predicting the respective symptom construct, were as follows: depression (Prospective = –.11, ns; Inhibitory = .72, p < .01); generalized anxiety (Prospective = .16, p < .05; Inhibitory = .52, p < .01); and obsessive–compulsive (Prospective = −.10, ns; Inhibitory = .65, p < .01). Because of the impact of each IUS–12 construct on one another in these models, a composite IUS–12 construct, which consisted of all 12 IUS–12 items loading a single latent construct, was used for the results reported from the structural regression analyses.

Fit statistics from the structural regression models, in which the correlated IUS–12, OBQ–P/C, and IUI–A constructs were allowed to predict the respective symptom construct, are presented in Table 1. With the exception of slightly elevated RMSEA statistics, all of the fit statistics met or exceeded the specified guidelines. Partial path coefficients from the structural regression models are presented in Table 4. As shown, the OBQ–P/C and IUI–A constructs both shared unique relations with each symptom construct. The IUS–12 construct, however, only shared a unique relation with the depression construct. Collectively, the IU constructs explained a large amount of variance in the symptom constructs.

Standardized Partial Path Coefficients From Structural Regression Models

Discussion

IU has received increased attention from clinical researchers in recent years, which highlights the need for valid assessment tools of this construct. Whereas in the existing literature Freeston et al.’s (1994) IUS has been the measure most often used to assess IU, additional self-report measures exist for researchers and clinicians to choose from when assessing this construct. Unfortunately, little systematic research has been completed to compare extant self-report measures of IU. The present research helped fill this gap in the literature by examining the structure and incremental explanatory power of three self-report measures that were developed to assess IU—the IUS–12 (Carleton et al., 2007; Freeston et al., 1994), OBQ–P/C (Moulding et al., 2011; Steketee, Frost, & OCCWG, 2005), and IUI–A (Gosselin et al., 2008). The present results indicated that all three of the measures can be conceptualized as assessing IU, although each measure appears to assess a distinct aspect of this construct.

The distinctiveness of the three measures of IU is not surprising, given the different operational definitions of IU used by researchers when creating the items of these measures. Gosselin et al. (2008) asserted that IU fundamentally consists of two defining features: (a) the tendency for an individual to find uncertainty intolerable or unacceptable and (b) reactions to uncertainty that may result from this tendency. As noted earlier, when developing the IUS items, Freeston et al. (1994) used a definition of IU that only pertained to reactions to uncertainty. Because the IUS– 12 is a short-form of the IUS, the items of the IUS–12, by extension, are also representative of Freeston et al.’s (1994) definition of IU. As noted by Carleton (2012), the IUS–12 items can be viewed as assessing cognitive (e.g., Unforeseen events upset me greatly) and behavioral (e.g., The smallest doubt can stop me from acting) reactions to uncertainty. The items assessing these two reactions to uncertainty were factorially distinct in the present research, as

represented by Prospective IU (i.e., cognitive reactions) and Inhibitory IU (i.e., behavioral reactions). However, the present results failed to replicate preliminary results that the separate scales of the IUS–12 share differential unique relations with criterion after the variance shared by the other IUS–12 scales has been accounted for (McEvoy & Mahoney, 2011). Moreover, the present results suggest that after other aspects of IU have been accounted for, the reactions to uncertainty assessed by the IUS–12 are particularly relevant to depression symptoms. The lack of a unique relation between the IUS–12 and generalized anxiety symptoms in the present research was particularly surprising, given the robust relations observed between worry and the IUS in prior research (Koerner & Dugas, 2006). However, Gentes and Ruscio (2011) noted that the bulk of existing research has examined IU in relation to worry, rather than to generalized anxiety symptoms more broadly. Spitzer et al.’s (2006) GAD–7 provides a broad assessment of generalized anxiety symptoms, as the items of this measure are not solely focused on worry. The use of this measure could help account for the lack of a unique relation between the IUS–12 and generalized anxiety symptoms found in the present research.

Whereas the items of the OBQ–P/C (e.g., I must keep working until it’s done exactly right) appear less conceptually linked to definitions of IU frequently cited in the existing literature relative to the items of either the IUS–12 or IUI–A, the OBQ–P/C was nevertheless found to belong to the same higher order factor as both of these other two measures. One tenable reason for this finding is that, similar to the items of the IUS–12, the items of the OBQ–P/C could be viewed as assessing reactions to uncertainty. For example, Birrell et al. (2011) stated that “individuals who are intolerant of uncertainty respond to situations in ways that reduce the level of uncertainty. This is often achieved through obtaining sufficient information for the individual to judge the situation as predictable (and, therefore, safe)” (p. 1205). Perfectionism has been conceptualized as a reaction to uncertainty that functions in this manner. Specifically, some theorists have viewed perfectionism within obsessive–compulsive disorder as an attempt to avoid feelings of uncertainty via striving to achieve a perfect state in perceptions and behavior (Frost, Novara, & Rhéaume, 2002). Feelings of uncertainty (e.g., uncertainty about risk of harm) might be reduced, at least temporarily, through perfection-seeking behavior that makes feared situations become more predictable. At the same time, Frost et al. (2002) noted that perfection- seeking behavior might also increase uncertainty, thereby engendering even greater distress for individuals with obsessive–compulsive disorder. Because IU and perfectionism both appear to have transdiagnostic importance (Carleton, Mulvogue, et al., 2012; Egan, Wade, & Shafran, 2011), perfectionism might be viewed as a reaction to uncertainty that spans across emotional disorders.

Consistent with assertions made by Gosselin et al. (2008), the IUI–A (e.g., I have difficulty

tolerating the possibility that a negative event may happen to me) would appear to be the only one of the three measures targeted in the present research that assesses the tendency for an individual to find uncertainty intolerable or unacceptable. This feature of IU is important, as Dugas et al. (2001) opined that “intolerance of uncertainty may be defined more specifically as the excessive tendency of an individual to consider it unacceptable that a negative event may occur, however small the probability of its occurrence” (p. 552). The present results (i.e., that the IUI–A shares unique relations with symptom measures while accounting for the variance shared with measures that seem to assess the other defining feature of IU—reactions to uncertainty) further support the importance of assessing the tendency for an individual to find uncertainty intolerable or unacceptable. Taken with the noted content differences across measures, the present results stand in contrast to Carleton’s (2012) assertion that the IUS–12 assesses the core aspects of IU that span across psychological disorders, but rather indicate that the IUS–12, OBQ–P/C, and IUI–A each assess a separate core aspect of IU that is relevant to symptoms of psychological disorders.

The future directions of research into the assessment of IU are plentiful and will help speak to the relative merits of the IUS–12, OBQ–P/C, and IUI–A. For example, it is important for researchers to examine whether these three measures demonstrate differential utility in clinical practice, as well as to examine the incremental explanatory power of these measures within prospective longitudinal studies. In addition, it might be informative to extend the present results by investigating whether the full-length IUS items (using Sexton & Dugas’s 2009 factor structure) function similarly as the IUS–12 items in relation to the items of the OBQ–P/C and IUI–A. Given the interest in examining the assessment of IU among racial/ethnic minorities (Norton, 2005), replication of these findings among respondents with greater racial/ethnic diversity is needed. Moreover, whereas the bulk of the extant literature concerning the assessment of IU has been completed in nonclinical samples (Berenbaum et al., 2008; Buhr & Dugas, 2002; Carleton et al., 2007, 2010; Freeston et al., 1994; Gosselin et al., 2008; Moulding et al., 2011; Norton, 2005; Sexton & Dugas, 2009), extending the present findings to samples that consistently score highly on both IU and symptom measures is warranted. Finally, depression, generalized anxiety, and obsessive–compulsive symptoms were specifically chosen because these symptom types have been most often targeted in available IU literature (Gentes & Ruscio, 2011). Nonetheless, it is important to broaden the symptom assessment in future research to include other symptom types of potential importance (e.g., panic, social anxiety; Carleton, Mulvogue, et al., 2012).

Although the present results suggest that the IUS–12, OBQ–P/C, and IUI–A each assess IU, the distinct content assessed by each measure warrants attention. Specifically, researchers and clinicians should carefully attend to the differences in the item content across these measures

when selecting one of the measures to assess IU. Future research efforts examining the relative merits of the IUS–12, OBQ–P/C, and IUI–A will be important in elucidating whether one of these measures might be considered a preferred measure of IU. Through such efforts, the assessment of IU can be refined, and the transdiagnostic importance of IU better understood.

Footnotes An exploratory factor analysis (EFA) yielded an identical pattern of results as the results

obtained when using the CFA approach described in this article. Specifically, a four-factor solution was obtained when using an EFA, which consisted of principal axis factoring and oblique (oblimin) rotation. A parallel analysis for both mean and 95th percentile eigenvalues (using O’Connor’s, 2000, syntax) indicated the appropriateness of the four-factor solution. The first five eigenvalues were 15.91, 1.82, 1.68, 1.44, and 0.96. Each item loaded saliently (i.e., factor loading > .40) on the expected factor (i.e., IUS–12-Prospective, IUS–12–Inhibitory, OBQ– P/C, or IUI–A), and there were no salient cross-loadings within the four-factor EFA solution.

Bryant and Satorra (2012) outlined a modified method for LISREL users to compute the SDCS. This modified method allows LISREL users to compute a SDCS in a manner that is analogous to the method used by users of other SEM software packages. Bryant and Satorra’s method involves computing a scaling correction factor (c) for each model by dividing the normal theory weighted least squares chi-square value by the Satorra–Bentler chi-square value. All SDCS values reported in this article correspond to Bryant and Satorra’s method.

Brown (2006) stated that “when the higher order model is overidentified, the nested χ test can be used to determine whether the specification produces a significant degradation in fit relative to the first-order solution” (p. 332). Following Brown, the SDCS was used to compare the second-order model with the correlated four-factor first-order model in the present research.

The rationale for using item indicators instead of item parcels for these latent constructs was threefold. First, the measures used in the present research were generally too brief to form a sufficient number of item parcels. Second, it is generally unknown whether the underlying structure of the items in a parcel is unidimensional, whereas prior research has supported the items indicators used for each construct as being unidimensional. Third, research has shown that models with parcels do not outperform models based on individual items (see Brown, 2006). Because Abramowitz et al. (2010) found that a one-factor solution of the DOCS items provided a significantly poorer fit to the data relative to a four-factor solution, the four DOCS scales (as scaled scores) were used as indicators rather than using each DOCS item as a separate indicator.

1

2

3 2

4

Kline (2011) advocated using a two-step process when examining structural regression models, with the first step consisting of fitting measurement models to the data before proceeding to testing structural regression models. The benefit of using this two-step process is that it is easier to locate the source of any poorly fitting structural regression models. This two- step process was adhered to when testing the reported structural regression models. However, given the adequate fit of the structural regression models, only the fit statistics from the structural regression models are reported for ease of interpretation of the results from the structural regression models. The measurement models consisted of removing path coefficients from the exogenous to the endogenous constructs and freeing latent factor intercorrelations. The latent factor correlations reported in Table 3 were obtained from the initial measurement models.

References Abramowitz, J. S., Deacon, B. J., Olatunji, B. O., Wheaton, M. G., Berman, N. C., Losardo, D., . . .Hale, L. R. (2010). Assessment of obsessive-–compulsive symptom dimensions: Development and evaluation of the Dimensional Obsessive–Compulsive Scale. Psychological Assessment, 22, 180–198. doi:10.1037/a0018260

Antony, M. M., Bieling, P. J., Cox, B. J., Enns, M. W., & Swinson, R. P. (1998). Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychological Assessment, 10, 176–181. doi:10.1037/1040-3590.10.2.176

Berenbaum, H., Bredemeier, K., & Thompson, R. J. (2008). Intolerance of uncertainty: Exploring its dimensionality and associations with need for cognitive closure, psychopathology, and personality. Journal of Anxiety Disorders, 22, 117–125. doi:10.1016/j.janxdis.2007.01.004

Birrell, J., Meares, K., Wilkinson, A., & Freeston, M. (2011). Toward a definition of intolerance of uncertainty: A review of factor analytical studies of the Intolerance of Uncertainty Scale. Clinical Psychology Review, 31, 1198–1208. doi:10.1016/j.cpr.2011.07.009

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, Y: Guilford Press.

Brown, T. A., Chorpita, B. F., Korotitsch, W., & Barlow, D. H. (1997). Psychometric properties of the Depression Anxiety Stress Scales (DASS) in clinical samples. Behaviour Research and

5

Therapy, 35, 79–89. doi:10.1016/S0005-7967(96)00068-X

Bryant, F. B., & Satorra, A. (2012). Principles and practice of scaled difference chi-square testing. Structural Equation Modeling, 19, 372–398. doi:10.1080/10705511.2012.687671

Buhr, K., & Dugas, M. J. (2002). The Intolerance of Uncertainty Scale: Psychometric properties of the English version. Behaviour Research and Therapy, 40, 931–945. doi:10.1016/S0005- 7967(01)00092-4

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980

Carleton, R. N. (2012). The intolerance of uncertainty construct in the context of anxiety disorders: Theoretical and practical perspectives. Expert Review of Neurotherapeutics, 12, 937– 947. doi:10.1586/ern.12.82

Carleton, R. N., Gosselin, P., & Asmundson, G. J. G. (2010). The Intolerance of Uncertainty Index: Replication and extension with an English sample. Psychological Assessment, 22, 396– 406. doi:10.1037/a0019230

Carleton, R. N., Mulvogue, M. K., Thibodeau, M. A., McCabe, R. E., Antony, M. M., & Asmundson, G. J. G. (2012). Increasingly certain about uncertainty: Intolerance of uncertainty across anxiety and depression. Journal of Anxiety Disorders, 26, 468–479. doi:10.1016/j.janxdis.2012.01.011

Carleton, R. N., Norton, P. J., & Asmundson, G. J. G. (2007). Fearing the unknown: A short version of the Intolerance of Uncertainty Scale. Journal of Anxiety Disorders, 21, 105–117. doi:10.1016/j.janxdis.2006.03.014

Carleton, R. N., Weeks, J. W., Howell, A. N., Asmundson, G. J. G., Antony, M. M., & McCabe, R. E. (2012). Assessing the latent structure of the intolerance of uncertainty construct: An initial taxometric analysis. Journal of Anxiety Disorders, 26, 150–157. doi:10.1016/j.janxdis.2011.10.006

Coles, M. E., Cook, L. M., & Blake, T. R. (2007). Assessing obsessive compulsive symptoms

and cognitions on the Internet: Evidence for the comparability of paper and Internet administration. Behaviour Research and Therapy, 45, 2232–2240. doi:10.1016/j.brat.2006.12.009

Dugas, M. J., Gosselin, P., & Ladouceur, R. (2001). Intolerance of uncertainty and worry: Investigating specificity in a nonclinical sample. Cognitive Therapy and Research, 25, 551–558. doi:10.1023/A:1005553414688

Dugas, M. J., & Robichaud, M. (2007). Cognitive–behavioral treatment for generalized anxiety disorder: From science to practice. New York, NY: Routledge.

Egan, S. J., Wade, T. D., & Shafran, R. (2011). Perfectionism as a transdiagnostic process: A clinical review. Clinical Psychology Review, 31, 203–212. doi:10.1016/j.cpr.2010.04.009

Fergus, T. A., & Wu, K. D. (2011). Searching for specificity between cognitive vulnerabilities and mood and anxiety symptoms. Journal of Psychopathology and Behavioral Assessment, 33, 446–458. doi:10.1007/s10862-011-9245-6

Freeston, M. H., Rhéaume, J., Letarte, H., Dugas, M. J., & Ladouceur, R. (1994). Why do people worry?Personality and Individual Differences, 17, 791–802. doi:10.1016/0191-8869(94)90048-5

Frost, R. O., Novara, C., & Rhéaume, J. (2002). Perfectionism in obsessive–compulsive disorder. In R. O.Frost & G.Steketee (Eds.), Cognitive approaches to obsessions and compulsions: Theory, assessment, and treatment (pp. 91–105). Oxford, United Kingdom: Pergamon. doi:10.1016/B978-008043410-0/50007-6

Gentes, E. L., & Ruscio, A. M. (2011). A meta-analysis of the relation of intolerance of uncertainty to symptoms of generalized anxiety disorder, major depressive disorder, and obsessive–compulsive disorder. Clinical Psychology Review, 31, 923–933. doi:10.1016/j.cpr.2011.05.001

Gosselin, P., Ladouceur, R., Evers, A., Laverdiere, A., Routhier, S., & Tremblay-Picard, M. (2008). Evaluation of intolerance of uncertainty: Development and validation of a new self-report measure. Journal of Anxiety Disorders, 22, 1427–1439. doi:10.1016/j.janxdis.2008.02.005

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:

Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118

Hunsley, J. (2003). Introduction to the special section on incremental validity and utility in clinical assessment. Psychological Assessment, 15, 443–445. doi:10.1037/1040-3590.15.4.443

Jöreskog, K. G., & Sörbom, D. (2007). LISREL (Version 8.80) [Computer software]. Chicago, IL: Scientific Software.

Kertz, S., Bigda-Peyton, J., & Bjorgvinsson, T. (in press). Validity of the Generalized Anxiety Disorder–7 Scale in an acute psychiatric sample. Clinical Psychology and Psychotherapy. doi:10.1002/cpp.1802

Khawaja, N. G., & Yu, L. N. H. (2010). A comparison of the 27-item and 12-item Intolerance of Uncertainty Scales. Clinical Psychologist, 14, 97–106. doi:10.1080/13284207.2010.502542

Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

Koerner, N., & Dugas, M. J. (2006). A cognitive model of generalized anxiety disorder: The role of intolerance of uncertainty. In G. C. L.Davey & A.Wells (Eds.), Worry and its psychological disorders: Theory, assessment and treatment (pp. 201–216). Sussex, United Kingdom: Wiley. doi:10.1002/9780470713143.ch12

Lovibond, S. H., & Lovibond, P. E. (1995). Manual for the depression anxiety stress scales. Sydney: Psychology Foundation of Australia.

McEvoy, P. M., & Mahoney, A. E. J. (2011). Achieving certainty about the structure of intolerance of uncertainty in a treatment-seeking sample with anxiety and depression. Journal of Anxiety Disorders, 25, 112–122. doi:10.1016/j.janxdis.2010.08.010

Moulding, R., Anglim, J., Nedeljkovic, M., Doron, G., Kyrios, M., & Ayalon, A. (2011). The Obsessive Beliefs Questionnaire (OBQ): Examination in nonclinical samples and development of a short version. Assessment, 18, 357–374. doi:10.1177/1073191110376490

Norton, P. J. (2005). A psychometric analysis of the Intolerance of Uncertainty Scale among four

racial groups. Journal of Anxiety Disorders, 19, 699–707. doi:10.1016/j.janxdis.2004.08.002

O’Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments & Computers, 32, 396–402. doi:10.3758/BF03200807

Olatunji, B. O., Broman-Fulks, J. J., Bergman, S. M., Green, B. A., & Zlomke, K. R. (2010). A taxometric investigation of the latent structure of worry: Dimensionality and associations with depression, anxiety, and stress. Behavior Therapy, 41, 212–228. doi:10.1016/j.beth.2009.03.001

Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A.von Eye & C. C.Clogg (Eds.), Latent variable analysis: Applications for development research (pp. 399–419). Thousand Oaks, CA: Sage.

Sexton, K., & Dugas, M. (2009). Defining distinct negative beliefs about uncertainty: Validating the factor structure of the Intolerance of Uncertainty Scale. Psychological Assessment, 21, 176– 186. doi:10.1037/a0015827

Steketee, G., Frost, R., and Obsessive Compulsive Cognitions Working Group. (2001). Development and initial validation of the Obsessive Beliefs Questionnaire and the Interpretation of Intrusions Inventory. Behaviour Research and Therapy, 39, 987–1006. doi:10.1016/S0005- 7967(00)00085-1

Steketee, G., Frost, R., and Obsessive Compulsive Cognitions Working Group. (2003). Psychometric validation of the Obsessive Beliefs Questionnaire and Interpretation of Intrusions Inventory–Part 1. Behaviour Research and Therapy, 41, 863–878. doi:10.1016/S0005- 7967(02)00099-2

Steketee, G., Frost, R., and Obsessive Compulsive Cognitions Working Group. (2005). Psychometric validation of the Obsessive Belief Questionnaire and Interpretation of Intrusions Inventory–Part 2: Factor analyses and testing of a brief version. Behaviour Research and Therapy, 43, 1527–1542. doi:10.1016/j.brat.2004.07.010

Spitzer, R. L., Kroenke, K., Williams, J. B., & Lowe, B. (2006). A brief measure for assessing generalized anxiety disorder: The GAD–7. Archives of Internal Medicine, 166, 1092–1097. doi:10.1001/archinte.166.10.1092

Submitted: December 5, 2012 Revised: April 9, 2013 Accepted: July 11, 2013

This publication is protected by US and international copyright laws and its content may not be copied without the copyright holders express written permission except for the print or download capabilities of the retrieval software used for access. This content is intended solely for the use of the individual user.

Source: Psychological Assessment. Vol. 25. (4), Dec, 2013 pp. 1322-1331) Accession Number: 2013-28554-001 Digital Object Identifier: 10.1037/a0034103

Plum Print