Tests and measurements

profilevmklan
assignmentread.docx

NOT ONLY ITEM CONTENT BUT ALSO ITEM FORMAT IS IMPORTANT: TAXONOMIZING ITEM FORMAT APPROACHES JOHN F. RAUTHMANN

Leopold-Franzens University of Innsbruck, Innsbruck, Austria In this article I argue that as well as item content, item formats (i.e., phrasing and response formats) are also important. Most trait items can be mapped onto 4 dimensions: point of reference (first person, possessive, others, indicator), general item format (staticity, frequency, valency, frequency + valency), construct indicator (attributal, behavioral, mental, contextual), and conditionality (unconditional, conditional). An item taxonomy tree for the first person perspective is provided for an Openness to Experiences item, and NEO-PI-R Extraversion items are analyzed according to the 4-item format dimensions. Future lines of research on item phrasing are outlined. Keywords: item format, response/answering format, item generation, scale construction, psychological/personality/trait assessment. Validly and reliably assessing people’s traits is crucial to personality psychology and psychological assessment, and thus considerable effort has been put into constructing self-report measures. Researchers usually seek to maximize content validity of their measures, that is, each item’s content for a certain construct is diligently chosen in order to optimally capture the construct. However, effects of item wording, grammar, or syntax are generally neglected or underestimated, and items – even from the same construct or scale – show a variety of formats. Not only the content of items but also their formats (including answering scales) SOCIAL BEHAVIOR AND PERSONALITY, 2011, 39(1), 119-128 © Society for Personality Research (Inc.) DOI 10.2224/sbp.2011.39.1.119 119 John F. Rauthmann, Research Assistant, Department of Psychology, Leopold-Franzens University of Innsbruck, Innsbruck, Austria. Appreciation is due to reviewers including: Chao-Chien Chen, Office of Physical Education, Asia University, No. 500, Liufeng Road, Wufeng, Taichung, Taiwan 413, ROC, Email: apeter72@ hotmail.com Please address correspondence and reprint requests to: John F. Rauthmann, Department of Psychology, Leopold-Franzens University of Innsbruck, Innrain 52, Bruno-Sander-Haus, A-6020, Innsbruck, Austria. Phone: +43 512 507 5548; Fax: +43 512 507 2838; Email: [email protected] 120 PROPOSING AN ITEM FORMAT TAXONOMY are important and influence psychometric properties of a scale, such as construct and criterion validity. In this article I seek to (a) formulate common item format approaches and present an item taxonomy tree, (b) show how the proposed item format taxonomization is applied to analyzing one of the most widely used trait measures in personality psychology (the NEO-PI-R by Costa & McCrae, 1992) for a central trait dimension (Extraversion), and (c) provide an outlook on future lines of research concerning different item formats and how they might affect psychometric properties. A TAXONOMY FOR ITEM FORMATS Items aimed at capturing trait content are various in their structures and formats, yet they can be organized according to certain dimensions; namely, general item format, construct indicators, conditionality/contextuality, and perspective taken/ point of reference. In general, trait items use (a) static sentences or descriptions (for example, “I am outgoing”, “I go out and talk with people”, “I think of other people”), (b) frequency descriptions of behaviors and mental processes (for example, “I often go out and talk with people”, “I usually think of people”), and (c) descriptions concerning the valence of one’s feelings towards something (for example, “I enjoy going out and talking with people”, “I love socializing”, “I do not like being around too many people”). These three approaches can be deemed as general item formats into which most items fit. Static descriptions use a staticity approach, frequency descriptions a frequency approach, and valence descriptions a valency approach. They need not be indicated within the item, however, but can be found in the response scales (for example, when answering the item “I go out and talk with people” on a 5-point Likert scale ranging from 1= almost never to 5 = very often). Besides using a general approach to item formats, items also contain different construct-relevant indicators (related to the item content) which are mostly attributes (in other words, adjectives), behaviors, mental states and processes, and contexts or situations. Further, items can be conditional or unconditional: Conditional items use if- or when-clauses to give contextual specifications under which certain mental processes and behaviors occur (cf. if-then patterns of dispositions by Mischel & Shoda, 1995). Moreover, the point of reference or perspective of items can be distinguished: There can be a first person referring to his or her own attributes, mental processes, and behaviors (“I am/think/feel/do/behave...”), but also can “possessively” refer to his or her own attributes, mental processes, and behaviors (“My thoughts/feelings/ behaviors...”). Also, someone else can be referring to one’s own attributes, mental processes, and behaviors (“Others think/say that I/my...”). Additionally, the item need not explicitly refer to oneself in particular but rather to a construct- PROPOSING AN ITEM FORMAT TAXONOMY 121 relevant indicator (for example, when agreeing to the item “Most people impose on others’ kindness”, one could be described as cynical although the item per se does not refer in any way to the person answering the item but rather to the person’s attitude). Most items aimed at capturing trait-constructs can thus be described in terms of a four-way interaction of the following dimensions (see Figure 1): point of reference (first person, possessive, others, indicator) x general item format (staticity, frequency, valency, frequency + valency) x construct indicator (attributal, behavioral, mental, contextual) x conditionality (unconditional, conditional). However, not all interactions are possible, and although most items have a standing on all four of the dimensions, some items may possess two standings within a dimension. The item taxonomy tree (ITT; Figure 1) gives an overview of possible basic item categorizations for the first person perspective along with examples of how a single item (“I enjoy thinking of new ways to solve problems” from the FIRNI by Denissen & Penke, 2008) could be rephrased for each branch.1 ANALYSES OF NEO-PI-R EXTRAVERSION ITEMS All analyses were performed on the German version of Costa and McCrae’s (1992) NEO-PI-R by Ostendorf and Angleitner (2003), and thus results should not be rashly generalized to the English version although the basic steps of analysis can also be applied to it. The NEO-PI-R2 was chosen due to its wide application and comprehensiveness, the trait of Extraversion as it is a fundamental dimension of human personality (Wilt & Revelle, 2009, p. 27). METHOD AND PROCEDURE Item format analyses were carried out with respect to the previously presented four basic dimensions of items: reference point (about oneself in first person; about indicators pertaining to oneself; others about indicators pertaining to oneself; about indicators), general item approach (staticity; frequency; valency; frequency + valency), construct indicators (attributal; behavioral; mental; contextual), and conditionality (unconditional; conditional). There were several criteria for each dimension and its subdimensions that were used to evaluate each item’s standing on these (sub)dimensions. All items with “I . . .” were classified as first person reference items. Items with “My . . .” were classified as items about indicators 1

This particular item was chosen because it could be rephrased for nearly all branches and still remain sensible at most points. 2 Analyses of the NEO-FFI are also included as the items for the NEO-FFI are taken from the NEOPI-R. NEO-PI-R Extraversion has 48 items (8 items for 6 facets), the NEO-FFI 12 Items. 122 PROPOSING AN ITEM FORMAT TAXONOMY Figure 1. The item SEE attached pic:

PROPOSING AN ITEM FORMAT TAXONOMY 123 Figure 1 notes: [1] I am a good and creative problem solver. I am good and creative at problem solving. I am someone who solves problems well and creatively. [2] I am a good and creative problem solver if the problem is interesting. I am good and creative at problem solving if the problem is interesting. I am someone who solves problems well and creatively if the problem is interesting. [3] I solve problems. [4] I solve problems if they are interesting. [5] I think of new ways to solve problems. [6] I think of new ways to solve problems if they are interesting. [7] I am usually a good and creative problem solver. I am usually good and creative at problem solving. I am someone who usually solves problems well and creatively. [8] I am usually a good and creative problem solver if the problem is interesting. I am usually good and creative at problem solving if the problem is interesting. I am someone who usually solves problems well and creatively if the problem is interesting. [9] I often solve problems. [10] I often solve problems if they are interesting. [11] I often think of new ways to solve problems. [12] I often think of new ways to solve problems if they are interesting. [13] I find myself often in problems which need to be creatively solved. [14] I find myself often in problems if they are interesting. [15] I like being a creative problem solver. I like being creative at problem solving. I am someone who likes solving problems well and creatively. [16] I like being a creative problem solver if the problem is interesting. I like being creative at problem solving if the problem is interesting. I am someone who likes solving problems well and creatively if the problem is interesting. [17] I enjoy solving problems. [18] I enjoy solving problems if they are interesting. [19] I enjoy thinking of new ways to solve problems. [20] I enjoy thinking of new ways to solve problems if they are interesting. [21] I enjoy (being in) interesting problems. [22] I enjoy (being in) problems if they are interesting. [23] I usually like being a creative problem solver. I usually like being creative at problem solving. I am someone who usually likes solving problems well and creatively. [24] I usually like being a creative problem solver if the problem is interesting. I usually like being creative at problem solving if the problem is interesting. I am someone who usually likes solving problems well and creatively if the problem is interesting. [25] I often enjoy solving problems. [26] I often enjoy solving problems if they are interesting. [27] I often enjoy thinking of new ways to solve problems. [28] I often enjoy thinking of new ways to solve problems if they are interesting. [29] I often enjoy (being in) interesting problems. [30] I often enjoy (being in) problems if they are interesting. A staticity approach with contextual indicators was excluded as it cannot be formulated with a first person perspective (in other words, the acting “I”). However, it could be formulated somewhat like this with a construct indicator perspective (using the example item of Openness to Experiences): Problems are [often] solved by me (if they are interesting). 124 PROPOSING AN ITEM FORMAT TAXONOMY pertaining to oneself. Items such as “Others think/say/etc. that I/my . . .” were classified as items with others (as the reference point) about indicators pertaining to oneself. Also, there were items which were classified as being items that used an “indicator reference point”: The indicator for Extraversion in one item was that the person finds most people sympathetic – and this is the central point of reference in the item. Any item containing adverbs of frequency (for example, never, seldom, rarely, sometimes, usually, normally, often) were classified as frequency items. Items containing words that reflect a certain positive versus negative valence (for example, like, dislike, love, enjoy, prefer) were classified as valency items. All other items were classified as staticity items as they had neither indicators of a frequency nor a valency approach. Should an item contain an adverb of frequency and a word reflecting valence, then it was classified as an item using a frequency and valency approach at once. Attributal construct indicators were adjectives, behavioral indicators any forms of overt behavior that could be potentially perceived by others (without inference), mental indicators any forms of mental processes and states (that need to be inferred as they are not observable), and contextual indicators any forms of “circumstances” (for example, “people around me”) and situations. Conditionality was judged upon the presence or lack of if/when clauses: Any if/when denoted a conditional item; if there was none, then the item was categorized as an unconditional item. RESULTS Each item provided a score for all four dimensions. For example, an item saying “I am dominant and assertive” would be classified as an item in “staticity approach with attributal indicators, in an unconditional format, and with firstperson reference”. In a similar manner, all NEO-PI-R items were classified. Table 1 shows the results of item format analyses for NEO-PI-R facets of Extraversion (warmth, gregariousness, assertiveness, activity, experience seeking, positive emotions), NEO-PI-R Extraversion domain, and NEO-FFI Extraversion. Point of reference For NEO-PI-R Extraversion facets, the first person perspective was found in most items (M = 91.67%), whereas the NEO-FFI Extraversion had 100%. Only two items in NEO-PI-R Extraversion had other people as the reference point and also only two items had construct indicators as the reference point. Item approach The general item approach used the most in NEO-PI-R Extraversion facets was the staticity approach (M = 41.67%, range: 12.5 – 75%), followed by the frequency approach (M = 31.25%, range: 0 – 50%)3 , and then 3 The means reflect percentages of NEO-PI-R Extraversion domain. PROPOSING AN ITEM FORMAT TAXONOMY 125 TABLE 1 PERCENTAGES OF PERSPECTIVES, GENERAL ITEM APPROACHES, CONSTRUCT INDICATORS, AND CONDITIONALITY IN NEO-PI-R AND NEO-FFI EXTRAVERSION ITEMS Scale Point of reference (perspective) in % General item approach in % Construct indicators in % Conditionality in % 1 2 3 4 S F V F + V A B M C No Yes Warmth 87 0 12.5 12.5 75 0 25 0 25 37.5 37.5 0 100 0 Gregariousness 10 0 0 0 25 12.5 50 12.5 0 50 25 25 87.5 12.5 Assertiveness 87.5 0 12.5 0 25 50 25 0 12.5 87.5 0 0 100 0 Activity 10 0 0 0 62.5 37.5 0 0 25 62.5 12.5 0 87.5 12.5 Experience seeking 87.5 0 0 12.5 12.5 37.5 50 0 0 62.5 25 12.5 100 0 Positive emotions 100 0 0 0 50 50 0 0 37.5 37.5 25 0 100 0 Total (NEO-PI-R) 91.67 0 4.17 4.17 41.67 31.25 25 2.08 16.67 56.25 20.83 6.25 95.83 4.17 Total (NEO-FFI) 100 0 0 0 50 8.33 33.33 8.33 33.33 50 8.33 8.33 100 0 Notes: Facets of Extraversion (in the NEO-PI-R) = Warmth, Gregariousness, Assertiveness, Activity, Experience Seeking, Positive Emotions: 8 items each; NEO-PI-R Extraversion: 48 items (6 x 8 items); NEO-FFI Extraversion: 12 Items (from the NEO-PI-R: 1 x Warmth, 2 x Gregariousness, 1 x Assertiveness, 3 x Activity, 1 x Experience Seeking, 4 x Positive Emotions). 1 = about oneself in first person, 2 = (possessively) about own attributes, behaviors, and mental processes, 3 = others about oneself and one’s attributes, behaviors, and mental processes, 4 = about construct-relevant indicators; S = staticity approach, F = frequency approach, V = valency approach, F + V = frequency and valency approach mixed together; A = attributal indicators, B = behavioral indicators, M = mental indicators, C = contextual indicators. 126 PROPOSING AN ITEM FORMAT TAXONOMY the valency approach (M = 25%, range: 0 – 50%). NEO-FFI Extraversion had 50% of staticity items, 33.33% of valency items, and 8.33% frequency items. In both the NEO-PI-R and NEO-FFI, only one item had the frequency and valency approaches combined. Obviously, the majority of items of NEO-PI-R and NEO-FFI Extraversion are in a staticity format which makes no references to frequencies or valencies of behaviors (or any other indicators). Construct indicators Most indicators are behavioral in NEO-PI-R Extraversion facets (M = 56.25%, range 37.5–87.5%) and in NEO-FFI Extraversion (50%). NEO-PI-R Extraversion showed 16.67% of attributal (range 0–37.5%) and 20.83% of mental indicators (range 0–37.5%), whereas NEO-FFI Extraversion showed 33.33% and 8.33% (in other words, only one item), respectively. Contextual indicators were generally rare (6.25% in NEO-PI-R and 8.33% in NEO-FFI Extraversion). Conditional items were also very rare (two items in NEO-PI-R and no items in NEO-FFI Extraversion). DISCUSSION A special topic of classification should be raised critically: An item such as “I enjoy talking to people”, for example, would have been classified here as an item in “valency approach (enjoy) with a behavioral indicator (talking), in an unconditional format (no if- or when-clause), and with first-person reference (I)”. However, going strictly by the rules, the item could also have been classified as an item in “staticity approach (static verb “enjoy”) with a mental indicator (enjoy) and behavioral specification of the mental indicator (talking), in an unconditional format (no if- or when-clause), and with first-person reference (I)”. Although this possibility is also correct (and would have the advantage of dissolving the valency approach and thus reducing the complexity of the item format taxonomy proposed), this was not done for two reasons: First, the valency approach may still be seen as a distinct form as it only refers to mental indicators of positive versus negative valence, and the “valency” is a central point. Second, the valency approach is used very commonly and should thus be distinguished. Thus, although I recommend distinguishing the valency approach, it could also be described as a special form of staticity approach with mental indicators (denoting positive–negative valence), and future researchers might also use this alternate type of classification. PROSPECTS AND FUTURE LINES OF RESEARCH Knowing which item phrasings are beneficial to which traits and which psychometric abilities is very important for personality research and psychological assessment as data quality could be improved. Thus, the ITT provides a first guide in determining which kind of items a certain scale possesses (the example PROPOSING AN ITEM FORMAT TAXONOMY 127 used here was NEO-PI-R Extraversion) and which kinds of items there could potentially be for a first person perspective (the example used here was a Five Individual Reaction Norms Inventory (FIRNI) Openness to Experiences item). Some traits might show affinities to certain branches of the ITT, and some might even provide better or worse psychometric properties than others. Further research should thus determine which ITT branches (a) are better suited for which constructs (in other words, some constructs might be better captured with certain branches than others), and (b) enhance or decrease psychometric properties (reliability, validity) of a scale using either consistently one branch or mixing branches. Concerning branches and constructs, behaviorally manifestive traits such as Extraversion are probably better captured by behavioral items in staticity or frequency approaches, whereas more intrapersonal traits such as Neuroticism or Intellect might better use items with mental indicators. Most traits may require a first person perspective, but traits denoting attitudes, such as Machiavellianism, might even be well suited to other- or construct-perspectives. Also, not all branches fit equally well for a trait; indeed, most branches might make no sense, but one, two, or three could be highly relevant for a certain construct. It will thus be a future goal to determine optimal branches for certain constructs. In any case, though, the researcher must decide, based on his or her knowledge, which item content and format to use in a particular case. Figure 1, depicting the ITT, and its notes show which possibilities in phrasing there are for an Openness to Experiences item and how these possibilities can be systematically organized in a taxonomy of item formats. Researchers aiming to construct new scales should diligently choose the item formats for their items as the format will likely have effects on psychometric properties of the scale4 . It is thus poorly understood how different item phrasing will affect reliability and validity, and future researchers could start comparing branches systematically for different constructs in order to evaluate which lead to better results and which do not. The ITT provides a systematic framework for such comparative research. With the emergence of large public-domain item pools such as the International Personality Item Pool (IPIP; Goldberg, 1999; Goldberg et al., 2006)5 , there is a trend to employ short phrases (in a staticity approach) rather than complete sentences as items. This gives the interesting opportunity to “outsource” the general item approach into the answering format. For example, the item “I 4 Indeed, preliminary analyses of my own still ongoing research (with Jaap Denissen) show that a frequency approach with behavioral indicators seems to have advantages over a valency approach to extraversion concerning most psychometric properties. Thus, it may be safer to ask people how often they engage in extraversion-relevant behaviors. 5 http://ipip.ori.org/ipip/, retrieved January, 2010. 128 PROPOSING AN ITEM FORMAT TAXONOMY socialize with people” can be equipped with a frequency-answering format (almost never to very often) or a valency-answering format (I dislike this very much to I enjoy this very much). This has the advantage of making items easier to understand and standardizing the frequency- or valency-based answering format rather than relying on different frequency and valency indicators in the items that might be perceived and interpreted idiosyncratically (for example, even though semantically “rarely” and “seldom” are equal, they might not be in people’s frequency perceptions and thus answering). Future researchers should determine how such items with outsourced general item approaches function in relation to those with the general item approach integrated into the items. CONCLUSION In this article, I provided an item taxonomy tree to systematize item construction concerning their phrasing. It should be a goal of future researchers to comparatively evaluate the different branches of the item taxonomy tree in relation to different traits and different psychometric criteria. Not only item content but also item formats are important.

REFERENCES Costa, P. T., & McCrae, R. R. (1992). Revised NEO personality inventory (NEO PI-R) and NEO Five-Factor Inventory (NEO-FFI): Professional manual. Odessa: Psychological Assessment Resources. Denissen, J. J. A., & Penke, L. (2008). Motivational individual reaction norms underlying the five factor model of personality: First steps towards a theory-based conceptual framework. Journal of Research in Personality, 42(5), 1285-1302. Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7-28). Tilburg, The Netherlands: Tilburg University Press. Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., et al. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40(1), 84-96. Mischel, W., & Shoda, Y. (1995). A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102(2), 246-268. Ostendorf, F., & Angleitner, A. (2003). NEO-Persönlichkeitsinventar nach Costa und McCrae, Revidierte Fassung (NEO-PI-R). [NEO Personality Inventory by Costa and McCrae, revised version]. Göttingen: Hogrefe. Wilt, J., & Revelle, W. (2009). Extraversion. In M. Leary & R. Hoyle (Eds.), Handbook of individual differences in social behavior (pp. 27-45). New York: Guilford.