Psychology : Psychometric Report
Kyra2795
PSY4046andPSY3018Week14-PsychometricsValidityandReliabilitytakingtest1.pdf
PSY4046 and PSY3018
SESSION 4 Psychometrics, Validity and Reliability
Neelam Ghuman
1
AND PSY4023 MASTERS
PSY4046 AND PSY4023 • To understand the basis of Psychometrics – Factor Analysis
• You will need to explain to the CLIENT in your introduction the basis behind the statistical techniques which underlie the psychometric profile they are getting.
• This can be a short paragraph that allows me to see that you understand it!
• To appreciate how and why Reliability and Validity are crucial to psychometric measures • Reassure the client that these aspects of tests mean they can be confident in the outcome of the test process
• To take the first test… and score it for homework! 2
PSY3018 • Your HR Report: • Section 1: How personality and emotional intelligence can impact on performance in the workplace (supported by citation) • You will be learning about this in subsequent sessions
• (However you will need to explain the basic principal of stable personality – this week and last week)
• Section 2: Techniques used in the workplace for job selection and appraisal, with a focus on psychometric methods • The material this week and last week will contribute to this section:
• Techniques in job selection (summary – last week) • Psychometric methods – statistical basis, norms and standardised procedures (this week) 3
The trait approach and measurement
• REMINDER FROM LAST WEEK… • Implicit in dispositional approaches to personality are several ASSUMPTIONS which allow for the possibility of SCIENTIFIC MEASUREMENT
• These are: • Dispositional roots – our behaviour is rooted largely in who we are, not
what happens around us • Consistency – stable across time • Predictability – of behaviour • Universality – we all demonstrate basic traits on a bipolar spectrum
• If these assumptions can be upheld, then universal traits that predict behaviour and are consistent can be measured using personal responses to questions linked to trait related adjectives...
• How? Using the psychometric process 4
Raw and converted scores
• A psychometric test will produce a RAW SCORE • Raw scores are fine for comparing average scores across populations
• Where an individual’s responses are to be interpreted, their raw scores must be compared to an average group’s performance in the same area • Otherwise it is meaningless; just an indication of where the person chose to respond
• Possible to score up to 100%...
5
We will be converting your RAW scores into NORMALISED scores for the report
Normalisation of raw scores
• For example in an intelligence test your initial RAW score will be compared to the body of people like yourself (age, gender, maybe other aspects) who have taken the test previously • A process is called STANDARDISATION or NORMALISATION.
• Using formulae, a new score will be calculated that represents how well the individual performed compared to what was expected of them
• This STANDARDISED/NORMALISED score is your true IQ score • Theoretically, a higher IQ score translates to greater intelligence
• When compared to your expectations of your norm (or peer) group
6
What test process is used? • As a general rule, psychometric tests utilise SELF-REPORT QUESTIONNAIRES to provide an inventory of personality traits, aptitudes and personal preferences on topics such as music, the need to be outdoors, social interaction, etc
• Self-report can be useful if it corresponds to ABILITY. A good psychometric test will enable an employer to pinpoint specific areas of compatibility or non-compatibility in the hiring process • Relating personality to essential components of a specific job.
• Self-report tests are only valuable if they: 1. measure what they are supposed to measure 2. can predict performance or consistency
• These are aspects of VALIDITY and RELIABILITY
7
Self-Report Tests – the Likert Scale
• Self-report questionnaires are generally constructed with a series of items with a range of response options know as a ‘Likert scale”
• Rensis Likert developed the Likert scale in 1932, and used it to identify the extent of a person’s beliefs, attitudes, or feelings towards some objects.
• The traditional Likert scale asks people the extent to which they agree or disagree with a statement on a 5-point scale. The scale ranges from “strongly agree” to “strongly disagree.”
• A Likert scale is therefore a psychometric scale commonly involved in research using questionnaires • Likert scales generally range from 5 to an infinite number of options for response – 5
to 9 are popular numbers. • Underlying each response option is a numerical value which is treated as categorical
or scale data for analysis 8
How are questionnaires constructed? • Psychometric tests are designed using an INDUCTIVE statistical process.
• This means a large amount of data is collected – in this case responses to many questions related to the area • Possibly without any presumed hypotheses
• Variables within the dataset are correlated (co-related) using a process similar to regression called FACTOR ANALYSIS
• Variables are retained which explain the scoring of the population
• These variables then form the basis for measurement of the theoretical area.
9
What is factor analysis?
— Factor analysis is process used to determine a small number of underlying factors which explain the relationship between a much larger number of quantitative variables • such as items on a questionnaire
— Unlike variables which can be directly measured such as speed, height, weight, etc., some variables such as extraversion, creativity, happiness and even IQ are not a single measurable entity.
— These are all constructs that are derived from the measurement of other directly observable variables • For example a whole range of abilities in IQ
10
EXAMPLE: Constructing an Aggression Questionnaire • A 200 item questionnaire is prepared with statements that all THEORETICALLY relate to aggression.
• The items on the questionnaire are related to aggressive feelings and behaviour
• 500 participants of mixed sex and over 18 are recruited. • Responses of 500 participants to 200 items on a questionnaire about aggression are collected and entered into a statistical software package (SPSS)
• TARGET: To find out if there are any underlying traits which might explain the way people are scoring on the questionnaire • And therefore reflect their attitudes and feelings
• PROCESS: A statistical technique called Factor Analysis which identifies questionnaire items which ALWAYS score the same way across all participants.
11
The process of selecting factors
• Here are our item responses, plotted on a graph.
• Factor analysis has grouped each response alongside other responses which co-vary – so always score the same way
• This is a type of REGRESSION. • This technique identifies every occasion when items are scored in the same way by most participants • For example an item relating to
swearing when angry is always positively correlated with an item about shouting in arguments, and negatively correlated with telling people how you feel when they upset you...
12
The process of selecting factors
• These items are regularly ‘co-vary’ or are ‘co-relating’ – correlation.
• This suggests they may be measuring an underlying trait or tendency • Which is causing people to score the
way they are
• In factor analysis scores are ‘rotated’ around the axes until a best fit is found.
• A number of possible underlying factors which may explain the responses people make have been detected – now we must identify them! 13
The process of selecting factors • When you run a Factor Analysis in SPSS you will get a little graph called a Scree Plot. • "scree" is a geological term referring to
the debris that collects on the lower part of a rocky slope.
• No more than the number of factors to the left of this point should be retained.
• Cattell (1966) proposed that this scree plot can be used to graphically determine the optimal number of factors to retain. • The eigenvalues (strength of
correlations) for successive factors will be displayed in a simple line plot.
14
In this illustration, the steep drop has an ‘elbow’ at the 3rd factor – this suggests a 3 factor solution to the data.
The process of selecting factors • Here we see a graph of how much of the
total correlation of items (variance in the way people score) in a dataset were be explained by each of the factors extracted.
• Factor 1 explains 25%, • Factor 1 + factor 2 explains 50% • Factors 1, 2 & 3 explain nearly 70% • Factors 1,2,3 & 4 together explain over
75% of the way people score in this dataset
• All the variation can be explained with 8 factors – but in reality not all these factors may be strong enough to be retained in a model
• How do we decide?
15
How well do factors explain scoring
Total Variance Explained
2.600 32.502 32.502 2.600 32.502 32.502 2.070 25.877 25.877 1.772 22.149 54.651 1.772 22.149 54.651 1.770 22.119 47.996 1.079 13.486 68.137 1.079 13.486 68.137 1.611 20.141 68.137
.827 10.332 78.469
.631 7.888 86.358
.487 6.087 92.445
.333 4.161 96.606
.272 3.394 100.000
Component 1 2 3 4 5 6 7 8
Total % of Variance Cumulative % Total % of Variance Cumulative % Total % of Variance Cumulative % Initial Eigenvalues Extraction Sums of Squared Loadings Rotation Sums of Squared Loadings
Extraction Method: Principal Component Analysis.
16
Using the output there were 3 eigenvalues greater than 1.0.
The latent root criterion for number of factors to derive would indicate that there were 3 components to be extracted for these variables.
Kline proposed that all factors with eigenvalues of over 1.0 should be retained. This is called the LATENT ROOT CRITERIA. However, the scree plot is commonly used if making a decision – otherwise SPSS uses the latent root criteria by default.
How well do factors explain scoring
Total Variance Explained
2.600 32.502 32.502 2.600 32.502 32.502 2.070 25.877 25.877 1.772 22.149 54.651 1.772 22.149 54.651 1.770 22.119 47.996 1.079 13.486 68.137 1.079 13.486 68.137 1.611 20.141 68.137
.827 10.332 78.469
.631 7.888 86.358
.487 6.087 92.445
.333 4.161 96.606
.272 3.394 100.000
Component 1 2 3 4 5 6 7 8
Total % of Variance Cumulative % Total % of Variance Cumulative % Total % of Variance Cumulative % Initial Eigenvalues Extraction Sums of Squared Loadings Rotation Sums of Squared Loadings
Extraction Method: Principal Component Analysis.
17
In addition, the cumulative proportion of variance criteria can be met with 3 components to satisfy the criterion of explaining 60% or more of the total variance.
A 3 components solution would explain 68.137% of the total variance.
The meaning of factors
• What does this mean? It means that 3 underlying factors best explains how people score in our aggression questionnaire.
• What these factors might be can only be decided by looking at the WORDING of the items which have correlated – SPSS will tell you this.
• Factors must be labelled to make sense of the underlying trait or principle • This is done by analysing the wording or content of the
items each factor
• In our example, the three factors may have been ‘Verbal Aggression’, ‘Physical Aggression’ and ‘Passive Aggression’.
• People will either score high or low or average for all items on each of these scales.
18
Factor analysis is at the heart of psychometrics
• The process of Factor Analysis underlies ALL psychometric tests. • In Trait theory, research has extracted different numbers of factors relating to basic aspects of personality • Cattell identified 16 factors • Eysenck identified 3 factors • Costa & McCrae, Goldberg and other have identified 5 factors
• The five factor model is the most popular in psychology, but both Cattell and Eysenck’s models are still used in psychometric testing
• In order for a model to be accepted, it has to have good RELIABILITY and VALIDITY...
19
RELIABILITY AND VALIDITY
20
VALIDITY AND RELIABILITY
• Validity and reliability are fundamental to the development of robust selection tools. • For example in job selection they are designed to assess a candidate’s suitability for a job role.
• A decision may then be based on the results of their performance in these tools.
• Psychometric assessments must therefore: • measure the candidate in sufficient breadth for the role • be accurate and trustworthy • Not be prone to fluctuating over time.
• VALIDITY = APPROPRIATENESS of what is being measured • RELIABILITY = ACCURACY of what is being measured.
21
VALIDITY
• A standard definition of validity is: • ‘the agreement between a test score or measure and the quality it is believed to measure’ (Kaplan & Saccuzzo, 2001).
• So the FIT between what a test CLAIMS to measure and what it ACTUALLYmeasures.
• There are 4 main types of validity.
22
Types of Validity 1. Face Validity – the appearance of a test – superficial, but important 2. Content Validity – adequacy of coverage of the theoretical domain – does it
measure everything it should – for example the range of symptoms of depression.
3. Criterion Validity – how well the test relates to an external, real-life criterion value. This includes: 1. Predictive Validity – how well a test does in predicting an outcome – for example
predictive validity would be important for an employment test, because employment tests are designed to predict performance.
2. Concurrent Validity - where a test correlates well with a measure that has previously been validated or matches criterion measures are obtained at the same time as the test scores – for example the test scores accurately estimate an individual’s current state (e.g. Depressed).
4. Construct Validity – the assurance that the test measures a valid construct; that it fits with theoretical propositions it should – specifically: 1. Discriminant Validity – the extent to which it correctly discriminates from other
theory 2. Convergent Validity – how well the test concurs with other measures of the same
criterion or theoretical area 23
VALIDITY
• However, validity of a test may vary depending upon the purpose to which is it put. • (Vernon, 1963) • EXAMPLE: the MBTI is a personality assessment tool and is argued to be a valid test but for development purposes only. The test developers themselves acknowledge that it is not a selection tool because it does not include a faking measure that would indicate whether a person was trying to lie. It is therefore not considered useful in selection as its purpose is to facilitate greater self insight and understanding. It is valid only in a development context.
• KLINE (1998) argues that the inclusion of this context specificity dimension is unnecessary and that a truly scientific psychometric test would always be valid.
24
RELIABILITY • Reliability concerns the ACCURACY of the tool.
• Reliability and Validity are distinct and are both vital considerations when choosing assessment tools.
• A reliable test will be consistent in the measuring what it should.
• There are 4 main types of reliability.
25
Types of Reliability • Test-Retest reliability • Parallel Forms Reliability • Internal Consistency Reliability • Inter-Rater Reliability • SO....
• Reliability as ACCURACY • Reliability as STABILITY (Test-retest accuracy) • Reliability as CONSISTENCY (internal reliability) • Reliability as EQUIVALENCE (correlation with other forms of the test – parallel forms)
• Reliability as DISCRIMINATION (inter-rater reliability) 26
RELIABILITY
• Increasingly reliability is an issue which is becoming a legal requirement for selection tests.
• An unreliable test is an instrument that has a questionable role in selection because it is not clear precisely what its results mean.
• A number of governments across the world demand that a test’s reliability is demonstrated before it can be used in assessment (HUEBERT AND HAUSER, 1999).
• Establishing the reliability of a selection tool involves three main elements: stability, consistency and equivalence of the results (or parallel form).
27
STABILITY
• Stability involves the repeatability of a test outcome over time.
• This can be done by getting the same person to complete the same measure on two separate occasions to produce a Pearson Product-Moment Correlation.
• This is termed test-re-test reliability. • If the test is taken twice the results should be pretty consistent if we are measuring stable traits • some error is acceptable
• The length of time between sessions is important, otherwise: • Practice effects • Carry-over effects
• Changes in circumstance must be considered…
28
CONSISTENCY
• Measuring reliability also needs to take account of consistency of the measure.
• This is concerned with reliability through the internal consistency of a test rather than the temporal change and learning of the test taker.
• One of the easiest methods of assessing reliability is split half testing.
• This involves dividing the test into two and comparing the results from each half.
• When correlated do they both give the same profile?
29
Cronbach’s a } Cronbach's alpha = best know index of internal reliability
} The internal consistency of scale items is based on the average inter-item correlation
} The higher the score, the more reliable scale • .90 or > = high reliability • .80-.89 = good reliability • .70-79 = acceptable reliability • .65-.69 = marginal reliability
} Lower thresholds are sometimes acceptable where more variation is appropriate.
30
Calculating with SPSS
31
Cronbach’s a
Chronbach's Alpha is located at the bottom of the output. An alpha of 0.60 or higher is the minimum acceptable level. Preferably, alpha will be 0.70 or higher, as it is in this case.
Cronbach’s a
If alpha is too small, this column may suggest which variable should be removed to improve the internal consistency of the scale variables. It tells us what alpha we would get if the variable listed were removed from the scale.
Item-total Statistics
Scale Scale Corrected Mean Variance Item- Alpha
if Item if Item Total if Item Deleted Deleted Correlation Deleted
MATHS1 25.2749 25.5752 .6614 .8629 MATHS2 25.0333 26.5322 .6235 .8661 MATHS3 25.0192 30.5174 .0996 .9021 MATHS4 24.9786 25.8671 .7255 .8589 MATHS5 25.4664 25.6455 .6707 .8622 MATHS6 25.0813 24.9830 .7114 .8587 MATHS7 25.0909 26.4215 .6208 .8662 MATHS8 25.8699 25.7345 .6513 .8637 MATHS9 25.0340 26.1201 .6762 .8623 MATHS10 25.4642 25.7578 .6495 .8638
Reliability Coefficients
N of Cases = 1353.0 N of Items = 10
Alpha = .8790
Internal Reliability – Quality of Maths Class Example
Item-total Statistics
Scale Scale Corrected Mean Variance Item- Alpha
if Item if Item Total if Item Deleted Deleted Correlation Deleted
MATHS1 22.2694 24.0699 .6821 .8907 MATHS2 22.0280 25.2710 .6078 .8961 MATHS4 21.9727 24.4372 .7365 .8871 MATHS5 22.4605 24.2235 .6801 .8909 MATHS6 22.0753 23.5423 .7255 .8873 MATHS7 22.0849 25.0777 .6166 .8955 MATHS8 22.8642 24.3449 .6562 .8927 MATHS9 22.0280 24.5812 .7015 .8895 MATHS10 22.4590 24.3859 .6524 .8930
Reliability Coefficients
N of Cases = 1355.0 N of Items = 9
Alpha = .9024
36
You Design a Questionnaire
VALID? RELIABLE?
Face
Content
Criterion
Predictive
Concurrent
Construct Convergent
Discriminant
Discrimination
Accuracy
Equivalence
Parallel forms =
Stability=
Test-retest
Consistency
Internal Reliability
Reliability vsValidity
Reliability is about the consistency of a measure. Validity is about whether a test actually measures what its meant to measure. Validity requires reliability, but reliability alone is not sufficient for validity...
Figure from http://www.socialresearchmethods.net/kb/relandval.php
SUMMARY • All psychometric tests must be constructed to reflect theoretical principles.
• In order for a test to be accepted and widely employed, it has to be proven to be reliable and valid • This process is known as validation of a test.
• Both the reliability and validity of an assessment tool can change over time because of: • Changes in context • Changes in people - new personnel in a firm, for example.
• Validity studies therefore need to be undertaken on an ongoing basis and test reliabilities checked.
• External distortions can also affect reliability and validity: • these may have been inadvertently introduced by the test and may be
related to internal effects from test takers themselves. • Practice effects, for example
• the test taker can distort – consciously or unconsciously – their responses. 38
SUMMARY
• There has been growing interest in issues such as deliberate response sabotage or unconscious response biasing in which the test taker wishes to appear better, or in some cases worse than they are. • The inclusion of lie scales and social desirability scales into the measure can help to offset this effect.
• These are all issues which influence the reliability of the test. • Test users cannot control everything but they need to be aware of how practice effect, honesty and motivation will alter the score obtained.
• Reliability and Validity are crucial... • Most problems in assessment are the fault of the test itself and failure to ensure these principles are upheld.
• Validity and reliability provide different indications of a test’s value but they are different issues...
39
YOUR FIRST TEST
40
BEFORE YOU START
• You will need to explain the process of administering psychometric tests in your assessment, so think carefully:
1. What are standardised instructions and why might they be important?
2. What extra instructions might be important when describing a test?
3. What might some of the important administrative and participant issues be with self-report tests?
41
STANDARDISED INSTRUCTIONS FOR THE IPIP
• Contained in the IPIP there are phrases describing people's behaviours.
• Please use the rating scale to describe how accurately each statement describes you.
• Describe yourself as you generally are now, not as you wish to be in the future.
• Describe yourself as honestly as possible. • So that you can describe yourself in an honest manner, be
assured your responses are completely confidential. • Please read each statement carefully and then put a cross
in the box under the option that best corresponds to yourself.
42 42
THE RESPONSE OPTIONS
• Very Inaccurate • Moderately Inaccurate • Neither Inaccurate nor Accurate • Moderately Accurate • Very Accurate
43 43
OTHER INSTRUCTIONS
• Please work in silence • If you need help, please do not consult anyone near you but raise your hand for assistance.
• There is no time limit for this test but do not spend too long thinking about your answer.
44 44
YOU MAY START
Post Administration Question: • Why was this instruction important?
• “There is no time limit for this test but do not spend too long thinking about your answer.”
45
PSY3018 –YOUR ASSESSMENT
• In your second section you will need to write about the importance of: • Using Norms to calculate scores in psychometric tests • Using standardised procedures, including instructions to
candidates. • Your aim is to present the positive aspects of using
psychometrics in a ‘career counselling’ environment • You should be able to use the material in this session
to do this. • It is to your advantage to start doing this task, rather
than leave it until later! 46
PSY4046 –YOUR ASSESSMENT
• In your methodology section you will need to write about the importance of: • Using Norms to calculate scores in psychometric tests • Using standard procedures, including instructions to
candidates. • Your aim is to inform the client as to the robust and
valid nature of the psychometrics process and why certain procedures were employed.
• You should be able to use the material in this session to do this.
• It is to your advantage to start doing this task, rather than leave it until later!
47
SCORING YOUR TEST
48
Transfer your scores onto a grid, in exactly the same places
49
Completing each grid – by hand
• Multiply the number of ticks in each column by the value, for example, in Section 1, three ‘X’s in the ‘Moderately Accurate’ column = 4 x 3 = 12.
• Add the column totals to make the section score
• Do the same for Section 2.
• Then add the two Section totals to make your Extraversion score, which in this example is 43.
50 Spot the calculation error ;-)
In Excel, all scoring is automatic
Place the number value of the column in the appropriate boxes on the grid, NOT an ‘X” All totals will be calculated for you – for example in top column 1 insert 1 instead of x, in the second column the value is 2, so insert 2…
This is your table (in the template):
52
IPIP SCORES
Raw score
Sample Statistics (n=xxx)
Z score T score
Mean SD
Extraversion
Agreeableness
Conscientiousness
Neuroticism
Openness
PSY4046 & PSY4023 - ENGAGING WITH MEANING
• The scores you will calculate are RAW SCORES only • They have no meaning until they are ‘normalised’ • Do not try to interpret your scores!
• You will also need to read through and consider the factor explanations for the BIG 5 before your session in 2 weeks time. • ALSO ON MOODLE
• Remember too that personality is a matter of individual differences, not ‘good’ or ‘bad’ • Every aspect of personality has strengths and weaknesses
53
PSY3018 – overlap with PSY3009 Module
• Finalists can calculate their RAW scores too, using the same spreadsheet
• You can use your scores and profile as part of your assessment for PSY3009 in the second term.
• Other students in PSY3009 will be taking an online test which scores for them; your involvement in this assessment task should enable you to write a stronger profile in your Individual Differences assessment
• Some of the module material about Psychometrics will cover the same material as this module
54
BEFORE NEXT WEEK
• Before next week you need to SCORE THE IPIP using the EXCEL sheet available on MOODLE
• The procedure is the same as the hand-scoring shown in the following slides
• However the table is SELF SCORING – when you add your responses scores will be generated for you.
• Please save the Excel file to your own computer to do this and for future reference
• DO NOT print and fill in by hand – there are formulae in some of the cells.
• You MUST bring your RAW scores for the 5 FACTORS to next week’s workshop.
55
