Human Resource Questions
mihkano
selectionpt1-mgmt351-fall2015.pptxSELECTION: PART ONE
October 12, 2015
Week 3
MGMT 351
Fall 2015
David Caughlin, Ph.D.
1
Discussion
In terms of selection tests, what does Google care about?
Google Doesn’t Care Where You Went to College
April 9, 2015
2
Outline
Overview of Selection
Quality Standards of Selection Measures
Project Questions
3
Talent Flows
Staffing Processes
Supply-Chain Perspective
Potential Labor Pool
Labor Pool
Applicant Pool
Candidates for Further Evaluation
Offer Candidates
New Hires
Building & Planning
Recruiting
Screening
Selecting
Offering & Closing
Cascio & Boudreau (2010)
Talent Flows
Staffing Processes
Supply-Chain Perspective
Potential Labor Pool
Labor Pool
Applicant Pool
Candidates for Further Evaluation
Offer Candidates
New Hires
Building & Planning
Recruiting
Screening
Selecting
Offering & Closing
Cascio & Boudreau (2010)
Overview of Selection
6
Overview of Selection
Definition: process by which companies decide who will or will not be permitted to work in the organization
Takes the applicant pool developed during recruitment and selects a subset of them
General Rule
Always begin with a rigorous job analysis!
Overview of Selection
Based on the information gathered during job analysis:
Measure people on the critical KSAOs by assigning each applicant a number or value (using some kind of measure, test, tool, or instrument)
Example: 3.5 out of 7 on a Conscientiousness scale
Example: 73 out of 100 on Creativity task
Select those applicants who have the desired levels on the critical KSAOs
Quality Standards of Selection Measures
10
Quality Standards of Selection Measures
To be considered of high quality, selection measures should be:
Reliable
Valid
Generalizable
Useful
Legal
Mini Review: Correlation (r)
Tests the relationship between variables
Magnitude of relationships
Sign of relationships (+/-)
Correlation ranges from -1.00 to +1.00, where 0.00 indicates no relationship and values closer to +/-1.00 indicate stronger relationship
Correlation: Scatterplot
1 2 3 4 5
5
4
3
2
1
Y
X
Correlation: Additional Scatterplot Examples
Reliability
15
Reliability
Definition: consistency of a measure; degree to which a measure is free from random error
An observed score on a measure has two components:
True Score
Measurement Error
True Score Theory
Observed Score = True Score + Measurement Error
All measures have error associated with them—goal is to minimize error
Reliability
Test-Retest Reliability
Inter-Rater Reliability
Internal Consistency Reliability
Test-Retest Reliability
Example of very high test-retest reliability:
| Participant | Measure A @ Time 1 | Measure A @ Time 2 |
| Sally | 78 | 79 |
| Susan | 62 | 59 |
| John | 99 | 99 |
| Enrique | 97 | 94 |
| Sonqui | 33 | 37 |
| Erik | 45 | 46 |
| Thom | 63 | 54 |
| Correlation (T1 & T2): | .99 |
18
Test-Retest Reliability
Example of low test-retest reliability:
| Participant | Measure A @ Time 1 | Measure A @ Time 2 |
| 001 | 56 | 40 |
| 002 | 30 | 63 |
| 003 | 95 | 82 |
| 004 | 85 | 83 |
| 005 | 22 | 68 |
| 006 | 45 | 38 |
| 007 | 99 | 71 |
| Correlation (T1 & T2): | .51 |
19
Inter-Rater Reliability
Example of very high inter-rater reliability:
| Participant | Rater A (RA) | Rater B (RB) |
| Jim | 78 | 79 |
| Doug | 62 | 59 |
| Francis | 99 | 99 |
| Dominique | 97 | 94 |
| Xavier | 63 | 54 |
| Correlation (RA & RB): | .99 |
20
Internal Consistency
Example of “bad” item in a scale (correlation matrix):
| Item 1 | Item 2 | Item 3 | Item 4 | |
| Item 1 | 1.00 | |||
| Item 2 | .72 | 1.00 | ||
| Item 3 | .35 | .40 | 1.00 | |
| Item 4 | .79 | .80 | .25 | 1.00 |
21
Validity
22
Reliability & Validity
To demonstrate high validity, a measure must demonstrate high reliability.
Intuitively: If an unreliable measure is comprised mostly of random error, how can we expect it to consistently relate to some other measure?
Statistically: The observed correlation between two measures is equal to their “true correlation” multiplied by the square root of their reliabilities.
Example: If the “true correlation” between two measures is .50 and both measures’ reliabilities are .70, then the observed correlation is:
23
Criterion-Related Validity
Definition: in the context of selection, validity refers to the extent to which performance on the measure is related to a criterion
A criterion could include:
Manager ratings of job performance
Objective measures of job performance
Customer satisfaction ratings
Absenteeism
Theft
Accidents
Criterion-Related Validity
A generally acceptable level of validity for most selection measures is r = .20 - .30
This may seem low, however:
Unreliability of measure attenuates affects
Most selection systems employ multiple measures whose validities accumulate
Most criteria (performance, theft) depend on all kinds of factors tangential to whatever a selection measure assesses
Example: Criterion-Related Validity
Selection Test
Job Performance
r = .36
Predictor
Criterion
Specific Types of Criterion-Related Validity
Concurrent Validity
Predictive Validity
Concurrent Validity
For a concurrent validation study, give the selection measure to people who are already on the job and correlate it with the criterion at the same time
Example: The SAT and college GPA
28
Predictive Validity
For a predictive validation study, give the selection measure to job applicants but don’t use it to hire (just hire at random or use something else that is unrelated to the measure), and several months later, correlate it with performance
Example: The SAT and college GPA
29
Advantages & Disadvantages of Predictive & Concurrent Validity
Concurrent validity:
Easier and faster
Good for situations in which the organization is not comfortable with the idea of not immediately using test scores to make decisions
Evidence indicates that in practice concurrent validity yields validity results that are as good as those found with predictive validity (Schmitt et al., 1984)
Predictive validity:
Involves validating the test on the population for which you plan to use it (i.e., applicants)
Less susceptible to a statistical issue called range restriction wherein there is reduced variability in test scores
Illustration of Range Restriction
Job Performance
Full range of test scores and job performance
Selection Test Score
Illustration of Range Restriction
Job Performance
Range restriction from the elimination of people who did not score well on the test and then employees who did not perform well on the job
Selection Test Score
Content Validity
Job Performance Domain (KSAOs)
Sample of Job Performance Domain (KSAOs) via Selection tool
Definition: degree to which a selection procedure has been developed to sample the criterion in terms of the required KSAOs
Generalizability
34
Generalizability
Does the validity of the selection measure generalize to other:
jobs?
organizations?
kinds of people?
time frames?
Validity Generalization vs. Situational Specificity
Validity Generalization
assumption that a test that is valid for one job will be valid for other, similar jobs
VS.
Situational Specificity
belief that just because a test was been shown to be valid in one setting, you cannot assume that it will be valid in other settings, even if the two situations are similar
Validity Generalization vs. Situational Specificity
After accumulated meta-analytic evidence, experts generally agree that test validities are likely to generalize across situations, as long as the jobs are similar in terms of the KSAOs required for each (SIOP Principles, 2003)
The courts and Uniform Guidelines, however, are in favor of a situational specificity approach, such that tests should be validated for each situation in which they are used (Gatewood et al., 2011)
Utility
38
Utility
How useful is the measure in terms of:
Increasing overall profitability
Depends on economic consequences of failure versus success
Allowing us to accurately select those who will succeed on the job
Depends on selection ratio and validity
Depends on the accuracy in prediction
Accuracy in Prediction
Selection procedures help organizations make better hiring decisions, but the quality of those decisions will be based on validity of the procedures
Selection decision can be categorized according to:
False Negatives
False Positives
True Positives
True Negatives
Four Quadrants of Selection Decisions
Job Performance
Test Score
True Positives
False Negatives
True Negatives
False Positives
The goal is to maximize true positives and true negatives (“hits”)
We also want to minimize false positives and false negatives (“misses”)
Four Quadrants of Selection Decisions
Job Performance
Test Score
True Positives
False Negatives
True Negatives
False Positives
When validity is very high, the number of “misses” falls close to zero
Thus, we are very likely to make optimal decisions
Four Quadrants of Selection Decisions
Job Performance
Test Score
True Positives
False Negatives
True Negatives
False Positives
When validity is very low, we are just as likely to make good decisions as we are to make bad decisions
Four Quadrants of Selection Decisions
Job Performance
Test Score
True Positives
False Negatives
True Negatives
False Positives
Increasing the passing score decreases false positives, but increases false negatives
That being said, almost all employees who pass the test would be successful on the job
Four Quadrants of Selection Decisions
Job Performance
Test Score
True Positives
False
Negatives
True
Negatives
False Positives
Decreasing the passing score increases false positives, but decreases the false negatives
In this case, all applicants that might be successful on the job would be hired
Utility
So, all else equal, utility improves when…
Decreasing the selection ratio
Decreasing the number of applicants selected and/or increasing the size of the applicant pool
Increasing the cut score of the selection measure used
Increasing the accuracy of the selection measure
Note: Selection ratio = number of applicants selected/number of applicants in applicant pool
Legality
47
Legality
Does the measure discriminate against any group?
Civil Rights Act
Age Discrimination in Employment Act
American with Disabilities Act
48
Outline
Overview of Selection
Quality Standards of Selection Measures
Project Questions
49
SELECTION: PART ONE
October 12, 2015
Week 3
MGMT 351
Fall 2015
David Caughlin, Ph.D.
50
