presentation
annyzyl123
RaterIssuesAmanda1.pptRater Issues in Performance Management
By Amanda Deane
*
Ch. 6 p. 134-143 and Ch. 7 p. 161- 168
Overview
- Rater error overview
- FOR training and rater idiosyncrasies
- The effect of rater’s goals on performance ratings
- Rater errors in 360 degree feedback
- Conclusion
*
FOR training and rater idiosyncrasies
- Using Frame of Reference Training to Understand the Implications of rater Idiosyncrasy for Rating Accuracy
Uggerselev & Sulsky (2008)
What is FOR training?
Define theory Idiosyncrasy
Why are raters idiosyncratic? – performance dimensions and performance-related behaviors
Error of Omission v. Error of Commission
*
FOR- enhances rater accuracy by calibrating raters such that they have a common conceptualization of what constitutes performance effectiveness across the performance continuum.
Levels of theory idiosyncrasy= the difference between raters implicitly held theories and the performance theopry imparted during FOR training (normative training theory)
Raters are idiosyncratic because of the dimensions they use to evaluate performance- they might be different than the dimensions the organization uses – can consider various aspects of performance when making an overall conclusion
Error of omission- when the normative theory contains performance dimensions not included in the rater’s implicit performance theory/ rater’s dimensional schema does not contain behaviors included in the normative training theory
Error of commission- when a rater’s implicit theory contains additional dimensions to those comprising the normative theory
Types of Rater Error
- Similar to me
- Contrast
- Leniency *
- Severity*
- Central tendency*
- Halo
- Primacy
- Recency
- Negativity
- First impression
- Spillover
- Stereotype
- Attribution
*
* intentional
Types of Rater Error Training Programs
- Frame of Reference Training
- Behavioral Observation Training
- Self Leadership Training
Hypothesis
- H1- Prior to training, rating accuracy will be negatively correlated with rater idiosyncrasy
- H2- Rating accuracy improvements following FOR training will be greater for trainees with higher omission idiosyncrasy than higher commission idiosyncrasy
- H3- Training will improve rating accuracy for all FOR trainees, such that trainees with relatively high performance theory idiosyncrasy will improve significantly more than trainees with lower idiosyncrasy
*
Rating accuracy depends on conforming to the normative theory that informs the development of the comparison scores used in the computation of accuracy
Perseverance Effect- People are less willing to accept information that is counter to info already in their schemas, being confronted with contrary info can even make them more accepting of their original schema- increasing resistance to new info
Articulating the performance theory may be a benefit for rating accuracy in and of itself
Results
- FOR training program was effective in teaching raters to evaluate professor lecturing performance more accurately than control training.
- Found a negative relationship between idiosyncrasy and rating accuracy prior to rater’s receiving training.
- Relationship between idiosyncrasy and rating accuracy may depend on both the degree of idiosyncrasy and the form of idiosyncrasy- omission v. commission
- For ¾ idiosyncrasy measures raters with low idiosyncrasy still improved, just not as much as raters with high idiosyncrasy following FOR training
*
Continuous measure of rater idiosyncrasy developed for performance dimensions by comparing the dimensions students identified with the dimensions from the normative training theory created for the study
First study to address the relationship between rater’s performance theory idiosyncrasy and rating accuracy
Study showed that mere exposure or practice to rating forms was not sufficient to increase accuracy, the FOR training itself is the mechanism for accuracy improvement.
Individual idiosyncrasy measures did not reliably predict rating accuracy, however when the measures are combined the expected relationship emerges
At dimension level training was more effective in improving accuracy for trainees with initially higher levels of omission when combined with lower commission than with higher commission- perseverance effect may extend only to situations where an individual is asked to remove something from a preexisting schema (error of commission)
At behavior level- trainees with higher levels of both forms of idiosyncrasy had the highest accuracy improvements- perhaps there is a provision in an individual’s performance schema that there is no absolute or limited set of behaviors that link to a dimension- rater schemas at the behavioral level may include the provision that there are multiple paths to the same end
No trainee held a performance theory that was not at least mildly idiosyncratic to the normative theory- there was also considerable variability among trainees in terms of the dimensions and behaviors they identified as part of their implicit performance theories – all trainees may be idiosyncratic!
For training was initially intended only for raters who hold performance theories that are highly idiosyncratic to the organizationally adopted theory.
How rater goals effect rater error
- Raters Who Pursue Different Goals Give Different Ratings
Murphy, Cleveland, Skattebo & Kinney (2004)
Rater goals influence the ratings they give- they provide ratings consistent with their goals
Goal of study- to provide an empirical test of the proposition that the goals raters claim to emphasize when evaluating performance are related to the ratings they give.
*
Rater errors and other psychometric deficiencies in ratings might not be the result of errors or limitations in the rater’s capacity, but rather might reflect the effects of strategic decisions on the part of raters about the sorts of ratings they should record.
Raters pursue a number of goals when completing performance appraisals – using ratings to maintain harmony, motivate subordinates to perform better)
The goals raters actually pursue are not always the same as the goals the organization would like them to pursue and conflicts between the official purpose of performance appraisal systems and the ways raters actually use these systems can substantially affect the utility of performance appraisal.
If raters goal is to motivate employees he will give higher rating that encourage them, even if the ratings aren't accurate
Many of the support systems and interventions in performance appraisal appear to be based on the questionable assumption that raters are trying their best to provide accurate ratings but they lack the skill and knowledge to do the job- more likely that they are able but chose to give ratings that advance their goals
Hypothesis
- H1- Measures of rating goals most strongly emphasized by raters will account for a substantial portion of the variance in the performance ratings they assign
- H2- Measures of goal importance obtained after the rater has observed the ratee will account for variance in performance ratings not accounted for by ratings of goal importance collected before observing the ratees performance
*
University teacher evaluation- conservative test of link between goals and evaluations because the ratings don’t have immediate consequences for the rater, they are anonymous and the rater has little to lose or gain by giving overly positive or negative ratings. Produces weaker goal effects- even stronger links when raters have strong incentives to give positive or negative ratings
In this context can partially control for ratee performance on raters goals and on the relationship between goals and ratings
Info about raters goals obtained before they observed performance and after
Rater goals would be influenced by ratees level of performance so hypothesized that importance of rating goals would change over time and these changes would be related to the rater’s final evaluation of the ratee.
Results
- Ratings of goal importance obtained at the beginning of the semester before students observed teacher performance predicted ratings of teacher performance collected at the end of the semester. Both in the individual and pooled sample.
- Corelational study between classes found that rater goals might in part might reflect stable orientations on the part of the rater even without external incentives
- Changes in ratings of goal importance might in part reflect the rater’s evaluation of the ratees performance (strength only)
*
Pilot study identified 4 goals- identifying strength and weaknesses, giving fair evaluations and motivating the ratee.
Pilot study- students filled out goal questionnaire at same time of evaluation ½ before, ½ after- goals were related to ratings given however because both were done at the same time it is possible that the links were spurious due to priming effects or common method effects
All had high ratings, differences across teachers accounted for 14% of variance in teacher ratings.
Goal importance ratings collected at the end of the semester were better predictors of performance than goal ratings collected at the beginning of the semester
Only modest stability in the goals raters pursue over time
Sample was pooled to control for teacher differences
Because of how study was set up reverse causation is not likely nor is priming effects
Raters who evaluated the ratees performance more favorably were more likely to rate the goal of conveying info about strengths more important at end of semester than at the beginning
Are raters at different organizational levels rating the same constructs?
- Measurement Equivalence of 360 degree Assessment Data: Are Different Raters Rating the Same Constructs?
Hannum (2007)
-This study used data collected using a 360 assessment instrument to investigate the structural equivalence of ratings according to rater type, controlling for organizational level.
-Raters inappropriately classified in previous research.
-Rater group differences due to three things
-Construct, scaling and reliability
*
Concerning ranks- researchers use the evaluator’s relationship to the individual being rated (peer, boss/supervisor) as a proxy for organizational level- not good because rater type alone is not a measure of organizational level with respect to either member of the dyad.
When organizational level is termed by rater type, researchers have found little evidence that rating source matters as differences of individual rater effects, rating differences could still be organizational level but when rater type is used a s a proxy the effect of organizational level is spread out across rater types.
Unless the same constructs are being evaluated it would be inappropriate to compare mean scores
Rater group differences may be attributable to real differences in the behavior of the ratees, different understanding of constructs, and differences associated with a difference of application of measurement ratings
For performance ratings to be directly comparable we must rule out that differences are attributable to construct, scaling or reliability differences across groups.
Method and Results
- Rater types included in sample: boss, peer and direct report.
- Only ratings of upper middle managers were used
- Multi-group SEM model demonstrated marginally-adequate fit across rater types at different organizational levels
- Information from the various rating sources can be combined
*
** in order to control variance associated with organizational level
Majority if sample was white males
360 degree assessment instrument was the Prospector
Boss ratings tended to be slightly higher than the other two
Rated on seven scales: learns from people and events, acts with integrity, adapts to cultural differences, is committed to making a difference, seeks broad business knowledge, is insightful: sees things from new angles, and has the courage to take risks.
SEM was used to test the equivalence of the proposed models for reach rater group – systematic plan for investigating the structure of variance across groups
They analyzed model fit
The multi group SEM models were employed to determine the degree to which data from the rater groups were structurally simillar
They created a hypothesized rater for each rater type for each manager in order to control for sample size – the sample size for each rater group was equal to the number of target managers
Why was there marginal evidence of invariance?
- Complexity theory
- Contingency theory
*
Jacques and Clement (1994) complexity theory- as leaders rise through the ranks of an organization they obtain a broader and more complex understanding of the organization and thus what it takes to be successful within the organization. The impact of this altered perspective may be a slightly different interpretation of rating schemes employed in 360 assessment,- reinforces organizational level as a variable of interest separate from rater type because level is a closer approximation of work complexity
Fiedler and Chemers (1982)- managers need to behave differently in different situations in order to be effective- because leaders may act and therefore may be perceived differently in different settings ratings from different sources may be based on incongruent evidence
Rater group effects are marginal
Case Study
- Macaroni Grill
- IFCO
- Life Wire
*
The most important goal is make the process transparent so that employees
understand what they are being asked to perform. Particularly when filling
out performance appraisals, ratees should understand exactly what they are
being rated on and how ratings were arrived at. Their perception of
fairness and accuracy means everything to their acceptance and subsequent
action to improve performance (or maintain high performance).
Type of error that is most common in your experience?
The most common error I've experienced/witnessed is rater's tendency to rate
everyone favorably. This eliminates the chance of distinguishing between
truly excellent performance, and average or poor performance.
And how ratings at different levels of the organization differ?
Ratings naturally change at different levels of the organization. Ratings
of front line employees are likely to focus predominantly on specific job
tasks and roles as well as adherence to organizational values. As you
ascend the organizational ranks, the focus tends to shift from specific
tasks to more general performance standards like management, leadership and
strategic thinking skills. In practice, there is a large amount of variability in how this
manifests itself in different organizations.
Summary
- FOR training improves rating accuracy even for people who weren’t idiosyncratic
- Rater goals predict the performance rating they give. Goals can change in time.
- Information from raters at different organizational levels can be combined
