HRM

Ch07_EvaluatingHRDTrainingProgram.pptx

Home >Human Resource Management homework help >HRM

Chapter 7 Evaluating HRD Programs

Werner

© 2017 Cengage Learning. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part, except for use as permitted in a license distributed with a certain product or service or otherwise on a password-protected website for classroom use.

Learning Objectives

Define evaluation and explain its role in HRD

Discuss the various types of evaluation information available and compare the methods of data collection

Explain the role of research design in HRD evaluation

Describe the ethical issues involved in conducting HRD evaluation

Learning Objectives (cont.)

Identify and explain the choices available for translating evaluation results into dollar terms

Calculate a utility estimate for a target organization.

Discuss how technology impacts HRD evaluation

Questions to Consider

How do you evaluate training and HRD?

What measures can be used to evaluate training?

Is there one best way to evaluate training?

What should be considered as one prepares to evaluate HRD?

What are the ethical issues involved in evaluating HRD?

How can the value of HRD be expressed in terms of costs and benefits, or dollars and cents?

Key Terms

Training effectiveness: benefits receive from training

Training evaluation: process of determining training effectiveness

Training outcomes: measures to evaluate training effectiveness

Evaluation design: data will be collected and which data will be collected for training evaluation

Training and HRD Process

Evaluation Can Help

identify a program’s strengths and weaknesses

assess what features of training content and context matter

identify which trainees benefited

gather information for marketing training

determine financial benefits and costs

Why Not Done Frequently?

Conducting an evaluation is not easy

Many external factors can affect whether employee performance improves

Makes it difficult to evaluate the impact of just the training

HRD managers afraid of criticism and program cuts

Kirkpatrick’s Framework

Reaction

Did trainees like program?

Did trainees think it valuable?

Learning

Did they learn what objections said they should learn?

Job Behavior

Did they use learning back on job?

Results

Has HRD improved organization’s effectiveness?

Most organizations do not collect information on all four types of outcomes

About one-third of organizations use Kirkpatrick’s model

Some feel it only measures after training

Others feel it is more of a taxonomy of outcomes

Kirkpatrick’s Shortcomings

Lack of explicit causal relationships among the different levels

Lack of specificity in dealing with different types of learning outcomes

Lack of direction concerning which measures are appropriate to assess which outcome measures

Training Outcomes

Results

Learning & Cognitive Outcomes

Behavior & Skill-Based Outcomes

Reactions

Return on Investment

Reaction

Trainees’ perceptions of the training experience relating to the content, facilities, trainer, and methods

How did participants react to the program?

Typically measured through attitude questionnaires at the end of training

In the first evaluation level, students are asked to rate the training after completing the program. These are sometimes called smile sheets because in their simplest form, they ask students how well they liked the training. This level is often measured through attitude questionnaires that are distributed at the end of training. It can also be done through focus groups of training participants.

This level measures reaction only; learners identify if they were satisfied with the training. It does not indicate if learners acquired any knew knowledge or skills, nor does it indicate that any new learning will be carried back to the workplace.

If learners react poorly to the training and indicate dissatisfaction at this evaluation level, trainers must determine if the negative results are due to poor program design or unskilled delivery.

Source: Clark, D. R. (2008). Instructional System Design. Retrieved 09/03/08 from

http://www.nwlink.com/~donclark/hrd/sat6.html.

“Why Measure Training Effectiveness?” (2008) Retrieved 09/03/08 from

http://www.businessperform.com/html/evaluating_training_effectiven.html

Learning & Cognitive Outcomes

Relate to familiarity with information, including principles, facts, techniques, procedures, and processes

Typically measured via paper-and-pencil tests and self-assessments

Tests often preferred over self-assessments

Cognitive learning

How much did trainees learn from the training?

Post-training learning

How much learning does the trainee demonstrate immediately after training?

Retention

How much learning does the trainee demonstrate back on the job?

The second evaluation level is used to determine learning results. Did students actually learn the knowledge, skills and attitudes the program was supposed to teach? It asks the questions:

What knowledge was acquired?

What skills were developed or enhanced?

What attitudes were changed?

The results are usually determined by pre-and post-test scores and on-the-job assessments or reports from supervisors. The second evaluation level is not as widely used as the first level, but it is still very common.

“Why Measure Training Effectiveness?” (2008) Retrieved 09/03/08 from

http://www.businessperform.com/html/evaluating_training_effectiven.html

Clark, D. R. (2008), “Instructional System Design;” Retrieved 09/03/08 from

http://www.nwlink.com/~donclark/hrd/sat6.html

Behavior & Skill-Based Outcomes

Relate to proficiency with technical skills, motor skills, and behavior

Include learning and transfer

Learning often assessed via work samples

Transfer often assessed via observation or managerial/peer ratings

Behavior

What behavior change occurred as a result of training?

Training performance

How well can trainees demonstrate the newly acquired skills at the end of training?

Transfer performance

How well can trainees demonstrate the newly acquired skills back on the job?

Kirkpatrick’s third evaluation level explores the consequences of the learner’s behavior. Has the learner transferred the learning back to changed performance in the workplace? Can the learner actually put the newly acquired skills to use on the job?

This is referred to as transfer of training. No matter how good the training program was, if participants cannot (or will not) use the new skills and knowledge on the job, the training has little value to the employer.

Ideally, this evaluation is conducted three to six months after completion of the training program. This allows time for learners to implement new skills, and retention rates can be evaluated. Evaluation is done by observation of learners on the job, or through self-evaluation or evaluation from supervisors, peers or others who work directly with the learner.

Source: Clark, D. R. (2008).

Kruse, K. Evaluating e-Learning: Introduction to the Kirkpatrick Model. Retrieved 09/02/08 from http://www.e-learningguru.com/articles/art2_8.htm.

Results

Outcomes used to determine the benefits of training to the company

e.g., reduced costs, increased employee retention, increased sales, improved quality or customer service

Results

What tangible outcomes or results occurred as a result of training?

Kirkpatrick’s level four evaluates the final results of the training. It asks the question – What effect has the training achieved? Effects can include such things as morale, teamwork, and most certainly, the monetary effect on the organization’s bottom line. Management wants to know if they received value for the training dollars spent and what their return on investment was.

Collecting and analyzing evaluation at this level can be difficult and time-consuming. Part of the difficulty comes from the challenge of isolating the training variable from other factors in the organization that may also affect learners’ behaviors. When employee behavior changes, it is difficult to know if the change is the result of training or the result of some other environmental factor.

Level four evaluations are done through financial reports, quality inspections and interviews with management personnel.

Source: Clark, D. R. (2008).

Kruse, K. Evaluating e-Learning: Introduction to the Kirkpatrick Model. Retrieved 09/02/08 from: http://www.e-learningguru.com/articles/art_2_8.htm.

____. (2008). Why Measure Training Effectiveness? Retrieved 09/03/08 from http://www.businessperform.com/html/evaluating_training_effectiven.html.

Levels of Evaluation vs. Value

Kirkpatrick, & Kirkpatrick, 2006

©SHRM 2009

The difficulty and cost of conducting evaluations increases as you move up the levels. Organizations and trainers must carefully consider which levels of evaluations are appropriate for which training programs. Most commonly, level one evaluations are conducted for all training. Level two--learning evaluations--are generally conducted for skills training programs. Level three evaluations–behavior—for strategic programs and level four--results evaluations—are appropriate only for broad-based, high-budget training programs.

Unfortunately, the easy evaluation instruments used at level one don’t give results that have much value to the organization. The value of the information obtained from the process increases as evaluation moves to higher levels. Level four–results–is the most difficult to assess and yet reveals the most valuable information.

Source: Kirkpatrick, D.L, Kirkpatrick, J. D. (2006). Evaluating Training Programs: The Four Levels. Barrett-Koehler.

Return on Investment

Involves comparing the training program's benefits to its costs in monetary terms

Benefits: value the company gains

Direct costs: salaries for employees involved in training, program materials, facilities, and travel

Indirect costs: costs not related directly to design and delivery

Discussion

Describe the four levels of evaluation that make up Kirkpatrick’s evaluation framework. Identify one example of data at each level that might be collected to provide evidence for the effectiveness of a class or training program in which you have participated.

Discussion

Which Outcomes?

Relevance

Reliability

Discrimination

Practicality

Relevance

The extent to which training outcomes are related to the learned capabilities emphasized in the program

Contamination refers to the inclusion of inappropriate or irrelevant outcomes

Deficiency refers to the omission of important information

Deficiency, Relevance, & Contamination

Reliability

The extent to which outcomes can be measured consistently over time

Evaluators are concerned with consistency over time, such that items do not change in meaning or interpretation over time

Discrimination

The extent to which measured performance reflects a true difference

We want tests that discriminate between high and low performance

A test that is too easy may not discriminate

In this instance, both high and low performers would do well and appear “good” even though they are not

Practicality

The extent to which outcomes can be easily measured and collected

Companies often claim that measurement is too burdensome

Discussion

Data Collection

Method Description

Interview

Conversation with one or more individuals to assess their opinions, observations, and beliefs

Questionnaire

A standardized set of questions intended to assess opinions, observations, and beliefs

Direct observation

Observing a task or set of tasks as they are performed and recording what is seen

Tests and simulations

Structured situation to assess an individual’s knowledge or proficiency to perform some task or behavior

Archival performance data

Use of existing information, such as files or reports

Participant Reaction Questionnaire

Measures immediate reaction to program

Transforms “feelings” into numbers

Likert scale

Allows for numerical analysis

Mean (Average)

Standard deviation (Spread)

Helps compare instructors and programs

Discussion

Identify and describe three potential problems with using self-report measures (e.g., participant questionnaires) in HRD evaluation. How can these problems be minimized?

Use of Self-Report Data

Can provide:

Personality data

Attitudes

Perceptions

Potential problems:

Mono-method bias

Socially desirable responses

Response-Shift Bias

Mono-method bias

If both reports in a before-and-after evaluation come from the same person at the same time (say, after training), conclusions may be questionable

The respondents may be more concerned about being consistent in their answers than about providing accurate responses

Socially desirable responses

Respondents may report what they think the researcher (or boss) wants to hear rather than the truth

Employees may be fearful or embarrassed to admit that they learned nothing in a training program

Respondents’ perspectives of their skills before training change during the training program and affect their after-training assessment

Other methods may yield better results

Direct observation by trained observers like supervisors

Tests

Simulations

Post-test Only

Involves collecting only post-training outcomes

Appropriate when trainees can be expected to have similar levels of proficiency prior to training

Strengthened with a control group

Control Group

Group of employees similar to those who receive training

Don’t receive training at the same time as those who are trained

Receives the same evaluation measures as the group that is trained

Allows for a comparison of their scores

The ideal scenario— Scores for the training group increase after training, while those of the control group remain constant

This provides fairly strong evidence that the training (and not some other factor) was responsible for the changes on the outcome measures

Pre-test / Post-test

Involves collecting both pre-training and post-training outcomes to determine if a change has occurred

Is there a difference in outcomes before and after training?

Pre-test / Post-test with Comparison Group

Includes pre-training and post-training outcomes and use of a control group

If the post-training improvement is greater for the training group, there is evidence that training was responsible

Strong Factors for Good Research

Pretest-posttest with control group

Random assignment between training and control group

Collection of data over time

Time-series design

Number of respondents should be over thirty to be accurate

Time-series design

Involves collecting measures at periodic intervals pre- and post-training

A comparison group may be used

Discussion

Suppose you have been asked to design a program intended to train airline flight attendant trainees in emergency evacuation procedures. You are now designing the evaluation study to show that the flight attendants understand the procedures and use them on the job.

Which data collection methods do you think would be the most useful in providing this evidence?

How might a type of learning outcome affect your choice(s) of how to measure learning? Support your choices.

©SHRM 2009

Students should design evaluation instruments that reflect Kirkpatrick’s four levels of evaluation.

Ethical Issues in Evaluation Research

Confidentiality

Informed consent

Withholding training

Pressure to produce positive results

Withholding training

Use lottery to assign to control groups

Assure training will be provided if found to be effective

Train both groups but at different time

Assessing HRD in Dollar Terms

Effect of an HRD program on the organization’s effectiveness

Measured in terms of increased

Productivity

Timeliness

Cost savings

Increased competition for investment dollars requires organizations to decide whether to invest in training or to invest in something else. A well-designed benefit-cost ratio analysis can aid in the decision-making process by allowing several different investment options to be compared with each other. The problem is that some benefits derived from training can be intangible and difficult to quantify. How do you measure and put a dollar value on increased morale or better teamwork? Consequently, gathering and compiling the information needed for an accurate benefit-cost analysis can be a complicated task.

Source: U.S. Dept. of Labor Retrieved 09/02/08, from http://www.careertools.org/pdf/AdvancedROI.pdf.

Making HRD programs investments leading to measurable payoffs in the future

Two practical methods

Evaluation of training costs (return on investment (ROI))

Utility analysis

Cost-Benefit Analysis

Companies may desire to quantify whether the benefits of training outweigh the costs

Cost-benefit analysis

Process of determining the economic impact of training using accounting methods that look at training costs and benefits

Cost Categories

program development or purchase

instructional materials

equipment and hardware

facilities

travel and lodging

salary of the trainer and support staff

cost of lost productivity or replacement workers while trainees are away

Calculate direct costs

Calculate indirect costs

Calculate development costs

Determine overhead costs

Determine compensation for participants

Sum total costs

Divide by number or trainees to get cost per participant

Overhead costs, often referred to as overhead or operating expenses, refer to those expenses associated with running a business that can't be linked to creating or producing a product or service. They are the expenses the business incurs to stay in business, regardless of its success level.

https://en.wikipedia.org/wiki/Overhead_(business)

Financial Benefits Accrued from Training

Increases in quality and profits

Reduction in waste and processing time

A number of methods can be used to identify benefits

literature that summarizes benefits

pilot training programs

observing successful job performers asking trainees and managers for estimates

Return on Investment (ROI)

Most common business ratio for determining performance

ROI = Net Benefits Training Costs

Process:

Identify annual change in outcomes

Place a monetary value on the outcomes

Determine training costs

Calculate net benefit

Divide net benefits by costs

If ROI < 1, training costs more than benefits accrues

If ROI > 1 benefits accrue

Greater the ratio, the better the benefit

Increasing ROI Credibility

Use conservative cost estimates (error on high side)

Find reliable estimate sources

Explain all assumptions and techniques used to calculate costs

Rely on hard data whenever possible

Use the “Balanced Scorecard” shown earlier

Training appropriate for ROI analysis:

Clearly identified outcomes.

Not one-time events.

Broad-based and highly visible in the organization.

Strategically focused.

Training effects can be isolated.

Remember from the Kirkpatrick model that the higher the level of evaluation, the more costly and difficult it will be to conduct the evaluation. Therefore, it is important to remember that ROI analysis may not be appropriate for all training programs.

Training programs best suited for ROI analysis must have clearly identified outcomes from which the benefit can be determined. They should be a reflection of the goals set in the organization’s strategic plan. These are broad-based across the organization and not one-time training events. The effects of training can be isolated to ensure that the benefit is not a reflection of other organizational factors.

Source: Noe, R. A. (2008).

Utility Analysis

Assessing the dollar value of training based on:

estimates of the difference in performance between trained and untrained employees

number of individuals trained

length of time training is expected to influence performance

variability in performance in the untrained group of employees

ΔU = (N)(T)(dt)(SDv) – C

Where

N = number of trainees

T = Time benefit expected to last

dt = Effect size

True size of difference in Std Dev terms

SDy = Dollar value of job performance in Std Dev terms

C = costs of training

Compute minimum annual benefits needed to break even

Use break even analysis to determine minimum effect size (dy) that will yield required minimum benefit

Use the results from meta-analytic analysis to determine expected cost and expected payoff

NOTE: Use the company statisticians and financial staff to help (and they take partial ownership of results)

How Technology Impacts Evaluation

Reaction

Easy to gather continuous feedback online

Could use a discussion thread or “chat room” to allow trainees to discuss their experiences with online learning

Learning

Very easy to test trainees electronically

Can also link to a learning management system

Behavior

Very hard to capture electronically

Some relevant data may be available in other information systems, e.g., appraisals, promotions, turnover, and discipline data

Results—Even harder to do online than traditionally, without face-to-face interaction, feedback, and buy-in

Big Data & Training

Big data help make decisions about human capital based on data, rather than intuition and conventional wisdom

Big data can be used to:

evaluate the effectiveness of programs

determine their impact on business results

develop predictive models for forecasting training needs, course enrollments, and outcomes

How Evaluation SHOULD Be Conducted

Perform needs analysis

Develop explicit evaluation strategy

Have specific training objectives

Obtain participant reactions

Develop criterion instruments

Plan and execute evaluation

When Evaluation May Not Be Necessary

Time constraints

Managers and trainees may lack expertise

The company may view training as an investment from which it expects little or no return

When Evaluation is a Must

The training is ongoing and has the potential to affect many employees

The training program involves multiple classes and a large number of trainees

The expertise exists to evaluate

The cost of training is significant

There is sufficient time and interest

Note: Not all reasons from the text are listed.

Discussion

Assessment

Assess needs

DesignImplementationEvaluation

Prioritize

needs

Define

objectives

Select

evaluation

criteria

Develop

lesson plan

Develop/acquire

materials

Select

trainer/leader

Select methods

and techniques

Schedule the

program/intervention

Determine

evaluation

design

Conduct

evaluation

of program

or intervention

Deliver the

HRD program

or intervention

Interpret

results