MBA 635 Week 6 Discussion

Rymario1

ch10_final.pptx

Home >Business & Finance homework help >Management homework help >MBA 635 Week 6 Discussion

PowerPoint Presentation for Managing Performance through Training and Development

Adapted by

Alan Saks

University of Toronto

10-1

Chapter 10

Training Evaluation

10-2

Learning Outcomes

After reading this chapter, you should be able to:

Define “training evaluation” and the main reasons for conducting evaluations

Discuss the barriers to evaluation and the factors that affect whether it is conducted

Describe the different types of evaluations

10-3

Learning Outcomes

Discuss the models of training evaluation and their interrelationships

Describe the main variables to measure in a training evaluation and how they are measured

Discuss the types of designs for training evaluation, their requirements, their limits, and when they should be used

10-4

Introduction

This chapter explores several training evaluation models, types of evaluation, variables to measure and how to do so, and the strengths and weaknesses of different data collection designs used in evaluations

10-5

Training Evaluation

A process to assess the value or worthiness of training programs to employees and to organizations

A continuum of techniques, methods, and measures

One needs to balance the informational needs of decision makers with the difficulty and costs of obtaining that information

10-6

Assesses this value by analyzing data collected from trainees, supervisors, or others familiar with the trainees and with the job context

Using a variety of techniques, objective and subjective data may be gathered before, during, and after the training to provide the data required to estimate the value of the training program

Why Conduct Training Evaluation?

It is of value to organizations in order to:

Improve training

Determine the programs most useful to employees and determine who should be trained

Determine cost benefits and cost-effectiveness of a program

Determine if a program has achieved expected results or solved the problem it was meant to solve

10-7

Management has to ensure the resources invested in training are worthwhile, because resources are scarce and the time available for training is limited

Why Conduct Training Evaluation?

Diagnose the strengths and weaknesses of a program and identify needed improvements

Use the data to justify and reinforce (if merited) the value and credibility of the training function to the organization

10-8

Do Organizations Conduct Training Evaluation?

By 2000, most organizations in North America were conducting some form of evaluation of their training programs

10-9

The Conference Board of Canada indicates that 56 percent of organizations now report evaluating their training programs (2012–2013)

Research suggests that Canadian organizations are increasingly aware of the importance of training evaluation

Organizations with strong learning cultures do better on the evaluation dimensions discussed by the Conference Board of Canada than do those whose learning cultures are weaker

Do Organizations Conduct Training Evaluation?

Most organizations that do evaluate training rely principally on a simpler evaluation strategy

The more complex the evaluation outcome assessed, the fewer the organizations that conduct them

10-10

Do Organizations Conduct Training Evaluation?

Organizations with stronger learning cultures conduct more evaluations at all levels, including the more sophisticated ones

10-11

The Conference Board of Canada indicates that 56 percent of organizations now report evaluating their training programs (2012–2013)

Research suggests that Canadian organizations are increasingly aware of the importance of training evaluation

Organizations with strong learning cultures do better on the evaluation dimensions discussed by the Conference Board of Canada than do those whose learning cultures are weaker

Barriers to Training Evaluation

Barriers to training evaluation fall into two categories:

Pragmatic barriers

Political barriers

10-12

Despite the pragmatic barriers, training evaluation has been unduly mystified. The principles, techniques, and procedures of training evaluation are logical, straightforward, and implementable

With the progress of technology and new evaluation models and designs, the disruptive impact and costs of data collection can be eased significantly

Pragmatic Barriers to Training Evaluation

Evaluation requires knowledge of evaluation models, research design, measurement, questionnaire construction, and data analysis (may be intimidating)

Time and effort required

Financial costs

10-13

Despite the pragmatic barriers, training evaluation has been unduly mystified. The principles, techniques, and procedures of training evaluation are logical, straightforward, and implementable

With the progress of technology and new evaluation models and designs, the disruptive impact and costs of data collection can be eased significantly

Political Barriers to Training Evaluation

Ethical grounds

Evaluation can be threatening

Investment implications

Accountability

10-14

Training will serve organizational success more clearly when trainees, their managers, and those who develop and administer training programs are more accountable for results

Managers need to stress evaluation

Types of Training Evaluation

The data that are collected and analyzed

Trainee perceptions at the conclusion of training

Psychological forces

Work environment

The purpose for which the evaluation is conducted

Formative evaluations

Summative evaluations

10-15

The Data Collected

Evaluations differ with respect to the type of information that is gathered and how that is accomplished:

Trainee perceptions at the conclusion of the training

The psychological forces during the program that affect learning and behaviour change

Work environment—transfer climate and learning cultures can affect training choices and effectiveness

10-16

The data that are collected and analyzed

Trainee perceptions at the conclusion of training (did they like it)

Psychological forces (affective, cognitive, and skills based)

Work environment (transfer climate and learning cultures)

The Purpose of the Evaluation

Evaluation differs in terms of the purpose of the evaluation:

Formative evaluations

Summative evaluations

10-17

The purpose for which the evaluation is conducted

Formative evaluations: data about various aspects of a training program

Summative evaluations: data about the worthiness or effectiveness of a training program

Formative Evaluation

Formative evaluations provide data about various aspects of a training program and the value of training processes and materials

The key goal of identifying improvements to the instructional experience

10-18

The purpose for which the evaluation is conducted

Formative evaluations: data about various aspects of a training program

Summative evaluations: data about the worthiness or effectiveness of a training program

Summative Evaluation

Summative evaluations provide data about the worthiness or effectiveness of a training program, e.g., cost–benefit analyses

Economic indices are often an integral and important part of these types of evaluations

10-19

The purpose for which the evaluation is conducted

Formative evaluations: data about various aspects of a training program

Summative evaluations: data about the worthiness or effectiveness of a training program

Descriptive Evaluations

Descriptive evaluations provide information describing trainees once they have completed a training program

Most evaluation designs have some descriptive component

10-20

Is the trainee knowledgeable?

Is the trainee confident about using the skill?

Is the trainee confident about using the skill on the job?

Most evaluation designs have descriptive components

Causal Evaluations

Causal evaluations provide information to determine whether training caused the post-training behaviours

Causal evaluations require more complex data gathering and statistical procedures

10-21

Was the performance improvement caused by the training program?

Causal evaluations require more sophisticated experimental and statistical procedures

Models of Training Evaluation

Models of training evaluation specify the information (the variables) to be measured and their interrelationships

Three models of training evaluation:

Kirkpatrick’s hierarchical model

The COMA model

Decision-Based Evaluation model

10-22

Dominant training evaluation model is Kirkpatrick’s hierarchical model

However, research and practical experience has improved upon this model

The COMA model and the Decision-Based Evaluation model are two recent efforts in that direction

Kirkpatrick’s Hierarchical Model: The Four Levels of Training Evaluation

Kirkpatrick’s hierarchical model of training evaluation is the oldest, best known, and most frequently used training evaluation model

It identifies four fundamental measures, called levels, to assess training

10-23

Kirkpatrick’s model is the oldest, best known, and most frequently used training evaluation model

It identifies four fundamental features

Kirkpatrick’s Hierarchical Model: The Four Levels of Training Evaluation

According to this model, a training program is “effective” when:

L1. Reactions: Trainees report a positive reaction to a training program

L2. Learning: Trainees learn the material

10-24

Kirkpatrick’s model is the oldest, best known, and most frequently used training evaluation model

It identifies four fundamental features

Kirkpatrick’s Hierarchical Model: The Four Levels of Training Evaluation

L3. Behaviours: Trainees apply on the job what they learn in training

L4. Results: Training has a positive effect on organizational outcomes

L5. Return on investment—cost–benefit analysis

10-25

In a more recent version, an additional level (Level 5) has been added

It is designed to assess the financial benefit to the organization when compared to costs incurred

Chapter 11 is devoted to the 5th level, cost–benefit analysis of training programs

Kirkpatrick’s Hierarchical Model: The Four Levels of Training Evaluation

The Kirkpatrick model states that the four levels are arranged in a hierarchy, such that each succeeding level provides more important (though more difficult to obtain) information than the previous one

The model suggests that each level has a causal link to the next level

10-26

Kirkpatrick’s model is the oldest, best known, and most frequently used training evaluation model

It identifies four fundamental features

Research on Kirkpatrick’s Model

Little correlation between the four levels of the model

Not being hierarchical seriously limits the model’s usefulness for organizations interested in formative evaluations

The Kirkpatrick model lacks precision

10-27

Kirkpatrick’s model is the oldest, best known, and most frequently used training evaluation model

It identifies four fundamental features

COMA Model

A training evaluation model designed for formative evaluation

Focus is on variables important for transfer

COMA suggests the measurement of variables that fall into four categories:

Cognitive

Organizational

Motivation

Attitudes

10-28

Enhances the usefulness of training evaluation questionnaires by identifying and measuring those variables that research has shown to be important for the transfer of training

These questionnaires can be administered before and immediately after the training session

COMA Model

Cognitive variables refer to the level of learning gained from the training (both declarative and procedural)

Organizational environment refers to the variables in the work environment that impact transfer of training (e.g., the learning culture, the opportunity to practise, the degree of support)

10-29

Cognitive variables refer to the level of learning gained from the training (both declarative and procedural)—the latter is more important because it is more strongly related to the transfer than the former

Organizational environment refers to the variables in the work environment that impact transfer of training—these include the learning culture, the opportunity to practise, the degree of support that is expected, and the level of support provided to trainees once they return to the job

COMA Model

Motivation refers to the desire to apply the learned skill on the job (e.g., training motivation and motivation to transfer)

Attitudes refers to individuals’ feelings and thinking processes (e.g., self-efficacy, perceptions of control, and expectations about self and the environment)

10-30

Motivation refers to the desire to learn and transfer on the job what was presented in training—training motivation (measured at the beginning of the program) and motivation to transfer (measured after the program) should both be measured

Attitudes refers to individuals’ feelings and thinking processes—these beliefs include self-efficacy, perceptions of control, and expectations about self and the environment

COMA Model

Training evaluation should assess the degree to which:

Trainees have mastered the skills (C)

The organizational environment will support and help trainees apply the skills (O)

10-31

COMA Model

Trainees are motivated to learn and apply the skills on the job (M)

Whether trainees have developed attitudes and beliefs that allow them to feel capable of applying their newly acquired skills on the job (A)

10-32

At the conclusion of the training, the trainees answer questionnaires that assess the COMA components

Questions are custom fitted because they list the trained skills

The participants indicate whether they know the learned skills (C), are motivated to display them (M), are confident that they display them (A), and are being realistic to the degree to which the work environment is supportive of their efforts to do so (O)

COMA Model

At the conclusion of training, the trainees answer questionnaires that assess the COMA components

When scores on the COMA variables are positive, the chances of transfer are greater

When scores are negative on one or more COMA dimensions, transfer is likely to be weaker

10-33

COMA Model

COMA improves upon the Kirkpatrick model in four ways:

It transforms and improves the typical reaction questionnaires by incorporating a greater number of measures

It is especially useful for formative evaluations

The variables it measures are causally related to transfer of training

It defines these new variables with greater precision

10-34

COMA Model

The COMA model does have limits:

COMA is focused exclusively on an analysis of the factors that affect transfer

It is not well suited for summative evaluation purposes

Different questionnaires must be constructed for different training programs

10-35

Decision-Based Evaluation Model

Evaluators custom fit their evaluations to the requirements of the situation with respect to the target, focus, and methods of evaluation:

Targets: What do we want to find out from the evaluation?

Focus: What are the variables we will measure?

Methods: What methods may be appropriate for conducting the evaluation?

10-36

Invites evaluators to select their evaluation techniques and variables as a function of the uses to which the results of the evaluation will be put

Evaluation choices depend on the decisions needed

Decision-Based Evaluation Model

The model specifies three potential “targets” for the evaluation:

Trainee change

Organizational payoff

Program improvement

10-37

Once these are determined, the evaluator specifies the focus of the change

Once the focus is selected, the model suggests the appropriate data collection method (e.g., surveys, job sample information, objective data, questionnaires, interviews)

It does not suggest a single best way for training evaluation

Decision-Based Evaluation Model

Once the target is decided, the evaluator specifies the “focus” of the change (e.g., level of trainee changes with respect to learning, behaviours, or the psychological states)

Once the focus or foci are selected, the model suggests the appropriate data collection method (e.g., surveys, job sample information, objective data, questionnaires, interviews)

10-38

Once these are determined, the evaluator specifies the focus of the change

Once the focus is selected, the model suggests the appropriate data collection method (e.g., surveys, job sample information, objective data, questionnaires, interviews)

It does not suggest a single best way for training evaluation

Decision-Based Evaluation Model

DBE is the only training evaluation model that clearly specifies that evaluations must always be guided by several key questions

DBE is more flexible, being useful for both formative and summative evaluations

10-39

Once these are determined, the evaluator specifies the focus of the change

Once the focus is selected, the model suggests the appropriate data collection method (e.g., surveys, job sample information, objective data, questionnaires, interviews)

It does not suggest a single best way for training evaluation

Training Evaluation Variables

10-40

Training evaluation requires the measurement of variables

Table 10.1 provides a list of the main variables that can be measured in training evaluation

Reactions

Trainee opinions and attitudes about a training program

The most common variables measured in evaluation studies

Two types of reaction measures have received the most attention: affective and utility reaction measures

10-41

Although there are many types of learning outcomes that can be measured, most training evaluations measure declarative learning

It refers to the acquisition of facts and information

It is usually assessed with multiple choice or true-false type questions

Reactions

Affective reactions are measures that assess trainees’ likes and dislikes of a training program

Utility reactions are measures that assess the perceived usefulness of a training program

10-42

Reaction measures are easy to administer, collect, and analyze, and the questions may focus on the trainees’ overall reactions to a training program and/or specific elements of the program

Typically measured by questionnaires right after training; trainees answer questions using rating scales

Research has shown that affective reaction measures bear little relationship to other training outcomes, including learning and behaviour

On the other hand, utility reaction measures demonstrate some relationship to learning and behaviour

Therefore, collecting utility reactions is important because they indicate to some degree whether transfer is likely or unlikely

Sample Reactions Rating Form

10-43

Training evaluation requires the measurement of variables

See also Table 10.1, which provides a list of the main variables that can be measured in training evaluation

Reactions

Trainee reaction measures are extremely popular because:

They provide trainers with immediate feedback on their course

They are easy to collect and analyze, and are easily understood by managers and employees

Trainees who have had a chance to comment on a program and make suggestions for improvements might be more motivated to transfer their learning than others who leave a program without providing input

10-44

Although there are many types of learning outcomes that can be measured, most training evaluations measure declarative learning

It refers to the acquisition of facts and information

It is usually assessed with multiple choice or true-false type questions

Learning

Although many types of learning outcomes can be measured, most training evaluations measure declarative learning

In rare cases, some evaluators also assess procedural learning

10-45

Although there are many types of learning outcomes that can be measured, most training evaluations measure declarative learning

It refers to the acquisition of facts and information

It is usually assessed with multiple choice or true-false type questions

Learning

Declarative learning is measured when trainees are assessed to determine if they have acquired facts and information (most frequently assessed learning measure)

10-46

Although there are many types of learning outcomes that can be measured, most training evaluations measure declarative learning

It refers to the acquisition of facts and information

It is usually assessed with multiple choice or true-false type questions

Learning

Procedural learning is measured when trainees are required to organize facts and information into a smooth behavioural sequence (e.g., a police officer learning the do’s and don’ts of how to protect a crime scene)

10-47

Involves the organization of facts and information into a smooth behavioural sequence

Procedural learning is strongly related to a number of training outcomes, including transfer of training

Behaviour

Behaviour refers to the display of the newly learned skills or competencies on the job or the transfer of training

The most important of all training effectiveness criteria

10-48

Behaviour refers to the display of the newly learned skills or competencies on the job (transfer of training)

Arguably the most important of all training effectiveness criteria

Self-reports remain the most frequently used measures of behaviour (accuracy may be questionable although valid)

Observations by mainly supervisors are sometimes used; observation is more reliable and useful when observer has extensive contact with trainee

Performance indices are considered to be objective measures (such as sales performance) and can provide highly precise data; however, not always the best measure since they sometimes contaminate individual performance with other events that impact performance

Behaviour

Three basic approaches:

Self-reports—Trainee indicates if and/or how often newly trained behaviours are used on the job; most frequently used measure

Observations—Others observe and record whether and/or how often trainee used the newly trained behaviours on the job

Production indicators—Objective output assessed through productivity records

10-49

Behaviour refers to the display of the newly learned skills or competencies on the job (transfer of training)

Arguably the most important of all training effectiveness criteria

Self-reports remain the most frequently used measures of behaviour (accuracy may be questionable although valid)

Observations by mainly supervisors are sometimes used; observation is more reliable and useful when observer has extensive contact with trainee

Motivation

Two types of training motivation in training:

Training motivation: Usually administered before the start of the training program

Motivation to transfer: Typically assessed after training is completed

10-50

Self-Efficacy

Assess a person’s confidence in engaging in specific behaviours or achieving specific goals

Measures focus on assessing trainees’ level of confidence for performing specific tasks and behaviours

10-51

Self-efficacy refers to beliefs that trainees hold about their ability to perform the behaviours that were taught in a training program

Perceived and/or Anticipated Support

Two important measures of support are:

Perceived support

Anticipated support

10-52

The measurement of perceived and/or anticipated support can be easily constructed for any training program

Specific questions can be designed to include the source of the support and the support received in applying the training in general or in transferring specific aspects of the training program

Perceived and/or Anticipated Support

Perceived support is the degree to which the trainee reports receiving support in her attempts to transfer the learned skills

Anticipated support is the degree to which the trainee expects to be supported in her attempts to transfer the learned skills

10-53

The measurement of perceived and/or anticipated support can be easily constructed for any training program

Specific questions can be designed to include the source of the support and the support received in applying the training in general or in transferring specific aspects of the training program

Perceived and/or Anticipated Support

Anticipated support measures are collected before and/or immediately following training

Actual support is measured weeks or months after the trainees have returned to work

10-54

The measurement of perceived and/or anticipated support can be easily constructed for any training program

Specific questions can be designed to include the source of the support and the support received in applying the training in general or in transferring specific aspects of the training program

Organizational Perceptions Variables

Several scales have been designed to measure perceptions of the transfer climate, a continuous learning culture, and the transfer system:

Transfer climate questionnaire (Rouiller & Goldstein)

Continuous learning culture (Tracey, Tannenbaum, & Kavanagh)

LTSI approach assesses factors associated with training transfer (Holton)

10-55

Organizational Results

The effects of training on the organization rather than on the trainee

Results criteria are often viewed as the ultimate criteria and may include measures such as productivity, quality, profitability, customer satisfaction, and accidents

Hard data: Results that are assessed objectively

Soft data: Results that are assessed through perceptions and judgments

10-56

Organizational Results: Hard Data

Hard data are objective measures that fall in one of four categories:

Quantity (the number of items sold or produced)

Quality (scrap rates and product returns)

Time (downtime or time to complete assignments)

Costs (sales expenses, benchmarks)

10-57

An evaluation may also focus on “results criteria” related to the effects of the training on the organization instead of the trainee

Questions may include, How has the organization benefited from the training program? Have profits increased?

Hard (objective) data and soft (subjective) data can be used

Return on expectations involves those who are involved in the training deciding exactly what they expect from the training, and sometime later deciding whether the performance results are in line with expectations

Organizational Results: Soft Data

Soft data are measures of beliefs, attitudes, and perceptions and usually involve judgments, observations, or perceptions of an outcome

Self-reports are frequently measured when it is difficult to measure some skills, such as decision making

10-58

An evaluation may also focus on “results criteria” related to the effects of the training on the organization instead of the trainee

Questions may include: How has the organization benefited from the training program? Have profits increased?

Hard (objective) data and soft (subjective) data can be used

Organizational Results

In some cases, it is difficult to assess the impact of training directly; therefore, the focus is on expectations of training instead

Return on expectations is the measurement of a training program’s ability to meet managerial expectations

10-59

An evaluation may also focus on “results criteria” related to the effects of the training on the organization instead of the trainee

Questions may include: How has the organization benefited from the training program? Have profits increased?

Hard (objective) data and soft (subjective) data can be used

Data Collection Designs in Training Evaluation

All training evaluations involve one or more of these three comparisons:

Trainee states relative to a predetermined criterion (e.g., after studying the chapter, can you pass a knowledge test?)

Trainee changes (e.g., after studying the chapter, is your knowledge of training evaluation greater than it was before?)

10-60

A number of alternatives are available to the evaluator

The data may be subjective or objective and may be gathered from one or more sources, including the trainees, supervisors, or performance records

Data Collection Designs in Training Evaluation

Trainees compared to non-trained people (e.g., compared to students who have not read the chapter, do you know more about training evaluation than they do?)

10-61

A number of alternatives are available to the evaluator

The data may be subjective or objective and may be gathered from one or more sources, including the trainees, supervisors, or performance records

Data Collection Designs in Training Evaluation

Descriptive data collection designs:

Post-only data collection design

The pre-post design

The time series design

Causal evaluation designs

Internal Referencing Strategy (IRS)

10-62

A number of alternatives are available to the evaluator

The data may be subjective or objective and may be gathered from one or more sources, including the trainees, supervisors, or performance records

Descriptive Training Evaluation Models

Descriptive data collection designs indicate the post-training states of trainees in terms of attitudes, knowledge and/or behaviours and include:

Post-only data collection design

The pre-post design

The time series design

10-63

A number of alternatives are available to the evaluator

The data may be subjective or objective and may be gathered from one or more sources, including the trainees, supervisors, or performance records

Descriptive Training Evaluation Models: Post-Only Design

Evaluation only after the training has occurred

Often for certification purposes, the organization is required to demonstrate that its employees have attained a pre-determined level of proficiency after training

Most organizations use the post-only design and measure trainee reactions

10-64

Often, for certification purposes the organization is required to demonstrate that its employees have attained, after training, a predetermined level of proficiency

In these cases, a post-only design may be used

Post-only design is the simplest and easiest method to use

Descriptive Training Evaluation Models: Pre-Post Design

Trainees are measured twice, once before (pre) and once after (post)

Used when the goal of the training evaluation is to assess trainee improvement in knowledge, skills, and/or performance

Training effectiveness is inferred when the post-training scores are significantly higher than the pre-training ones

10-65

If the goal is to assess trainee improvement in knowledge, skills, and/or performance, a pre-post design is indicated

Measurement takes place twice (before and after the training)

The training program is declared successful when the post-training scores are significantly higher than the pre-training ones

Limitations to this process include:

1. There are many reasons that might explain a pre-post change (and training may be only one of them)

2. Since only one post-training measure is taken, it is not possible to know whether the change is long lasting

Descriptive Training Evaluation Models: Time Series Design

An extension of pre-post design

Trainees are assessed several different times both before and after the training experience

Most time series–based evaluations rely on objective performance records as this information is repeatedly collected without disturbing the employees

10-66

Instead of measuring trainees once before and once after training, trainees are assessed at several times both before and after the training experience

This model is most practical when outcomes measured are job performance indicators such as sales or production losses, since these data can be collected repeatedly before and after without disturbing the employees

Descriptive Training Evaluation Models: Time Series Design

Effectiveness is indicated when a significant improvement is noted immediately following training

The time series is an improvement over the pre-post as it assesses the degree to which that improvement persists

10-67

Instead of measuring trainees once before and once after training, trainees are assessed at several times both before and after the training experience

Limitations of Descriptive Evaluation Models

Practical to implement and they can provide some useful information

Many events coincide with training; descriptive models cannot isolate the effects of training with these events

Therefore, these models are called non-causal training evaluation designs because you can’t know if the training caused changes in trainees

10-68

These designs all provide useful information and are relatively practical to implement

However, there are limitations

It is important to consider whether the results obtained would have occurred without the training

Isolating the effects of other potential causes is the issue

Causal Evaluation Designs

Causal evaluation designs collect data from two “identical” groups of employees: one is trained and one, called a control group, is not

Compare the learning, skills, and/or performance of those who receive training to a comparable group of employees who do not

Training is judged to be effective when the trained group outperforms the untrained group on the relevant post-training measures

10-69

The development of causal training evaluation requires five steps:

Select the basic designs (post-only, pre-post, time series) as a function of the constraints and informational needs of the organization

Identify a group of employees whose jobs are similar and for whom the need for training has been identified

Randomly divide these employees into two groups: one group is to attend training, the other, called the control group, is not

Measure both groups on the same outcome variables at the same time

Statistically compare the outcomes achieved by both groups

Causal Evaluation Designs

The development of a causal training evaluation requires five steps.

Select the basic designs (post only, pre-post, time series) as a function of the constraints and of the informational needs of the organization

Identify a group of “identical” employees for whom the need for the specific training is pertinent

10-70

The development of causal training evaluation requires five steps:

Select the basic designs (post-only, pre-post, time series) as a function of the constraints and informational needs of the organization

Identify a group of employees whose jobs are similar and for whom the need for training has been identified

Randomly divide these employees into two groups: one group is to attend training, the other, called the control group, is not

Measure both groups on the same outcome variables at the same time

Statistically compare the outcomes achieved by both groups

Causal Evaluation Designs

Randomly divide these employees into two groups. One group is to attend training, the other, called the control group, is not

Measure both groups on the same outcome variables at the same time

Statistically compare the outcomes achieved by both groups

Evaluation designs that follow these five steps are known as “experimental”

10-71

The development of causal training evaluation requires five steps:

Select the basic designs (post-only, pre-post, time series) as a function of the constraints and informational needs of the organization

Identify a group of employees whose jobs are similar and for whom the need for training has been identified

Randomly divide these employees into two groups: one group is to attend training, the other, called the control group, is not

Measure both groups on the same outcome variables at the same time

Statistically compare the outcomes achieved by both groups

Causal Evaluation Designs

When you cannot randomly assign trainees to the two groups you have a quasi-experimental design

Quasi-experimental designs have a trained group that is compared to a group of employees who do a similar job in similar circumstances—a comparison group

10-72

The development of causal training evaluation requires five steps:

Select the basic designs (post-only, pre-post, time series) as a function of the constraints and informational needs of the organization

Identify a group of employees whose jobs are similar and for whom the need for training has been identified

Randomly divide these employees into two groups: one group is to attend training, the other, called the control group, is not

Measure both groups on the same outcome variables at the same time

Statistically compare the outcomes achieved by both groups

The Internal Referencing Strategy (IRS)

Data are collected from the trained group only using a pre-post design

On both pre- and post- measures, two types of outcomes are measured:

Relevant outcome (why training is provided)

Non-germane outcome (one for which training could have been provided but was not)

If training was effective, there should be greater pre-post change on the relevant outcome vs. the non-germane outcome

10-73

This is a hybrid quasi-experimental design that allows causal inference when the simple pre-post design (with no control group) is used

Research has shown that IRS provides inferences equivalent to those that can be drawn from the more complex pre-post experimental models

Provides a major improvement to the pre-post design at the cost of being slightly more complex

The Internal Referencing Strategy (IRS)

Causality is shown when the pre-post change is statistically larger for the relevant outcomes than is the case for the irrelevant but germane outcomes

If the pre-post change is equivalent on both the relevant and the irrelevant contents, we cannot claim that this was “caused” by the training and we would need to conclude that training was not effective

10-74

This is a hybrid quasi-experimental design that allows causal inference when the simple pre-post design (with no control group) is used

Research has shown that IRS provides inferences equivalent to those that can be drawn from the more complex pre-post experimental models

Provides a major improvement to the pre-post design at the cost of being slightly more complex

Summary of Data Collection Designs

More complex designs yield much better data and more convincing conclusions; however, using them is more complicated, difficult and costly

The evaluator will need to choose the design that provides management the information it needs

10-75

Models, Outcomes, and Data Collection Designs

Trainers might begin by using Decision-Based Evaluation

Knowing the target will dictate whether a post-only or pre-post design is required

Then consider whether a causal or non-causal data collection design is required

10-76

Models, Outcomes, and Data Collection Designs

Then focus on the variables to be measured

Next consider the type and the source of the data

These decisions will be driven by the information needs and by practical considerations

The evaluation choices will have an impact on the legitimacy of the conclusions reached

10-77

Summary

Considered main reasons for evaluating training programs as well as barriers to training evaluation

Described the types of data collected and the purpose of evaluation

Presented models of training evaluation including Kirkpatrick’s model, COMA, and Decision-Based Evaluation

Examined variables required for an evaluation and methods and techniques required to measure variables

10-78

Summary

Described descriptive and causal data collection designs and the internal referencing strategy

Discussed advantages and disadvantages of each

10-79

Key Terms

Affective reactions

Anticipated support

Causal evaluations

COMA

Decision-Based Evaluation (DBE)

Declarative learning

Descriptive evaluations

Formative evaluations

Hard data

Perceived support

Procedural learning

Return on expectations

Soft data

Summative evaluations

Training evaluation

Utility reactions

10-80