Training & Development Course Project

Chapter Six

Training Evaluation

Objectives

After reading this chapter, you should be able to

1. Explain why evaluation is important.

2. Identify and choose outcomes to evaluate a training program.

3. Discuss the process used to plan and implement a good training evaluation.

4. Discuss the strengths and weaknesses of different evaluation designs.

5. Choose the appropriate evaluation design based on the characteristics of the company and the importance and purpose of the training.

6. Conduct a cost-benefit analysis for a training program.

7. Explain the role of Big Data workforce analytics and dashboards in determining the value of training practices.

Evaluation Helps Ensure Guests Find Paradise on Hawaii’s North Coast

Turtle Bay Resort is located on the stunning coastline of Oahu, Hawaii. Guests can choose many activities while enjoying the beautiful surroundings at the at the resort including dining on farm-to-table meals, enjoying the ocean while surfing and paddling, relaxing with a massage, or taking an invigorating horseback ride. New management made a $40 million investment and renovations to revitalize the resort and inspire its guests to make them part of the local community. They recognized that the physical changes to the resort were necessary and important. But they also believe that investing in training leads to happier and more engaged employees, and, in turn, leads to satisfied guests. Training at Turtle Bay includes classroom learning, role-plays, and social learning. All training incorporates Turtle Bay’s 6 Values that provide the standard by which employees work and serve guests. The values relate to time (Manawa), goodness (Pono), caring for others (Malama), support of family (Hanai), Aloha (kindness), and local engagement and culture (Kama’aina). The values include underlying behaviors and practices such as greeting guests promptly (Manawa), hold others accountable (Malama), demonstrating interest in peers (Hanai), engage guests and peers (Aloha), and treat locals as guests and guests as246locals (Kama’aina). Every employee is required to attend a training program that focuses on the values. Managers are asked to complete an individual development plan based on their self-rating as well as ratings from employees, peers, and their manager on how well they applied these values at work. Employees also complete a self-assessment and personal improvement plan based on the values.

Laulima (many hands working together), a service quality training program, is an extension of the values. The program includes modules on greeting guests, service delivery, service recovery, and knowledge of service, food and beverages, history, and culture. The program was developed using input from employees who were chosen as the best service providers at Turtle Bay. Each module has a workbook that guides employees through a series of exercises. New employees attend a scavenger hunt to help them understand the property and its plants and animals. Managers are expected to help teach employees and reinforce what they learn. Employees also have to learn how to use Guidepost, a lobby experience center that provides concierge and guest services in an interactive space. Guidepost includes iPads and touchscreen panels for viewing activities and reviewing, learning about, and booking local activities.

To reinforce delightful customer service and emphasize the importance of training, Turtle Bay has several rewards programs. The Ho’ohana Awards recognize employees for exceptional service for guest and employees. Ali’i and llima Awards for exceptional service are given each quarter to a manager and two other employees. The Best of the Best Award is given to outstanding employees who continuously demonstrate exceptional service.

Turtle Bay collects several different types of data to determine the success of training. The most important measure is guest satisfaction, which includes using social media tools like TripAdvisor, Revinate, and Market Metrix. A values feedback system is used to determine how well employees are applying the Value practices. This data is evaluated for employees, managers, department, and functional areas. Occupancy rates, market share, sales performance, internal promotions, and turnover are used as financial measures. To assess employee engagement, two surveys are conducted each year.

Source: L. Freifeld, “Turtle Bay turnaround,” training (January/February 2015): 120–123.

INTRODUCTION

As the opening vignette illustrates, Turtle Bay Resort wants to show that the time, money, and effort devoted to training makes a difference. That is, the training function was interested in assessing the effectiveness of training programs. Training effectiveness refers to the benefits that the company and the trainees receive from training. Benefits for trainees may include learning new skills or behaviors. Benefits for the company may include increased sales and more satisfied customers. A training evaluation measures specific outcomes or criteria to determine the benefits of the program. Training outcomes or criteria refer to measures that the trainer and the company use to evaluate training programs. To determine the effectiveness of training, an evaluation needs to occur. Training evaluation247refers to the process of collecting the outcomes needed to determine whether training is effective. For Turtle Bay, the outcomes included engagement, guest satisfaction, and financial measures such as occupancy rates. Although not discussed in the vignette, Turtle Bay also has to be confident that the data its information-gathering process is providing accurate data for making conclusions about the effectiveness of its training programs. The evaluation design refers to the collection of information—including what, when, how, and from whom—that will be used to determine the effectiveness of the training program. Any organization that evaluates training has to be confident that training—rather than same other factor—is responsible for changes in the outcomes of interest (e.g., turnover, productivity). The degree of confidence that any changes in the outcomes of interest is due to training depends on the type of evaluation design that is used.

Recall the Instructional Systems Design model shown in  Figure 1.1  and the topics covered in  Chapters Two  through  Five . The information from the needs assessment, the characteristics of the learning environment, and the steps taken to ensure transfer of training should all be used to develop an evaluation plan. In order to identify appropriate training outcomes, a company needs to look at its business strategy, its organizational analysis (Why are we conducting training? How is it related to the business?), its person analysis (Who needs training?), its task analysis (What is the training content?), the learning objectives of the training, and its plan for training transfer.

This chapter will help you understand why and how to evaluate training programs. The chapter begins by discussing the types of outcomes used in training program evaluation. The next section of the chapter discusses the practical factors to consider when choosing an evaluation design. An overview of the types of designs is presented. The chapter reviews the process involved in conducting a program evaluation. The chapter concludes with a discussion of metrics that can be used to evaluate the strategic value of the training function.

REASONS FOR EVALUATING TRAINING

Companies are investing millions of dollars in training programs to help gain a competitive advantage. Companies invest in training because learning creates knowledge; often, it is this knowledge that distinguishes successful companies and employees from those who are not. Research summarizing the results of studies that have examined the linkage between training and human resource outcomes (such as attitudes and motivation, behaviors, and human capital), organizational performance outcomes (performance and productivity), or financial outcomes (profits and financial indicators) has found that companies that conduct training are likely to have more positive human resource outcomes and greater performance outcomes. 1  The influence of training is largest for organizational performance outcomes and human resource outcomes and weakest for financial outcomes. This result is not surprising, given that training can least affect an organization’s financial performance and may do so through its influence on human resource practices. As emphasized in  Chapter Two , “Strategic Training,” training is more strongly related to organizational outcomes when it is matched with the organization’s business strategy and capital intensity. Because companies have made large dollar investments in training and education and view training as a strategy to be successful, they expect the outcomes or benefits related to training to be measurable.

248

Asurion developed a leadership development program for high-potential employees. 2  The program includes on-the-job rotations, mentoring and coaching from senior leaders, and classroom and on-the-job training. Asurion conducted an evaluation to determine the program’s influence on both high-potential employees and business results. Evaluation showed that since it was developed, 71 percent of program participants have received promotions. Also, program participants have rated professional development as one of the top three strengths on the company’s engagement surveys. Departments have found that projects completed during the job rotations have led to positive business results, including reducing product handling time and improved quality scores. At Jiffy Lube, training evaluation involves more than just counting the number of programs employees attend each year. 3  Jiffy Lube’s business depends on properly servicing customers’ cars so they are satisfied and will be repeat customers. As a result, Jiffy Lube employees must be certified to perform a service. Jiffy Lube tracks both certifications and their relationship to business results. They track both employee and store-level certifications. Entry-level certifications must be completed within thirty days after an employee is hired. Seventy-six percent of Jiffy Lube’s stores are at 80 to 100 percent certification. One-third of the stores with 100 percent certification have average customer sales 9 percent higher than all stores.

Training evaluation provides a way to understand the investments that training produces and provides information needed to improve training. 4  If the company receives an inadequate return on its investment in training, the company will likely reduce its investment in training or look for training providers outside the company who can provide training experiences that improve performance, productivity, customer satisfaction, or whatever other outcomes the company is interested in achieving. Training evaluation provides the data needed to demonstrate that training does offer benefits to the company. Training evaluation involves both formative and summative evaluation. 5

Formative Evaluation

Formative evaluation refers to the evaluation of training that takes place during program design and development. That is, formative evaluation helps ensure that (1) the training program is well organized and runs smoothly, and (2) trainees learn and are satisfied with the program. Formative evaluation provides information about how to make the program better; it usually involves collecting qualitative data about the program. Qualitative data include opinions, beliefs, and feelings about the program. Formative evaluations ask customers, employees, managers, and subject-matter experts (SMEs) their opinions on the description of the training content and objectives and the program design. These people are also asked to evaluate the clarity and ease of use of a part of the training program that is demonstrated to them in the way that it will be delivered (e.g., online, face-to-face, or video). 6  The formative evaluation is conducted either individually or in groups before the program is made available to the rest of the company. Trainers may also be involved to measure the time requirements of the program. As a result of the formative evaluation, training content may be changed to be more accurate, easier to understand, or more appealing. The training method can be adjusted to improve learning (e.g., provide trainees with more opportunities to practice or give feedback). Also, introducing the training program as early as possible to managers and customers helps in getting them to buy into the program, which is critical for their role in helping employees learn and transfer skills. It also allows their concerns to be addressed before the program is implemented.

249

Formative evaluation involves pilot testing. Pilot testing refers to the process of previewing the training program with potential trainees and managers or with other customers (persons who are paying for the development of the program). Pilot testing can be used as a “dress rehearsal” to show the program to managers, trainees, and customers. It should also be used for formative evaluation. For example, a group of potential trainees and their managers may be asked to preview or pilot test a web-based training program. As they complete the program, trainees and managers may be asked to provide their opinions about whether graphics, videos, or music used in the program contributed to (or interfered with) learning. They may also be asked how easy it was to move through the program and complete the exercises, and they may be asked to evaluate the quality of feedback the training program provided after they completed the exercises. The information gained from this preview would be used by program developers to improve the program before it is made available to all employees. St. George Bank developed a new web-based training system for bank tellers. 7  Before the program was provided to all bank tellers, it was reviewed by a small group of them who were considered to be typical users of the program. The tellers provided suggestions for improvement, and the instructional designers incorporated their suggestions into the final version of the program.

Summative Evaluation

Summative evaluation refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program. That is, have trainees acquired knowledge, skills, attitudes, behavior, or other outcomes identified in the training objectives? Summative evaluation may also include measuring the monetary benefits (also known as return on investment or ROI) that the company receives from the program. Summative evaluation usually involves collecting quantitative (numerical) data through tests, ratings of behavior, or objective measures of performance such as volume of sales, accidents, or patents.

From the discussion of summative and formative evaluation, it is probably apparent to you why a training program should be evaluated:

1. To identify the program’s strengths and weaknesses. This includes determining if the program is meeting the learning objectives, if the quality of the learning environment is satisfactory, and if transfer of training to the job is occurring.

2. To assess whether the content, organization, and administration of the program—including the schedule, accommodations, trainers, and materials—contribute to learning and the use of training content on the job.

3. To identify which trainees benefit most or least from the program.

4. To assist in marketing programs through the collection of information from participants about whether they would recommend the program to others, why they attended the program, and their level of satisfaction with the program.

5. To determine the financial benefits and costs of the program.

6. To compare the costs and benefits of training versus nontraining investments (such as work redesign or a better employee selection system).

7. To compare the costs and benefits of different training programs to choose the best program.

250

OVERVIEW OF THE EVALUATION PROCESS

Before the chapter explains each aspect of training evaluation in detail, you need to understand the evaluation process, which is summarized in  Figure 6.1 . The previous discussion of formative and summative evaluation suggests that training evaluation involves scrutinizing the program both before and after the program is completed.  Figure 6.1  emphasizes that training evaluation must be considered by managers and trainers before training has actually occurred. As was suggested earlier in this chapter, information gained from the training design process shown in  Figure 6.1  is valuable for training evaluation.

FIGURE 6.1 The Evaluation Process

Sources: Based on D. A. Grove and C. Ostroff, “Program Evaluation,” in Developing Human Resources, ed. K. N. Wexley (Washington, D. C.: Bureau of National Affairs, 1991): 5-185–5-220; K. Kraiger, D. McLinden, and W. Casper, “Collaborative Planning for Training Impact,” Human Resource Management (Winter 2004): 337–351.

The evaluation process should begin with determining training needs (as discussed in  Chapter Three , “Needs Assessment”). Needs assessment helps identify what knowledge, skills, behavior, or other learned capabilities are needed. Needs assessment also helps identify where the training is expected to have an impact. Needs assessment helps focus the evaluation by identifying the purpose of the program, the resources needed (human, financial, and company), and the outcomes that will provide evidence that the program has been effective. 8  The next step in the process is to identify specific, measurable training objectives to guide the program. The characteristics of good objectives are discussed in  Chapter Four , “Learning and Transfer of Training.” The more specific and measurable these objectives are, the easier it is to identify relevant outcomes for the evaluation. Besides considering the learning and program objectives in developing learning outcomes, it is also important to consider the expectations of those individuals who support the program and have an interest in it (stakeholders such as trainees, managers, and trainers). 9  If the needs assessment was done well, the stakeholders’ interests likely overlap considerably with the learning and program objectives. Analysis of the work environment to determine transfer of training (discussed in  Chapter Five , “Program Design”) can be useful for determining how training content will be used on the job. Based on the learning objectives and analysis of transfer of training, outcome measures are designed to assess the extent to which learning and transfer have occurred.

251

Once the outcomes have been identified, the next step is to determine an evaluation strategy. Factors such as expertise, how quickly the information is needed, change potential, and the organizational culture should be considered in choosing a design. Planning and executing the evaluation involves previewing the program (formative evaluation), as well as collecting training outcomes based on the evaluation design. The results of the evaluation are used to modify, market, or gain additional support for the program. The results of the evaluation should also be used to encourage all stakeholders in the training process—including managers, employees, and trainers—to design or choose training that helps the company meet its business strategy and helps managers and employees meet their goals. 10

OUTCOMES USED IN THE EVALUATION OF TRAINING PROGRAMS

To evaluate its training program, a company must decide how it will determine the program’s effectiveness; that is, it must identify what training outcomes or criteria it will measure.

Table 6.1  shows the six categories of training outcomes: reaction outcomes, learning or cognitive outcomes, behavior and skill-based outcomes, affective outcomes, results, and return on investment. 11

TABLE 6.1 Evaluation Outcomes

Sources: Based on K. Kraiger, J. K. Ford, and E. Salas, “Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation,” Journal of Applied Psychology, 78 (2) (1993): 311–328; K. Kraiger, “Decision-Based Evaluation,” in Creating, Implementing, and Managing Effective Training and Development, ed. K. Kraiger (San Francisco: Jossey-Bass, 2002): 331–375; D. Kirkpatrick, “Evaluation,” in The ASTD Training and Development Handbook, 2nd ed., ed. R. L. Craig (New York: McGraw-Hill, 1996): 294–312.

Table 6.1  shows training outcomes, the level they correspond to in Kirkpatrick’s evaluation model, a description of each of the outcomes and how they are measured, and the question that each outcome can help answer. Kirkpatrick’s original evaluation model included only four levels (reaction, learning, behavior, and results) but recent thinking suggests a fifth level, return on investment (ROI), is necessary to demonstrate the financial value of training. Both level 1 and level 2 outcomes (reactions and learning) are collected at the completion of training, before trainees return to the job. Level 3 outcomes (behavior/skills) can also be collected at the completion of training to determine trainees’ behavior or skill level at that point. To determine whether trainees are using training content back on the job (i.e., whether transfer of training has occurred), level 3, level 4, and/or level 5 outcomes can be collected. Level 3 criteria can be collected to determine whether behavior/skills are being used on the job. Level 4 and level 5 criteria (results and return on investment) can also be used to determine whether training has resulted in an improvement in business results, such as productivity or customer satisfaction. These criteria also help determine whether the benefits of training exceed their costs. Keep in mind that the levels do not indicate the importance of the outcomes or that lower-level outcomes cause higher-level outcomes. 12  That is, reactions cause learning, which in turn influences skills and results. The outcomes that are collected in evaluation are based on training needs, program objectives, and the strategic reasons for training. We discuss this in the section “Evaluation Practices,” later in the chapter.

Reaction Outcomes

Reaction outcomes refer to trainees’ perceptions of the program, including the facilities, trainers, and content. (Reaction outcomes are often referred to as a measure of “creature comfort.”) They are often called class or instructor evaluations. This information is typically collected at the program’s conclusion. You probably have been asked to complete class or instructor evaluations either at the end of a college course or a training program at work. Reactions are useful for identifying what trainees thought was successful or what inhibited learning. Reaction outcomes are level 1 (reaction) criteria in Kirkpatrick’s framework.

252

Reaction outcomes are typically collected via a questionnaire completed by trainees. A reaction measure should include questions related to the trainee’s satisfaction with the instructor, training materials, and training administration (e.g., ease of registration and accuracy of course description), as well as the clarity of course objectives and usefulness of the training content. 13   Table 6.2  shows a reaction measure that contains questions about253these areas. Reaction measures can also include open-ended questions that ask learners about the experience. 14  Example of open-ended questions include “What did you learn that you are most likely to try on the job?” and “What topics covered in the class still seem confusing?” These types of questions will take longer to review than analyzing items such as are shown in  Table 6.2 , but they can potentially provide more detailed suggestions about how to improve program design and delivery.

TABLE 6.2 Sample Reaction Measure

Read each statement below. Indicate the extent to which you agree or disagree with each statement using the scale below.
Strongly Disagree	Disagree	Neither	Agree	Strongly Agree
1	2	3	4	5
1. I had the knowledge and skills needed to learn in this course. 2. The facilities and equipment made it easy to learn. 3. The course met all of the stated objectives. 4. I clearly understood the course objectives. 5. The way the course was delivered was an effective way to learn. 6. The materials I received during the course were useful. 7. The course content was logically organized. 8. There was enough time to learn the course content. 9. I felt that the instructor wanted us to learn. 10. I was comfortable asking the instructor questions. 11. The instructor was prepared. 12. The instructor was knowledgeable about the course content. 13. I learned a lot from this course. 14. What I learned in this course is useful for my job. 15. The information I received about the course was accurate. 16. Overall, I was satisfied with the instructor. 17. Overall, I was satisfied with the course.

An accurate evaluation needs to include all the factors related to a successful learning environment. 15  Most instructor or class evaluations include items related to the trainer’s preparation, delivery, ability to lead a discussion, organization of the training materials and content, use of visual aids, presentation style, ability and willingness to answer questions, and ability to stimulate trainees’ interest in the course. These items come from trainer’s manuals, trainer certification programs, and observation of successful trainers. Conventional wisdom suggests that trainees who like a training program (who have positive reactions) learn more and are more likely to change behaviors and improve their performance (transfer of training). Is this really the case? Recent studies suggest that reactions have the largest relationship to changes in affective learning outcomes. 16  Also, research has found that reactions are significantly related to changes in declarative and procedural knowledge, which challenges previous research suggesting that reactions are unrelated to learning. For courses such as diversity training or ethics training, trainee reactions are especially important because they affect learners’ receptivity to attitude change. Reactions have been found to have the strongest relationship with post-training motivation, trainee self-efficacy, and declarative knowledge when technology is used for254instructional delivery. This suggests that for online or e-learning training methods, it is important to ensure that it is easy for trainees to access them and the training content is meaningful (i.e., linked to their current job experiences, tasks, or work issues).

Learning or Cognitive Outcomes

Cognitive outcomes are used to determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, and processes emphasized in the training program. Cognitive outcomes measure what knowledge trainees learned in the program. Cognitive outcomes are level 2 (learning) criteria in Kirkpatrick’s framework. Typically, pencil-and-paper tests or self-assessments are used to assess cognitive outcomes. Self-assessments refer to learners’ estimates of how much they know or have learned from training. Tests and quizzes rather than self-assessments are the preferred measures of learning. This is because self-assessments are only moderately related to learning and influenced by how much learners liked the course or were motivated to learn rather than what they actually learned. 17   Table 6.3  provides an example of items from a pencil-and-paper test used to measure trainees’ knowledge of decision-making skills. These items help measure whether a trainee knows how to make a decision (the process that he or she would use). They do not help to determine if the trainee will actually use decision-making skills on the job.

TABLE 6.3 Sample Test Items Used to Measure Learning

For each question, check all that apply. 1. If my boss returned a piece of work to me and asked me to make changes on it, I would: __ Prove to my boss that the work didn’t need to be changed. __ Do what the boss said, but ask where the boss thought changes were needed __ Make the changes without talking to my boss. __ Request a transfer from the department. 2. If I were setting up a new process in my office, I would: __ Do it on my own, without asking for help. __ Ask my boss for suggestions. __ Ask the people who work for me for suggestions. __ Discuss it with friends outside the company.

Source: Based on A. P. Carnevale, L. J. Gainer, and A. S. Meltzer, Workplace Basics Training Manual (San Francisco: Jossey-Bass, 1990): 8–12.

Consider how Grant Thornton, the public accounting and consulting firm, used tests to evaluate one of its training programs. The company introduced a new tax services methodology and tools to approximately 1,400 tax professionals in more than fifty offices. 18  The training solution, known as Tax Symphony, involved blended learning, which included a three-day national- and local-office classroom programs, web-based performance support, and webcasts. The new methodology was introduced during a tax conference. After the conference, the content was taught at local offices using avatars presenting prerecorded audio clips combined with group application activities. Evaluation results showed that 86 percent of the program participants completing the post-test passed it with an average passing score of 85 percent. Average post-test scores improved by 11 percentage points over national classroom post-test scores. Using avatars instead of real instructors in local offices saved $72,000 in costs.

255

Behavior and Skill-Based Outcomes

Skill-based outcomes are used to assess the level of technical or motor skills and behaviors. Skill-based outcomes include acquisition or learning of skills (skill learning) and use of skills on the job (skill transfer). Skill-based outcomes relate to Kirkpatrick’s level 2 (learning) and level 3 (behavior). The extent to which trainees have learned skills can be evaluated by observing their performance in work samples such as simulators. Skill transfer is usually determined by observation. For example, a resident medical student may perform surgery while the surgeon carefully observes, giving advice and assistance as needed. Trainees may be asked to provide ratings of their own behavior or skills (self-ratings). Peers, managers, and subordinates may also be asked to rate trainees’ behavior or skills based on their observations. Because research suggests that the use of only self-ratings likely results in an inaccurately positive assessment of skill or behavior transfer of training, it is recommended that skill or behavior ratings be collected from multiple perspectives (e.g., managers and subordinates or peers). 19   Table 6.4  shows a sample rating form. This form was used as part of an evaluation of a training program developed to improve school principals’ management skills. To evaluate several of its training programs, Orkin, a pest control company, requires managers to observe each employee’s performance on the job and complete a detailed checklist. 20  The checklist helps determine that behaviors introduced in training are being used at work.

256

TABLE 6.4 Sample Rating Form Used to Measure Behavior

Rating task: Consider your opportunities over the past three months to observe and interact with the principal/assistant principal you are rating. Read the definition and behaviors associated with the skill. Then complete your ratings using the following scale:
Always	Usually	Sometimes	Seldom	Never
1	2	3	4	5

I.	Sensitivity: Ability to perceive the needs, concerns, and personal problems of others; tact in dealing with persons from different backgrounds; skill in resolving conflict; ability to deal effectively with people concerning emotional needs; knowing what information to communicate to whom.

To what extent in the past three months has the principal or assistant principal:

__  1.	Elicited the perceptions, feelings, and concerns of others?
__  2.	Expressed verbal and nonverbal recognition of the feelings, needs, and concerns of others?
__  3.	Taken actions that anticipated the emotional effects of specific behaviors?
__  4.	Accurately reflected the point of view of others by restating it, applying it, or encouraging feedback?
__  5.	Communicated all information to others that they needed to perform their job?
__  6.	Diverted unnecessary conflict with others in problem situations?

II.	Decisiveness: Ability to recognize when a decision is required and act quickly. (Disregard the quality of the decision.)

To what extent in the past three months has this individual:

__  7.	Recognized when a decision was required by determining the results if the decision was made or not made?
__  8.	Determined whether a short- or long-term solution was most appropriate to various situations encountered in the school?
__  9.	Considered decision alternatives?
__10.	Made a timely decision based on available data?
__11.	Stuck to decisions once they were made, resisting pressures from others?

Affective Outcomes

Affective outcomes include attitudes and motivation. Affective outcomes that might be collected in an evaluation include tolerance for diversity, employee engagement, motivation to learn, safety attitudes, and customer service orientation. Affective outcomes can be measured using surveys.  Table 6.5  shows an example of questions on a survey used to measure career goals, plans, and interests. The specific attitude of interest depends on the program objectives. Affective outcomes relate to Kirkpatrick’s level 2 (learning), level 3 (behavior), or level 4 (results), depending on how they are evaluated. If trainees were asked about their attitudes on a survey, that would be considered a learning measure. For example, attitudes toward career goals and interests might be an appropriate outcome to use to evaluate training focusing on employees self-managing their careers. Walgreens included engagement survey scores as one of the outcomes in its new Assistant Store Manager training program. 21  The program included action learning projects (work in a small team with other trainees to solve a business problem), videos, on-the-job projects, and online learning. Trainees’ engagement scores improved 15 percent a year after attending the program.

TABLE 6.5 Example of Affective Outcomes: Career Goals, Plans, and Interests

1. At this time, I have a definite career goal in mind. 2. I have a strategy for achieving my career goals. 3. My manager is aware of my career goals. 4. I have sought information regarding my specific areas of career interest from friends, colleagues, or company career sources. 5. I have initiated conversations concerning my career plans with my manager.

Results

Results are used to determine the training program’s payoff for the company. Examples of results outcomes include increased production and reduced costs related to employee turnover rates of top talent (managers or other employees), accidents, and equipment downtime, as well as improvements in product quality or customer service. 22  Results outcomes are level 4 (results) criteria in Kirkpatrick’s framework. For example, Aetna, Inc. determines the effectiveness of performance support tools based on how it impacts the client experience (Kirpatrick’s Level 4 criteria). 23  Customer service at Aetna depends on the successful training of employees on fourteen different products that are provided to customers. Aetna developed a wiki so that each of its product areas would be responsible for uploading content. The wiki provided performance support to employees who might need access to information about each of the fourteen products. Feedback from customers indicated that they felt the tool was useful and effective. USAA, a company who provides insurance, banking, investments, and retirement products to current and former members of the U.S. military and their families, designed a new e-learning program to help its new sales team members learn about its products and systems. 24  Sales data was used to evaluate effectiveness of the program. Program graduates exceeded sales goals by257over 22 percent three to six months after completing the program. WakeMed Health and Hospitals, a health-care system located in North Carolina, uses a human patient simulator, known as Stan, for training. 25  Stan can blink, breathe, and be given oxygen, CPR, and medications. Stan was used in simulations attending by nurses and other patient care and emergency staff involved in transporting critically ill heart patients to WakeMed’s Heart Center to help them understand how to improve the efficiency and care administered during the transportation process and mortality rates. Results of the simulation found that the simulation helped decrease mortality rates, prevent delays in catheter placement, and improve efficiency.

Return on Investment

Return on investment (ROI) refers to comparing the training’s monetary benefits with the cost of the training. ROI is often referred to as level 5 evaluation (see  Table 6.1 ). ROI can be measured and communicated based on a percentage or a ratio. For example, assume that a new safety training program results in a decline of 5 percent in a company’s accident rate. This provides a total annual savings (the benefit) of $150,000 in terms of lost workdays, material and equipment damage, and workers’ compensation costs. The training program costs $50,000 to implement (including both direct and indirect costs). To calculate the ROI, you need to subtract the training costs from the benefits, divide by the costs, and multiply by 100. That is, ROI = ((150,000 – 50,000) ÷ 50,000) × 100% = 200%. The ROI for this program is 200 percent. Another way to think about ROI is to consider it as ratio based on the return for every dollar spent. In this example, the company gained a net benefit of $2 for every dollar spent. This means the ROI is 2:1. Training costs can be direct and indirect. 26  Direct costs include salaries and benefits for all employees involved in training, including trainees, instructors, consultants, and employees who design the program; program material and supplies; equipment or classroom rentals or purchases; and travel costs. Indirect costs are not related directly to the design, development, or delivery of the training program. They include general office supplies, facilities, equipment, and related expenses; travel and expenses not directly billed to one program; training department management and staff salaries not related to any one program; and administrative and staff support salaries. Benefits are the value that the company gains from the training program.

Tata Consultancy Services LTD, a global information technology service company headquartered in India, measures ROI for its technology training programs. 27  To calculate the ROI, revenues earned as a result of training are calculated based on the billing rates of participants who attend the training and use the new skills. Then, training costs are subtracted from the revenues. ROI for the technical programs is 483 percent.

DETERMINING WHETHER OUTCOMES ARE APPROPRIATE

An important issue in choosing outcomes is to determine whether they are appropriate. That is, are these outcomes the best ones to measure to determine whether the training program is effective? Appropriate training outcomes need to be relevant, reliable, discriminative, and practical. 28

258

Relevance

Criteria relevance refers to the extent to which training outcomes are related to the learned capabilities emphasized in the training program. The learned capabilities required to succeed in the training program should be the same as those required to be successful on the job. The outcomes collected in training should be as similar as possible to what trainees learned in the program. That is, the outcomes need to be valid measures of learning. One way to ensure the relevancy of the outcomes is to choose outcomes based on the learning objectives for the program. Recall from  Chapter Four  that learning objectives show the expected action, the conditions under which the trainee is to perform, and the level or standard of performance.

Figure 6.2  shows two ways that training outcomes may lack relevance. Criterion contamination refers to the extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions. For example, if managers’ evaluations of job performance are used as a training outcome, trainees may receive higher ratings of job performance simply because the managers know they attended the training program, believe the program is valuable, and therefore give high ratings to ensure that the training looks like it positively affects performance. Criteria may also be contaminated if the conditions under which the outcomes are measured vary from the learning environment. That is, trainees may be asked to perform their learned capabilities using equipment, time constraints, or physical working conditions that are not similar to those in the learning environment.

FIGURE 6.2 Criterion Deficiency, Relevance, and Contamination.

For example, trainees may be asked to demonstrate spreadsheet skills using a newer version of spreadsheet software than they used in the training program. This demonstration likely will result in no changes in their spreadsheet skills from pretraining levels. In this case, poor-quality training is not the cause for the lack of change in their spreadsheet skills. Trainees may have learned the necessary spreadsheet skills, but the environment for the evaluation differs substantially from the learning environment, so no change in skill level is observed.

259

Criteria may also be deficient. Criterion deficiency refers to the failure to measure training outcomes that were emphasized in the training objectives. For example, the objectives of a spreadsheet skills training program emphasize that trainees both understand the commands available on the spreadsheet (e.g., compute) and use the spreadsheet to calculate statistics using a data set. An evaluation design that uses only learning outcomes such as a test of knowledge of the purpose of keystrokes is deficient because the evaluation does not measure outcomes that were included in the training objectives (e.g., use a spreadsheet to compute the mean and standard deviation of a set of data).

Reliability

Reliability refers to the degree to which outcomes can be measured consistently over time. For example, a trainer gives restaurant employees a written test measuring knowledge of safety standards to evaluate a safety training program that they attended. The test is given before (pretraining) and after (post-training) employees attend the program. A reliable test includes items for which the meaning or interpretation does not change over time. A reliable test allows the trainer to have confidence that any improvements in post-training test scores from pretraining levels are the result of learning that occurred in the training program, not test characteristics (e.g., items are more understandable the second time) or the test environment (e.g., trainees performed better on the post-training test because the classroom was more comfortable and quieter).

Discrimination

Discrimination refers to the degree to which trainees’ performance on the outcome actually reflects true differences in performance. For example, a paper-and-pencil test that measures electricians’ knowledge of electrical principles must detect true differences in trainees’ knowledge of electrical principles. That is, the test should discriminate on the basis of trainees’ knowledge of electrical principles. (People who score high on the test have a better understanding of the principles of electricity than do those who score low.)

Practicality

Practicality refers to the ease with which the outcome measures can be collected. One reason companies give for not including learning, performance, and behavior outcomes in their evaluation of training programs is that collecting them is too burdensome. (It takes too much time and energy, which detracts from the business.) For example, in evaluating a sales training program, it may be impractical to ask customers to rate the salesperson’s behavior because this would place too much of a time commitment on the customer (and probably damage future sales relationships).

EVALUATION PRACTICES

Figure 6.3  shows outcomes used in training evaluation practices. Surveys of companies’ evaluation practices indicate that reactions (an affective outcome) and cognitive outcomes are the most frequently used outcomes in training evaluation. 29  Despite the less frequent use of cognitive, behavioral, and results outcomes, research suggests that training can have a positive effect on these outcomes. 30  Keep in mind that while most companies are conducting training evaluations, some surveys indicate that 20 percent of all companies are not!

260

FIGURE 6.3 Training Evaluation Practices

Note: Respondents were 704 high-level business, HR and learning professionals participating in the ASTD/i4cp study, “The Value of Evaluation: Making Training Evaluations More Effective.”

Source: Based on L. Patel, 2010 State of the Industry Report (Alexandria, VA: American Society for Training and Development, 2010).

There are a number of reasons why companies don’t evaluate training. Learning professional reports with access to results and tools needed to obtain them are the most significant barriers. 31  Access to results is often determined by the extent to which managers and leaders understand the need for evaluation and support it. Getting key stakeholders (managers, executives) to understand the importance of training and its link to business goals and their involvement in designing training (recall our discussion in  Chapters Two  and  Three ) can help you get access to the type of data needed to conduct an evaluation. But to gain managers’ support for evaluation and conduct an effective evaluation, you need to identify and assess relevant and important outcomes and choose and implement the appropriate evaluation design.

Which Training Outcomes Should be Collected?

From our discussion of evaluation outcomes and evaluation practices, you may have the mistaken impression that it is necessary to collect all five levels of outcomes to evaluate a training program. While collecting all five levels of outcomes is ideal, the training program objectives determine which ones should be linked to the broader business strategy, as discussed in  Chapter Two . To ensure adequate training evaluation, companies should collect outcome measures related to both learning (levels 1 and 2) and transfer of training (levels 3, 4, and 5).

It is important to recognize the limitations of choosing to measure only reaction and cognitive outcomes. Consider the previous discussions of learning and transfer of training in  Chapters Four  and  Five . Remember that for training to be successful, learning and transfer of training must occur.  Figure 6.4  shows the multiple objectives of training programs and their implication for choosing evaluation outcomes. Training programs usually have objectives related to both learning and transfer. That is, they want trainees to acquire knowledge and cognitive skills and also to demonstrate the use of the knowledge or strategies they learned in their on-the-job behavior. As a result, to ensure an adequate training evaluation, companies must collect outcome measures related to both learning and transfer.

261

FIGURE 6.4 Training Program Objectives and Their Implications for Evaluation

Verizon Wireless developed Tech U to prepare its technological service representatives to support both current and future technologies. 32  Verizon created Tech U to provide courses in deductive reasoning, device trouble shooting, and network provisioning. An activity called “Prove It” was developed to determine representatives’ skill and behavior on their jobs after training. Their managers observe the demonstration, score it, and provide feedback to the service representative. Verizon evaluated the program using Level 2, 3, 4, and 5 outcomes. The outcomes included course pass rates (Level 2), completion and evaluation of “Prove Its” (Level 3), repair ticket volume and repeat calls (Level 4), and ROI (Level 5).

Note that outcome measures are not perfectly related to each other. That is, it is tempting to assume that satisfied trainees learn more and will apply their knowledge and skills to the job, resulting in behavior change and positive results for the company. However, research indicates that the relationships among reaction, cognitive, behavior, and results outcomes are small. 33

Which training outcomes measure is best? The answer depends on the training objectives. For example, if the instructional objectives identified business-related outcomes such as increased customer service or product quality, then results outcomes should be included in the evaluation. As  Figure 6.4  shows, both reaction and cognitive outcomes may affect learning. Reaction outcomes provide information regarding the extent to which the trainer, facilities, or learning environment may have hindered learning. Learning or cognitive outcomes directly measure the extent to which trainees have mastered training content. However, reaction and cognitive outcomes do not help determine how much trainees actually use the training content in their jobs. As much as possible, evaluation should include behavior or skill-based, affective, or results outcomes to determine the extent to which transfer of training has occurred—that is, whether training has influenced262a change in behavior, skill, or attitude or has directly influenced objective measures related to company effectiveness (e.g., sales).

How long after training should outcomes be collected? There is no accepted standard for when the different training outcomes should be collected. In most cases, reactions are usually measured immediately after training. 34  Learning, behavior, and results should be measured after sufficient time has elapsed to determine whether training has had an influence on these outcomes. Positive transfer of training is demonstrated when learning occurs and positive changes in skill-based, affective, or results outcomes are also observed. No transfer of training is demonstrated if learning occurs but no changes are observed in skill-based, affective, or learning outcomes. Negative transfer is evident when learning occurs but skills, affective outcomes, or results are less than at pretraining levels. Results of evaluation studies that find no transfer or negative transfer suggest that the trainer and the manager need to investigate whether a good learning environment (e.g., opportunities for feedback and practice) was provided in the training program, trainees were motivated and able to learn, and the needs assessment correctly identified training needs.

EVALUATION DESIGNS

The design of the training evaluation determines the confidence that can be placed in the results, that is, how sure a company can be that training is either responsible for changes in evaluation outcomes or has failed to influence the outcomes. No evaluation design can ensure that the results of the evaluation are completely due to training. What the evaluator strives for is to use the most rigorous design possible (given the circumstances under which the evaluation occurs) to rule out alternative explanations for the results of the evaluation.

This discussion of evaluation designs begins by identifying these “alternative explanations” that the evaluator should attempt to control for. Next, various evaluation designs are compared. Finally, this section discusses practical circumstances that the trainer needs to consider in selecting an evaluation design.

Threats to Validity: Alternative Explanations for Evaluation Results

Table 6.6  presents threats to validity of an evaluation. Threats to validity refer to factors that will lead an evaluator to question either (1) the believability of the study results or (2) the extent to which the evaluation results are generalizable to other groups of trainees and situations. 35  The believability of study results refers to internal validity. The internal threats to validity relate to characteristics of the company (history), the outcome measures (instrumentation, testing), and the persons in the evaluation study (maturation, regression toward the mean, mortality, initial group differences). These characteristics can cause the evaluator to reach the wrong conclusions about training effectiveness. An evaluation study needs internal validity to provide confidence that the results of the evaluation (particularly if they are positive) are due to the training program and not to another factor. For example, consider a group of managers who have attended a communication skills training program. At the same time that they attend the program, it is announced that the company will be restructured. After the program, the managers may become better263communicators simply because they are scared that if they don’t they will lose their jobs. Perhaps no learning actually occurred in the training program at all.

TABLE 6.6 Threats to Validity

Threats to Internal Validity	Description
Company
History	Event occurs, producing changes in training outcomes.
Persons
Maturation	Changes in training outcomes result from trainees’ physical growth or emotional state.
Mortality	Study participants drop out of study (e.g., leave company).
Initial group differences	Training group differs from comparison group on individual differences that influence outcomes (knowledge, skills, ability, and behavior).
Outcome Measures
Testing	Trainees are sensitized to perform well on post-test measures.
Instrumentation	Trainee interpretation of outcomes changes over course of evaluation.
Regression toward the mean	High-and low-scoring trainees move toward the middle or average on post-training measure.
Threats to External Validity	Description
Reaction to pretest	Use of a test before training causes trainees to pay attention to material on the test.
Reaction to evaluation	Being evaluated causes trainees to try harder in training program.
Interaction of selection and training	Characteristics of trainees influence program effectiveness.
Interaction of methods	Results of trainees who received different methods can be generalized only to trainees who receive same training in the same order.

Source: Based on T. D. Cook, D. T. Campbell, and L. Peracchio, “Quasi-Experimentation,” in Handbook of Industrial and Organizational Psychology, 2d ed., Vol. 1, eds. M. D. Dunnette and L. M. Hough (Palo Alto, CA: Consulting Psychologists Press, 1990): 491–576.

Trainers are also interested in the generalizability of the study results to other groups and situations (i.e., they are interested in the external validity of the study). As shown in  Table 6.6 , threats to external validity relate to how study participants react to being included in the study and the effects of multiple types of training. Because evaluation usually does not involve all employees who have completed a program (or who may take training in the future), trainers want to be able to say that the training program will be effective in the future with similar groups.

Methods to Control for Threats to Validity

Because trainers often want to use evaluation study results as a basis for changing training programs or demonstrating that training does work (as a means to gain additional funding for training from those who control the training budget), it is important to minimize the threats to validity. There are three ways to minimize threats to validity: the use of pretests and post-tests in evaluation designs, comparison groups, and random assignment.

264

Pretests and Post-tests  One way to improve the internal validity of the study results is to first establish a baseline or pretraining measure of the outcome. Another measure of the outcomes can be taken after training. This is referred to as a post-training measure. A comparison of the post-training and pretraining measures can indicate the degree to which trainees have changed as a result of training.

Use of Comparison Groups  Internal validity can be improved by using a control or comparison group. A comparison group refers to a group of employees who participate in the evaluation study but do not attend the training program. The comparison employees have personal characteristics (e.g., gender, education, age, tenure, and skill level) as similar to the trainees as possible. Use of a comparison group in training evaluation helps rule out the possibility that changes found in the outcome measures are due to factors other than training. The Hawthorne effect refers to employees in an evaluation study performing at a high level simply because of the attention they are receiving. Use of a comparison group helps show that any effects observed are due specifically to the training rather than the attention the trainees are receiving. Use of a comparison group helps control the effects of history, testing, instrumentation, and maturation because both the comparison group and the training group are treated similarly, receive the same measures, and have the same amount of time to develop.

For example, consider an evaluation of a safety training program. Safe behaviors are measured before and after safety training for both trainees and a comparison group. If the level of safe behavior improves for the training group from pretraining levels but remains relatively the same for the comparison group at both pretraining and post-training, the reasonable conclusion is that the observed differences in safe behaviors are due to the training and not some other factor, such as the attention given to both the trainees and the comparison group by asking them to participate in the study.

Keep in mind that a comparison group can be naturally available for training evaluation because all employees may not receive training at the same time or training may be attended by employees at one or several company locations but not at all locations. Employees who do not initially receive training can serve as a comparison group. These employees can be scheduled to receive training at a later time after the evaluation is completed.

Random Assignment  Random assignment refers to assigning employees to the training or comparison group on the basis of chance alone. That is, employees are assigned to the training program without consideration of individual differences (ability or motivation) or prior experiences. Random assignment helps ensure that trainees are similar in individual characteristics such as age, gender, ability, and motivation. Because it is often impossible to identify and measure all the individual characteristics that might influence the outcome measures, random assignment ensures that these characteristics are equally distributed in the comparison group and the training group. Random assignment helps reduce the effects of employees dropping out of the study (mortality) and differences between the training group and comparison group in ability, knowledge, skill, or other personal characteristics.

Keep in mind that random assignment is often impractical. Companies want to train employees who need training. Also, companies may be unwilling to provide a comparison group. One solution to this problem is to identify the factors in which the training265and comparison groups differ and control for these factors in the analysis of the data (a statistical procedure known as analysis of covariance). Another method is to determine trainees’ characteristics after they are assigned and ensure that the comparison group includes employees with similar characteristics.

Types of Evaluation Designs

A number of different designs can be used to evaluate training programs. 36   Table 6.8  compares each design on the basis of who is involved (trainees or comparison group), when measures are collected (pretraining, post-training), the costs, the time it takes to conduct the evaluation, and the strength of the design for ruling out alternative explanations for the results. As shown in  Table 6.7 , research designs vary based on whether they include pretraining and post-training measurement of outcomes and a comparison group. In general, designs that use pretraining and post-training measures of outcomes and include a comparison group reduce the risk that alternative factors (other than the training itself) are responsible for the results of the evaluation. This increases the trainer’s confidence in using the results to make decisions. Of course, the trade-off is that evaluations using these designs are more costly and take more time to conduct than do evaluations not using pretraining and post-training measures or comparison groups.

TABLE 6.7 Comparison of Evaluation Designs

Post-test Only

The post-test-only design refers to an evaluation design in which only post-training outcomes are collected. This design can be strengthened by adding a comparison group (which helps rule out alternative explanations for changes). The post-test-only design is appropriate when trainees (and the comparison group, if one is used) can be expected to have similar levels of knowledge, behavior, or results outcomes (e.g., same number of sales or equal awareness of how to close a sale) prior to training.

266

Consider the evaluation design that the Mayo Clinic used to compare two methods for delivering new manager training. 37  The Mayo Clinic is one of the world’s leading centers of medical education and research. Recently, Mayo has undergone considerable growth because a new hospital and clinic have been added in the Phoenix area (the Mayo Clinic is also located in Rochester, Minnesota). In the process, employees who were not fully prepared were moved into management positions, which resulted in increased employee dissatisfaction and employee turnover rates. After a needs assessment indicated that employees were leaving because of dissatisfaction with management, Mayo decided to initiate a new training program designed to help the new managers improve their skills. There was some debate whether the training would be best administered in a classroom or one-on-one with a coach. Because using coaching would be more expensive than classroom training, Mayo decided to conduct an evaluation using a post-test comparison group design. Before training all managers, Mayo held three training sessions with no more than 75 managers included in each. Within each session, managers were divided into three groups: a group that received four days of classroom training, a group that received one-on-one training from a coach, and a group that received no training (a comparison group). Mayo collected reaction (did the trainees like the program?), learning, transfer, and results outcomes. The evaluation found no statistically significant differences in the effects of coaching compared to classroom training and no training. As a result, Mayo decided to rely on classroom courses for new managers and to consider coaching only for managers with critical and immediate job issues.

Pretest/Post-test

The pretest/post-test refers to an evaluation design in which both pretraining and post-training outcome measures are collected. There is no comparison group. The lack of a comparison group makes it difficult to rule out the effects of business conditions or other factors as explanations for changes. This design is often used by companies that want to evaluate a training program but are uncomfortable with excluding certain employees or that intend to train only a small group of employees.

Pretest/Post-test with Comparison Group

The pretest/post-test with comparison group refers to an evaluation design that includes trainees and a comparison group. Pretraining and post-training outcome measures are collected from both groups. If improvement is greater for the training group than the comparison group, this finding provides evidence that training is responsible for the change. This type of design controls for most threats to validity.

Table 6.8  presents an example of a pretest/post-test comparison group design. This evaluation involved determining the relationship between three conditions or treatments and learning, satisfaction, and use of computer skills. 38  The three conditions or treatments (types of computer training) were behavior modeling, self-paced study, and lecture. A comparison group was also included in the study. Behavior modeling involved watching a video showing a model performing key behaviors necessary to complete a task. In this case the task was procedures on the computer. (Behavior modeling is discussed in detail in  Chapter Seven , “Traditional Training Methods.”)

TABLE 6.8 Example of a Pretest/Post-test Comparison Group Design

Source: Based on S. J. Simon and J. M. Werner, “Computer training through behavior modeling, self-paced, and instructional approaches: A field experiment,” Journal of Applied Psychology, 81 (1996): 648–659.

Forty trainees were included in each condition. Measures of learning included a test consisting of eleven items designed to measure information that trainees needed to know267to operate the computer system (e.g., “Does formatting destroy all data on the disk?”). Also, trainees’ comprehension of computer procedures (procedural comprehension) was measured by presenting trainees with scenarios on the computer screens and asking them what would appear next on the screen. Use of computer skills (skill-based learning outcome) was measured by asking trainees to complete six computer tasks (e.g., changing directories). Satisfaction with the program (reaction) was measured by six items (e.g., “I would recommend this program to others”).

As shown in  Table 6.8 , measures of learning and skills were collected from the trainees prior to attending the program (pretraining). Measures of learning and skills were also collected immediately after training (post-training time 1) and four weeks after training (post-training time 2). The satisfaction measure was collected immediately following training.

The post-training time 2 measures collected in this study help determine the occurrence of training transfer and retention of the information and skills. That is, immediately following training, trainees may have appeared to learn and acquire skills related to computer training. Collection of the post-training measures four weeks after training provides information about trainees’ level of retention of the skills and knowledge.

Statistical procedures known as analysis of variance and analysis of covariance were used to test for differences between pretraining measures and post-training measures for each condition. Also, differences between each of the training conditions and the comparison group were analyzed. These procedures determine whether differences between the groups are large enough to conclude with a high degree of confidence that the differences were caused by training rather than by chance fluctuations in trainees’ scores on the measures.

Time Series

Time series refers to an evaluation design in which training outcomes are collected at periodic intervals both before and after training. (In the other evaluation designs discussed here, training outcomes are collected only once after and maybe once before training.) The strength of this design can be improved by using reversal, which refers to a time period in which participants no longer receive the training intervention. A comparison group can also be used with a time series design. One advantage of the time series design is that it allows an analysis of the stability of training outcomes over time. Another advantage is that using both the reversal and comparison group helps rule out alternative explanations for the evaluation results. The time series design is frequently used to evaluate training programs that focus on improving readily observable outcomes (such as accident rates,268productivity, and absenteeism) that vary over time. CHG Healthcare Services evaluates the effects of new hire training by first establishing a performance baseline that includes measures of total billing, applications received, and interviews. 39  As new training courses are introduced, CHG can track their influence on new hires performance on these measures.

Figure 6.5  shows a time series design that was used to evaluate how much a training program improved the number of safe work behaviors in a food manufacturing plant. 40  This plant was experiencing an accident rate similar to that of the mining industry, the most dangerous area of work. Employees were engaging in unsafe behaviors, such as putting their hands into conveyors to unjam them (resulting in crushed limbs).

FIGURE 6.5 Example of a Time Series Design

Source: J. Komaki, K. D. Badwick, and L. R. Scott, “A behavioral approach to occupational safety: Pinpointing safe performance in a food manufacturing plant,” Journal of Applied Psychology, 63 (1978). Copyright 1978 by the American Psychological Association. Adapted by permission.

To improve safety, the company developed a training program that taught employees safe behaviors, provided them with incentives for safe behaviors, and encouraged them to monitor their own behavior. To evaluate the program, the design included a comparison group (the Makeup department) and a trained group (the Wrapping department). The Makeup department is responsible for measuring and mixing ingredients, preparing the dough, placing the dough in the oven and removing it when it is cooked, and packaging the finished product. The Wrapping department is responsible for bagging, sealing, and labeling the packages and stacking them on skids for shipping. Outcomes included observations of safe work behaviors. These observations were taken over a twenty-five-week period.

The baseline shows the percentage of safe acts prior to introduction of the safety training program. Training directed at increasing the number of safe behaviors was introduced after approximately five weeks (twenty observation sessions) in the Wrapping department and ten weeks (fifty observation sessions) in the Makeup department. Training was withdrawn from the Wrapping and Makeup departments after approximately sixty-two observation sessions. The withdrawal of training resulted in a reduction of the work incidents269performed safely (to pretraining levels). As shown, the number of safe acts observed varied across the observation period for both groups. However, the number of safe behaviors increased after the training program was conducted for the trained group (Wrapping department). The level of safe acts remained stable across the observation period. (See the intervention period.) When the Makeup department received training (at ten weeks, or after fifty observations), a similar increase in the percentage of safe behaviors was observed.

Solomon Four-Group

The Solomon four-group design combines the pretest/post-test comparison group and the post-test-only control group design. In the Solomon four-group design, a training group and a comparison group are measured on the outcomes both before and after training. Another training group and control group are measured only after training. This design controls for most threats to internal and external validity. However, it is not frequently used in training evaluation because of its complexity and the number of groups required.

An application of the Solomon four-group design is shown in  Table 6.9 . This design was used to compare the effects of training based on integrative learning (IL) with traditional (lecture-based) training of manufacturing resource planning. Manufacturing resource planning is a method for effectively planning, coordinating, and integrating the use of all resources of a manufacturing company. 41  The IL-based training differed from traditional training in several ways. IL-based training sessions began with a series of activities intended to create a relaxed, positive environment for learning. The students were asked what manufacturing resource planning meant to them, and attempts were made to reaffirm their beliefs and unite the trainees around a common understanding of manufacturing resource planning. Students presented training material and participated in group discussions, games, stories, and poetry related to the manufacturing processes.

TABLE 6.9 Example of a Solomon Four-Group Design

	Pretest	Training	Post-test
Group 1	Yes	IL-based	Yes
Group 2	Yes	Traditional	Yes
Group 3	No	IL-based	Yes
Group 4	No	Traditional	Yes

Source: Based on R. D. Bretz and R. E. Thompsett, “Comparing traditional and integrative learning methods in organizational training programs,” Journal of Applied Psychology, 77 (1992): 941–951.

Because the company was interested in the effects of IL compared to traditional training, groups that received traditional training were used as the comparison groups (rather than groups who received no training). A test of manufacturing resource planning (knowledge test) and a reaction measure were used as outcomes. The study found that participants in the IL-based learning groups learned slightly less than participants in the traditional training groups. However, IL-group participants had much more positive reactions than did those in the traditional training groups.

Considerations in Choosing an Evaluation Design

There is no one appropriate evaluation design. An evaluation design should be chosen based on an evaluation of the factors shown in  Table 6.10 . There are several reasons why270no evaluation or a less rigorous evaluation design may be more appropriate than a more rigorous design that includes a comparison group, random assignment, or pretraining and post-training measures. First, managers and trainers may be unwilling to devote the time and effort necessary to collect training outcomes. Second, managers or trainers may lack the expertise to conduct an evaluation study. Third, a company may view training as an investment from which it expects to receive little or no return. A more rigorous evaluation design (pretest/post-test with comparison group) should be considered if any of the following conditions is true: 42

TABLE 6.10 Factors That Influence the Type of Evaluation Design

Factor	How the Factor Influences the Type of Evaluation Design
Change potential	Can the program be modified?
Importance	Does ineffective training affect customer service, safety, product development, or relationships among employees?
Scale	How many trainees are involved?
Purpose of training	Is training conducted for learning, results, or both?
Organization culture	Is demonstrating results part of company norms and expectations?
Expertise	Can a complex study be analyzed?
Cost	Is evaluation too expensive?
Time frame	When is the information needed?

Source: Based on S. I. Tannenbaum and S. B. Woods, “Determining a strategy for evaluating training: Operating within organizational constraints,” Human Resource Planning, 15 (1992): 63–81.

1. The evaluation results can be used to change the program.

2. The training program is ongoing and has the potential to have an important influence on employees or customers.

3. The training program involves multiple classes and a large number of trainees.

4. Cost justification for training is based on numerical indicators. (Here, the company has a strong orientation toward evaluation.)

5. Trainers or others in the company have the expertise (or the budget to purchase expertise from outside the company) to design and evaluate the data collected from an evaluation study.

6. The cost of the training creates a need to show that it works.

7. There is sufficient time for conducting an evaluation. Here, information regarding training effectiveness is not needed immediately.

8. There is interest in measuring change (in knowledge, behavior, skill, etc.) from pretraining levels or in comparing two or more different programs.

For example, if the company is interested in determining how much employees’ communications skills have changed as a result of a training program, a pretest/post-test comparison group design is necessary. Trainees should be randomly assigned to training and no-training conditions. These evaluation design features offer a high degree of confidence that any communication skill change is the result of participation in the training program. 43  This type of evaluation design is also necessary if the company wants to compare the effectiveness of two training programs.

271

Evaluation designs without pretest or comparison groups are most appropriate in situations in which the company is interested in identifying whether a specific level of performance has been achieved. (For example, are employees who participated in training able to communicate their ideas adequately?) In these situations, companies are not interested in determining how much change has occurred, but rather whether the trainees have achieved a certain proficiency level.

Sometimes naturally occurring comparison groups are available, which provides the opportunity to use the pretest and the post-test with comparison group or the post-test with comparison group evaluation designs. This can occur because of the realities of scheduling employees to attend training (all employees cannot attend training at the same time) or when new training is implemented. For example, some employees may be scheduled to receive training later than others. The employees who do not initially receive training can be considered the comparison group. Outcomes can be measured and comparisons made between the employee group who received training and the employees who are waiting to receive the training. Consider how Mountain America Credit Union evaluated the effectiveness of a revised sales training program. 44  Mountain American tracked of the average monthly sales of thirty new employees during their first two months of employment. Ten of the thirty employees attended training before it was revised (they called this the Traditional Group). Twenty employees attended the program after it was revised (the Express Group). The revised program included more interactions with a variety of customers with different needs. Monthly sales were compared between the Express Group and the Traditional Group. Sales in the Express Group exceeded sales in the Traditional Group in both the first (11.4 versus 3.5 average sales) and second month (34.83 versus 5.5) of employment.

One company’s evaluation strategy for a training course delivered to the company’s tax professionals shows how company norms regarding evaluation and the purpose of training influence the type of evaluation design chosen. 45  This accounting firm views training as an effective method for developing human resources. Training is expected to provide a good return on investment. The company used a combination of affective, cognitive, behavior, and results criteria to evaluate a five-week course designed to prepare tax professionals to understand state and local tax law. The course involved two weeks of self-study and three weeks of classroom work. A pretest/post-test comparison design was used. Before they took the course, trainees were tested to determine their knowledge of state and local tax laws, and they completed a survey designed to assess their self-confidence in preparing accurate tax returns. The evaluators also identified the trainees’ (accountants’) billable hours related to calculating state and local tax returns and the revenue generated by the activity. After the course, evaluators again identified billable hours and surveyed trainees’ self-confidence. The results of the evaluation indicated that the accountants were spending more time doing state and local tax work than before training. Also, the trained accountants produced more revenue doing state and local tax work than did accountants who had not yet received the training (comparison group). There was also a significant improvement in the accountants’ confidence following training, and they were more willing to promote their expertise in state and local tax preparation. Finally, after fifteen months, the revenue gained by the company more than offset the cost of training. On average, the increase in revenue for the trained tax accountants was more than 10 percent.

272

DETERMINING RETURN ON INVESTMENT

Return on investment (ROI) is an important training outcome. This section discusses how to calculate ROI through a cost-benefit analysis. Cost-benefit analysis in this situation is the process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits. Training cost information is important for several reasons: 46

1. To understand total expenditures for training, including direct and indirect costs

2. To compare the costs of alternative training programs

3. To evaluate the proportion of money spent on training development, administration, and evaluation, as well as to compare monies spent on training for different groups of employees (exempt versus nonexempt, for example)

4. To control costs

There is an increased interest in measuring the ROI of training and development programs because of the need to show the results of these programs to justify funding and to increase the status of the training and development function. 47  Most trainers and managers believe that there is a value provided by training and development activities, such as productivity or customer service improvements, cost reductions, time savings, and decreased employee turnover. ROI provides evidence of the economic value provided by training and development programs. However, it is important to keep in mind that ROI is not a substitute for other program outcomes that provide data regarding the success of a program based on trainees’ reactions and whether learning and transfer of training have occurred.

Typically, ROI is used to show a training program’s cost effectiveness after it has been delivered. However, ROI is also useful for forecasting the potential value of a new training program, choosing the most cost-effective training method by estimating and comparing the costs and benefits of each approach, and making decisions about whether to fund and offer training programs in the future. 48

Consider the use of ROI at LensCrafters. LensCrafters brings the eye doctor, a wide selection of frames and lenses, and the lens-making laboratory together in one location. 49  LensCrafters has convenient locations and hours of operations, and it has the ability to make eyewear on-site. Emphasizing customer service, the company offers a one-stop location and promises to make eyewear in one hour. Dave Palm, a training professional at LensCrafters, received a call from a concerned regional manager. He told Palm that although company executives knew that LensCrafters employees had to be well trained to design eyewear and that employees were satisfied with the training, the executives wanted to know whether the money that they were investing in training was providing any return. Palm decided to partner with the operations people to identify how to link training to measurable outcomes such as profitability, quality, and sales. After conversations with the operations employees, he decided to link training to waste from mistakes in quality and remakes, store performance and sales, and customer satisfaction. He chose two geographic regions for the evaluation study and compared the results from these two regions with results from one that had not yet received the training. Palm found that all stores in the two regions that received training reduced waste, increased sales, and improved customer satisfaction. As a result, LensCrafters allotted its training department more financial273resources—$10 million a year for training program development and administration—than any other optical retail competitor. Because the training department demonstrated that it does contribute to business operations, it also received money to develop a multimedia-based training system.

The process of determining ROI begins with an understanding of the objectives of the training program. 50  Plans are developed for collecting data related to measuring these objectives. The next step is to isolate, if possible, the effects of training from other factors that might influence the data. Last, the data are converted to a monetary value and ROI is calculated. Choosing evaluation outcomes and designing an evaluation that helps isolate the effects of training were explained earlier in this chapter. The following sections discuss how to determine costs and benefits and provide examples of cost-benefit analysis and ROI calculations.

Because ROI analysis can be costly, it should be limited only to certain training programs. ROI analysis is best for training programs that are focused on an operational issue (measurable identifiable outcomes are available), are linked to a companywide strategy (e.g., better customer service), are expensive, are highly visible, have management interest, are attended by many employees, and are permanent. 51 At Deloitte, the tax and auditing firm, managers don’t require analysis of ROI for many training programs. 52  Because knowledge is the product at Deloitte, investment in training is seen as an important part of the business. Deloitte makes money through the billable hours that its consultants provide to clients. Training helps prepare the consultants to serve clients’ needs. ROI is primarily calculated for courses or programs that are new or expensive. For example, ROI analysis was conducted for a simulation designed to help new employees learn more quickly how to service clients. At Deloitte, use of the simulation has resulted in new hires being 30 to 40 percent faster in serving clients—resulting in an ROI of over $66 billion after subtracting program costs.

Determining Costs

One method for comparing costs of alternative training programs is the resource requirements model. 53  The resource requirements model compares equipment, facilities, personnel, and materials costs across different stages of the training process (needs assessment, development, training design, implementation, and evaluation). Use of the resource requirements model can help determine overall differences in costs among training programs. Also, costs incurred at different stages of the training process can be compared across programs.

Accounting can also be used to calculate costs. 54  Seven categories of cost sources are costs related to program development or purchase, instructional materials for trainers and trainees, equipment and hardware, facilities, travel and lodging, salary of trainer and support staff, and the cost of lost productivity while trainees attend the program (or the cost of temporary employees who replace the trainees while they are away from their jobs). This method also identifies when the costs are incurred. Onetime costs include those related to needs assessment and program development. Costs per offering relate to training site rental fees, trainer salaries, and other costs that are realized every time the program is offered. Costs per trainee include meals, materials, and lost productivity or expenses incurred to replace the trainees while they attend training. For example, consider the costs for virtual training compared to instructor-led training. Aetna has moved274from face-to-face instructor-led classroom training to virtual training for its sixteen-week program for new customer service representatives. 55  The cost to bring a trainer to locations without one onsite was more than $27,000. The comparable costs for the same training program conducted virtually are approximately $3,000.

Determining Benefits

To identify the potential benefits of training, the company must review the original reasons that the training was conducted. For example, training may have been conducted to reduce production costs or overtime costs or to increase the amount of repeat business. A number of methods may be helpful in identifying the benefits of training:

1. Technical, academic, and practitioner literature summarizes the benefits that have been shown to relate to a specific training program.

2. Pilot training programs assess the benefits from a small group of trainees before a company commits more resources.

3. Observance of successful job performers helps a company determine what successful job performers do differently than unsuccessful job performers. 56

4. Trainees and their managers provide estimates of training benefits.

For example, a training and development consultant at Apple Computer was concerned with the quality and consistency of the training program used in assembly operations. 57  She wanted to show that training was not only effective but also resulted in financial benefits. To do this, the consultant chose an evaluation design that involved two separately trained groups—each consisting of twenty-seven employees—and two untrained groups (comparison groups). The consultant collected a pretraining history of what was happening on the production line in each outcome that she was measuring (i.e., productivity, quality, and labor efficiency). She determined the effectiveness of training by comparing performance between the comparison and training groups for two months after training. The consultant was able to show that the untrained comparison group had 2,000 more minutes of downtime than the trained group did. This finding meant that the trained employees built and shipped more products to customers—showing definitively that training was contributing to Apple’s business objectives.

To conduct a cost-benefit analysis, the consultant had each employee in the training group estimate the effect of a behavior change on a specific business measure (e.g., breaking down tasks will improve productivity or efficiency). The trainees assigned a confidence percentage to the estimates. To get a cost-benefit estimate for each group of trainees, the consultant multiplied the monthly cost-benefit by the confidence level and divided by the number of trainees. For example, one group of 20 trainees estimated a total overall monthly cost benefit of $336,000 related to business improvements and showed an average 70 percent confidence level with that estimate. The calculation is as follows: 70 percent multiplied by $336,000 gave a cost-benefit of $235,200. This number was divided by 20 ($235,200/20 trainees) to give an average estimated cost benefit for each of the trainees ($11,760). To calculate ROI, follow these steps: 58

1. Identify outcomes (e.g., quality, accidents).

2. Place a value on the outcomes.275

3. Determine the change in performance after eliminating other potential influences on training results.

4. Obtain an annual amount of benefits (operational results) from training by comparing results after training to results before training (in dollars).

5. Determine the training costs (direct costs + indirect costs + development costs + overhead costs + compensation for trainees).

6. Calculate the total benefits by subtracting the training costs from benefits (operational results).

7. Calculate the ROI by dividing operational results by costs. The ROI gives an estimate of the dollar return expected from each dollar invested in training.

Example of a Cost-Benefit Analysis

A cost-benefit analysis is best explained by an example. 59  A wood plant produced panels that contractors used as building materials. The plant employed 300 workers, 48 supervisors, 7 shift superintendents, and a plant manager. The business had three problems. First, 2 percent of the wood panels produced each day were rejected because of poor quality. Second, the production area was experiencing poor housekeeping, such as improperly stacked finished panels that would fall on employees. Third, the number of preventable accidents was higher than the industry average. To correct these problems, the supervisors, shift superintendents, and plant manager attended training in (1) performance management and interpersonal skills related to quality problems and poor work habits of employees, and (2) rewarding employees for performance improvement. Training was conducted in a hotel close to the plant. The training program was a purchased videotape, and the instructor for the program was a consultant.  Table 6.11  shows each type of cost and how it was determined.

TABLE 6.11 Determining Costs for a Cost-Benefit Analysis

Direct Costs
Instructor	$       0
In-house instructor (12 days @ $125 per day)	1,500
Fringe benefits (25% of salary)	375
Travel expenses	0
Materials ($60 × 56 trainees)	3,360
Classroom space and audiovisual equipment (12 days @ $50 per day)	600
Refreshments ($4 per day × 3 days × 56 trainees)	672
Total direct costs	$ 6,507
Indirect Costs
Training management	$       0
Clerical and administrative salaries	750
Fringe benefits (25% of salary)	187
Postage, shipping, and telephone	0
Pre- and post-training learning materials ($4 × 56 trainees)	224
Total indirect costs	$ 1,161
Development Costs
Fee for program purchase	$ 3,600
Instructor training
Registration fee	1,400
Travel and lodging	975
Salary	625
Benefits (25% of salary)	156
Total development costs	$ 6,756
Overhead Costs
General organizational support, top management time (10% of direct, indirect, and development costs)	$ 1,443
Total overhead costs	$ 1,443
Compensation for Trainees
Trainees’ salaries and benefits (based on time away from job)	$16,969
Total training costs	$32,836
Cost per trainee	$ 587

The benefits of the training were identified by considering the objectives of the training program and the type of outcomes the program was to influence. These outcomes included the quality of panels, housekeeping in the production area, and the accident rate.  Table 6.12  shows how the benefits of the program were calculated.

TABLE 6.12 Determining Benefits for a Cost-Benefit Analysis

Source: Adapted from D. G. Robinson and J. Robinson, “Training for impact,” Training and Development Journal (August 1989): 30–42.

Once the costs and benefits of the program are determined, ROI is calculated by dividing return or benefits by costs. In this example, ROI was 5.72. That is, every dollar invested in the program returned almost $6 in benefits. How can the company determine if the ROI is acceptable? One way is for managers and trainers to agree on what level of ROI is acceptable. Another method is to use the ROI that other companies obtain from similar types of training.  Table 6.13  provides examples of ROIs obtained from several types of training programs.

TABLE 6.13 Examples of ROIs

Industry	Training Program	ROI
Bottling company	Workshops on managers’ roles	15:1
Large commercial bank	Sales training	21:1
Electric and gas utility	Behavior modification	5:1
Oil company	Customer service	4.8:1
Health maintenance organization	Team training	13.7:1
Health medical services	Coaching for Leaders	2:1

Source: Based on Top 125 Rankings 2015, training (January/February 2015): 62–101. J. J. Philips, “ROI: The search for best practices,” Training and Development (February 1996): 45.

Recall the discussion of the new manager training program at the Mayo Clinic. 60  To determine Mayo’s ROI, the human resource department calculated that one-third of the eighty-four employees retained (twenty-nine employees) would have left Mayo as a result of dissatisfaction. The department believed that this retention was due to the impact of the training. Mayo’s cost of a single employee turnover was calculated to be 75 percent of average total compensation, or $42,000 per employee. Multiplying $42,000 by the twenty-nine employees retained equals a savings of $609,000. However, the cost of the training program needs to be considered. If the annual cost of the training program ($125,000) is subtracted from the savings, the new savings amount is $484,000. These numbers are276based on estimates, but even if the net savings figure were cut in half, the ROI is still over 100 percent. By being able to quantify the benefits delivered by the program, Mayo’s human resource department achieved greater credibility within the company.

Other Methods for Cost-Benefit Analysis

Other more sophisticated methods are available for determining the dollar value of training. For example, utility analysis is a cost-benefit analysis method that involves assessing the dollar value of training based on estimates of the difference in job performance between trained and untrained employees, the number of individuals trained, the length of time a training program is expected to influence performance, and the variability in job performance in the untrained group of employees. 61  Utility analysis requires the use of277a pretest/post-test design with a comparison group to obtain an estimate of the difference in job performance for trained versus untrained employees. Other types of economic analyses evaluate training as it benefits the firm or the government using direct and indirect training costs, government incentives paid for training, wage increases received by trainees as a result of completion of training, tax rates, and discount rates. 62

Practical Considerations in Determining ROI

As mentioned earlier in the chapter, ROI analysis may not be appropriate for all training programs. Training programs best suited for ROI analysis have clearly identified outcomes, are not onetime events, are highly visible in the company, are strategically focused, and have effects that can be isolated. In the examples of ROI analysis in this chapter, the outcomes were very measurable. That is, in the wood plant example, it was easy to see changes in quality, to count accident rates, and to observe housekeeping behavior. For training programs that focus on soft outcomes (e.g., attitudes or interpersonal skills), it may be more difficult to estimate the value.

278

Showing the link between training and market share gain or other higher-level strategic business outcomes can be very problematic. These outcomes can be influenced by too many other factors not directly related to training (or even under the control of the business), such as competitors’ performance and economic upswings and downturns. Business units may not be collecting the data needed to identify the ROI of training programs on individual performance. Also, the measurement of training can often be very expensive. Verizon Communications employs over 200,000 people. 63  The company estimates that it spends approximately $5,000 for an ROI study. Given the large number of training programs that the company offers, it is too expensive to conduct an ROI for each program.

Companies are finding that, despite these difficulties, the demand for measuring ROI is still high. As a result, companies are using creative ways to measure the costs and benefits of training. 64  For example, to calculate ROI for a training program designed to cut absenteeism, trainees and their supervisors were asked to estimate the cost of an absence. The values were averaged to obtain an estimate. Cisco Systems tracks how often its partners return to its website for additional instruction. A. T. Kearney, a management consulting firm, tracks the success of its training by how much business is generated from past clients.

Success Cases and Return on Expectations

One way to establish the value of training and overcome the difficulty of evaluating training using a design that can rule out and isolate its effects on results is to rely on return on expectations or success cases. Return on expectations (ROE) refers to the process through which evaluation demonstrates to key business stakeholders, such as top-level managers, that their expectations about training have been satisfied. 65  ROE depends on establishing a business partnership with business stakeholders from the start of a training program through its evaluation.

Rather than relying on ROI, Verizon Communications uses training ROE. Prior to training, the senior managers who are financially accountable for the training program are asked to identify their expectations regarding what the training program should accomplish, as well as a cost estimate of the current issue or problem. After training, the senior managers are asked whether their expectations have been met, and they are encouraged to attach a monetary value to those met expectations. The ROE is used as an estimate in an ROI analysis. Verizon Communications continues to conduct ROI analysis for training programs and courses in which objective numbers are available (e.g., sales training) and in which the influence of training can be better isolated (evaluation designs that have comparison groups and that collect pretraining and post-training outcomes).

Success cases or stories refer to concrete examples of the impact of training that show how learning has led to results that the company finds worthwhile and the managers find credible. 66  Success cases do not attempt to isolate the influence of training, but rather to provide evidence that it was useful. Both Agilent Technologies, a high-tech measurement equipment company, and Standard Chartered Bank, an international bank, use success stories. 67  Agilent provides senior managers with the use of specific example, written by trainees, of what they are doing better and differently as they apply what they learned to their work. For example, a manufacturing engineer realized that, as a result of attending a training program, they could expand their area of influence outside of their own team to solve a financial reporting problem. Through expanding the area of influence, the engineer is now more aware and in control of expenses, which has improved the bottom line279of the business unit. At Standard, one hundred days after training the trainees connect on a conference call to share stories about changes that have occurred. Trainees are asked to prepare for the call by considering four question: What learning did you apply? What achievement or result are you most proud of? What have been the personal and business benefits? What advice do you have for how others can use what they learned?

MEASURING HUMAN CAPITAL AND TRAINING ACTIVITY

So far, this chapter has focused on how to evaluate training programs. It is important to remember that evaluation can also involve determining the extent to which learning and training activities and the training function contribute to the company strategy and help achieve business goals.  Chapter One  emphasized that there is increased recognition that training, development, and learning can and should contribute to employee performance and business goals (see  Figure 1.1 ). We discussed the role of metrics in the strategic training and development process in  Chapter Two . Metrics are used to determine the value that learning activities or the training function provide to the company.

One way to understand the value that learning activities or the training function provides is through comparisons to other companies. Each year, the American Society of Training and Development (ASTD, now the Association for Talent Development [ATD]) prepares a report that summarizes company-provided training in the United States. This report provides information about training hours and delivery methods that companies can use to benchmark, or compare themselves to other companies in similar industries or size.

Table 6.14  provides examples of different measurements, or metrics. These metrics are valuable for benchmarking purposes, for understanding the current amount of training activity in a company, and for tracking historical trends in training activity. However, collecting these metrics does not address such issues as whether training is effective, or whether the company is using the data to make strategic training decisions. 68

TABLE 6.14 Training Metrics

Expenditure per employee Learning hours received per employee Expenditure as a percentage of payroll Expenditure as a percentage of revenue Cost per learning hour received Percentage of expenditures for external services Learning hours received per training and development staff member Average percentage of learning activities outsourced Average percentage of learning content by content area (e.g., basic skills, customer service, executive development) Average percentage of learning hours provided via different delivery methods (instructor-led, technology-based, etc.)

Sources: L. Miller, 2014 State of the Industry (Alexandria, VA: Association for Talent Development, 2014); T. Wilk, “How to Run Learning Like a Business,” Chief Learning Officer (June 2014): 48–60.

Big Data and Workforce Analytics

Big data refers to complex data sets developed by compiling data across different organizational systems including marketing and sales, human resources, finance, accounting, customer service, and operations. Three dimensions characterize big data: volume, variety,280and velocity. 69  Volume refers to the large amount of available data. Variety includes the large number of sources and types of data that are available come from. Velocity refers to the huge amount of data that is being generated and the speed with which it must be evaluated, captured, and made useful. Big data can come from many different sources including transactions, business applications, e-mails, social media, smartphones, and even sensors embedded in employees’ identification badges or company products.

The goal of big data is to make decisions about human capital based on data rather than intuition or conventional wisdom, which likely lead to incorrect conclusions and recommendations. Big data can be used for many purposes, including to evaluate the effectiveness of learning and development programs, determine their impact on business results, and develop predictive models that can be used for forecasting training needs, course enrollments, costs, and outcomes. For example, big data can help predict when employees will take training in their careers and how that relates to their retention and promotion. Including learning-related data as part of big data can help show the strategic value of learning for a company. Using big data requires the use of workforce analytics. Workforce analytics refers to the practice of using quantitative methods and scientific methods to analyze data from human resource databases, corporate financial statements, employee surveys and other data sources to make evidence-based decisions and show that human resource practices (including training, development, and learning) influence important company metrics. 70

Several examples show how companies are using big data and workforce analytics. 71  To analyze data about how learning contributed to the success of their salesforce, SuccessFactors used data from three systems: customer relationship management, learning and performance management, and employee records. Customer relations data included number and size of sales. Learning data included courses taken and self-evaluation of training effectiveness. Performance data included managers’ performance ratings and learning plans. Employee records included sales experience and hire date. They analyzed the performance and number and value of sales made by employees who completed training compared to those who did not. Defense Acquisition University (DAU) used big data to determine the effectiveness of its learning programs. DAU provides training for active and reserve information technology employees in the U.S. military. DAU integrated data from human resources, budgeting, and accounting systems with learning data and trainee information. For example, some of the training data includes post-training surveys of course quality and instructor evaluations completed by trainees immediately after they complete a course. Also, two months later trainees are asked to assess how the course affected their job performance and business outcomes. This data sets allows DAU to determine the trends in courses, quality, preferred course location, business unit participation rates in training, and how the number of employees enrolling in a course relates to business performance. For example, one of the analyses showed that guest speakers in courses were highly related to individual learning, which subsequently influenced job performance and business results. Another analysis of room cost, instructor salary and travel costs, expected attendance, and travel costs for students helped DAU determine the least expensive places to conduct classroom training. Juniper Networks uses LinkedIn to track and analyze the skills, knowledge, experience, and career paths of potential, current, and former employees. Juniper was interested in determining how long it took new salespersons to generate expected revenue and what types of training might shorten that time period. TELUS, a telecommunications company that uses online, face-to-face, and281informal learning, created a measurement system that assesses accessing training (clicking, opening, attending), usage (viewing, staying, reading, participating), grade (knowledge acquisition), evaluation (learner assessment), and return (performance impact). TELUS’s learning team collects data from different tracking systems (individual, corporate, department, business). All employees can access a learning dashboard that displays learning investments, assessment results, and analytics by work team, region, or division. To determine return every quarter, TELUS randomly surveys employees to ask about their learning activities from the past ninety days. Return on learning (ROL) is determined by the percentage of employees who respond that learning had a positive impact on their knowledge and skills. In 2013 the average ROL was 75 percent.

Some companies are using dashboards to help measure the effectiveness, financial benefits, and relationship of learning activities to business strategy and goals. A dashboard refers to a computer interface designed to receive and analyze the data from departments within the company to provide information to managers and other decision-makers. 72  Dashboards can draw their data from different sources, such as the company’s human resource information system or learning management or course enrollment system. Dashboards are useful because they can provide a visual display using charts of the relationship between learning activities and business performance data, including measures related to intangible assets such as human capital (recall the discussion of human capital in  Chapter One ). The dashboard allows the user to access and use the data in different ways, including isolating problem areas and creating categories based on precalculated formulas (such as cost per learning hour received). Jiffy Lube provides store managers, company leaders, franchisees, and learning professionals with access to an online report that shows completion of certifications by employee, store, franchisee, and region. 73  Color-coding is provided to make it easy to understand. Completions at or above target are shown in green, 50 to 99 percent in yellow, and less than 50 percent in red.

Key Bank, headquartered in Cleveland, Ohio, is one of the largest bank-based financial services companies in the United States. 74  Its business strategy is to build client relationships through client-focused solutions and extraordinary service. Key Bank provides a wide variety of training and development programs to help employees grow personally and professionally. To help ensure the efficiency and effectiveness of its training and development activity, Key Bank uses a dashboard that allows access to real-time data providing access to information about the strategic alignment, effectiveness, and investment in training programs across the company. The training activity dashboard provides historical information on learning and training (a “rear-view” perspective). The user identifies the time period that he or she is interested in and then can select companywide or business-specific information. The dashboard offers information on total training activities, the distribution of training activities across the company or business, identifies which training areas are receiving the most funding, and can identify employee progress in completing training and acquiring skills. The dashboard also can be used to provide a perspective of future learning (a “windshield” perspective). Users can look at completion rates for training courses and programs and identify the progress rates at Key Bank locations across the United States. This information is useful for training managers to understand whether local training goals are met, in progress, or behind completion dates, helping them better plan future training and adjust training goals with local managers.

282

Summary

Evaluation provides information used to determine training effectiveness. Evaluation involves identifying the appropriate outcomes to measure. The chapter notes that a good evaluation requires thinking about evaluation before conducting training. Information from the needs assessment and specific and measurable learning objectives help identify outcomes that should be included in the evaluation design. The outcomes used in evaluating training programs include trainees’ satisfaction with the training program, learning of knowledge or skills, use of knowledge and skills on the job, and results such as sales, productivity, or accident prevention. Evaluation may also involve comparing the costs of training to the benefits received (ROI). Outcomes used in training evaluation help determine the degree to which the program has resulted in both learning and transfer of training. Evaluation also involves choosing the appropriate design to maximize the confidence that can be placed in the results. The design is based on a careful analysis of how to minimize threats to internal and external validity, as well as the purpose, expertise, and other company and training characteristics. The types of designs used for evaluation vary on the basis of whether they include pretraining and post-training measures of outcomes and use of a training group and a comparison group. The chapter concludes with a discussion of how to use big data, workforce analytics, and dashboards to measure, analyze, and display the contribution of training to a company’s human capital assets.

Key Terms

training effectiveness,  247

training outcomes,  247

criteria,  247

training evaluation,  247

evaluation design,  248

formative evaluation,  249

pilot testing,  250

summative evaluation,  250

reaction outcomes,  252

cognitive outcomes,  255

self-assessments,  255

skill-based outcomes,  256

affective outcomes,  257

results,  257

return on investment (ROI),  258

direct costs,  258

indirect costs,  258

benefits,  258

criteria relevance,  259

criterion contamination,  259

criterion deficiency,  260

reliability,  260

discrimination,  260

practicality,  260

threats to validity,  263

internal validity,  263

external validity,  264

pretraining measure,  265

post-training measure,  265

comparison group,  265

Hawthorne effect,  265

random assignment,  265

post-test-only,  266

pretest/post-test,  267

pretest/post-test with comparison group,  267

time series,  268

reversal,  268

Solomon four-group,  270

cost-benefit analysis,  273

utility analysis,  277

return on expectations (ROE),  279

success cases or success stories,  279

big data,  280

workforce analytics,  281

self-evaluation,  281

dashboard,  282

Discussion Questions

1. What can be done to motivate companies to evaluate training programs?

2. What do threats to validity have to do with training evaluation? Identify internal and external threats to validity. Are internal and external threats similar? Explain.283

3. What are the strengths and weaknesses of each of the following designs: post-test-only, pretest/post-test with comparison group, and pretest/post-test only?

4. What are results outcomes? Why do you think most organizations don’t use results outcomes to evaluate their training programs?

5. This chapter discussed several factors that influence the choice of evaluation design. Which of these factors would have the greatest influence on your choice of an evaluation design? Which would have the least influence? Explain your choices.

6. How might you estimate the benefits of a training program designed to teach employees how to use the Internet to monitor stock prices?

7. A group of managers (N = 25) participated in the problem-solving module of a leadership development program two weeks ago. The module consisted of two days in which the group focused on the correct process to use in problem solving. Each manager supervises fifteen to twenty employees. The company is willing to change the program, and there is an increasing emphasis in the company to show that training expenses are justifiable. You are asked to evaluate this program. Your boss would like the results of the evaluation no later than six weeks from now. Discuss the outcomes you would collect and the design you would use. How might your answer change if the managers had not yet attended the program?

8. What practical considerations need to be taken into account when calculating a training program’s ROI?

9. What is return on expectations (ROE)? How can it be used to show the costs and benefits of training without collecting statistics and conducting analyses? Explain its strengths and weaknesses compared to a cost-benefit analysis.

10. What are the characteristics of big data? Explain how big data could be used to show that learning influences business outcomes.

Application Assignments 

1. Consider this course as a training program. In teams of up to five students, identify (a) the types of outcomes you would recommend to use in evaluating this course and (b) the evaluation design you would use. Justify your choice of a design based on minimizing threats to validity and practical considerations.

2. Domino’s Pizza was interested in determining whether a new employee could learn how to make a pizza using a computer-based training method (CD-ROM). The CD-ROM application addresses the proper procedure for “massaging” a dough ball and stretching it to fit a 12-inch pizza pan. Domino’s quality standards emphasize the roundness of the pizza, an even border, and uniform thickness of the dough. Traditionally, on-the-job training is used to teach new employees how to stretch pizza dough to fit the pizza pan.

b. What outcomes or criteria should Domino’s Pizza measure to determine if CD-ROM training is an effective method for teaching new employees how to stretch pizza dough to fit a 12-inch pan? Who would be involved in the evaluation?

b. Describe the evaluation design that you would recommend using to determine if CD-ROM training is more effective than on-the-job training.

284

1. Ask your instructor for a copy of the evaluation form, survey, or rating sheet that is used by your college, university, or business to evaluate the course or program in which you are using this text. As you look over the evaluation, answer the following questions:

c. What are the strengths and weaknesses of the evaluation form?

c. What changes would you suggest to improve the evaluation form (e.g., different questions, additional questions)?

c. How should the evaluation be used to actually improve the instruction that you receive?

1. Sears designed a training program to improve tool and hardware sales. The two-hour program involved distance learning and was broadcast from the Sears training facility to fifty salespersons at ten store locations in the United States. The salespersons are paid $15 per hour. The program involved training salespeople in how to set up merchandise displays so they attract buyers’ attention. Sales of tools and merchandise at the ten stores included in the program averaged $5,000 per week before the program and $6,500 per week after the program. Program costs included:

Instructor	$10,000
Distance learning (satellite space rental)	5,000
Materials ($100 per trainee @ 50 trainees)	5,000
Trainees’ salaries and benefits (50 trainees with wages of $15 per hour in a 2-hour training program)	1,500

1. What is the ROI from this program?

1. Cablevision developed an e-learning course that taught salespersons how to increase the number of cable television subscribers, thereby increasing revenue. The company wants to know if salespersons will increase upselling of cable television services (e.g., premium channels) and will try to sell other products (e.g., e-mail and web access). The company also wants to know the ROI of this training program.

f. What training outcomes should the company collect? From whom should the outcomes be collected?

f. What evaluation design would you recommend? Defend your recommendation.

f. Show how Cablevision can conduct an ROI analysis. Describe the information that the company should collect and how it should be collected.

1. The 100-employee information technology department of a financial services company had a high turnover rate. A survey of employees revealed that the reason that most of them left was dissatisfaction with the level of training. The average turnover rate was 23 percent per year. The cost to recruit and train one new employee was $56,625. To address the turnover problem, the company developed a skills training program that averaged 80 hours per year per employee. The average employee wage was $35 per hour. Instructor, classroom, and other costs were $170,000.

g. What is the total cost of training? The total cost of turnover?

g. If the turnover rate dropped 8 percent (from 23 percent to 15 percent), what was the financial benefit of the training program?285

g. What was the ROI of the training program?

g. How much would the turnover rate have to be reduced (from 23 percent) for the training program to show a benefit?

1. Go to  www.roiinstitute.net , the website for ROI Institute, Inc., the leading resource on research, training, and networking for practitioners of the Phillips ROI MethodologyTM. Click on “Free Tools,” Review the “Nations Hotels––Measuring the ROI in Business Coaching.” What are the strengths of this approach for determining ROI? What are the weaknesses?

1. Watch the video about how Skillsoft and IBM are using big data at  http://www.youtube.com/watch?v=Texn4xpaZ0w . How is using big data to analyze employees learning usage patterns useful for companies? For employees?

Case:

Developing Financial Planners at AMP

AMP, an Australian-based financial services company, recognized that the greatest challenge to company growth was attracting and developing the best financial planners. As a result, AMP developed the Career Changer Program, a twelve-month learning program for individuals who desire to be financial advisors but lack financial experience. Aspiring financial planners participate in both online and face-to-face learning during the first ten weeks of the program. In the classroom, instructors provide knowledge about finance, financial products, and selling. Also, to enhance selling skills and customer service, learners engage in role-plays. The next nine months of the program include mentoring and on-the-job experiences. Learners work with financial planners in their practices, providing real advice to clients. During this nine months learners are provided with on-the-job coaching, professional development, and complete compliance training. After successfully completing the program, learners achieve the status allowing them to start their own join one of AMP’s personal financial practices.

What outcomes should AMP collect to determine the effectiveness of the Career Changer Program? What evaluation design should it use? Explain your choice of outcomes and design.

Source: “Training Broadens Career Horizons,” T+D (July 2014): 80.

Endnotes

1 .P. Tharenou, A. Saks, and C. Moore, “A review and critique of research on training and organizational-level outcomes,” Human Resource Management Review, 17 (2007): 251–273; S. Sung and J. Choi, “Do organizations spend wisely on employees? Effects of training and development investments on learning and innovation in organizations,” Journal of Organizational Behavior, 35 (2014): 393–412; H. Aguinis and K. Kraiger, “Benefits of training and development for individuals, teams, organizations, and society,” Annual Review of Psychology, 60 (2009): 451–474; Y. Kim and R. E. Ployhart, “The effects of staffing and training on firm productivity and profit growth before, during, and after the Great Recession,” Journal of Applied Psychology, 99 (2014): 361–389.

2 .“Leaders take flight,” T+D (February 2014): 72.

3 .J. Dearborn, “Big data: A quick start guide for learning practitioners,” T+D (May 2014): 52–57.

4 .A. Purcell, “20/20 ROI,” Training and Development (July 2000): 28–33.

5 .M. Van Wart, N. J. Cayer, and S. Cook, Handbook of Training and Development for the Public Sector (San Francisco: Jossey-Bass, 1993).

286

6 .K. Brown and M. Gerhardt, “Formative evaluation: An integrative practice model and case study,” Personnel Psychology, 55 (2002): 951–983.

7 .J. Salopek, “BEST 2005: St. George Bank,” T + D (October 2005): 68.

8 .D. Russ-Eft and H. Preskill, “In search of the Holy Grail: Return on investment evaluation in human resource development,” Advances in Developing Human Resources, 7 (February 2005): 71–85.

9 .K. Kraiger, D. McLinden, and W. Casper, “Collaborative planning for training impact,” Human Resource Management, 43 (4) (2004): 337–351; F. Nickols, “Why a stakeholder approach to evaluating training,” Advances in Developing Human Resources, 7 (February 2005): 121–134.

10 .K. Kraiger, D. McLinden, and W. Casper, “Collaborative planning for training impact,” Human Resource Management (Winter 2004): 337–351; R. Brinkerhoff, “The success case method: A strategic evaluation approach to increasing the value and effect of training,” Advances in Developing Human Resources, 7 (February 2005): 86–101.

11 .K. Kraiger, J. K. Ford, and E. Salas, "Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation," Journal of Applied Psychology, 78 (1993): 311–328; J. J. Phillips, "ROI: The search for best practices," Training and Development (February 1996): 42–47; D. L. Kirkpatrick, “Evaluation of Training,” in Training and Development Handbook, 2d ed., ed. R. L. Craig (New York: McGraw-Hill, 1976): 18-1–18-27; J. Kirkpatrick and W. Kirkpatrick, “Creating a post-training evaluation plan,” T+D (January 2013): 26–28.

12 .K. Kraiger, J. K. Ford, and E. Salas, “Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation,” Journal of Applied Psychology, 78 (1993): 311–328; J. J. Phillips, “ROI: The search for best practices,” Training and Development (February 1996): 42–47; G. M. Alliger et al., “A meta-analysis of the relations among training criteria,” Personnel Psychology, 50 (1997): 341–355; K. Kraiger, “Decision-Based Evaluation,” in Creating, Implementing, and Managing Effective Training and Development, ed. K. Kraiger (San Francisco: Jossey-Bass, 2002): 331–375; G. Alliger and E. Janek, “Kirkpatrick’s levels of training criteria: Thirty years later,” Personnel Psychology, 42 (1989): 331–342.

13 .R. Morgan and W. Casper, “Examining the factor structure of participant reactions to training: A multidimensional approach,” Human Resource Development Quarterly, 11 (2000): 301–317; K. Brown, “An examination of the structure and nomological network of trainee reactions: A closer look at ‘smile sheets,’ ” Journal of Applied Psychology, 90 (2005): 991–1001; G. Vellios, “On the level,” T+D (December 2008): 26–29.

14 .A. Beninghof, “How’s My Training”, TD (January 2015): 28–31.

15 .G. Hopkins, “How to design an instructor evaluation,” Training and Development (March 2000): 51–53.

16 .T. Sitzmann et al., “A review and meta-analysis of the nomological network of trainee reactions,” Journal of Applied Psychology, 93 (2008): 280–295.

17 .T. Sitzmann, K. Ely, K. Brown, and K. Bauer, “Self-assessment of knowledge: A cognitive learning or affective measure,” Academy of Management Learning & Education, 9 (2010): 169–191.

18 .“Grant Thornton,” T+D (October 2010): 75.

19 .P. Taylor, D. Russ-Eft, and H. Taylor, “Transfer of management training from alternative perspectives,” Journal of Applied Psychology, 94 (2009): 104–121.

20 .M. Weinstein, “Taking training’s measure,” training (November/December 2014): 42–43.

21 .“Walgreens: Leadership leap,” training (January/February 2014): 113.

22 .J. J. Phillips, “Was it the training?” Training and Development (March 1996): 28–32.

23 .G. Dutton, “Training at your fingertips,” training (March/April 2014): 26–30.

24 .“USAA,” TD (October 2014): 98.

25 .J. Salopek, “Tackling business-critical issues through training,” TD (October 2014): 65–68.

26 .J. J. Phillips, “ROI: The search for best practices,” Training and Development (February 1996): 42–47.

27 .P. Harris, “Short can be oh, so sweet,” T+D (October 2013): 66–69.

28 .D. A. Grove and C. Ostroff, “Program Evaluation.” In Developing Human Resources, ed. K. N. Wexley (Washington, D.C.: Bureau of National Affairs, 1991): 5-185–5-220.

287

29 .H. J. Frazis, D. E. Herz, and M. W. Horrigan, “Employer-provided training: Results from a new survey,” Monthly Labor Review, 118 (1995): 3–17.

30 .W. Arthur, Jr. et al., “Effectiveness of training in organizations: A meta-analysis of design and evaluation features,” Journal of Applied Psychology, 88 (2003): 234–245.

31 .C. Anderson, “Bad measurement affects training impact,” Chief Learning Officer (May 2014): 44–46; J. Marshall and A. Rossett, “Perceptions of barriers to the evaluation of workplace learning programs,” Performance Improvement Quarterly, 27 (2014): 7–26.

32 .“Verizon: Tech University (Tech U) – Deductive reasoning,” training (January/February 2015): 57.

33 .G. Alliger and E. Janek, “Kirkpatrick’s levels of training criteria: Thirty years later,” Personnel Psychology, 42 (1989): 331–342.

34 .W. Arthur, Jr., W. Bennett, P. Edens, and S. Bell, “Effectiveness of training in organizations: A meta-analysis of design and evaluation features,” Journal of Applied Psychology, 88 (2003): 234–245.

35 .T. D. Cook, D. T. Campbell, and L. Peracchio, “Quasi-experimentation.” In Handbook of Industrial and Organizational Psychology, 2d ed., Vol. 1, eds. M. D. Dunnette and L. M. Hough (Palo Alto, CA: Consulting Psychologists Press, 1990): 491–576.

36 .Ibid.; J. J. Phillips, Handbook of Training Evaluation and Measurement Methods, 2d ed. (Houston, TX: Gulf Publishing, 1991).

37 .D. Sussman, “Strong medicine required,” T+D (November 2005): 34–38.

38 .S. J. Simon and J. M. Werner, “Computer training through behavior modeling, self-paced, and instructional approaches: A field experiment,” Journal of Applied Psychology, 81 (1996): 648–659.

39 .CHG Healthcare Services, “2011 Training Top 125,” Training (January/February 2011): 87.

40 .J. Komaki, K. D. Bardwick, and L. R. Scott, “A behavioral approach to occupational safety: Pinpointing and reinforcing safe performance in a food manufacturing plant,” Journal of Applied Psychology, 63 (1978): 434–445.

41 .R. D. Bretz and R. E. Thompsett, “Comparing traditional and integrative learning methods in organizational training programs,” Journal of Applied Psychology, 77 (1992): 941–951.

42 .S. I. Tannenbaum and S. B. Woods, “Determining a strategy for evaluating training: Operating within organizational constraints,” Human Resource Planning, 15 (1992): 63–81; R. D. Arvey, S. E. Maxwell, and E. Salas, “The relative power of training evaluation designs under different cost configurations,” Journal of Applied Psychology, 77 (1992): 155–160.

43 .P. R. Sackett and E. J. Mullen, “Beyond formal experimental design: Toward an expanded view of the training evaluation process,” Personnel Psychology, 46 (1993): 613–627.

44 .“Mountain American Credit Union: Flow philosophy training,” training (January/February 2015): 103.

45 .B. Gerber, “Does your training make a difference? Prove it!” Training (March 1995): 27–34.

46 .A. P. Carnevale and E. R. Schulz, “Return on investment: Accounting for training,” Training and Development Journal (July 1990): S1–S32.

47 .J. Phillips and P. Phillips, “Distinguishing ROI myths from reality,” Performance Improvement (July 2008): 12–17.

48 .J. Mattox, “ROI: The report of my death is an exaggeration,” T+D (August 2011): 30–33.

49 .A. Purcell, “20/20 ROI,” Training and Development (July 2000): 28–33.

50 .J. Phillips and P. Phillips, “Using action plans to measure ROI,” Performance Improvement, 42 (2003): 22–31.

51 .K. Ellis, “What’s the ROI of ROI?” Training (January 2005): 16–21; B. Worthen, “Measuring the ROI of training,” CIO (February 15, 2001): 128–136; D. Russ-Eft and H. Preskill, “In search of the Holy Grail: Return on investment evaluation in human resource development,” Advances in Developing Human Resources (February 2005): 71–85.

52 .J. Gordon, “Eye on ROI,” Training (May 2007): 43–45.

53 .A. P. Carnevale and E. R. Schulz, “Return on investment: Accounting for training,” Training and Development Journal (July 1990): S1–S32; G. Kearsley, Costs, Benefits, and Productivity in Training Systems (Boston: Addison-Wesley, 1982).

288

54 .S. D. Parry, “Measuring training’s ROI,” Training and Development (May 1996): 72–77.

55 .Aetna Inc., “Training Top 125 Rankings,” training (January/February 2013): 81.

56 .D. G. Robinson and J. Robinson, “Training for impact,” Training and Development Journal (August 1989): 30–42; J. J. Phillips, “How much is the training worth?” Training and Development (April 1996): 20–24.

57 .A. Purcell, “20/20 ROI,” Training and Development (July 2000): 28–33.

58 .J. J. Phillips, Handbook of Training Evaluation and Measurement Methods, 2d ed. (Houston, TX: Gulf Publishing, 1991); J. J. Phillips, “ROI: The search for best practices,” Training and Development (February 1996): 42–47; J. Phillips and P. Phillips, “Moving from evidence to proof,” T+D (August 2011): 34–39.

59 .D. G. Robinson and J. Robinson, “Training for impact,” Training and Development Journal (August 1989): 30–42.

60 .D. Sussman, “Strong medicine required,” T+D (November 2005): 34–38.

61 .J. E. Matheiu and R. L. Leonard, “Applying utility analysis to a training program in supervisory skills: A time-based approach,” Academy of Management Journal, 30 (1987): 316–335; F. L. Schmidt, J. E. Hunter, and K. Pearlman, “Assessing the economic impact of personnel programs on workforce productivity,” Personnel Psychology, 35 (1982): 333–347; J. W. Boudreau, “Economic considerations in estimating the utility of human resource productivity programs,” Personnel Psychology, 36 (1983): 551–576.

62 .U. E. Gattiker, “Firm and taxpayer returns from training of semiskilled employees,” Academy of Management Journal, 38 (1995): 1151–1173.

63 .B. Worthen, “Measuring the ROI of training,” CIO (February 15, 2001): 128–136.

64 .D. Abernathy, “Thinking outside the evaluation box,” Training and Development (February 1999): 19–23; E. Krell, “Calculating success,” Training (December 2002): 47–52; D. Goldwater, “Beyond ROI,” Training (January 2001): 82–90.

65 .J. Kirpatrick and W. Kirkpatrick, “Creating ROE: The end is the beginning,” T+D (November 2011): 60–64; J. Kirkpatrick and W. Kirkpatrick, “ROE’s rising star: Why return on expectations is getting so much attention,” T+D (August 2010): 34–38.

66 .R. Brinkerhoff, “The success case method: A strategic evaluation approach to increasing the value and effect of training,” Advances in Developing Human Resources, 7 (February 2005): 86–101; J. Kirkpatrick and W. Kirkpatrick, “ROE’s rising star: Why return on expectations is getting so much attention,” T+D (August 2010): 34–38.

67 .C. Wick and M. Papay, “Feasting on achievement,” T+D (January 2013): 56–60.

68 .E. Holton and S. Naquin, “New metrics for employee development,” Performance Improvement Quarterly, 17 (2004): 56–80.

69 .E. Massie, “Big learning data,” (Alexandria, VA: American Society for Training & Development, 2013); L. Miller, Big Data, Better Learning? How Big Data is Affecting Organizational Learning (Alexandria, VA; American Society for Training & Development, 2014).

70 .E. Frauenheim, “Numbers game,” Workforce Management (March 2011): 20–21; P. Gallagher, “Rethinking HR,” Human Resource Executive (September 2, 2009): 1, 18–23; B. Roberts, “How to put analytics ‘on your side,’” HR Magazine (October 2009): 43–46; A. Rossett, “Metrics matter,” T+D (March 2010): 64–69.

71 .G. Dutton, “What’s the big deal about big data?” training (March/April 2014): 16–19; B. Roberts, “The benefits of big data,” HR Magazine (October 2013): 20–24; J. Dearborn, “Big data: A quick start guide for learning practitioners,” T+D (May 2014): 52–57.

72 .D. Galloway, “Achieving accurate metrics using balanced scorecards and dashboards,” Performance Improvement (August 2010): 38–45.

73 .J. Dearborn, “Big Data: A Quick Start Guide for Learning Practitioners,” T+D (May 2014): 52–57.

74 .D. Mallon, “At a glance,” T+D (December 2009): 66–68;  www.key.com , webpage for Key Bank, accessed September 15, 2011.

 

289

Case 2

Learning in Practice

Business Goals Drive Learning at Verizon

Verizon’s three business goals for 2011 were to build a business and a workforce that were as good as its networks, to lead in shareholder value creation, and to be recognized as an iconic technology company. The company’s strategic business units align their priorities with the overall company business goals. The Learning and Development section of Verizon establishes training priorities and initiatives that support the business unit and overall company goals. To help create shareholder value, a customized executive education development program sponsored by the company president and CEO was developed to help senior managers understand how to drive long-term value creation. The program provides them with tools, processes, and metrics to help them understand how to positively influence shareholder value. As part of the program, senior managers work in cross-functional teams and are given an assignment to identify barriers to creating more shareholder value. At the end of the session, each team presents its recommendations to top executives. Many of these recommendations have been implemented, resulting in such changes as new budgeting processes and new process improvement programs. Also, as part of the program, each manager chooses one or two actions that they will commit to for positively influencing shareholder value. These actions are part of the senior manager’s performance review. Because innovation and new technology drives Verizon’s success, the company is using social media for training and on-demand learning. Videos distributed through the company’s YouTube site are one of the employees’ favorite ways to learn. For example, videos are used to demonstrating system processes for sales teams. Many videos go “viral,” with employees recommending to their peers a video they may have just watched. Verizon also expanded its internal social network for peer-to-peer collaboration. Also,  Yammer.com  was recently launched. It has more than 8,000 members and over 400 groups using it. Sales teams use the site to post questions and share best practices. In 2013 Verizon implemented tablet computers for performance support and training of retail store employees. An app puts all the data that retail store employees need to learn about devices, service plans, promotions, and policies on the easy to use and readily accessible tablet computer. The app called SIMON (Simplified Information for the Moment of Need) runs on both Apple and Android devices and is designed to be used when interacting with a customer. Also, Verizon provided its field technicians with tablets to access product knowledge and fixes to service problems. Previously, each day local managers gave them handouts with this information. Verizon envisions that by 2022, its learning organization will have a greater emphasis on facilitating and moderating user-generated content rather than generating or providing content itself. The company expects that its workforce will use mobile online performance support on an as-needed basis into the future.

Questions

1. Do you think it is easier for a company like Verizon, which emphasizes technology and innovation, to adopt and use new technologies such as social media for training? Why?

2. What metrics should Verizon use to show the effectiveness of its internal YouTube video site?

3. Should all training that Verizon offers be available to employees on an “as-needed” basis? Explain your answer. Is performance support the same as learning?

Sources: Based on L. Freifeld, “Verizon’s new # is 1,”training (January/February 2012): 28–30; M. Weinstein, “Verizon Connects to Success,”training (January/February 2011): 40–42; J. Salopek, “Good Connections,” T+D (October 2014): 48–50.