write a review

EvaluationCases.pdf

Home >Business & Finance homework help >Economics homework help >write a review

Caterpillar won the overall Corporate University Best In Class (CUBIC) award in 2004. In addition, it received the CUBIC awards for Evaluation Technique and the CUX Xchange Best Measurement. This case study describes the program that was one of the reasons for the awards. It evaluated this program at levels 1 (Reaction), 3 (Behav- ior), and 4 (Results). It will be of interest to the readers for both the subject content and the forms and procedures that can be adapted to organizations of all sizes and types.

Caterpillar, Inc. Caterpillar University

Merrill C. Anderson, Ph.D. CEO, MetrixGlobal, LLC Chris Arvin, Dean, Leadership Development

Peoria, Illinois

Introduction

The announcement of Caterpillar’s business growth goals thrust its leaders into a world of paradoxes: operate with autonomy to run a business unit but in a way that collaborates with other business unit leaders; be accountable to drive higher business unit profits but in a way that does not suboptimize profits in other business units; maxi-

Chapter 17

Evaluating a Leadership Development Program

186

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed u nd er U .S . or a pp li ca bl e co py ri gh t la w.

EBSCO : eBook Collection (EBSCOhost) - printed on 9/21/2019 4:51 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 260710 ; Kirkpatrick, Donald L., Kirkpatrick, James D..; Evaluating Training Programs : The Four Levels Account: s3654090.main.ehost

Evaluating a Leadership Development Program 187

mize the near-term value of current assets but be prepared to make investments that take advantage of global opportunities.

This leadership challenge was not just to develop more leaders, it was to develop different leaders: leaders who epitomize collaboration, business acumen, and a global mind-set. Meeting this challenge to develop a new kind of leader also required new ways of thinking about leadership development.

Caterpillar has a rich history of growing its own leaders. In the 1970s and 1980s the annual management course at the Starved Rock State Park in Illinois exposed leaders to the latest thinking about lead- ing people and organizations.This course evolved into the Caterpillar Advanced Management Program that prepared leaders to effectively expand Caterpillar’s business base.With the establishment of Caterpil- lar University and the College of Leadership in 2001, Caterpillar had an exciting new capability to develop leaders. Building a unified approach to leadership development across Caterpillar became the focus.

The Leadership Development Pilot

This new leadership initiative, launched in 2002, represented a bold departure for Caterpillar with the intention of creating a new kind of leadership. The initiative featured multisource feedback, a two- day workshop and a follow-up session to further drive application and business impact. Participants in this initiative received multi- source feedback that was structured around the new leadership framework. They reflected upon this feedback to chart their own unique course of development. The workshops deepened their understanding about how they needed to change and how to make this change happen.

The centerpiece of the Leadership Development initiative was a two-day experiential workshop for department heads and their intact leadership teams. These workshops featured feedback on individual and organization climate surveys to develop self-awareness, and action planning to apply key insights to improve performance. Each partici- pant completed an action plan. Over the course of three months the participant (and others) took actions to remedy this issue and docu- ment their actions in the form of a case study.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

188 Case Studies of Implementation

A second, one-day session was then conducted with the leader and his or her intact team three months after their initial two-day work- shop.The intention of this session was to reinforce and accelerate how participants applied what they learned to their work environment. Case studies were reviewed, obstacles were identified, potential solu- tions were brainstormed, and successes were highlighted. Participants also explored the potential impact of their case studies on the perfor- mance of people and the organization.

The Caterpillar CEO and his senior leaders decided to first con- duct a pilot of this new approach to leadership development. Evaluat- ing the results of this pilot was critical to learning how best to deploy leadership development throughout Caterpillar.

Evaluation Approach

The evaluation plan consisted of three elements that were organized according to the four Kirkpatrick (1998) levels (Table 17.1):

Table 17.1.The Evaluation Plan for the Leadership Development Pilot

Level Activity Description

1 Leadership Develop- The evaluation was conducted at the ment Feedback conclusion of the two-day workshop and (Exhibit 17.1) addressed the quality of facilitation,

workshop content, relevance of the workshop and additional items.

3 Quick Wins Score The evaluation was conducted about two Sheet (Exhibit 17.2) months after the workshop had been

completed and just prior to participation in a one-day follow-up session.This evaluation addressed how well leaders applied what they learned in the work- shop, their assessment of improved effectiveness, and areas of business impact.

3, 4 Value Narratives This evaluation was conducted about four months after the one-day follow-up session and probed specific examples of application and business impact. Business impact was captured in terms of mone- tary as well as intangible benefits.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Leadership Development Program 189

Level 1. Reaction data were gathered via a questionnaire completed by each pilot participant at the conclusion of the workshop (Exhibit 17.1). Areas addressed included the quality of facilitation, workshop content, relevance of the workshop, and additional items.

Level 2. Learning data was not formally collected as part of the evaluation plan. Given the senior levels of the leaders in the organiza- tion, it was felt that a learning comprehension test would not be appropriate. Learning data were collected as part of the value narra- tives, in addition to application examples and business impact as part of the storytelling process.

Level 3. Change in behavior data were collected via the Quick Wins Score Sheet about two months after the completion of the workshop and about one week prior to participation in a one-day follow-up meeting (Exhibit 17.2). The score sheet began by asking for an example of how the participants applied what they learned in the workshop.Then, based on this example, participants offered their assessment of improved effectiveness on their performance, the per- formance of their teams and the performance of the organization. If respondents indicated that performance had improved as a result of their participation in the LD initiative, then they checked off one or more of the particular areas of the business they thought were impacted. Examples of these areas included productivity, employee engagement, product quality, and other areas.

Level 4. Business results data were collected about four months after the one-day follow-up session. Specific examples of behavior change and business results were probed in one-on-one interviews according to an innovative values narrative process. A value narrative is defined as the written representation of events and people producing value in an organization. It is, in essence, a short story. There are three main elements to these stories:

1. The first element is to capture background information about the leaders and the particular situation that they faced.

2. The second element describes what leaders did as a result of their participation in the Leadership Development Initiative. Actions must be specific enough to support further probing into business impact.

3. The third element probes the impact that the leaders’ actionsCo py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

190 Case Studies of Implementation

Exhibit 17.1. Leadership Development Workshop: Feedback

Instructions: We appreciate your participation in the pilot workshop. Please com- plete this questionnaire so that we may learn from you about how to improve the content and delivery of the Leadership Development Workshop. Space is provided to give feedback on each facilitator.Thank you!

Please select a response category for each item that best reflects your views:

1 Strongly Disagree

2 Disagree

3 Somewhat Disagree

4 Somewhat Agree

5 Agree

6 Strongly Agree

Items 1 2 3 4 5 6

Facilitator Name:

1. The facilitator was prepared and organized for the workshop.

2. The facilitator was responsive to participants’ needs and questions.

3. The facilitator kept all participants actively engaged.

Facilitator Name:

1. The facilitator was prepared and organized for the workshop.

2. The facilitator was responsive to participants’ needs and questions.

3. The facilitator kept all participants actively engaged.

Workshop Content

4. The objectives for the workshop were clearly explained.

(continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Leadership Development Program 191

5.The workshop content/materials were sufficient to achieve the workshop objectives.

6.The length of the workshop was appropriate for the workshop objectives.

Relevance of the Workshop

7.This workshop was relevant to my work.

8. I have gained new skills and knowledge that will improve my effectiveness.

9. I will apply what I have learned to my job.

Additional Items

10. I would recommend this workshop to my colleagues and co-workers.

11.What was the most valuable piece of new learning you received in this program?

12. How could this workshop be improved?

Exhibit 17.1. Leadership Development Workshop: Feedback (continued)

have had on the business. Results were captured in terms of monetary as well as intangible benefits.

Results of the Evaluations

Level 1: Reaction of Leaders to the Workshop

Overall, the leaders rated the workshop highly, averaging 87 percent favorable (defined as either a 6 or a 5 favorable response on the six- point scale.) Lowest rated was the workshop content (79 percent aver-

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

192 Case Studies of Implementation

Exhibit 17.2. Quick Wins Score Sheet

Name: ___________________________________

Please respond to the following questions in preparation for the one-day Leadership Development follow-up session. In addition to helping you prepare for this session, your responses will help us to better understand how you have applied what you have learned. This information will help us to learn from the pilot experience and ultimately improve the full deployment of the Leadership Development initiative.

1. What are you doing differently as a result of what you have learned from Leadership Development?

________________________________________________________________

2. Have these actions improved:

a. Your effectiveness as a leader? Yes No Not Sure

b. Your team’s effectiveness? Yes No Not Sure

c. Your organization’s performance? Yes No Not Sure

3. If you feel that your actions have improved effectiveness, please indicate in what areas:

i. Productivity

ii. Employee engagement

iii. Quality of work

iv. Decision making

v. Clarity about priorities

vi. Communications

vii. Collaboration

viii. Time to complete projects

ix. Other:

4. What other benefits have you, your team and/or the organization realized so far from Leadership Development?

________________________________________________________________

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Leadership Development Program 193

Exhibit 17.2. Quick Wins Score Sheet (continued)

________________________________________________________________

Thank you!

age for the three items), in particular, leaders felt that the workshop objectives could have been better explained (73 percent favorable). Workshop relevance was rated high (93 percent) and almost all lead- ers (96 percent) would recommend the workshop to colleagues. The level 1 data suggested several enhancements to the workshop. These enhancements were made and reaction scores soared to over 95 per- cent favorable in all three areas.

Level 3: Change in Behavior

Leadership participants from two Leadership Development pilots indi- cated that they were able to readily apply what they learned from the leadership sessions to create meaningful impact on the organization. Heightened employee engagement and increased clarity of leadership team members on strategic priorities topped the list of eight impact areas.All but two leaders cited examples of how they were able to apply what they learned from LD to their workplace. These actions were credited with improving their effectiveness as leaders by 81 percent of the respondents, improving team effectiveness by 56 percent, and improving organization performance by 44 percent of the respondents.

Respondents identified specific ways in which their LD experi- ences increased effectiveness. Figure 17.1 presents the percent of respondents who identified one or more of eight effectiveness cate- gories.Topping the list of eight was engagement with 81 percent. One team leader reported taking actions to improve organization climate and provide employees with greater recognition for their efforts. Greater engagement seemed to be extended to leadership teams as well. Respondents reported encouraging more open dialogue in lead- ership team meetings, allowing more input by team members, and pro- viding greater latitude for the teams to be engaged in problem solving.

Three additional impact areas were selected by over 50 percent ofCo py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

194 Case Studies of Implementation

0 20 40 60 80 100

Percent

Productivity

Engagement

Quality of Work

Decision-making

Clarity about priorities

Communications

Collaboration

Time to complete projects

Figure 17.1.The Percent of Leadership Development Participants Whose Actions Impacted the Eight Business-Impact Categories

the respondents: clarity about priorities, communications, and collabo- ration. Written comments by the participants indicated that leaders had increased alignment of their leadership teams to strategy and busi- ness direction. Some leadership teams were reorganized to accomplish this alignment. Improved communication effectiveness also facilitated strategic alignment. Better delegation, improved listening, and increas- ing the quality of leadership team meetings were cited as examples of improved communication. Many respondents indicated that they were spending less time in meetings and yet getting more accomplished. Coaching skills had improved. Leaders reported that they were not necessarily acting in a more collaborative way. Rather, they were using collaboration more effectively and perhaps even more sparingly. For example, one respondent wrote that there was in his leadership team less collaboration on areas requiring his direct decision.

Other intangible benefits were cited by the respondents. These included having one common language to talk about leadership and a shared understanding of the required competencies, having their lead- ership group being seen by stakeholders as more cohesive and more capable of leading the growth strategy, team members gaining a better understanding of each other, and an increased energy and focus on developing leaders.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Leadership Development Program 195

Level 4: Business Results

Given the very senior levels of the participants and the need for high- quality data, it was decided to conduct value narratives with a 25 per- cent sample of leaders from the two pilot groups. Four leaders who participated in one of two pilots were interviewed to further explore how they applied what they learned to impact their respective busi- nesses.These interviews were written up as brief narratives or stories. Intangible benefits and, when appropriate, monetary benefits were documented as part of the narrative-building process. Monetary benefits were expressed in dollars or hours and directly attributed to actions taken as a result of participation in the leadership development initiative. Of course, there were many other potential influencing fac- tors on producing these benefits, so when monetary benefits were identified, the leaders were also asked two additional questions (Anderson 2003; Phillips 1997).The first question required leaders to attribute a percentage of the monetary benefit directly to their LD experience. The leaders were then asked to express as a percentage their confidence in this attribution. The monetary benefit was dis- counted by these two factors (attribution and confidence). This resulted in monetary benefits that were qualified and conservative in nature.

Two value narratives are excerpted and offered as examples of how these data were collected (see Exhibits 17.3 and 17.4). The leaders identified many important intangible benefits that were produced by their actions.These included:

1. Improved strategic focus in decision making, enabling leaders to focus on the most strategically critical decisions, and not just those decisions that were the most urgent and not neces- sarily the most strategic.

2. Improved performance management of subordinate leaders, as clearer expectations for performance were set and more effective leadership styles were adopted.

3. Increased accountability for results, as leaders became more involved in setting performance targets and their personal roles in achieving these targets were given greater visibility.

4. Increased insights into personal development needs, as lead- ers better grasped how their actions impacted the climate of

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

196 Case Studies of Implementation

Exhibit 17.3. Value Narrative No. 1

Background Diane’s (a fictitious name) predecessor ran the marketing group with a strong team concept that emphasized consultation and involvement in all aspects of decision mak- ing. The group’s high employee engagement scores were in large part attributed to the highly consultative team environment created by the group’s leaders. Diane con- tinued this style when she took over the group about a year ago, although it was at times frustrating. She felt that her group could be more responsive to changes in the external environment. Her participation in the LD initiative helped her explore the downside of this highly consultative style. Diane’s key learning was that consultation, while important, needed to be better focused on only those decisions that required a broader base of information, and not just reflexively applied to all decisions.

Change in Behavior Encouraged by her LD experience, Diane implemented better screening of issues that required decisions. Specific accountabilities for making the decisions were clarified. Ground rules for bringing certain kinds of decisions to the attention of decision- making bodies were specified. Decision-making bodies such as Diane’s leadership team gained added focus as their time was better spent on more strategic issues. Leaders were consulted when their specific expertise and knowledge were required. Decisions that Diane needed to make that did not require other leaders’ input did not go through the gauntlet of consensus building. Meetings and the topics covered were streamlined.The team concept continued to flourish and engagement levels remained high.

Business Results Diane estimated that at least a 10 percent to 15 percent improvement in team produc- tivity was realized for herself and her team of ten direct reports. She attributed 100 percent of this productivity gain to LD and was 70 percent confident in this estimate.

The monetary benefits of the productivity gain were:

11 people × $85 hour × 40 hours per week = $37,400 total per team per week

$37,400 × 10% productivity gain = $3,740 productivity gain per week

$3,740 × 48 weeks = $179,520 of annualized benefit

$179,520 × 100% (attribution) × 70% (confidence) = $125,664

A total monetary benefit of $125,664 was gained from LD through increased productivity.

Intangible benefits included:

• Improved strategic focus in decision making

• Improved and more efficient communications

• Clearer expectations

• Better risk management

• Increased insights into personal development needs

• Stronger teams

• Facilitated culture change

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Leadership Development Program 197

Exhibit 17.4. Value Narrative No. 2

Background The timing of Frank’s (a fictitious name) leadership development experience was excellent, given that Frank’s team had gone through a process of great change. Frank recently replaced someone who was in the role for twenty years.While the group was considered to be successful, it had, in the last few years, become rather set in its ways. The employee opinion survey results were trending down for the group, and the group did not seem strongly connected with the broader business enterprise.

Frank assumed his new role and immediately led a change in the group’s approach to working with business partners. Frank’s approach was to roll up his sleeves and manage every aspect of the group’s business. His strong orientation to detail enabled him to set the pace in working with his people so that they understood what Frank expected from them. Frank’s hands-on management style was successful. During this transition phase, the group went from being perceived to be on the fringe of the core business to becoming a more vibrant and central partner to the other business units. Frank’s business grew as dealers were reengaged and stronger partnerships with deal- ers were forged.

Frank’s style of personally setting the pace was effective during the transition phase. However, with the transition completed, a different approach to leadership was required. While relationships with other business units and the dealers were improved, Frank’s own team was becoming dispirited. They often felt that they needed to wait for Frank in order to make the right decision.Teamwork was low and employee engagement was trending downward.

Change in Behavior Frank participated in the LD initiative and learned that his strong pacesetting style was no longer the appropriate style for his group. In lieu of any overarching strategy, people did not feel empowered to make decisions. His weekly staff meetings, which had become quite lengthy, were nothing more than data dumps so that Frank could make the appropriate decisions. Encouraged by LD, Frank decided to take a more strategic approach. He stopped his weekly staff meetings and instead facilitated monthly off-sites.The purpose of these off-sites was to delve more deeply into strate- gic issues. Frank began engaging his people in creating the group’s strategy so that they could make decisions independently and still know that these decisions were in line with the strategy. Decision making improved. Employee engagement jumped to 72 percent from 37 percent, and Frank attributed a significant chunk of this increase to his leadership development experience.The team went from running hard to run- ning smart.

Business Results According to Frank, these actions freed up at least two to three hours of Frank’s time per week. He attributed 50 percent of this gain to LD and was 90 percent confident in this estimate. Monetary benefits were determined as follows:

2 hours per week × 48 weeks × $85 per hour = $8,160 in annualized benefits

$8,160 × 50% (attribution) × 90% (confidence) = $3,672 in qualified benefits

(continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

198 Case Studies of Implementation

Exhibit 17.4. Value Narrative No. 2 (continued)

Intangible benefits included:

• Improved decision making

• Higher employee engagement

• Increased teamwork and enthusiasm

• Increased empowerment

• Increased strategic focus

• Improved communications

the organization and the performance of their teams and managers.

5. Higher employee engagement, as the organizational climate improved and people were able to make a stronger link from their behavior to achieving the organizational goals. People felt more empowered to act without necessarily having to go through a series of approval steps. Teamwork improved and communications became more effective and focused.

In addition to these benefits, a total of $141,576 in qualified, annual- ized, monetary benefits were identified by a 25 percent sample of the leaders included in the value narrative process. These benefits com- pare favorably with the out-of-pocket cost of $134,000 for conduct- ing the sessions with both of the pilot teams. It is fair to say, based on the sample data collected, that the two Leadership Development pilots more than paid for themselves while delivering substantial, strategic, and sustainable value.

Conclusion

The value narratives completed the successful story of the Leadership Development pilot at Caterpillar. The story began with the leaders’ initial workshop experience being rated very favorably. The story continued with the Quick Wins Score Sheet, which documented sig- nificant examples of application to the work environment.The value narratives enabled leaders to tell their stories in a way that really res-

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Leadership Development Program 199

onated with others.While monetary benefits were only one element of these stories, the monetary benefits that accrued from the leader- ship development initiative more than paid for the investment in the initiative.

References

Anderson, Merrill C. Bottom-Line Organization Development. Boston: Butterworth-Heinemann, 2003.

Kirkpatrick, Donald L. Evaluating Training Programs: The Four Levels. 2nd ed. San Francisco: Berrett-Koehler, 1998.

Phillips, Jack J. Handbook of Training Evaluation and Measurement Methods. 3rd ed. Houston: Gulf Publishing, 1997.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

This case study from Australia concentrates on the evaluation of desk- top application courses, including Word, Excel, and Power Point. The training company also teaches and evaluates soft skill courses. It evalu- ated at all four Kirkpatrick levels. Particular emphasis is placed on communicating with the participants before, during, and after the courses. Of special interest will be the many contacts between the trainers and the learners and their bosses.

Pollak Learning Alliance Heather Bolster, General Manager, Professional Services

Antoinette Chan, General Manager, Marketing Sydney, Australia

Description of Our Organization

Pollak Learning Alliance is a large and long-standing Australian train- ing company that has been providing learning solutions to both cor- porate and government for over twenty years. We train in excess of 35,000 participants a year in learning centers located throughout the country, with a head office in Sydney.We have a staff of around sixty.

We provide learning solutions in both desktop and soft skills, sup- plemented with e-learning tools. Our focus is on learning transfer to

Chapter 18

Evaluating Desktop Application Courses

200

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

EBSCO : eBook Collection (EBSCOhost) - printed on 9/21/2019 4:55 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 260710 ; Kirkpatrick, Donald L., Kirkpatrick, James D..; Evaluating Training Programs : The Four Levels Account: s3654090.main.ehost

Evaluating Desktop Application Courses 201

improve performance in the workplace.We support the training with a range of consulting services, including training-needs analysis, appli- cations development, and tailored training.

Description of the Evaluation Program

Evaluating the effectiveness of training has long been a key focus for us, and we have many stories to tell in the measurement area. But for the purpose of this exercise we would like to describe the evaluation pro- gram we have put in place across the board for our desktop application courses—Word, Excel, PowerPoint, and so on, which are mostly one- or two-day courses. (We train soft skills as well but have a different method- ology for evaluating in that domain.) We feel we have broken some ground here and are pleased to take this opportunity to describe it.

Background

It began with a marketing event we hosted for key clients at the Syd- ney Opera House.We decided to have an open discussion with them about return on investment—to hear from our clients what they had done, their case studies, war stories, key challenges, things still unre- solved.We wanted a chance to hear them, really, and to let them hear each other about this alone-in-the-wilderness topic.

If you are reading this book, you are someone who can imagine the themes that emerged—primarily of course the extreme difficulty of providing evidence that the training they do actually produces results for the business.Their budgets, they said, go elsewhere, where evidence can be provided. And they have no real way of measuring the effective- ness of their training providers.We found that although these were large corporations and government agencies, there actually were very few ROI case studies, and not even a lot of war stories. It’s just too hard.

So we decided to take it on. Our Managing Director, Steve Lem- lin, had been at the seminar and was piqued by the topic. As an ex- accountant, he had been involved always in eliciting proof of return on investment. And he knew there had to be a way to measure return on training investment.

Thus began our journey.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

202 Case Studies of Implementation

Our Objectives

We based our thinking from the beginning around Kirkpatrick’s four levels of assessment. And we decided (controversially?) that when it comes right down to it what any business is interested in is not fundamentally the first three levels; it’s great if people have enjoyed the training, it’s great if they learn and if they change behaviors, given spending all that time on the course, but it’s not central to the objectives of the business. What is important to the CEO is whether the training has actually impacted the business objectives and driven the organization closer in some way to reach- ing its desired outcomes.

So we decided to focus on level 4: has the training impacted the business objectives and how can that be measured? And equally important, if we’re going to do all this measuring and evaluation, how can we make it actually add to the process, increasing the effectiveness of the training rather than just driving everyone mad?

When we looked at the financial drivers of a business (simply put, to earn more or spend less), we saw that what we could actually mea- sure is time saved. This would be our primary ROI measure, and we would enrich it with a variety of other “softer” measures also known by a business to be important.

Specifics of the Evaluation

The process we have developed is called Prepare—Learn—Apply— Measure (known affectionately as PLAM internally.) It works like this:

Before the Course

1. When participants register for any of our courses (e.g.,Word, Excel, Access, at intro, intermediate, or advanced levels), they are asked to appoint a “sponsor.” This is a learning sponsor, someone who will support their learning back in the work- place. Often it is their manager.

2. They are also e-mailed a link to a Web site.They go online to an attractive screen that asks them to specify their learning outcomes for the course. They are also asked whether their

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating Desktop Application Courses 203

outcomes have been discussed with their sponsor, and if the outcomes are aligned to their job role.

3. Our trainers review these outcomes in advance of the course.

During the Course

1. Participants go online after each and every module of the course and are asked to speculate briefly about how they will use the software’s feature after the program, and how much time they may save by using the feature after the course.

2. They are also asked at the end of the day to rate any changes in their confidence, the potential quality of their work, their ability to problem-solve, and their attitude to their job.

3. They go online and create an action plan. 4. They complete a traditional evaluation of the program (level 1).

After the Course

1. They are reminded to work with their sponsor toward accomplishing their action plan.

2. Four weeks after, they are asked to go online again and com- plete a survey about time saved and changes in the way they are approaching their work.

Summary of the Levels Being Evaluated

Level 1. The process evaluates level 1 in the usual way, with a com- prehensive questioning about their reaction to the trainer, the course, the service, and the like.

Level 2. Level 2 evaluation is done during the training itself, indi- rectly, with exercises built in to the program.We also have online test- ing software and occasionally clients will take advantage of the offer to test participants before and/or after their training.

Level 3. Participants define outcomes before the course and create an action plan during the course. Afterward, we find out if they have achieved their outcomes and accomplished their action plan.

Level 4. Level 4 is at the heart of what we’re up to with this pro- gram.The data we collect is about time saved, and how that translates

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

204 Case Studies of Implementation

into dollars, as well as about what participants and their sponsors see in regard to how the training has impacted their roles and results and the business overall.

Results of the Evaluation

Because this is a new process for us, results are only beginning to flow in. Some of our observations at this stage:

• Reporting: We can of course turn this data into a wide spec- trum of reports for our clients.The most popular reports are:

1. Participation Reports (e.g., who has done what stages of the process, who the popular sponsors are)

2. Perceived Value Reports (summarizing the “soft data” from the process—increase in confidence, increase in quality of work, and so on)

3. Dollars Saved Reports (summarizing clients’ speculation about how much time they will save and actually have saved in using the features, and translating that into dollars)

• Client reaction: Our client contacts at senior levels in the orga- nization love it. Response to the reports, in particular the Dollars Saved Reports, has been very positive, with strong feedback by both HR managers and CEOs.

• Participant reaction: Both trainers and participants report favor- ably about the process. The extra discipline of thinking through how the software is going to be used back at work, and how their jobs will be impacted by it, is generally seen as a valuable investment.As one of our trainers put it, the biggest impact is probably people’s buying into the fact that they’ve actually got an impact on the business.

• Participation: Currently about half the participants are engag- ing in the precourse stage (remembering that ours is a large, across-the-board client base, this is not too bad a result— which we’re working to improve).Virtually all participants do the on-the-day stage. And, finally, it’s too early to have good statistics on the postcourse stage. This will be the tough one and will demand our attention in making it happen.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating Desktop Application Courses 205

Our Communication

During the pilot process, several client focus groups were formed to gather feedback.This group consisted of senior HR professionals from some of our top corporate clients, who were very keen to participate.

Upon completion of the pilot, e-mail marketing was executed to our top-tier corporate clients, advising them of the initiative. Infor- mation has also been posted on our Web site and collateral (brochures) to incorporate the new training methodologies. We also communicated the new tools and measures internally, reinforcing our objective of strengthening long-term relationships with our clients by undertaking this initiative.

The results have been very positive, with clients responding favor- ably.We will be convening another focus group sometime in the next two months to review the reports and the response to the program generally.

We are enormously excited by the possibilities opening up for us and for our clients from this initiative.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Chapter 19

Evaluating an Orientation Program for New Managers

206

This program, referred to as “jump start,” was evaluated at all four Kirkpatrick levels.The content of the program will be of interest to all readers who are eager to get their new managers off to a good start. Also, the evaluation approaches and forms will be of special interest and adaptable to organizations of all sizes and types.

Canada Revenue Agency, Pacific Region David Barron, Regional Learning and Development Advisor

Vancouver, British Columbia

Introduction

Canada Revenue Agency (CRA) forms part of the Federal Public Ser- vice of Canada. Nationwide the agency has approximately 40,000 employees. The agency’s Pacific Region, covering the province of British Columbia and the Yukon Territory, has roughly 4,600 employees.

CRA Pacific Region’s Jump Start to Management Program was developed to provide new managers throughout Pacific Region with the opportunity to learn what they need to perform effectively in their new roles. The design and development of the program was undertaken in collaboration with all prospective stakeholders in response to clearly defined urgent regional needs.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

EBSCO : eBook Collection (EBSCOhost) - printed on 9/21/2019 4:56 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 260710 ; Kirkpatrick, Donald L., Kirkpatrick, James D..; Evaluating Training Programs : The Four Levels Account: s3654090.main.ehost

Evaluating an Orientation Program for New Managers 207

In order to most effectively meet the learning needs of new man- agers a four-phase model was developed:

Phase I: A Local Orientation Checklist was developed Phase II: A three-day Regional Orientation Session was designed

and delivered Phase III: A Compendium of Core Learning Events was developed Phase IV: An inventory of Advanced Learning Events was pro-

jected and initial development work was undertaken.

This case study focuses on the evaluation process used to assess the effectiveness of Phase II: the Regional Orientation Session. The theme of the Regional Orientation Session was “Balancing manage- ment with leadership,” and great stress was laid on effective people management as the key to effective program management.The session contained modules on values and ethics, inspirational leadership, self- assessment, achieving corporate goals, coaching as a management practice, priority management and meeting management, as well as a full-day hands-on exercise on managing performance and the oppor- tunity to learn from a senior manager in an informal armchair session. An overview of a typical Regional Orientation Session can be found in Appendix 19.1.

Four three-day sessions were held between September 2003 and February 2004.

Evaluation Approach

An in-depth formal evaluation strategy was developed for the Regional Orientation Session to assess its effectiveness at all of Kirk- patrick’s four levels: reaction, learning, behavior, and results. Instru- ments used included Learner Reaction Questionnaires, Content Analysis Questionnaires, random follow-up contacts, and postsession focus groups.

Level 1 Reaction: Relevance and Satisfaction

Participants were asked to evaluate how relevant they found the con- tent of the Regional Orientation Session to their jobs and to rate

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Appendix 19.1. Overview of Jump Start Regional Orientation Session

DAY 1 DAY 2 DAY 3

8:30–9:00 Introductions 8:30–9:00 Networking and recap 8:30–9:00 Networking and recap of of and links to day 1 and links to days 1 and 2

Facilitated by Learning and Facilitated by Learning and Facilitated by Learning and Development Team Development Team. Development Team

9:00–10:30 Opening Remarks 9:00–12:00 Performance Management 9:00–11:15 Coaching Practices for Managers Workshop

Frankly Speaking: Exploring Presented by Learning and Presented by National the Leadership Mindset Development Team supported Manager’s Community

by various HR Subject representatives or . Matter Experts alternate manager

Guest: Senior Manager

Performance Coaching Management . . . continued continued

Participants examine the leader/manager’s role in the Performance Management process through work-related scenarios.

10:30–10:45 Health Break 10:30–10:45 Health Break 10:30–10:45 Health Break (Included in Coaching Session)

208

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

Appendix 19.1. Overview of Jump Start Regional Orientation Session (continued)

10:45–12:00 The Corporate World Performance 11:15–12:00 Meetings Bloody of the CCRA: An overview Management . . . continued Meetings: Meeting

Management and the use of Bob Chartier’s Tools for getting the most out of meetings

Facilitated by Intergovernmental Presented by Learning and Corporate Affairs and Development Team

12:00–1:00 Lunch: a networking 12:00–1:00 Lunch: a networking 12:00–1:00 Lunch: a networking opportunity opportunity opportunity

1:00–2:00 Inspirational Leadership 1:00–2:45 Performance 1:00–2:15 Managing Priorities: Facilitated by George Matthews Management . . . continued the Key to Time

Management

2:00–2:45 Balancing the Role of Presented by Learning Leader and Manager and Development Team

Facilitated by Learning and Development Team

2:45–3:00 Health Break 2:45–3:00 Health Break 2:15–2:30 Health Break

3:00–4:30 Self-Assessment: 3:00–4:30 Performance 2:30–3:50 Ethics and Values A Time to Reflect Management . . . continued

Facilitated by Learning Facilitated by a and Development Team Senior Manager

3:50–4:30 Concluding Remarks: Summary and Transfer of Learning Plan

By Learning and Development Team

209

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

210 Case Studies of Implementation

their overall satisfaction with the session in terms of a five-point scale where 5 is the highest and 1 is the lowest. A copy of the form used to evaluate level 1 can be found in Appendix 19.2.

Level 2 Learning

For the content sessions on Day 1 and Day 3 participants were asked to complete content evaluation questionnaires designed to capture what they thought they had learned. The hands-on performance management simulation on Day 2 was evaluated separately by narra- tive report. Copies of the forms used to evaluate level 2 can be found in Appendix 19.3a and Appendix 19.3b.

Level 3 Behavior

In order to assess transfer of learning, two focus groups were held with participants in April/May 2004—that is, some considerable time after they had attended Jump Start Phase II. At these focus group events participants were asked what they had been able to apply on the job from that which they had learned in Jump Start. A question schedule for these focus groups can be found in Appendix 19.4.

Level 4 Results

In an attempt to gain insight into how participation in Jump Start Phase II could positively impact business results, focus group partici- pants were asked to gauge the effect of implementing what they had learned from Jump Start Phase II in terms of:

• Morale • Teamwork • Turnover • Production

Results

Level 1

Over 80 percent of the participants found the topics covered in the session either relevant or very relevant to their jobs and were satisfied

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Orientation Program for New Managers 211

Appendix 19.2. Example of Learner Reaction Questionnaire (Level 1 Evaluation)

Jump Start to Management Regional Orientation Session February 10–12, 2004

Learner Reaction Questionnaire

Your feedback will be used to help us continually improve our products and ser- vices.

1. Position Title: Level: Work Location:

2. How much total experience have you had in a management role (including acting)? _____years _____months

3. Did you complete Jump Start Phase I (Local orientation) before attending this session? YES NO

4. Why did you attend this session?

5. How would you rate the importance level of the topics covered in this session to your job? Low _____________High

1 2 3 4 5

6. To what extent was your learning enhanced by the opportunities to engage in activities with senior managers and HR representatives? Low _____________High

1 2 3 4 5

7. Overall, what was your level of satisfaction with this session? Low _____________High

1 2 3 4 5

8. What is your confidence level in applying to your job what you learned through your participation in this learning event? Low _____________High

1 2 3 4 5

(continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

212 Case Studies of Implementation

9. Please describe aspects of this learning event you found particularly meaningful.

10. What specific elements of the three-day session had the most positive impact on you and why?

11. Is there anything else that would have facilitated your learning? If so, please describe.

12. Are there any changes you feel we need to make to this workshop? If so, please describe.

Name: (optional) ____________________________________

Thank You!

Appendix 19.2. Example of Learner Reaction Questionnaire (Level 1 Evaluation) (continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Orientation Program for New Managers 213

Appendix 19.3a. Example of Content Evaluation Form Day 1 (Level 2 Evaluation)

November 18, 2003 Name:____________________________

Session: The Corporate World of the CCRA

Objective

• To give new managers an understanding of how and what they contribute to the organization as a whole

After participating in this session do you feel you now have a better understanding of how your work as a manager contributes to the achievement of corporate goals? Please explain.

Session: Balancing Management and Leadership

Objectives

• To examine the leadership expectations of a CCRA manager • To illustrate why a CCRA manager must balance management and leadership

roles in order to be successful.

After participating in this session do you now better understand why you need to bal- ance management and leadership? Please explain.

November 18, 2003 Name:____________________________

Session: Understanding the Possibilities for Leadership

Objectives

• To energize • To inspire • To motivate • To reflect

(continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

214 Case Studies of Implementation

After participating in this session, how energized, inspired, and/or motivated do you feel about your new role? Please explain.

Did you have an opportunity to reflect on your new role? What was the outcome of your reflection?

Session: Ethics and Values

Objectives

• To raise awareness of the roles values and ethics play in effective leadership • To raise awareness of the Public Service Values • To profit from the experience of senior managers

After participating in this session, do you now better understand the role that ethics and values play in effective leadership? Please explain.

How useful did you find the case studies?

Appendix 19.3a. Example of Content Evaluation Form Day 1 (Level 2 Evaluation) (continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Orientation Program for New Managers 215

November 18, 2003 Name:____________________________

Session: Self-Assessment and Reflection

Objectives

• To introduce participants to the Managing for Success instrument • To develop self-awareness and self-understanding prerequisites to effective man-

agement • To introduce the concept of reflective practice

After participating in this session, what have you learned about yourself as a manager?

How useful did you find the Managing for Success instrument?

Appendix 19.3a. Example of Content Evaluation Form Day 1 (Level 2 Evaluation) (continued)

or very satisfied with the workshop. On their own, these are very high average figures. They might well have been even higher had not a number of more experienced managers been sent to a workshop designed for new managers.

Level 2

The results received from the content evaluation questionnaires illus- trate that the overwhelming majority of Jump Start to Management participants reported that due to having taken part in a Phase II ses- sion they now felt better equipped to do their jobs. Specifically: 83 percent of participants reported that they now felt they could manage meetings more effectively, 82 percent reported a better understanding

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

216 Case Studies of Implementation

Appendix 19.3b. Example of Content Evaluation Form Day 3 (Level 2 Evaluation)

November 20, 2003 Name:____________________________

Session: Coaching Practices for Managers

Objectives

• To familiarize participants with Paul Lefebvre/National Managers Network coaching tool kit

• To provide practice on how to use the tool

After participating in this session, do you feel you now have a better understanding of coaching as a management practice? Please explain.

To what extent do you feel that you can apply what you have learned in this ses- sion?

Session: Meeting Management

Objective

• To rethink the role and use of meetings so the time is used more effectively

After participating in this session, do you feel you can now manage meetings more effectively? Please explain.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Orientation Program for New Managers 217

Session: Priority Management—the Key to Time Management

Objective

• To familiarize participants with the Covey model of priority setting

After participating in this session do you now feel that you are better equipped to manage your time more effectively? Please explain.

Session: Armchair Session (Innovation)

Objective

• To allow participants to profit from the experience of an experienced senior manager (knowledge transfer)

What was the most valuable learning you gained from this session?

Appendix 19.3b. Example of Content Evaluation Form Day 3 (Level 2 Evaluation) (continued)

of ethics and values’ role in leadership, 79 percent felt they were now better equipped to manage time, 77 percent found the self-assessment exercise useful, 73 percent found the inspirational leadership session energizing and inspirational, 72 percent felt they better understood how their work linked to the achievement of corporate goals, and 64 percent reported that they now better understood the need to balance management and leadership. In addition, many participants com- mented on the value of the armchair session, and indeed on the posi- tive effects of senior management’s demonstrated support of the program.

The hands-on one-day practical performance management work- shop, which formed Day 2 of Phase II, was evaluated by narrative

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

218 Case Studies of Implementation

Appendix 19.4. Focus Group Question Schedule (Levels 3 and 4)

Jump Start to Management Regional Orientation Session Focus group questions

1. What did you learn in Jump Start that you found relevant to your work?

2. From what you learned in Jump Start, what have you been able to apply?

3. What has been the effect of applying what you learned in terms of:

• morale

• teamwork

• turnover

• production

4. What, if anything, has made it difficult for you to apply what you learned in Jump Start?

5. Since participating in Jump Start, have you identified further learning needs?

6. What help do you need to meet your ongoing learning needs?

response. Participants reported learning from the personal experiences of others: they commented on the way that colleagues’ issues and sug- gestions were very useful in putting things into perspective. Participants also reported learning from the various Human Resources subject mat- ter experts from Staff Relations, Alternative Dispute Resolution, Employee Assistance Program and Competencies who were on hand to help them work through the scenarios. Participants found these resource persons very beneficial in discussing issues that had, or could potentially, come up in the workplace, while their presence also rein- forced the fact that managers can turn to Human Resource advisers when in need of help. Various tools such as the 5Ps, Appreciative Inquiry, and SMART Goals were found to be very useful and of great potential in the workplace since these tools were important in under- standing people’s values, interests, and passions. Additionally, the exercise on Vision, Goals, and Objectives was regarded as a potentially powerful way to help differentiate corporate goals from smaller team goals.

Level 3

As can be expected, the answers differed from individual to individ- ual. In general, however, the participants found that Jump Start to Management Phase II helped them realize the importance of taking

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Orientation Program for New Managers 219

the time to get to know their employees as individuals. Their work was put into perspective, affecting a decrease in stress level when deciding the urgency of demands from HQ versus those that existed in their work areas. They also reported that attending the session resulted in great improvements in team communication. With the tools that they have learned from Jump Start, they had been able to involve their team members in coming up with different and better solutions to problems that their teams might be facing.

Specifically, as illustration of how they had transferred what they had learned in Phase II into practice, participants made reference to:

Appreciative Inquiry

• Discovering new ways of doing things • Reflecting more

Managing Priorities

• Learning how to say no • Beating the terror of e-mails • Learning to distinguish between urgent and important • Being better organized • Being more available for team members

Managing Meetings

• Encouraging participation • Rotating meeting functions • Doing joint minutes with a related team • Being better organized

Performance Management

• Being more effective at career management • Using 5 Ps as a good tool to create more interesting and effec-

tive feedback sessions • Helping to get buy-in to performance management process

Armchair Session

• Being yourself • Treating others the way you want to be treated

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

220 Case Studies of Implementation

Coaching

• Investigating the commitment behind the complaint

Level 4

Most comments recorded focused on the increase in team morale through better people management. In one focus group almost every- one reported a noticeable increase in morale, which they attributed to their changed behavior (e.g., modeling values, more “hands-off ” management) as a result of having participated in Jump Start. Several participants commented on the close link between an increase in morale and improved teamwork, which itself was reflected in improved production. In one case an example was cited where two closely related teams, divided by a wall, had now learned to work around this wall.Turnover was not found to be an issue.

Communication of Results

A full evaluative report was compiled on the entire Jump Start pro- gram. This report was presented to CRA Pacific Region’s senior management, distributed among the stakeholders who had collabo- rated in the design of the program, and submitted to Training and Learning Directorate of Canada Revenue Agency in Ottawa. Jump Start has since been recognized by the National Managers Council of the Public Service of Canada as a best practice.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

This comprehensive case study describes in detail the strategy, plan- ning, implementation, and evaluation of the training program at levels 1 (Reaction), 2 (Learning) and 3 (Behavior). In addition to using Kirk- patrick’s “four levels” as the basis for evaluation, it used the work of three other authors in developing an effective training program.

PacifiCorp Dan Schuch, Power Learning Training Developer

Portland, Oregon

PacifiCorp is a large, internationally owned utility, based in Portland, Oregon, that generates more than 8,400 megawatts (mw) and delivers power to 1.5 million customers in six western states. The company has 15,000 miles of transmission line, 44,000 miles of overhead distri- bution line, and 13,000 miles of underground distribution line. Paci- fiCorp operates as Pacific Power in Oregon, Washington, California, and Wyoming and as Utah Power in Utah and Idaho. There are approximately 6,140 employees within the company whose duties range from those involving the maintenance and operation of electri- cal power lines to those normally found in a large business.

PacifiCorp is committed to the professional and personal develop- ment of its employees and community members. We firmly believe that, in a constantly changing environment, continuous learning and

Chapter 20

Evaluating Training for an Outage Management System

221

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

EBSCO : eBook Collection (EBSCOhost) - printed on 9/21/2019 4:57 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 260710 ; Kirkpatrick, Donald L., Kirkpatrick, James D..; Evaluating Training Programs : The Four Levels Account: s3654090.main.ehost

222 Case Studies of Implementation

the acquisition of new skills and knowledge are essential for personal development and the overall success of the company.To this aim, Paci- fiCorp has built an extensive and varied training program, including an extensive distance education program, a number of separate train- ing facilities, computer-based training opportunities, and relationships with universities and colleges located in our service territory. Power- Learning is one of the training branches of PacifiCorp.This past year PowerLearning conducted over 750 courses equating to over 15,000 training days to its employees.

Early in 2004, PowerLearning reexamined its training program with the intent to improve it and to better match training with on- the-job performance.This strategy was in line with the leading work in effective training programs. A comprehensive training and evalua- tion strategy was developed. It was based on the leading research and best practices in the design and development of effective training and includes Kirkpatrick’s work, Shrock and Coscarelli’s book on criterion-referenced test development, Dick and Carey’s instruction design model, Robinson and Robinson’s work in performance, and learning theories from Gagné.

Kirkpatrick’s four levels of evaluation for training programs was the standard selected for the evaluation component. Specifically, Kirk- patrick’s levels 1 to 3 (Reaction, Learning, and Behavior) were inte- grated into our training strategy. Business results and return on investment issues were separated from our basic training strategy. In this chapter, a specific training class will be described in detail com- plete with level 2 and 3 evaluation outcomes highlighted. Discussion will follow how Kirkpatrick’s evaluation levels were integrated into our training strategy, outcomes from this training, and benefits received—especially in reference to the integration of level 3 activi- ties early in the training development process.

New Outage Management System Training—Case Study

In March 2004, PowerLearning developed and conducted training on a new system. This event has turned out to be an ideal case study when determining the effects of training on job performance.

Early in 2004, PacifiCorp facilities in California were upgraded to a new outage management system.This new computer software pro- vided an important link between the centralized dispatch centers andCo

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating Training for an Outage Management System 223

the various offices scattered across our service territory. Company offices in the other states had already been using this system for some time. Upgrading the California offices to the new system would enable the entire company to be using the same outage management system.

However, once the new system was implemented the outdated one would be turned off. It was not possible to run the old and the new systems simultaneously. The company’s California employees using this new system were required to master the new system before the previous one was permanently shut down. None of the employees had any previous experience with this new system prior to the train- ing. Needless to say, training was critical. Mistakes made using the new system could result in delays in service during outages to the com- pany’s California customers or could place our employees working in the field at risk of injury. This training took place from March 8 to March 10.The new system was activated on March 15, 2004.

The training team included the company subject matter expert on the new system, a representative from the central dispatch group, and the trainer.This team was assembled to address any questions or prob- lems pertaining to the new system, interactions between the field and the central dispatch office, or the training itself.These resources were provided so that the right people were available to handle any possible problem or question that might arise with the system or groups affected by the new system. In addition, all supervisors of the employ- ees participating in the training were present during the training and participated as well.

Training consisted of demonstration of the system, discussion of the impacts and risks, followed by the students practicing and demonstrat- ing proficiency of the specific tasks to the instructors. Each participant was provided with documentation on the system as well as a job aid describing the process step by step.This training lasted a day and a half.

It was important to accurately assess the performance of the employees taking this training in order to identify any gaps in compe- tence and close them. All employees participating in this training completed level 1 and level 2 assessments upon completion of the training.The level 2 assessment was developed such that each question simulated actual performance on the system or checked critical verbal information material required to operate the new system. Immedi- ately after the class completed the level 2 assessment, the training team reviewed the questions and answers and used this time as a learningCo

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

224 Case Studies of Implementation

opportunity. Though the performance on the assessment was out- standing, all found this immediate feedback to be invaluable and help- ful in clearing up all outstanding issues. Both learners and supervisors strongly felt that the training more than adequately prepared them to use the new system—which was scheduled to be implemented the very next week.

Here’s the rest of the story. During the first day that the new system had gone live, a transmission line went down in Crescent City, causing a large power outage in the area that affected numerous customers.As a result of some unforeseen factors, this outage quickly became a complex one. However, as a result of the thorough training, the employees handled the situation smoothly and efficiently. They were confident in their abilities and performed them with the new system flawlessly. In this instance, there was no time gap between the training and the major outage in which the participants had time to practice their new skills.The transfer between training and performance of the job was clearly evident. In this specific instance, the effectiveness of the training and comprehensive assessment strategy without any con- founding variables can be clearly demonstrated.

Structured level 3 (Behavior) evaluations were conducted via interviews over the phone with these supervisors of the employees of the California offices who participated in the training. All expressed favorable performance results from their employees. The manager from the office experiencing the large outage stated that their employees were well able to handle the outages with the new system. He was very satisfied with their training and confident that they would be able to use the system. He also mentioned that a couple of his employees had expressed appreciation to him for the training.

Why the Training Was Successful

The success of this training was a result of a number of factors.While it is difficult to identify the specific contribution of any one of these factors, it can be confidently stated that the outcomes from the train- ing were very successful and the attitude about the training from the participants and management was great.

When developing the training, the entire system involving and sur- rounding the new outage management system was considered.Train- ing included more than just learning the specifics of the newCo

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating Training for an Outage Management System 225

application; it also included content about other computer systems interacting with the new one and interactions with other groups in the organization affected by the new system.

The training involved true simulation training. Each participant worked in a training environment identical to production, with one person per computer.The learners worked through realistic scenarios. The training tasks provided matched the actual ones performed in the field.

The training team received complete management buy-in and involvement. Supervisors participated in all phases of training devel- opment, including the development of specific job tasks, identifica- tion of performance objectives, approving training materials, signing off on level 2 and 3 assessments, and even participating in the actual training along with their employees.

The right resources were made available for training. Training activities matched actual job performance. The computer training environment mirrored the production environment. Subject matter experts in all areas of the new system were present during the actual training. A comprehensive training development strategy was imple- mented to develop the training, including a thorough job task analy- sis, sound behavioral objectives, and well-written assessment items.

Evaluation Strategy as a Component of a Broader Training Strategy

Our training development model was designed using the leading evaluation, instructional design, and performance improvement mod- els. Dick and Carey’s instruction design model, The Systematic Design of Instruction, provides the overall training development strategy and foundation from which the model was built. Kirkpatrick’s book, Eval- uating Training Programs:The Four Levels, tells us what type of evaluation questions to ask and who should answer them. Shrock and Coscarelli’s book, Criterion-Referenced Test Development, provides sound advice on the specifics of how to develop the evaluations. Our model also inte- grated learning theories from Gagné’s book, The Conditions of Learn- ing, as well as Robinson and Robinson’s work on performance, Performance Consulting: Moving Beyond Training.

The evaluation strategy used at PacifiCorp is a subset of a broader training strategy. PacifiCorp PowerLearning has developed a TrainingCo

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

226 Case Studies of Implementation

Development Model that outlines the training development strategy in a ten-step process, and the evaluation component is an integral part of this model. The steps of the training strategy are provided here as well as details of their evaluation component.

Ten-Step Model of Training Development

1. Identify Business Goals 2. Assess Needs 3. Conduct Instructional Analysis 4. Conduct Learner Analysis 5. Write Performance Objectives 6. Develop Assessment Instruments 7. Develop Instructional Strategy 8. Develop and Select Instructional Materials 9. Implement Instruction

10. Evaluate Program

Step 1: Identify Business Goals. When first meeting with the manager or supervisor, it is important to provide an overview of the training development process. A part of this debriefing will include a descrip- tion of the evaluation components. Evaluation is important to the manager/supervisor/client because it will tie the training to the per- formance on the job. In short, the evaluation strategy will determine if the participants were satisfied with the training (level 1), how much they learned in the class (level 2), and how well they are now applying this new knowledge to their job performance (level 3). PowerLearn- ing’s level 1 evaluation is provided in Exhibit 20.1.

In the specific case study presented earlier, once the decision was made to implement the new outage management system, the author communicated with the supervisors to discuss the project and the training requirements. Attention was given to the outcomes of the training and the supervisors’ responsibilities before, during, and after the training took place. Step 4: Conduct Learner Analysis. A learner analysis is conducted to identify the characteristics of those who will be trained. The client/manager/supervisor will work with training personnel to iden- tify the target audience for a training course or program. Together they will also identify existing learner skills, behaviors, and general ability level of any participant. This information will help define theCo

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Exhibit 20.1.Training Strategy

In our model, the manager/client (business owners of the training) is actively involved in all aspects of the training. Note that sign-off occurs at steps 1, 2, 4, 7, 8, and 10.

PacifiCorp “Power Skills”Training Strategy

Step 1: Identify Business Goals Step 7: Develop Instructional Strategy Meet with Business Owners Meet with Business Owners Get sign-off Get sign-off

Step 2: Assess Needs Step 8: Develop and Select Instructional Materials Meet with Business Owners Meet with Business Owners Get sign-off Get sign-off

Step 3: Conduct Instructional Analysis Step 9: Implement Instruction

Step 4: Conduct Learner Analysis Step 10: Evaluate Program Meet with Business Owners Meet with Business Owners Get sign-off Get sign-off

Step 5: Write Performance Objectives

Step 6: Develop Assessment Instruments

Assess Needs

Identify Business Goals

Analyze Learners

and Context

Conduct Instructional

Analysis

Develop Assessment Instrument

Write Performance Objectives

Develop and Select

Instructional Materials

Evaluate

Implement Instruction

227

Develop Instruc- tional

Strategy

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

228 Case Studies of Implementation

parameters of the planned training. Identifying the motivation of the participants, as well as interests, attitudes, and learning preferences will help determine how the training will be conducted.

In some instances, a pretest may be given to determine compe- tency levels of the participants. Outcomes from the pretest could affect whether a person is exempt from taking the course or from some components of the course. It could also highlight possible con- tent areas to be included in the training.

At this stage in the new outage management system training devel- opment, important decisions were made about the parameters of the training based on the skill levels of the employees who would be par- ticipating.These decisions affected the instructional strategy. It was at this time that decisions were made to include subject matter experts from the business areas affected by the new system in the actual train- ing sessions.The skill level of the participants dictated the detail of the instructional content that was required. Step 6: Develop Assessment Instruments. At this point in the process, the training team has worked with the client to develop the performance objectives of the training. At the same time, the level 2 and 3 evalua- tions will be developed. The performance objectives reflect the behavioral purpose of the training. The level 2 assessment simply determines whether the learner has mastered these objectives. The level 3 assessment simply determines whether the learner has trans- ferred this new knowledge, skill, or attitude to the job. For most train- ing organizations, it is assumed that level 1 assessments have previously been developed. However, if one is not available, then it would have to be developed as well.Two different types of level 2 assessments are provided (see Exhibits 20.2 and 20.3).

Because of the timing and importance of the new outage manage- ment system, a strategy was developed in which the participants would complete two types of level 2 assessments. Each participant was required to demonstrate proficiency on all tasks and corresponding objectives on the system to the instructor and also to complete a level 2 paper-and-pencil assessment specifically designed for this course. Every objective was included in the assessments, and there were no items not reflected in the objectives. Step 7: Develop Instructional Strategy. From the input previously gathered of the learner analysis, performance objectives, and assessments, an instructional strategy can be developed. The instructional strategy

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating Training for an Outage Management System 229

Exhibit 20.2. Level 2 Assessment—netCADOPS

Answer the following questions using the System Outages Overview menu.

5) List the district(s) that show outages with hazards? 6) Circle the button on the display above to show the outages for the district with

non-customer calls.

includes details of how the training will be delivered. Factors consid- ered include length, location, delivery method, and materials provided.

Once the instructional strategy has been determined, then the busi- ness owner will agree to the instructional strategy and all assessments developed and will “sign off ” on them.The Assessment Sign-off Form has been developed to serve as the sign-off sheet (see Exhibit 20.4).

In the new outage management system course, the training team worked together to determine the most appropriate instructional strategy. Based on the information learned from the specific goals of

One form of the level 2 assessment for this training of a new computer system was given as a paper-and-pencil test.The questions were carefully written to best assess if the learner knew the proper procedures, application keystrokes, and verbal infor- mation required to successfully use the application.To simulate the important tasks presented in this training, screen captures of the application were taken and the learners were asked to simulate the appropriate responses. For example, a screen capture is made of a display from the application and specific questions are asked to make a response close to the actual response. Question 6, as shown, requires the learner to circle the button on the diagram to perform a task. In actual application, the learner would actually push the button to perform the task. Both the question and actual task require the learner to accurately process the information on the page in order to make the correct decision.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Exhibit 20.3. Level 2 Assessment—EMS SCADA Basic Navigation

The level 2 assessment for this training of a new computer system was given as a competency check list.The questions carefully matched the objectives of the course. Each person taking the class was required to demonstrate competency to the instructor on each specific task listed. Various factors required this training to be conducted one on one.This task list was given out to all the learners even prior to the training, and a blank assessment was provided to all learners after completion. Distributing the checklist before the training provides the person with the important elements of the training before it starts.The learner can also use this checklist to supplement the training to help verify abilities after the training. A section of this assessment is provided here.

WS500 Navigation—Performance Assessment

The student will achieve the goal of the course by completing the presented objectives.These objectives are achieved by demonstrating competency to the instructor in the specific behaviors assigned to each objective. Students must demonstrate mastery in each objective to earn credit for this course.

Procedure Objectives Tasks Demonstrated To accomplish . . . Demonstrate the By showing you can . . .

ability to . . .

Logging into the Log In Launch WS500 from desktop system Shift Change Log In Log in while another operator is already logged in

Log Out Log out of the WS500

Change Password Change the WS500 password

Working with Open Displays Using the Open the master menu (MSTRMENU) in a new window using Displays File Method the filter and wildcard characters

Open a substation index display in the same window from a poke point on the Master Menu

230

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

Exhibit 20.3. Level 2 Assessment—EMS SCADA Basic Navigation (continued)

Procedure Objectives Tasks Demonstrated To accomplish . . . Demonstrate the By showing you can . . .

ability to . . .

Working with Open Displays Using the Open a substation one-line display in a new window from a poke Displays (cont.) File Method (cont.) point on the substation index display

Navigate Between Navigate to a display previously open in the active window using Displays Display Recall buttons

View the display history for the window using Display Recall and select a display to open

Navigate to another open display using the Window drop down menu

231

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

232 Case Studies of Implementation

Signatures of Approval

Business Owners: Date:

Power Skills: Date:

TSF 5 Rev 8/04

Assessment Instrument Form

Level 2 Evaluation (attached)

Level 3 Evaluation (attached)

Estimated Date for Level 3 Evaluation:

Project Name: Date: Business Owner: PowerLearning Manager:

Department: Project Assigned To:

Exhibit 20.4. PacifiCorp Assessment Instrument Form

During step 7 of our training strategy, we meet with the business clients to review the level 2 assessments developed, approve them, and collaboratively develop the items for the level 3 assessment. We ask our clients to sign the Assessment Instru- ment Form on these items.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating Training for an Outage Management System 233

the course, the learner characteristics, and the development of the objectives and corresponding assessments, it was determined that the training had to be hands-on at a company facility in close proximity to the company offices in California. It was determined that the train- ing team would include a lead trainer and subject matter experts from company areas affected by the new system.

Management also signed off on the assessments and worked with the training team to develop together the appropriate level 3 assess- ment items. It was decided that the level 3 assessment would be con- ducted by the lead trainer in the form of an informal interview within a few weeks after the training. Furthermore, it was decided that this time would also be used to determine the next course of action if there were deficiencies found in the transfer of the learning to the workplace. Step 10: Evaluate program.Training has been given. Level 1 (Reaction) and 2 (Learning) evaluations will be conducted immediately after the training. Feedback on the evaluations will be provided to the business owner. A time will be established to administer the level 3 (Behavior) evaluation to the business owner or designate. The purpose of the level 3 evaluation is solely to determine if there has been a transfer of training from the class to the job performance. This date should be sufficiently long enough after the end of training in order for the supervisor to determine if the skills learned in the course have been transferred to the workplace.Training staff will conduct level 3 evalu- ation at a later date after the end of training (see Exhibit 20.5). Training staff will meet with business owners to review level 3 results, acquire approval signatures, and determine next steps.

An interesting thing happened during the administration of the paper-and-pencil assessment during the new outage management class. Upon completion of the assessment, the class, including the supervisors, reviewed each of the questions and answers. The assess- ment turned into a valuable learning tool, and the participants gath- ered some valuable insights. The instructors, supervisors, and class participants left the class with a newfound appreciation for level 2 assessments and how they can be used as additional training tools.

Because of the unique situation that occurred after the training, the level 3 evaluation was conducted very shortly after the class.The man- agers were delighted beyond measure by the performance of the class participants on the job.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Exhibit 20.5. PacifiCorp Level 1 Assessment

234

A significant amount of effort was put into the development of our level 1 assessment provided below.

Course:

Instructor:

Date:

Location:

It is our sincere desire to provide you with the best possible learning experience. We take our responsibility to help you perform your job better very seriously. Please take a few moments to complete this survey about your training experience. Thanks from the entire PowerLearning training team.

About this Learning Activity...

This learning activity met my expectations. This activity will help me to perform my job better. The materials used in this activity helped my understanding. I feel that I have learned something from this activity.

Was the length of the activity appropriate? (please circle) Too Short Please provide a suggestion for improving the course (use back of sheet for additional suggestions):

Just Right Too Long

About the Facilitator...

The facilitator was effective presenting the material. The facilitator was knowledgeable in the subject matter. The facilitator involved me in learning. The facilitator managed time well. The facilitator provided applicable examples/demonstrations. The course objectives were clearly stated. The course objectives were fully met.

Strongly Disagree

1 1 1 1 1 1 1

Disagree

2 2 2 2 2 2 2

Neutral

3 3 3 3 3 3 3

Agree Strongly Agree

4 4 4 4 4 4 4

5 5 5 5 5 5 5

What did the faciliator do well in the class that really helped your learning?

Strongly Disagree Neutral Agree Strongly Disagree Agree

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

W Y

Exhibit 20.5. PacifiCorp Level 1 Assessment (continued )

235

What can the facilitator do to improve the learning experience:

About the Learning Experience...

Rate the overall ease & clarity of the enrollment process. Rate the overall training facility

What other suggestions would you have for improving your learning experience?

About What You Learned...

Rate your productivity, BEFORE TRAINING, on a scale of 0 to 10, on the skills/knowledge you learned in this course.

Predict your productivity, AFTER TRAINING, on a scale of 0 to 10, on the skills/knowledge you learned in this course.

On a scale of 0 to 10, how much of your total working time will you be

spending on tasks that require the skills/knowledge you learned in this course?

On a scale of 0 to 10, rate the importance of the skills/knowledge you learned in this course as it relates to your specific job?

1 2

8 9

Poor

1 1

Fair

2 2

Good

3 3

Very Good

4 4

Excellent

5 5

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

236 Case Studies of Implementation

Benefits Experienced from Implementing Level 2 Assessments

PowerLearning has recently begun to implement level 2 assessments in our training. In addition to being able to measure effects of our training better, we have received a number of additional benefits as a result of implementing level 2 assessments in our training programs. We found that the level 2 assessment also serves as a teaching tool. In the case study presented earlier, the answers to the assessment were reviewed with the class upon completion of the assessment.We were delighted to discover that in a couple of instances material covered during the course was clarified. We noted that students would pose additional questions that were answered by both the instructors and other classmates and led to a richer training experience.

The level 2 assessment provided a content check for the instructors. In one specific instance it was identified during the debrief time that an important point covered in the assessment was not covered in the depth that it needed to be addressed during the training. A potential problem was averted by reviewing the assessment after the class. The assessment provided a valuable and time-saving check on the training.

The use of the level 2 assessments also improves the consistency of content presented by the different trainers, because, we have found, that having the different instructors use the same level 2 assessment for a given course as a benchmark has helped us to bridge the gaps in training and learning outcomes between instructors. Differences are quickly identified and resolved before the actual training begins.

The implementation of level 2 assessments has gone smoothly and there has been complete support from the class participants for the courses we have developed. Instructors and class participants have a better idea of what is important in the class, and the level 2 evalua- tions enforce consistency between the instructional content and the course objectives. Development of the level 2 assessment has helped focus the training development. Extraneous material is removed from the instruction, and objectives are added and refined to better match important instructional material. This has helped streamline our courses to include only the relevant and important content.

An additional benefit is that the level 2 assessment is also being used as a teaching tool. The level 2 assessment can help validate learners’ understanding and increase instructors’ confidence that the class par- ticipants have mastered the material covered in the class. In situationsCo

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating Training for an Outage Management System 237

when a student is not able to demonstrate competency, instructors are provided with a good opportunity to clarify and answer questions.

Benefits Experienced from Implementing Level 3 Assessments

It is important to note that even if students can demonstrate profi- ciency during training, it does not mean that they can perform the task on the job. Obviously, since job performance is more important than performance in the classroom, there is a need to check actual performance on the job after training to determine if transfer has taken place. Normally these evaluations take place long enough after the training to enable the supervisor or manager enough time to determine if the employee has transferred the skills onto the job. Kirkpatrick’s Level 3 evaluation is designed to do just that.

We have integrated the development of the level 3 assessment into our training development strategy. Once we have identified the skills and tasks to be included in the training, we develop the objectives (level 2 and level 3 questions).We have the person who has requested the class, usually a manager or supervisor, review these and sign off on them.This takes place before any instructional materials are developed.

The majority of the level 3 assessments we have given have been in the format of a structured interview. During this meeting, we discuss the level 3 questions (that we had previously developed collabora- tively) for each employee.We then explore next steps.We have found that we received tremendous support from management using this process and that there has been a strong sense of ownership and part- nership from management.When we have followed up with the level 3 evaluation after training, we found that managers were very recep- tive and provided specific and useful feedback.

We have found that we have been able to develop training that bet- ter addressed the rationale for training. By jointly developing the level 2 and 3 assessments and comparing them, we were to include content that was relevant and eliminate extraneous material before extensive development had occurred. As a result, the resulting training better matches the required job performance and often has saved the com- pany time and money.

We have also found that when these managers provided us with training development projects, they did so with an increased confidenceCo

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

238 Case Studies of Implementation

level. These managers are proactively involved in additional training projects and are able to better articulate the outcomes they are expect- ing as a result of having previously gone through level 3 evaluations.

By developing and receiving approval for the level 3 questions before development of the training materials, we received better sup- port for our training from management, developed and delivered bet- ter training, and saved the company time and money resources.These outcomes were achieved with minimal additional effort or cost.

Final Thoughts

Implementing a sound evaluation strategy into our training develop- ment has been highly effective.We agree with others that level 3 and 4 evaluations cannot be performed unless a level 2 evaluation has been previously performed. It makes sense to our training team and managers that one cannot determine if the training was effective without knowing the learning (level 2) outcomes of the training par- ticipants.We strongly believe that the success of our evaluation efforts is affected by implementing the sound methodology for developing the different assessments.We have experienced the fact that the busi- ness owners/managers’ confidence and satisfaction in our training organization increased as a result of involving them strategically throughout the development of training.We found that not only were our training efforts successful, but that our training group also experi- enced the benefits of increased responsibility and opportunity for developing bigger training initiatives for the company.

References

Dick, Walter, Lou Carey, and James O. Carey. The Systematic Design of Instruction. 6th ed. Boston: Pearson, 2005.

Gagné, Robert M. The Conditions of Learning. 4th ed. New York: Holt, Rinehart and Winston, 1985.

Kirkpatrick, Donald L. Evaluating Training Programs: The Four Levels. 2nd ed. San Francisco: Berrett-Koehler, 1998.

Robinson, Dana G., and James C. Robinson. Performance Consulting: Mov- ing Beyond Training. San Francisco: Berrett-Koehler, 1996.

Shrock, Sharon, and William Coscarelli. Criterion-Referenced Test Develop- ment. Boston: Addison Wesley, 2000.Co

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

This practical case study comes from Spain. It describes a program of great interest to many types and sizes of organizations where “coach- ing” has become a critical component of training. Moving from level 2 (Learning) to level 3 (Behavior) requires the manager to encourage and help learners apply what they have learned.You will find practical subject content as well as evaluation tools and techniques that you can use and/or adapt.

Grupo Iberdrola Gema Gongora, Training and Development Manager

Consultants Epise, Barcelona

Juan Pablo Ventosa, Managing Director Nuria Duran, Project Manager

Madrid, Spain

The Company

With more than 100 years of experience, Iberdrola is one of the main private electricity supply industries of the world. Its services, addressed to sixteen millions of clients—more than nine million just in Spain—are focused on the generation, transport, distribution and

Chapter 21

Evaluating a Coaching and Counseling Course

239

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

EBSCO : eBook Collection (EBSCOhost) - printed on 9/21/2019 4:58 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 260710 ; Kirkpatrick, Donald L., Kirkpatrick, James D..; Evaluating Training Programs : The Four Levels Account: s3654090.main.ehost

240 Case Studies of Implementation

marketing of electricity, and natural gas. There are more than 10,000 staff members at the Iberdrola offices in Spain and Latin America.

For Iberdrola, training has strategic relevance, since it is an essential function that helps to assure the competency levels demanded of the professionals, so they can fulfill the requirements of the Strategic Plan. In the year 2000 they did 400,000 hours of training, which make for an average of about forty-one training hours per person per year. To date, training has been evaluated exclusively in regard to the partici- pants’ reaction or satisfaction level.

The corporation asked whether there was the need for an integral evaluation system to form part of its strategic guidelines. This system would allow the evaluation of training’s impact on all of the com- pany’s business and units.

The Project

The Corporate Training Unit and the Training Services attached to the various companies of Iberdrola decided to attempt a common approach to the development and implementation of the guidelines for evaluation. A team of training specialists from the organization, with the collaboration of an external consultant, Epise, developed a project for the creation of a general handbook of procedures designed to evaluate training events.

Three training events were chosen to serve as a pilot, and an evalu- ation procedure was designed and applied to these events in accor- dance with Kirkpatrick’s four-level model. The training events were intended for business units and dealt with widely varying subjects so that they would provide a sufficient number of cases for the creation, based on the acquired experience, of a practical handbook that met the needs of the organization. One of these training events was a face- to-face course on coaching and counseling, administered at Iberdrola Engineering and Consultancy.

The Course

The characteristics of the course are given in Table 21.1.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Coaching and Counseling Course 241

Table 21.1.

Course Title: COACHING and COUNSELING Date: 28/03/01 to 30/03/01

Number of Participants: 11 Duration: 16 hours

Number of Assessed Participants: 10 Location: Madrid

Taught by: Euroresearch

Profile of the participants

• People who are going from performing the function of junior engineer to that of senior engineer.They have at least two years of experience in the company.

• They will go on to coordinate small work teams.

Course objectives

1. To make the participants aware of the importance of directing their col- leagues by using a style of constant listening and personal attention.

2. To provide training in the skill of developing collaborators for the position. 3. To develop active listening skills in order to confront problems of perfor-

mance or motivation. 4. To develop the skills necessary to intervene in the event of emotional or

motivational conflict between colleagues.

Methodology

A completely participative method is employed. Three “coaching” and “counsel- ing” role-playing exercises are conducted, as well as two games, in order to demonstrate some key aspects. Participants complete three questionnaires about learning style, styles of response in emotional communication, and the opening of avenues for interpersonal communication. Each theoretical explanation is followed by a practical exercise of similar duration.

Evaluation

How Are the Criteria Defined?

Having been conducted in previous years, this training event had already been designed and the educational goals necessary for the level 2 evaluation were available, but the criteria for levels 3 and 4 were not. In order to obtain this information, a workshop was con- ducted with participants’ supervisors.

In the first part of the session, the project was presented, with an emphasis on the contribution expected from the supervisors and the

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

242 Case Studies of Implementation

benefits they would receive in exchange. In the second part, those in attendance responded collectively to the following questions:

• As regards the functions of the participants in the training event, what tasks are they responsible for that are related to the content of the course, and what criteria are used to evaluate their performance?

• What are the strategic goals of the department? • What factors, apart from the training of the staff, have an

influence on the performance of the department?

As seen in Table 21.2 the results of the workshop, were used to:

• develop tools to evaluate behavior (level 3), based on the criteria used to evaluate the tasks related to the course con- tent.

• select criteria for the evaluation of results (level 4).

What Tools Were Used?

Level 1 Reaction. The questionnaire usually used by the consulting firm responsible for teaching the course was employed.

Level 2 Learning. Because the educational goals of the course included not only knowledge but skills as well, the consulting firm that gave the course was asked to conduct one test of knowledge and another of skills. For this purpose, the firm designed a questionnaire and guidelines for observation. These can be seen in Exhibits 21.1 and 21.2.

Level 3 Behavior. A questionnaire was designed, with some generic questions and some specific ones based on the criteria for the evalua- tion of the tasks related to the content of the course. Exhibit 21.3 displays the most comprehensive version of this questionnaire, the one intended for the participant after the training event.

Level 4 Results. The level 4 criteria that were selected were those that corresponded to the strategic goals of the department that were most influenced by the tasks related to the content of the course (see chart in Table 21.2).They were:

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Coaching and Counseling Course 243

Table 21.2. Results of the Workshop with Supervisors

With regard to the functions of those attending the training event, what tasks, related to the training received, do they carry out and what are the criteria used to evaluate their performance?

Tasks Evaluation Criteria for the Task

- Degree of satisfaction of colleagues

1. Motivate - Complaints by colleagues - Dedication - Contribution of new ideas

- Correct the course of the project

2. Assign responsibilities - Avoidance of “bottlenecks” - Distribution of the workload - Knowledge of colleagues

- Be aware of information regarding: - Training of colleagues - Abilities of colleagues

3. Know colleagues - Relationship of colleagues with their

surroundings - Behavior of colleagues in extreme situations

- Data regarding performance assessments - Rotation index (unexpected) - Dissatisfaction expressed to the boss

- Knowledge of colleagues - Prevent conflicts from having an influence on

the course of the project 4. Resolve conflicts - As a rule, don’t receive complaints

from the group - Don’t avoid responsibilities (hot potato) - Don’t display lack of camaraderie

- If necessary: 5. Control resources - Number of rotations

(optimize) - Requests for inclusion - Offering of available resources

- Identification of profiles

Possess communication/

- Attitude adjustment (positive results)

6. negotiation skills

- Absence of “rumor mill” due to transparency of information

- Give Web-organized explanations - Brief and precise explanations

(continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

244 Case Studies of Implementation

7. Delegate - Don’t return the delegated “item” - Excessive workload for colleagues

8. Follow the progress - Provide feedback to colleagues of the project - Achieve the goals set out in the planning stages

- Redirect the project if necessary - Have up-to-date information

9. Assess performance - Results are coherent (assessor and assessed) - 360° feedback is carried out - Results can be justified

10. Make decisions - Result - On time - According to plan - Decisions don’t need to be retaken

11. Identify needs - Presentation of proposals - Training needs met - Knowledge of the technical requirements for

the project - No repetition in the meeting of needs - Results of the performance assessment

12. Train colleagues - Satisfaction of colleagues with performance assessments

- Display acquired knowledge and greater independence

• Index rotation • Meeting deadlines • Commercial activity • Profits • Training given • Internal client’s satisfaction index

In order to isolate the effect of the training, it was decided that a con- trol group would be used and that this group would be made up of individuals with characteristics similar to those of the participants in the training, and that they would be matched with each of the partic- ipants one to one. Unfortunately, it was impossible to carry out the evaluation at this level.

Table 21.2. Results of the Workshop with Supervisors (continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Coaching and Counseling Course 245

Chart of Goals/Tasks

Operative Tasks

Motivate Assign responsibilities Know colleagues

• Index Rotation Resolve conflicts Delegate Follow the progress of the project Assess performance

• Meeting deadlines Motivate Assign responsibilities Resolve conflicts Delegate Follow the progress of the project Take decisions

• New breakthroughs (R+D) Motivate

• Commercial activity • Enlargement strategies: Motivate

- Number of applicants Possess communication/negotiation skills - Number of offers

• Profits Control resources (optimize) Follow the progress of the project Take decisions

• Provide technical training Know colleagues Assess performance Identify needs Train colleagues

Corporate

• Internal client’s satisfaction index Motivate Resolve conflicts Possess communication/negotiation skills

• Meeting deadlines Motivate Assign responsibilities Resolve conflicts Delegate Follow the progress of the project Take decisions

• Scope of training (number of Identify needs hours of training per person)

• Degree to which plan is Identify needs successfully carried out

Table 21.2. Results of the Workshop with Supervisors (continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Exhibit 21.1. Knowledge Test for Level 2 Evaluation

(You must remember these numbers at the end of the course)

Coaching and Counseling

Please, fill in this questionnaire related to the Coaching and Counseling course that has as its exclusive purpose to determine the level of learning reached once the course is over.

The content of this questionnaire is totally confidential. The answers of all the group members will be compiled in one document in order to protect the identity of the authors.

At the top of the document, please enter a combination of four numbers (that you must remember at the end of the course) for identification purposes.

To answer the questionnaire, you must indicate (in every item) to which extent the item really fits to team direction.

----

For the team management this behavior is

Very Quite Not very Not suitable suitable suitable suitable

at all

1. Maintaining an open and personal communication with your colleagues

2. Putting yourself in others’ place and understanding their views

3. Being polite and distant in personal relations

4. Showing empathy to emotive expressions

5. Considering that personal life should not be taken into account in professional life

6. Respecting others’ opinion

7. Being inflexible with your thoughts and feelings

8. Providing your colleagues with solutions in conflict situations

9. Paying attention to others

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Coaching and Counseling Course 247

Exhibit 21.1. Knowledge Test for Level 2 Evaluation (continued)

For the team management this behavior is

Very Quite Not very Not suitable suitable suitable suitable

at all

10. Understanding the real difficulties of the work of your colleagues

11. Judging issues from your point of view the others’ opinions without consider- ing feelings and emotions

12. Showing indifference to the personal conflicts of your colleagues

13. Ignoring whenever you can the differences and brushes between team members

14. Communicating clearly and assertively

15. Creating a relaxed and pleasant atmosphere suitable for dialogue

16. Appearing to be perfect without having problems

17. Taking care of personal relations for colleagues to be fluent and positive

18. Trying to provide solutions in conflicts between per- sonal and corporate interests

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Exhibit 21.2. Guidelines for Level 2 Evaluation

Seminar-Workshop Techniques for people management:

Coaching and Counseling -------------------------------------------------------------------------------

Impulse management or counseling Observation notes

Name: .................................................................................................

Observe the manager’s behavior in regard to the verbal and the nonverbal spheres. Write down your comments for every item. At the end of the performance, grade the manager in every item and explain your scoring by writing constructive com- ments in the right. In the scale, 1 stands for “needs to improve substantially” and 5 stands for “excellent.”

EUROSEARCH CONSULTORES DE DIRECCIÓN

CHECK LIST COMMENTS EXAMPLES

Structure Has the skills developer followed all the stages of the skills development model?

*In accordance with the topic 1 2 3 4 5 *It identifies goals 1 2 3 4 5 *It encourages discoveries 1 2 3 4 5 *It establishes criteria 1 2 3 4 5 *It empowers and authorizes 1 2 3 4 5 *It recapitulates 1 2 3 4 5

Procedure Has the chief used the required preparation for the procedure?

*He has paid attention 1 2 3 4 5 carefully *He has asked questions 1 2 3 4 5 *He has made suggestions 1 2 3 4 5 *He has given feedback 1 2 3 4 5 *He has used “I statements” 1 2 3 4 5

Atmosphere Has the chief created a productive atmosphere?

*He has clarified purposes 1 2 3 4 5

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Coaching and Counseling Course 249

*He has avoided value 1 2 3 4 5 judgments *He has created a pleasant, 1 2 3 4 5 genuine, respectful and empathetic atmosphere *Good opening and closing 1 2 3 4 5

Summary *According to you, has this 1 2 3 4 5 been a successful “Skills Development” session?

Has the manager followed the basic counseling model?

*Exploration 1 2 3 4 5 *Finding new perspectives 1 2 3 4 5 *Action 1 2 3 4 5

How does the manager implement the basic skills of counseling?

*Paying attention 1 2 3 4 5 *Listening 1 2 3 4 5 *Visual contact 1 2 3 4 5 *Nonverbal communication 1 2 3 4 5 *In the sort of questions used 1 2 3 4 5

How does the manager handle the two core elements in the interview?

*Feelings/Emotions 1 2 3 4 5 *Empathy 1 2 3 4 5

Summary

*According to you, has this 1 2 3 4 5 been a successful counseling model session?

EUROSEARCH CONSULTORES

DE DIRECCIÓN

Exhibit 21.2. Guidelines for Level 2 Evaluation (continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Exhibit 21.3. Questionnaire About Learning Transference

Posttest: Coaching and Counseling

Participant Questionnaire—Learning Transference

Personal Particulars Name Position

Supervisor or Manager Data Name Position

We have contacted you again to ask for your collaboration in the fulfillment of this questionnaire.The purpose is to collect the necessary data to determine the rightness of the training that you have received at the Coaching and Counseling course as an Iberdrola holding employee.

Personal particulars are essential to manage properly the answer received and the data transferred. However, we assure you that the answers received will be totally confi- dential: the data will be used exclusively for statistical purposes.

Once you have finished the questionnaire, please send it to: E-mail address:

Question 1: In the last months you attended the Coaching and Counseling course. What have you been able to put into practice that you learned?

Nothing A few things A lot of things Almost everything/everything (1) (2) (3) (4)

Question 2: If you answered “Nothing” or “A few things” in the above question, what are the main reasons that led to this?

1. The skills I learned have proved to be insufficient. 2. I didn’t have the opportunity to put them into practice. 3. The supervisor didn’t facilitate the implementation of these skills. 4. I didn’t have the resources to put into practice what I learned. 5. Other reasons. (Please explain them).

Question 3: Do you feel your knowledge about coaching techniques is?

Insufficient Sufficient Good Very good (1) (2) (3) (4)

Question 4. Next you will find a list of different behaviors related to your job. Indi- cate in each case the frequency in which they occur.

Activity Almost Sometimes Often Almost No Answer Never Always

1. Provide feedback about the devel- opment of the project.

250

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Activity Almost Sometimes Often Almost No Answer Never Always

2. Present training proposals for your colleagues to management

3. Receive systematic (destructive) complaints from your colleagues.

4. Unexpected changes (leaves) in the team occur.

5. Colleagues extend their workday to complete tasks assigned.

Activity Almost Sometimes Often Almost No Answer Never Always

1. Assign tasks according to the training and abilities of your colleagues.

2. Plan explanations according to the level of the audience.

3. Synthesize and organize ideas in explanations/ negotiations.

4. Colleagues express their satisfaction with regard to the coordina- tion of the team.

5. Colleagues display greater independence in the completion of tasks.

In case you ticked off “No answer” for any of the items, please explain.

Date / /

Exhibit 21.3. Questionnaire About Learning Transference (continued)

Question 5. Next you will find another list of different behaviors related to your job. Evaluate in each case how you perform these tasks.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

252 Case Studies of Implementation

How Was the Evaluation Carried Out?

Level 1. The participants completed the reaction questionnaire at the end of the last session of the course.

Level 2. The participants took the knowledge test at the beginning and at the end of the training event.The trainer applied the observa- tion guidelines to the role-playing activities that were conduced dur- ing the course.

Level 3. The participants and their supervisors completed the ques- tionnaire before the training event and again three months after the completion of the event.

What Were the Results of the Evaluation?

Level 1. See Table 21.3.

Level 2. See Figures 21.1 and 21.2.

Level 3. Only three of the supervisors responded to the question- naire that was sent to them three months after the completion of the training event, so the study is limited to the data provided by the par- ticipants. The numbering of the questions in the presentation of the results corresponds to the questionnaire displayed in Exhibit 21.3. See Figures 21.3, 21.4, and 21.5, and Tables 21.4 and 21.5.

What Are the Conclusions?

From the questionnaires, it is evident that the training event received a very positive reaction from the participants. With regard to the learning evaluation, the results are positive because:

• 100 percent of the participants assessed received a score of 17 or 18 out of a possible 18 on the test administered after the course.The results show that the participants had considerable knowledge of the subject before the course, as 60 percent of the original scores were over 15 points. As a result, the increase in the level of knowledge was not very pronounced.

• In the level 3 questionnaire, 73 percent of the participants dis- played an increase in knowledge level with respect to their initial self-evaluations.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Coaching and Counseling Course 253

However, because the information about skills learning is unavailable, we cannot consider the results to be representative of the overall effi- ciency of the training event.

In regard to the evaluation of behavior, the results indicated a not very high degree of applicability of the knowledge acquired during the course. Of all of the participants, only 27 percent said they had been able to apply much of what they learned.The rest said they had applied little or none of what they learned, and, of these, 75 percent said this is because the opportunity had not presented itself.

With regard to concrete behaviors observed at work, it is not pos- sible to reach any conclusions because:

Table 21.3

1. Organization of the Course Media

1.1. Following of the planned agenda 7.3

1.2. Duration of the course 8.2

1.3. Quality of the materials (manuals, overheads, etc.) 9.0

2. Usefulness of the Course

2.1. Quality of the exercises 8.7

2.2. Number of exercises 8.7

2.3. Applicability at work of the knowledge obtained 7.5

3. Content of the Course

3.1. The content of the course met with expectations 7.2

4. Instructors

4.1. Knowledge of the subject 9.8

4.2. Quality of the explanations 9.5

4.3. Friendliness, holding of interest 9.8

5. Services of the Center

5.1. Training rooms 9.2

5.2. Support equipment (overhead projector,VCR,TV, etc.) 9.7

6. Time

6.1. Overall evaluation of the course 8.8

6.2. The stated objectives of the course have been achieved 9.0

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

254 Case Studies of Implementation

• The lack of application distorts the results obtained. The impact the training may have had on these changes is not sig- nificant if a large proportion of the participants indicate that they have been able to apply little or none of what they learned, making it possible that other factors might have played a part in bringing about the apparent changes.

• In addition, the evaluations by the superiors, which would have served as an alternate source of information, were not available.

The results can be considered satisfactory with regard to the learning achieved by the participants. However, the desired application of this learning in the workplace has not come about. In this case, it is pos- sible that more time and concrete opportunities might facilitate the application of the acquired knowledge.

Score range: 0–18 points

18 18 0

14 18 4

13 18 5

15 18 3

10 18 8

16 17 1

17 18 1

14 18 4

16 18 2

15 17.9 2.9

Participants

Average

Before After Difference

Le arning of Skills

Figure 21.1. Knowledge Learning

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Coaching and Counseling Course 255

30% 30%

30%

10%

Results Before

<11

11-12

13-14

15-16

17-18

Figure 21.2a. Results Before

<11

11-12

13-14

15-16

17-18

100%

Results After

Figure 21.2b. Results After

What Are the Strategic Goals of the Entity?

The objectives are described from two perspectives:

Operative

• Rotation index • Meeting deadlines • New breakthroughs (R+D) • Commercial activity

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Question 1: Have you been able to put what you learned in the course into practice?

Nothing 2 18%

A few things 6 55%

A lot of things 3 27%

Almost everything/everything 0 0%

Figure 21.3.

Have you been able to put what you learned in the course into practice?

P e rc

e n ta

g e

0 Nothing A few

things A lot

of things Almost

everything/ everything

256

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

Evaluating a Coaching and Counseling Course 257

Reasons for the lack of applicability of the knowledge obtained

P e rc

e n ta

g e

0 Reason 1: Reason 2: Reason 3:

Figure 21.4.

Question 2:What are the reasons for the lack of applica- bility of the knowledge obtained?

Reason 1: Don’t have people 13% working under them

Reason 2: Haven’t had the opportunity 75%

Reason 3: Workload 13%

• Enlargement strategies

• Number of applicants • Number of offers

• Profits • Provision of technical training

Corporate

• Internal client’s satisfaction index • Meeting deadlines • Scope of training (number of hours of training per person) • Degree to which plan is successfully carried out

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Insufficient Sufficient

Knowledge about coaching and counseling

P e rc

e n ta

g e

0 Good Very good

Before (n = 11) After (n = 11)

Figure 21.5.

Question 3: Do you feel your knowledge about coaching and counseling is:

Before (n = 11) After (n = 11)

Insufficient 64% 0%

Sufficient 27% 64%

Good 9% 9%

Very Good 0% 27%

(n = 11) N %

Increased 8 73%

Did not increase 3 27%

Difference between knowledge before and after the training event:

258

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

Table 21.4.

Question 4: In each case what is the frequency with which the following things occur?

4.1 Provide feedback about the development of the project

4.2 Present training proposals for your colleagues to management

Before (n = 11) After (n = 11)

No answer 18% 0% Almost never 0% 0% Sometimes 18% 27% Often 27% 18% Almost always 36% 55%

Before (n = 11) After (n = 11)

No answer 18% 9% Almost never 18% 9% Sometimes 36% 64% Often 27% 9% Almost always 0% 9%

4.3 Receive systematic complaints from your colleagues

Before (n = 11) After (n = 11)

No answer 0% 9% Almost never 91% 64% Sometimes 9% 27% Often 0% 0% Almost always 0% 0%

4.4 Unexpected changes in the team occur

Before (n = 11) After (n = 11)

No answer 9% 9% Almost never 64% 55% Sometimes 27% 27% Often 0% 9% Almost always 0% 0%

4.5 Colleagues extend their workday to complete tasks assigned

Before (n = 11) After (n = 11)

No answer 0% 9% Almost never 18% 9% Sometimes 36% 64% Often 36% 9% Almost always 9% 9%

259

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

Table 21.5.

Question 5: In each case, how will you perform these tasks?

5.1 Assign tasks according to the training and abilities of your colleagues.

5.2 Plan explanations according to the audience.

Before (n = 11) After (n = 11)

No answer 9% 18% Poor 0% 0% Fair 0% 0% Good 64% 45% Very good 27% 36%

Before (n = 11) After (n = 11)

No answer 9% 0% Poor 0% 0% Fair 18% 9% Good 64% 64% Very good 9% 27%

5.3 Synthesize and order ideas in explanations/negotiations.

Before (n = 11) After (n = 11)

No answer 9% 0% Poor 0% 0% Fair 27% 9% Good 45% 45% Very good 18% 45%

5.4 Colleagues express their satisfaction with regard to the coordination of the team.

Before (n = 11) After (n = 11)

No answer 9% 27% Poor 0% 0% Fair 36% 27% Good 45% 36% Very good 9% 9%

5.5 Colleagues display greater independence in the completion of tasks.

Before (n = 11) After (n = 11)

No answer 0% 18% Poor 0% 0% Fair 9% 0% Good 64% 36% Very good 27% 45%

260

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

Evaluating a Coaching and Counseling Course 261

Factors Apart from Training that Influence the Results

• Personality • Personal surroundings • Work environment • Changes in geographic location • Performance of business • Change in the organization’s strategic orientation • Turnover of key personnel • Assignment of resources by the organization • Interaction with other departments or with providers or

clients • IT operating systems

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Chapter 22

Evaluating a Performance Learning Model

262

Instead of evaluating a specific program, DAU evaluates all its programs within an enterprise learning framework they call the Performance Learning Model which includes evaluating all of its training courses, continuous learning modules, and performance support efforts total- ing over 103 thousand graduates per year. Details of this evaluation include all four Kirkpatrick levels. The figures will be of particular interest.

Defense Acquisition University (DAU) Christopher R. Hardy, Ph.D., Strategic Planner

Mark Whiteside, Director Performance and Resource Management

Ft. Belvoir, Virginia

Who We Are

The Defense Acquisition University (DAU) is a government “corpo- rate” university for the Department of Defense, managed by the Office of the Undersecretary of Defense (Acquisition, Technology, and Logistics) (DoD USD [AT&L]).To accomplish its mission of pro- viding practitioner training and consulting services to over 134,000 Depart- ment of Defense employees across fifteen career fields, DAU provides

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

EBSCO : eBook Collection (EBSCOhost) - printed on 9/21/2019 4:59 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 260710 ; Kirkpatrick, Donald L., Kirkpatrick, James D..; Evaluating Training Programs : The Four Levels Account: s3654090.main.ehost

Evaluating a Performance Learning Model 263

a full range of certification training (required for all 134,000 to qual- ify for advancement), assignment-specific training, performance sup- port, continuous learning opportunities, and knowledge sharing.

DAU was established in 1992 and funded by Congress. DAU head- quarters are located at Ft. Belvoir, Virginia, where DAU maintains a headquarters staff for centralized academic oversight, a robust cur- riculum development center, and an e-learning and technology development directorate. In addition, DAU has strategically located five regional campuses in areas where there is a high concentration of DoD AT&L workforce members:

• West—San Diego, California (serves workforce of 26,000) • Midwest—Wright Patterson Air Force Base, Ohio (serves

workforce of 20,000) • South—Huntsville, Alabama (serves workforce of 27,000) • Mid-Atlantic—Patuxent River, Maryland (serves workforce

of 23,000) • Capital and Northeast—Fort Belvoir, Virginia (serves

workforce of 37,000). We also have the Defense Systems Management College-School of Program Managers at Ft. Belvoir for executive and international training.

What We Do

DAU’s products and services include more than training. To meet all its learning and development requirements, DAU created an over- arching learning strategy, the Performance Learning Model (PLM), which promotes career-long learning and achievement. The PLM includes four main thrusts:

• Certification and assignment-specific training through resi- dent training, hybrid learning courses, and distance learning courses

• Continuous learning through DAU’s continuous learning modules

• Performance support through consulting, rapid deployment training, and targeted training

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

264 Case Studies of Implementation

• Knowledge sharing through the AT&L Knowledge Sharing System (AKSS) and Acquisition Community Connection (Communities of Practice).

By developing and adopting this new learning strategy, DAU rapidly changed the traditional training paradigm of instruction limited to the classroom, to one that provides learning solutions twenty-four hours a day, seven days a week—the concept of anytime, anywhere learning. With implementation of the PLM, the 134,000 workforce members now have more control over their career-long learning opportunities. Before adopting the PLM learning model, DAU would train on aver- age 30,000 students per year. Now with an expanded learning plat- form and providing alternative delivery methods for our training, DAU graduates over 90,000 students per year from our course instruc- tion, we support over 200,000 registered users to our continuous learning modules, and we have over 300,000 queries to our learning assets through our communities of practice. Clearly DAU has dramati- cally increased its reach to its customers and broadened the depth and scope of its training assets in a fairly short time frame.

Evaluation and Feedback—Our Commitment

We used to spend an inordinate amount of time and effort collecting data that left little time for actual analysis. In the last few years, we com- pletely modernized our “back room” infrastructure and invested in busi- ness tools.This has helped to reverse the trend so that we now spend the majority of our time analyzing and acting on data. (See Figure 22.1.)

At DAU, we strive to provide our customers with an experience marked by quality products, relevant learning solutions, a responsive faculty and staff, and a continuous connection to the broader AT&L community.To ensure we meet their needs, we have established robust evaluation and feedback mechanisms for each of our products and services. Comments and results are not just put on the shelf. Each quarter the senior leaders conduct a thorough review of the results of the quarter and, where appropriate, make timely, effective enhance- ments to our learning solutions.To earn a “premier corporate univer- sity” evaluation is not a choice, it is an imperative.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Performance Learning Model 265

20% Analysis

80% Data collection

80% Analysis

20% Data

collection

Figure 22.1. Reversing the Trend

DAU evaluates all of its products and services. Evaluation plays a significant role as part of our quality reviews and as a barometer of our mission success. DAU employs several systems and the resulting data to link and assess the impact of learning on individual and organizational performance. More important, the results of its eval- uation program are used to improve faculty performance, delivery, facilities, content, curricula, relevance to the job, applicability, and impact on the organization. Evaluation data, comparatives, bench- marking, and trends are also a key element in DAU’s overall perfor- mance measurement program. DAU uses a holistic approach, leveraging its time accounting system, evaluation results, and cost data to afford its leadership the actionable intelligence with which it effectively manages its enterprise. By using an industry leader for its evaluation services, DAU is also able to access benchmarking data from many other learning and development organizations (over 100) throughout the world to better calibrate its performance. As Figure 22.2 shows, DAU, using a data mart, can quickly evaluate mission performance and trends, comparing measures of efficiency and effectiveness mined from various tools and legacy systems.

In theory and in actual practice, DAU adapted Dr. Don Kirk- patrick’s four-level model for evaluating its learning products and services. Taking advantage of the speed and reach of technology as well as the real-time data analyses it affords, we have implemented a

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Cost Center

1-3 Users 5-10 Users 40-60 Users 540 Users 132,000 Info only

Time accounting data

Budget data

Evaluation data

Finance data

Human resources

Course schedule

Region/Dept. DAU Enterprise

DAU Data Mart

AT & L Extranet

What-if analysis Data exploration E-intelligence

Descriptive statistics Graphical reporting Data entry applications

Cost data

Figure 22.2. DAU Data Mart

266

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed

un de r U. S. o r ap pl ic ab le c op yr ig ht l aw .

Evaluating a Performance Learning Model 267

DAWIA Certification

AT&L Knowledge Sharing System (AKSS) Acquisition Community Connection DAU Virtual Library

Knowledge Sharing Consulting Rapid Deployment Training (RDT) Targeted Training

Performance Support

Continuous Learning Modules Conferences and Symposiums

Continuous Learning

Assignment- specific

Executive & International

Training Courses

L E A R N I N G

E X C ELLE

NC E IN A CQ U S IT I O

D E

F E

N SE

AC QU

ISITION UNIV E

R S

IT Y

Figure 22.3. Performance Learning Model (PLM)

tool that is a best-in-class web-enabled evaluation system (Metrics that Matter), developed by Knowledge Advisors. This web-based learning evaluation capability allows DAU leadership to quickly gauge how effective learning is and cost-effectively measure training impact on the individual, supervisors, and organizational performance in real time.This system provides a robust real-time analytical capabil- ity that enables leaders to make better business decisions in regard to learning products and services.

AT&L Performance Learning Model (PLM)

We evaluate and receive feedback for each of our products and serv- ices. The centerpiece of the DAU experience is the AT&L Perfor- mance Learning Model, depicted in Figure 22.3.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

268 Case Studies of Implementation

This award-winning, best-practice, and overarching learning strat- egy provides each member of the AT&L workforce with more con- trol over his or her career-long learning opportunities. The PLM extends the learning experience from traditional classroom instruc- tion to a variety of learning solutions that are available anytime, anywhere. The PLM components include: Training through web- enabled, hybrid, and classroom courses that include case-based instruction aimed at developing critical thinking skills; Continuous learning with self-paced, relevant training modules, available 24/7, to help meet continuous learning requirements and improve job perfor- mance; Performance support with rapidly delivered business solutions offered to students and their organizations and supplementing the classroom experience through on-site consulting, targeted training, and rapid deployment training; and Knowledge sharing through the AT&L Knowledge Sharing System and the Acquisition Community Connection, to connect with experts, peers, and technical resources.

Training Courses: Certification, Assignment-Specific, and Executive and International

Our more than 1,200 course offerings per year are delivered in a classroom setting at one of our five regional campuses, or ten satellite locations, or via the internet. Courses are also taught at customer sites. At the end of each course, we survey students on several aspects of the course, including course content, course work, faculty, and job appli- cability. For in-resident courses, students take the survey online while they are still in the classroom. In our distance learning courses (more than 300 offerings per year), students must answer the survey before they can print the course completion certificate. Faculty members and course managers have full access to the results for their courses. The faculty, academic deans, and course managers review results and work to address shortfalls. Upon completion of the course and after the students are back at work for sixty days, for selected courses we e- mail students and their managers follow-up surveys. After six months, for selected courses we attempt to determine the impact of the train- ing on the organization.

Continuous Learning: Continuous Learning Modules

Our Continuous Learning Center (CLC) contains self-paced contin- uous learning modules that are available 24/7 to help meet continu-Co

py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Performance Learning Model 269

ous learning requirements and improve job performance. At the end of each DAU-developed Continuous Learning Module is a course feedback survey. Students must complete this survey before they receive their completion certificate. Course owners review the survey results for their course and incorporate improvements. Noncourse comments or site-specific technical questions abut the CLC are e- mailed to our help desk.

Performance Support: Consulting, Targeted Training, and Rapid Deployment Training

Our performance support services are provided to DoD and other government agencies to help them resolve individual project and agency-level acquisition problems. We also provide immediate train- ing on new policy initiatives. At the end of each consulting effort, the customer is asked to provide feedback. Following each targeted train- ing event, students respond to an online course survey similar to the one used for our certification and assignment-specific courses. DAU reviews the results for both consulting and targeted training efforts and incorporates improvements.

Knowledge Sharing: AT&L Knowledge Sharing System, Acquisition Community Connection, and Virtual Library

DAU hosts the AT&L Knowledge Sharing System (AKSS) and the Acquisition Community Connection (ACC). AKSS is the central gateway for all AT&L resources and information. ACC is the collabo- rative arm of AKSS, with a variety of knowledge communities (of practice). Users of our knowledge-sharing systems are asked to pro- vide feedback through online comment areas. On the AKSS home page, they can just click the “We want your feedback—again” button. Users of the ACC can provide feedback through the “Contact us” link. DAU reviews comments weekly and incorporates improvements and suggestions as appropriate.

DAU Evaluation Survey Instruments

We adapted templates from Knowledge Advisors for Level 1, 3 and 4 and have their permission to share examples of our template.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

270 Case Studies of Implementation

Level 1—Reaction

End-of-class surveys are required from all our students in order to graduate from our courses, whether they be class, hybrid, or online. These surveys focus questions in specific subject areas, such as faculty performance in the classroom, course content/material, learning effectiveness, business results, and environment. Each focus area has multiple questions to provide for further drill down on the data, thus allowing DAU to make more informed decisions for its survey results. When DAU managers review the data from our class survey, they can do so in many different slices or dimensions. For example, DAU man- agement can drill down into course results by region or across all regions. Survey results can be distilled down by question category or down to specific question responses. Our survey results are accessible within twenty-four hours of completion of the class by the faculty who taught the class, course managers who are responsible for that specific course, center directors for curriculum development to assess the course content results, and our deans, who are responsible for the overall delivery of all of our classes. Paper copy examples of our level 1 survey instruments are displayed in Exhibit 22.1 for both class and web-based courses.

Level 2—Learning

For all our certifications courses, we require content testing—both precourse diagnostics and end-of-course assessments. Precourse diag- nostics are used to tailor delivery to meet the learning needs of the students and to help determine the extent of learning when com- pared to the end-of-course tests. Students must also achieve an 80 percent score on an end-of-course test to graduate. In addition, in senior-level courses that include case-based scenarios, individual stu- dent performance is evaluated by faculty with one-on-one feedback provided. Test scores are also analyzed and tracked for curricula and faculty development purposes. For low scores, we work with students to provide remediation as necessary.

Level 3—Behavior

To determine if a change in behavior has occurred attributable to the training program or learning activity, we also survey students (and

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Performance Learning Model 271

Please help us improve the training program by responding to our brief survey below.

1. What is your primary Career Field? (Please check only one)

2. How many years of experience do you have in your career field?

3. What level of DAWIA certification did you have in your primary career field prior to attending this course?

4. Are you:

5. Please identify from one of the three sections below your current Military, GS or ACQ DEMO position grade level:

Active duty military personnel:

GS Personnel:

Auditing

Facilities Eng.

Life Cycle Logistics

Business, CE & Fin Mgmt

Industrial/Contract Property Mgmt

Production, QA & Man.

Contracting

Information Technology

Program Management

Purchasing

Systems Engineering

SPRDE & S&T

Test & Evaluation

Don't Know or NA

Less than 1

10 to 15

RESPONDENT INFORMATION INSTRUCTOR (Continued)

INSTRUCTOR

Don't know

Military (Active)

E 1-5

GS 1-5

Acq Demo Personnel:

6. The instructor was knowledgeable about the subject. 7. The instructor was prepared and organized for the class. 8. Participants were encouraged to take part in class discussions.

Broadband: I Broadband: II Broadband: IV SES

Broadband: III

GS 6-8 GS 14-15 SES

GS 9-11 GS 12-13

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

7 6 5 4 3 2 1 n/a

9. The instructor was responsive to participants' needs and questions. 10. The instructor's energy and enthusiasm kept the participants actively engaged. 11. On-the-job application of each objective was discussed during the course.

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

7 6 5 4 3 2 1 n/a

ENVIRONMENT

12. The physical environment was conducive to learning.

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

COURSEWARE

13. The scope of the material was appropriate to meet my needs. 14. The material was organized logically. 15. The examples presented helped me to understand the content. 16. The participant materials (manual, presentation handouts, etc.) will be useful on the job. 17. The case studies and exercises added value to my learning.

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

LEARNING EFFECTIVENESS

18. I learned new knowledge and skills from this training. 19. The guest speakers (from outside DAU) were effective in contributing to my learning in this course. 20. Rate your INCREASE in skill level or knowledge of this content before versus after the training. A 0% is no increase and a 100% is very significant increase.

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

JOB IMPACT

21. I will be able to apply the knowledge and skills learned in this class to my job. 22. What percent of your total work time requires the knowledge and skills presented in this training? Check only one.

23. On a scale of 0% (not at all) to 100% (extremely critical), how critical is applying the content of this training to your job success? Check only one.

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

7 6 5 4 3 2 1 n/a

E 6-7 E 8-9 O 1-3 O 4-5 06 GO

Civilian Government Industry

None Level I Level II Level III

1 to 2

More than 15

2 to 5 5 to 10

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Exhibit 22.1. DAU Classroom Training—End-of-Class Survey*

*This survey is adapted from Knowledge Advisors.

their managers) when they are back on the job, usually three months after course completion.

Level 4—Results

Over the years, we have found that it is difficult to attribute an orga- nization’s success to just our training products and services. Other sig- nificant factors also contribute and might even be the key determinant, such as reorganization, recruiting, visionary leadership,

(continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

272 Case Studies of Implementation

Hawthorne effect, and so on. Survey questions remain fairly subjective even though quantified as value judgments. ROI data is equally suspect regardless of algorithms and clever manipulations. However, we are interested in ROI-type data, but it is not the sole determinant at DAU.

Each quarter during the Enterprise Performance Review and Analysis (EPRA), DAU reviews the quarter’s performance and progress for all performance targets and selected operational metrics. Our evaluation survey data is an important part of this review. An example of the type of report we use for our reviews is shown in Fig- ure 22.4.

JOB IMPACT (Continued)

24. What percent of new knowledge and skills learned from this training do you estimate you will directly apply to your job? Check only one.

BUSINESS RESULTS

25. This training will improve my job performance. 26. Given all factors, including this training, estimate how much your job performance related to the course subject matter will improve.

27. Based on your response to the prior question, estimate how much of the improvement will be a direct result of this training. (For example if you feel that half of your improvement is a direct result of the training, enter 50% here.)

28. This training will have a significant impact on: (check all that apply)

Yes No

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

RETURN ON INVESTMENT

29. This training was a worthwhile investment in my career development. 30. This training was a worthwhile investment for my employer.

What about this class was most useful to you?

What about this class was least useful to you?

How can we improve the training to make it more relevant to your job?

I would recommend that my colleagues take this course.

What is your overall opinion of this course?

Strongly Agree Strongly Disagree

6 5 4 3 2 1 n/a

7 6 5 4 3 2 1 n/a

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

0% 10%

increasing quality decreasing costs decreasing cycle time

increasing productivity increasing sales decreasing risk

increasing employee satisfaction increasing customer satisfaction

20% 30% 40% 50% 60% 70% 80% 90% 100%

Exhibit 22.1. DAU Classroom Training—End-of-Class Survey* (continued)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Performance Learning Model 273

Figure 22.4. DAU “Spider” Diagram Display

Value-Added Contributions

It is vital to show substantial evidence of “value-added contributions” to our stakeholders. We compile a compelling rationale using evi- dence from many sources to portray results.Though our survey data is important, summary data from many different sources is more impor- tant as we measure the total contribution across the enterprise.To link budget allocation with demonstrated results, DAU uses a formal enterprise performance review and analysis run by a resource council to review program performance based on past performance, observ- able results, and operational efficiencies. At the end of the year, an enterprise-wide program review is conducted during the last quarter’s EPRA, and management decisions are made in regard to past perfor- mance and in regard to next year’s annual performance plan and pro- gram funding and continuance. Improvements brought about by the management processes described in this chapter yield the timely and reliable financial management data necessary to achieve the remark- able results DAU has achieved. As a result, management decisions are

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

274 Case Studies of Implementation

made in the context of past trend data, and financial/personnel resources are allocated as appropriate.

An example of management’s response to trend data results is in the area of course content. If survey results consistently over time show that a specific course’s content is rated low in the area of cur- rency or value of the materials on the job, a team of curriculum developers, teaching faculty, and course managers meet to reconfigure the course content in time for the next scheduled class data. Some specifics that are addressed include tracking and reporting macro models, comparative analyses and benchmarking by other organiza- tions, and internally comparing courses, regions, faculty, delivery methods, and environments.

Building Compelling Evidence of Results for Our Stakeholders

By comparing trends in key areas such as evaluation scores, bench- marking metrics, growth in student throughput, student travel costs, and other comparatives, we look for the total enterprise’s multiyear results and trends.We have been benchmarked by over eighty organi- zations, and through our survey application service provider (as a part of their services), we also use their survey database for comparative purposes with other learning organizations. Able to successfully meet its challenge of serving significantly increasing numbers of students with no increase in budget, DAU has improved in many areas.

Since 1998 (with a relatively flat learning and development budget investment per year) we have increased our students trained from 33,000 to 92,000 per year while at the same time reducing the faulty/staff from 643 to 540 and our student travel costs from $31,000 to $17,000 per year. Over this time, the average training cost per stu- dent has declined by 42 percent—a reduction of $1,300 per student. This translates to a projected savings in the next five years of over $50 million and a faculty savings of $10 million, allowing us to reprioritize resources into e-learning initiatives, curricula modernization, and other greater reach initiatives. Even as our cost per student has been dramatically reduced, the courses have continued to receive high marks from students and supervisors in response to survey questions. This along with our sector leadership recognition as a best-in-class

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating a Performance Learning Model 275

corporate university provides our stakeholders with compelling evi- dence that DAU makes a considerable value-added contribution to the overall mission of the organization. Our sector leadership recogni- tion can be summarized:

2002 Corporate University Best-in-Class (CUBIC) Awards

• Leader of the Year • Best Overall Corporate University • Best Virtual Corporate University / Best Use of Technology • Runner-up Most Innovative Corporate University

2003—Gold Medal Winner of Brandon Hall Best Practices for e-Learning

2003 and 2004—Winner of the American Society of Training and Development BEST Award (2004 First place among eighty-three organizations worldwide)

2003 Winner of Corporate University Xchange Excellence Award for Measurement

2003—Selected in Training Magazine Top 100 2004—Selected by CLO Magazine for Two Best Practices 2004—Chief Learning Officer of the Year

Reference

Knowledge Advisors (Metrics That Matter). 222 S. Riverside, Suite 1700, Chicago, IL 60606. 312/423-8750. kbarnett@knowledgeadvisors .com.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Chapter 23

Evaluating an Information Technology Skills Training

Program

276

This evaluation will be of special interest to readers who are looking for concepts, principles, techniques, and forms that they can adapt to similar programs.The case study measures the effectiveness of the pro- gram by evaluating at level 1 (reaction), level 2 (learning), and level 3 (behavior).The Questionmark Corporation provided software to assist in the evaluation. Be sure to read the “Summary” and “Recommenda- tions” for ideas you may find useful.

The Regence Group James C. Baker, e-Learning Specialist, Organizational

Development Portland, Oregon

The Regence Group is the largest affiliation of health-care plans in the Pacific Northwest/Mountain State region. It includes Regence BlueShield of Idaho, Regence BlueCross BlueShield of Oregon, Regence BlueCross BlueShield of Utah, and Regence BlueShield (in Washington). Collectively, these four plans serve nearly three million people in four states, with more than $6.5 billion in combined rev- enue as of January 2004.

In 2003 Organizational Development (OD) at Regence gathered training evaluation data of an information technology skills training program for Regence Information Technology Services (RITS) using

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

EBSCO : eBook Collection (EBSCOhost) - printed on 9/21/2019 5:07 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 260710 ; Kirkpatrick, Donald L., Kirkpatrick, James D..; Evaluating Training Programs : The Four Levels Account: s3654090.main.ehost

Evaluating an Information Technology Skills Training Program 277

our online assessment system. RITS participated in a workforce and capacity-building training program administered by the Oregon Department of Community Colleges & Workforce Department (CCWD). The CCWD requested that both the training participants and their supervisors assess the training initially and in follow-up eval- uations with an evaluation tool developed by the American Society for Training & Development (ASTD) and based on Kirkpatrick’s four-level model. The responses about training, learning, and perfor- mance could be compared with other benchmark measures at Regence and at other organizations.

The Regence online training evaluation system employs the Ques- tionmark Perception authoring and web deployment software suite to manage a broad range of classroom and online learning activities, both for individuals or teams. Using the Perception browser-based assess- ment forms and SQL Server database system, OD conducted online training evaluations of seventeen classes, which instructors delivered in a classroom setting.The training evaluations spanned eight months and produced a total of 490 online assessments.

Regence Online Evaluation Method

In OD at Regence, several assessment authors use the Questionmark Perception software suite to build, deliver, and report training evalua- tions for internal clients such as RITS. We follow this engagement process with our internal clients:

1. Collaborate on a time frame with mutual roles and responsi- bilities;

2. Develop and import the question wording, choices, out- comes, and scoring into the authoring tools;

3. Select a template for page layout (images, instructions, questions, submit button, and optional jump-block ques- tions);

4. Test the evaluation in the development environment with the client;

5. Gain client approval and move the evaluation into our pro- duction environment for distribution;

6. Create an assessment schedule for participants;

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

278 Case Studies of Implementation

7. Turn on or off the settings for log-on (anonymous or user- name and password), limit of one try, and time limit;

8. Notify the participants about the schedule with a link to their online evaluation; and,

9. At the conclusion of the session, pull the respondent data out of the database management system for analysis and report- ing by participant, class, and instructor.

Online Evaluation of Levels 1, 2, and 3

Following our engagement process, OD and RITS Professional Development established a time line for the information technology skills classes with both immediate and follow-up evaluations. For the training assessments of levels 1 (reaction), 2 (learning), and 3 (behav- ior), we adopted the ASTD evaluation tool to benchmarking training evaluation data. Supplemental questions produced records for report- ing evaluation data by participant, class, and instructor for initial and follow-up training evaluations.

Immediately after each class, participants launched our Part A online assessment of level 1 and level 2. In Exhibit 23.1 of this case study, you can see the multilevel Part A evaluation tool that we adopted from ASTD and then supplemented with other questions for reporting. Part A questions consisted of a 1–5 scale to measure reac- tions to statements about these categories:

• administration and logistics (prerequisites, facilities and equip- ment);

• content (understood the objectives, the objectives were met); • design (method of delivery, materials, length of class time,

organization); • instruction (satisfaction with instructor); • perceived impact (knowledge and skills increased; applicability

to current job; applicability for preparing participant for other jobs in the company; training helped toward other jobs in the company); and,

• overall satisfaction with the class.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Information Technology Skills Training Program 279

Exhibit 23.1. Part A Online Assessment Levels 1 and 2

INSTRUCTIONS:When you have completed this evaluation, click Submit.

Class name and course objectives

Your name:

Instructor name:

Questions

Choices: a. Strongly Agree b. Agree c. Neither d. Disagree e. Strongly Disagree

1. I had the knowledge and/or skills required to start this course.

2. The facilities and equipment were favorable to learning.

3. I was able to take this course when I needed it.

4. I clearly understood the course objectives.

5. The course met all of its stated objectives.

6. The way this course was delivered (such as classroom, computer, and video) was an effective way for me to learn this subject matter.

7. Participant materials (handouts, workbooks, etc.) were useful during the course.

8. I had enough time to learn the subject matter covered in the course.

9. The course content was logically organized.

10. I had an opportunity to give input to the course design or content.

11. Overall, I was satisfied with the instructor(s).

12. My knowledge and/or skills increased as a result of this course.

13. The knowledge and/or skills gained through this course are directly applicable to my job.

14. This course has helped prepare me for other job opportunities within the com- pany or industry.

15. Overall, I was satisfied with this course.

Several weeks after each class we distributed the Part B assessment (see Exhibit 23.2). In our online Part B assessments, we asked partici- pants to provide their names and then to answer questions about their:

• use of skills from training (opportunity to use the training, actual use of the training);

• confidence in ability to perform (extent of increase in confi- dence resulting from this training);

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

280 Case Studies of Implementation

Exhibit 23.2. Part B Levels 1, 2, and 3 Online Assessment

INSTRUCTIONS:When you have completed this evaluation, click Submit.

Course name and objectives

Your name:

Questions

Choices: a.To a very great extent b.To a great extent c.To a moderate extent d.To a small extent e. Not at all/never/rarely applicable f. Not applicable

1. To what extent did you use the knowledge and/or skills prior to attending this course?

2. To what extent have you had the opportunity to use the knowledge and/or skills presented in this course?

3. To what extent have you actually used the knowledge and/or skills presented in this course, after completing the course?

4. To what extent has your confidence in using the knowledge and/or skills increased as a result of this course?

5. To what extent did you receive the assistance necessary in preparing you for this course?

6. To what extent has the content of this course accurately reflected what happens on the job?

7. To what extent have you had access to the necessary resources (e.g., equipment and information) to apply the knowledge and/or skills on your job?

8. To what extent have you received help, through coaching and/or feedback, with applying the knowledge and/or skills on the job?

9. As a result of this course, my performance on the course objectives has changed by (%).

10. As a result of this course, my overall job performance has changed by (%).

• barriers to and enablers of transfer (training accurately reflected the job, access to necessary resources to apply the training, extent of coaching and other assistance); and,

• measures of impact (percentage changes in production and performance).

Besides the online Part B follow-up data from participants, RITS gathered follow-up training evaluations from their supervisors as part of the ASTD evaluation tool.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Information Technology Skills Training Program 281

Question Choices with Scores Descriptive Statistics

Strongly Agree (5 points) Highest: 4.83

Agree (4 points) Lowest: 4.37

Neither (3 points) Mean: 4.77

Disagree (2 points) Mode: 4.83

Strongly Disagree (1 point)

Table 23.1. Part A Question Choices, Descriptive Statis- tics, and Significance

Training Evaluation Findings

Part A Findings

What are the results of the training evaluation in Part A? Scores indi- cate a consensus about the high ratings (see Table 23.1). The highest average (4.83) equals the most frequent value or mode (4.83). The mean or average (4.77) is only .05 points away from both.The results reflect very positive satisfaction on Kirkpatrick’s levels 1 and 2.

The Part A training evaluation consisted of fifteen questions (see Table 23.2). One hundred thirty-nine (139) of one hundred fifty-one (151) online assessments were returned for a response rate of 92.10 percent.This return rate was above our expected two–thirds response rate because RITS Professional Development monitored participants and encouraged completion.

The consensus about the multilevel evaluation questions in Part A forms a nearly straight-line graphic (see Figure 23.1), with a relatively narrow range of 4.83 to 4.63 on a five-point scale.The lowest average (4.37) was for question 3: “I was able to take the course when I needed it” is an exception.The training schedule for these participants did not provide them as much opportunity for personal choice as they wanted.

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

282 Case Studies of Implementation

Question Wording Average Ranking

1. I had the knowledge and/or skills required to 4.80 start this course.

2. The facilities and equipment were favorable to 4.80 learning.

3. I was able to take this course when I needed it. 4.37 Low

4. I clearly understood the course objectives. 4.63

5. The course met all of its stated objectives. 4.83

6. The way this course was delivered (such as 4.83 classroom, computer, and video) was an effective way for me to learn this subject matter.

7. Participant materials (handouts, workbooks, etc.) 4.83 were useful during the course.

8. I had enough time to learn the subject matter 4.77 covered in the course.

9. The course content was logically organized. 4.83

10. I had an opportunity to give input to the course 4.80 design or content.

11. Overall, I was satisfied with the instructor(s). 4.83

12. My knowledge and/or skills increased as a result 4.77 of this course.

13. The knowledge and/or skills gained through this 4.83 course are directly applicable to my job.

14. This course has helped prepare me for other job 4.83 opportunities within the company or industry.

15. Overall, I was satisfied with this course. 4.80

Table 23.2. Part A Question Score and Ranking

Part B Findings

The question items in Part B have different wording, choices, out- comes, and scoring than question items in Part A. For example, Part A contains five-scale scoring whereas Part B scoring uses six-scale scor- ing. Any attempts at comparative analysis of the two parts must over- come the differences in question items in Part A immediately after the training and in Part B several weeks or months later.The Part B train-

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Information Technology Skills Training Program 283 S

ca le

1 1

Part A Rating by Question

2 3 4 5 6 7 8

Question number

9 10 11 12 13 14 15

4.80 4.80 4.37

4.63 4.83 4.83 4.83 4.77 4.83 4.80 4.83 4.77 4.83 4.83 4.80

Figure 23.1. Part A Average Scores by Line Graph

ing evaluation consisted of questions: questions 1–8 and then questions 9–10 to measure performance impact (see Exhibit 23.2).

Questions 1–8. Regarding Kirkpatrick’s levels 1, 2, and 3, is there a positive consensus in the Part B 1–8 results? Maintaining a pattern from Part A, the participants show agreement and high ratings in the follow-up training evaluation (see Table 23.3).

Of the 336 online assessments, 333 were returned for a response rate of 98.10 percent for Part B questions 1–8.The rate is higher than the Part A response rate of 92.10 percent because of increased com- munication with participants and their supervisors by RITS Profes- sional Development.

The mean (5.68) and mode (5.68) are high on a six-scale and are the same value (see Table 23.4), which is only .04 points from the highest average (5.72).The results in the first eight questions in Part B reflect very positive satisfaction on Kirkpatrick’s levels 1, 2, and 3.

Table 23.3. Part B Questions 1–8: Question Choices, Descriptive Statistics, and Significance

Question Choices with Scores Descriptive Statistics

To a very great extent (5 points) Highest: 5.72

To a great extent (4 points) Lowest: 5.63

To a moderate extent (3 points) Mean: 5.68

To a small extent (2 points) Mode: 5.68

Not at all/never/rarely applicable (1 point)

Not applicable (0 points)

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

284 Case Studies of Implementation

Table 23.4. Part B Questions 1–8: Scores and Ranking

Question Wording Average Score Ranking

1. To what extent did you use the knowledge 5.68 and/or skills prior to attending this course?

2. To what extent have you had the opportunity 5.68 to use the knowledge and/or skills presented in this course?

3. To what extent have you actually used the 5.69 knowledge and/or skills presented in this course, after completing the course?

4. To what extent has your confidence in using 5.72 High the knowledge and/or skills increased as a result of this course?

5. To what extent did you receive the assistance 5.70 necessary in preparing you for this course?

6. To what extent has the content of this course 5.63 Low accurately reflected what happens on the job?

7. To what extent have you had access to the 5.65 necessary resources (e.g., equipment and information) to apply the knowledge and/or skills on your job?

8. To what extent have you received help, 5.67 through coaching and/or feedback, with applying the knowledge and/or skills on the job?

The graphical representation of this data forms a fairly straight line, one where the range is from 5.72 to 5.65 on a scale of 6. There is considerable agreement and a positive evaluation shown from the data questions 1–8 in Part B (see Figure 23.2).

Questions 9–10. Following questions 1–8 in Part B there are two questions on job performance (Table 23.5). For questions 9–10, 319 of 333 online assessments were returned for a response rate of 95.80 percent, another high response rate maintained by the good commu- nications from RITS Professional Development with particpants and their supervisors.

What are the performance impact measures? For question 9: “As a result of this course, my performance on the course objectives has changed by 54%.” For question 10: “As a result of this course, my overall job performance has changed by 24%.”

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

Evaluating an Information Technology Skills Training Program 285 S

ca le

1 1

Part B Ranking by Question

2 3 4 5 6 7 8

5.68 5.68 5.69 5.72 5.70 5.63 5.65 5.67

Question number

Figure 23.2. Average Scores by Line Graph

Table 23.5. Part B Questions 9–10: Scores and Ranking

9. As a result of this course, my performance on the course 54% objectives has changed by (%).

10. As a result of this course, my overall job performance has 24% changed by (%).

The findings allow us to conclude from the online training evalua- tions that those participants in Parts A and B and their supervisors in Part B both evaluated the training for RITS very positively. Using Kirkpatrick’s model, there is significant consensus that the training was satisfying (level 1), effective (level 2), and applicable (level 3) in improving the performance of the participants.

Summary

This case study employed an ASTD training evaluation tool and online assessment methods in assessing an information technology training program.These tools and methods combined to achieve sev- eral goals:

1. Standardized question items from ASTD, based on Kirk- patrick’s model, provided standardized data types for bench- mark comparisons by instructors, training managers, and budget analysts;

Co py ri gh t @ 20 06 . Be rr et t- Ko eh le r Pu bl is he rs .

17EvaluatingALeadersh_EvaluatingTrainingPro (1)
18EvaluatingDesktopAp_EvaluatingTrainingPro (1)
19EvaluatingAnOrienta_EvaluatingTrainingPro (1)
20EvaluatingTrainingF_EvaluatingTrainingPro (1)
21EvaluatingACoaching_EvaluatingTrainingPro (1)
22EvaluatingAPerforma_EvaluatingTrainingPro (1)
23EvaluatingAnInforma_EvaluatingTrainingPro (1)