article

February 2007 | T+D | 49Photos by Getty Images

When I ask workplace learning and performance (WLP) professionals about the levels of evaluation, they respond in near unison: “Level 1 is reaction, 2 is knowledge; 3 is behavior in the workplace; and Level 4 is results.” Regardless of the location, workshop, or conference, people involved with workforce learning can describe Donald Kirkpatrick’s work on evaluation. >>

By Allison Rossett

LEVELING THE LEVELS As technology transforms learning, it also can advance how, when, and where evaluations happen.

>>But just because they know about the levels, doesn’t mean they actu- ally measure them. A recent ASTD benchmarking forum study looked at course evaluations by level. While 94 percent of courses are examined for Level 1 and 34 percent are checked for Level 2, just 13 percent of courses are examined for Level 3, transfer of behavior. Only about 3 percent of courses reach Level 4, which mea- sures influence in the field.

50 | T+D | February 2007

It isn’t only ASTD that has noted the chasm between the many who articu- late the levels and the few who practice them. Decades ago, Kirkpatrick himself pointed to a lack of action. In 1968, he and Ralph Catalanello measured how much evaluation occurred in 110 busi- nesses. Most respondents (78 percent) reported that they actively question reactions to the learning experience; far less reported making progress in Levels 2, 3, and 4. But instead of lamenting about learn- ing and its influence on practice and outcomes, let’s look instead at what is emerging today, and how changes in

learning grant an opportunity to rethink the ways we plan and measure.

The new world of learning and performance It is a new world for learning, and stud- ies document it. Technology is playing a growing role in training and develop- ment. In 2000, almost 9 percent of cours- es were delivered through technology. By 2002, more than 15 percent were reported to be using technology for delivery. And by 2004, that number increased to 28 per- cent. While the Internet steadily advances how we gain knowledge and support, it also reduces the time participants spend with instructors in classrooms. ASTD’s 2005 State of the Industry Report (SOIR) described what BEST award-winning organizations are doing: “BEST organizations delivered 32 per- cent of all their learning content using technology. Approximately 75 percent of technology-based learning was online in 2004, and about 75 percent of online learning was self-paced.” This is where it gets interesting. It’s not the proliferation of technology that intrigues as much as what it enables.

Technology is changing the how, where, and when of learning. In the past, the word training con- jured up visions of a room with students and an instructor. Variety came from the number of students and the configura- tion of the seating. What is the fresh face of training? Here are several examples. In Hamburg Germany, three corpo- rate executives have turned to a Boston-based coach to help them make a presentation in English at a major international event. Before they meet online to practice their delivery, they listen to podcasts of similar speeches, review the coach’s comments, and then

create drafts that are shared via email with each other and their coach. After the drafts are polished, the executives participate in a synchronous online meeting with their coach. Each execu- tive delivers a speech and benefits from the immediate feedback from the group. Certified public accountants are taking an online class about looming changes in tax regulation. They com- plete an e-learning module, refer to a searchable knowledge base, and chat online with each other and a coach who helps them apply the new regulations to clients’ tax problems. Another example is the United States Coast Guard. Boarding officers now learn to inspect ships and file reports by rely- ing on a performance support tool on their personal digital assistants (PDAs). The PDAs go to sea with them and help them conduct better inspections and create more accurate, timely reports. With sales representatives all over the world and experts in short supply, how do global pharmaceutical organiza- tions roll out new products? Many now provide online briefings, sales scenarios, and databases with explanatory materi-

It’s not the proliferation of technology that intrigues as much as what it enables. Technology is changing the how, where, and when of learning.

February 2007 | T+D | 51

als for representatives and custom- ers. They also schedule synchronous meetings so salespeople can discuss the product and disease state with an expert, and they provide materials for sales supervisors to use in face-to-face conversations with their representa- tives. Learning efforts and use of these tools are tracked in the corporate learning management system (LMS). In another example, new supervi- sors at IBM turn to Basic Blue for their development. Basic Blue is a blended learning system, based on online self- assessments, vivid e-learning scenari- os, an online community and mentor, and workshops. These new supervi- sors gain knowledge over time through structured learning and coaching experiences that are embedded in the challenges faced on the job. Technology-enabled learning is making its way into the unlikeliest of locations, even the dugouts of Amer- ica’s national pastime. Sitting on the bench between innings at a baseball game, some pitchers review video podcasts for the batters due up in the next inning. If they are better able to tailor pitches to these hitters, it will be, in part, because their lessons are in closer proximity to the moment when they hurl the ball toward home plate. Timing is another distinctive trait of technology-based learning and support. For example, pharmaceutical sales representatives use their laptops in the waiting rooms to bone up on new products and review objections just before they meet with doctors. Mili- tary veterans, also eager to get the most from their time with their physicians, now look to a website (www.myhealth.va.gov) to prepare for appointments. The site clues them in on what questions to ask, new studies or medications related to their problems, necessary tests, and their medical histories. While these veterans still make and keep scheduled appointments with their caregivers, their health is managed informally through

the site’s calendars that record effort and progress. It also offers presentations about disease states, and opportunities to chat online, look up information, and communicate with experts.

New world measurement New forms of inquiry are required for development that will soon be characterized by on-demand growth, self-study, performance support, on- line communities, e-coaching, blogs, wikis, knowledge bases, e-learning modules, and blended learning. These are evolving forms. They make new demands on employees, supervisors, and executives. Here are five ways that these budding efforts will affect the workplace: Erase the dichotomy between learning and work. Training, histori- cally, has worked like a switch. Employ- ees were either in class or at work. They were not in both places at the same time. Kirkpatrick’s approach matched that historical reality. Levels 1 and 2 were conducted when employees were in class. They asked if participants enjoyed the class and learned the necessary information. Levels 3 and 4 measured transfer of learning to the workplace. They asked questions such as “Did employees do it on the job?” and “Did doing it matter?” Ruhe Hao, quality and productivity engineer executive for Bank of America,

Technology- enabled learning is making its way into the unlikeliest of locations, even the dugouts of America’s national pastime.

Not satisfied with your

sales?

Let KLA

Group help drive

your results

www.klagroup.com ¢ 303.741.6636

KLA Group works with all departments that interact with customers to speed sales and

improve customer interactions. For over 10 years we have

driven notable sales increases with our clients that range from small entrepreneurial companies to Fortune 500

enterprises worldwide.

How can we help you?

lead generation ¢ prospecting ¢ new business development

52 | T+D | February 2007

makes this point, “Kirkpatrick’s four levels provided us a taxonomy of evalu- ation and measurement that empha- sized learning transfer. When learn- ers move to center stage, this transfer becomes less visible because learning is on the job, for the job, and very often, it is the job.” As technology closes the chasm between learning and work, evaluation linked to defined time and place loses resonance. The levels are not levels or stages, and they do not necessarily hap- pen in a fixed order. Questions are bet- ter asked continuously and idiosyncrati- cally, as learning, support, and informa- tion are delivered when and where they are needed. Integrate measurement with learn- ing, support, and work. Methods for judging worth and planning subsequent efforts belong when and where the ac- tion is. If e-coaching comes to a physi- cian on her PDA or cell phone, then communication about it should occur on that device, immediately or soon after the advice is delivered. The same is true for the Coast Guard boarding officers mentioned earlier. Af- ter the inspection, officers use software on their PDAs to create and file reports. Questions about value and improvement also should be there, baked into the as- sets, not an add-on for which permission to administer must be secured. Diana Wright, senior instructional designer at Welkin Associates, says, “What use is data that takes weeks to gather when the world is web enabled? The [Internet generation] won’t stand still that long!” Go native to the process. In ASTD’s 2005 SOIR, nearly all organizations reported that they carried out Level 1 evaluations. You can imagine why that number is so high: The forms are there, on the table or near the door. All class participants are asked to fill out an eval- uation form before exiting the event. There is no similar anticipation for field-based evaluation. In a 2006 study of the levels of evaluation for the eL- earning Guild, Director of Research Joe Pulichino acknowledged the burden of gaining access to the field. His respond- ing professionals said they lacked the time and right of entry to examine the

ultimate influence of their products and services. Chalk it up to the gulf be- tween them and us, with learning in its place, and work in another. As our world changes, and as learn- ing and support mix in the workplace, inquiry can become native to the pro- cess. A select few questions can pop up on the screen, in the midst of an e-learning module, or an email can be triggered after access to the knowledge base or engagement with an e-coach. A woman at a financial institution got it immediately. “A few questions are bolted on to the process,” she says. “That’s the way it works at Wal-Mart. As I finish my purchases and the credit or debit process is over, a few questions about cleanliness and service pop up. It takes nothing to respond, so I do.” Western Michigan University Profes- sor Robert Brinkerhoff’s successful case approach points to the value of a deep, narrow inquiry that examines a few employees, their experiences, and their stories about their growth and results. On the other hand, the learning LMS can enable a broader reach because of its ability to track engagement, action, contribution, knowledge, choices, satis- faction, progress, and next steps. Appreciate appreciation. WLP pro- fessionals routinely dismiss Level 1 as “just questions about reactions.” Oddly, the most frequent form of evaluation is unpopular. If they are to benefit from workplace learning and support, however, employ- ees must see the value in the assets that surround them. Appreciation is a per- quisite to voluntary participation. Work- ers must choose to look something up, chat with their online community, listen to a coach, use a performance support tool, work a case with a supervisor, or engage with an online learning module. If they doubt the benefits of those tools, do not share the priority, question the message, or find the system clunky— they won’t like it or use it. Commit to doing it better. Jeffrey Berk, director of products and services for KnowledgeAdvisors, asked practi- tioners why they measure. This is how he summed up their responses: “Or- ganizations provide the results of their

WLP professionals routinely dismiss Level 1 as “just questions about reactions.” Oddly, the most frequent form of evaluation is unpopular.

February 2007 | T+D | 53

learning analytics for many reasons. The most popular reason is to showcase the training’s value to the organization. An- other common reason is to indicate the quality of the training services provided. Additional reasons are because stake- holders request it or need to justify large expenditures.” Documenting contributions is but one motive for measuring. There are other pressing reasons, such as using sys- tem performance to improve the assets and program, providing feedback about results, usage, progress, and next steps, and planning subsequent initiatives. Suppose you are the learning man- ager for a pharmaceutical company that is introducing a new product. You have decided to innovate. The experts will not be sent on the road. No more classes about the new product in hotels around the world. Instead, expert and sales- related messages have been captured in

e-learning modules, knowledge bases, and performance support tools. You have scheduled online, synchronous conver- sations so that salespeople can query product experts. You have developed briefings for sales supervisors to use. It’s been a big effort. Are you ready now to move to the next project? Do you dare to sit back and wait for the next quarter’s sales numbers? I think not. You must inquire if the sales reps are using the resources or downloading the materials to send to their customers. You must find out if the salespeople are satisfying customers. Are salespeople and supervisors attending online con- versations with experts? Are they adding ideas to the sales knowledge base? How can you learn from this experi- ence to make this effort and the next more effective? The time to know is now. As other workforce learning pro- fessionals bring fresh services and

products into the workplace, they have questions that must be answered. No longer can they glance around the room and pat themselves on the back be- cause people are present and engaged. They must use technology to help in counting, reaching, querying, and communicating. Kirkpatrick’s work has been the indus- try standard for half a century. But the numbers are in. Levels 3 and 4 languish. It is time to respectfully move forward, using this moment and these technolo- gies to jump-start measurement smack dab in the middle of work. T+D

Allison Rossett is a professor of educational technology at San Diego State University. She is the co-author of the new book, Job Aids and Performance Support; arossett@ mail.sdsu.edu.

01 07

25 .6

22 20

Staying Legal is the perfect guide for learning officers, authors, human resource professionals, business owners, and anyone else who is concerned—or should be concerned—about how copyright and trademark laws apply to their work. Based on the authors’ lively and instructive seminars, the interactive audio CD-ROM is the perfect way to gain this important and timely knowledge without leaving your office. A companion book included is designed to reinforce the material presented on the CD-ROM.

AVAILABLE NOW!

Product Order Code: 110701 ASTD Member Price: $59.95 List Price: $69.95

Order at store.astd.org

Staying LegaL A Guide to Copyright and Trademark Use