Organization Development paper

jperso

06ch_bierema_organization.pdf

Home >Human Resource Management homework help >Organization Development paper

6 Action Research: The Checking Phase

DragonImages/iStock/Thinkstock

Learning Objectives

After reading this chapter, you should be able to:

• Describe evaluation according to how it is defined and what steps encompass it.

• Identify the types and categories of evaluation.

• Examine different frameworks of evaluation.

• Determine how to plan and perform an evaluation, including defining its purpose, identifying appropri- ate measures, using the most appropriate data collection methods, analyzing data, sharing feedback, and anticipating and managing client resistance.

• Explore strategies for concluding the action research process, including terminating the consultant–client relationship or recycling the intervention.

co-cn

co-cr

co-box

co-photo

CO_CRD

CO_PT

CO_NLF

CO_NL

CO_NLL

bie81554_06_c06_181-216.indd 181 6/30/14 1:00 PM

In Chapter 5 we learned about the Public Health Leadership Academy, which was founded by a major university using funds from a federal grant to promote leadership development among public health employees in a southern state. The project involved developing a Leadership Academy for midlevel managers who exhibited potential to advance to higher levels of public health leadership in the state. The intervention was in response to a long-term need based on previous analyses of the state’s public health agency, including succession planning. This need had existed for many years because there were not enough public funds available to provide a comprehensive program. The grant finally created the opportunity to deliver this much-needed program. James (the client) worked with Leah (the external consultant) to plan and implement the program.

James and Leah engaged in action research to collect and analyze data about the needs of the target population (midlevel public health managers) using interviews and surveys to determine the content of the courses that would be offered in the Leadership Academy. The project had a 2-year implementation time line, with year 1 focused on planning and year 2 devoted to implementation. Evaluation would be ongoing and continue past year 2 with a new cohort starting in year 3, staffed by internal consultants.

During the year 1 planning phase, James and Leah were very involved in collecting data to inform the content and process of the Leadership Academy. They continually stopped to reflect on their decisions, plans, and processes and made adjustments to each as the project unfolded. They also piloted the first session among a small group of advisors to the Leadership Academy to make sure their design would resonate with the participants. They made more changes following the pilot to improve the program.

During year 2, 25 managers chosen for the academy participated in monthly leadership development experiences and seminars. The Leadership Academy began in September with these 25 managers, who had been competitively selected from across the state. The participants convened at a resort, and the program was kicked off by high-level state public health officials. The first session lasted 3 days, during which time the participants received the results of a leadership styles inventory, listened to innovative lectures and panels on leadership, planned an individual leadership project in their districts, and engaged with each other to deve- lop working relationships. The academy continued meeting monthly for a year and focused on a range of topics related to leadership that were prioritized based on prior data collection. The grant provided for an evaluator, so data were collected at each meeting.

The first 2 years of the project involved ongoing ass- essment of the academy’s plans and implementation, followed by appropriate adjustments. James and Leah included cycles of assessment and adjustment as a regular part of their agenda and conversation.

Kai Chiang/iStock/Thinkstock

The Leadership Academy is off to a lively start.

box-1

LO_BL

LO_BLL

BX_TX

BX1_H1

sec_n sec_t

bie81554_06_c06_181-216.indd 182 6/30/14 1:00 PM

Section 6.1Defining Evaluation

The evaluator observed all of the sessions and sent out formal evaluations after each monthly session. During the sessions, facilitators regularly asked participants to provide feedback. For example, they were asked to respond to questions like, “How did that exercise work for you?” “How are you looking at this now?” and “How could we do this better?” The evaluation data contributed to changes to the planned curriculum and program activities. For example, the participants took an inventory to assess leadership style and wanted to spend more time on the topic, so the next month’s agenda was adjusted to accommodate the request. Participants complained that the sequencing of topics was not logical, so the agenda for the second cohort to follow in year 3 of the project was adjusted.

The first cohort graduated at its final session, during which the cohort welcomed the members of the new cohort. Leah had worked with an internal team of consultants throughout the implementation, and the team was ready to take over the facilitation with the second cohort. Following the event, Leah met with James and the new team to tie up loose ends and make the transition. She met periodically with James during the third year to ensure that the Leadership Academy was running smoothly.

As this vignette illustrates, although checking is the third phase of the action research pro- cess, it takes place during the planning and doing phases as well. This chapter focuses on checking, in which the implementation is evaluated to see if it solved the problem (see Table 5.1).

6.1 Defining Evaluation The final phase of action research involves three steps. First, the consultant and client gather data about the key changes and learning that have occurred. This step is known as assess- ing changes. Next, the consultant uses these data to assess if the intended change occurred. Was the change implementation effective? Were the proposed outcomes met? As a result of this assessment, the consultant adjusts the intervention accordingly. This step is known as adjusting processes. The third step is to terminate the OD process or repeat it to correct or expand the intervention (known as recycling). Assessment, adjustment, and terminating or recycling are collectively known as evaluation.

Purposes of Evaluation

The overall purpose of conducting an evaluation is to make data-based decisions about the quality, appropriateness, and effectiveness of OD interventions. Evaluation helps us determine whether an intervention’s intended outcomes were realized and assess and adjust the intervention as needed. Evaluation helps ensure accountability and knowledge generation from the intervention.

bie81554_06_c06_181-216.indd 183 6/30/14 1:00 PM

Section 6.1Defining Evaluation

zimmytws/iStock/Thinkstock

Evaluation helps assess the effectiveness and impact of OD interventions by analyzing data such as employee satisfaction surveys.

An evaluation creates criteria of merit, constructs standards, measures per- formance, compares performance to standards, and synthesizes and inte- grates data into a judgment of merit or worth (Fournier, 1995). Evaluation find- ings help render judgments, facilitate improvements, or generate knowledge (Patton, 1997). Evaluations used to ren- der judgments focus on accountability for outcomes such as holding manage- ment responsible for making changes in leadership. Improvements concentrate on developmental processes such as cre- ating new learning and growth. Knowl- edge generation emphasizes academic contributions such as new insights that may change a process.

Establishing a Benchmark To illustrate how evaluation helps OD consultants assess and adjust an intervention, let us consider an organization that has conducted survey research to assess employee satisfaction. The first year creates a benchmark that can be used in future evaluations. Further, let us imag- ine that employee satisfaction is at a moderately satisfied level the first time it is measured. When the survey research instrument on employee satisfaction is replicated in future years, the level of satisfaction will be compared with the original baseline to evaluate whether the organization is doing worse, the same, or better than it had originally. The evaluation can help the organization identify key changes and learning that occurred as a result of the interven- tion. Then the organization can adjust practices accordingly.

The American Productivity and Quality Center developed a benchmarking definition that rep- resented consensus among 100 U.S. companies:

Benchmarking is a systematic and continuous measurement process; a pro- cess of continuously measuring and comparing an organization’s business process against business process leaders anywhere in the world to gain infor- mation, which will help the organization take action to improve its perfor- mance. (as cited in Simpson, Kondouli, & Wai, 1999, p. 718)

bie81554_06_c06_181-216.indd 184 6/30/14 1:00 PM

Section 6.1Defining Evaluation

Who Invented That? Benchmarking

When an organization compares its business processes, practices, and performance standards to other organizations that are considered best in class, they are engaging in benchmarking. The exact derivation of the term benchmarking is unknown. It is thought to have possibly originated from using the surface of a workbench in ancient Egypt to mark dimensional measurements on an object. Alternatively, surveyors may have used the term to refer to the process of marking cuts into stone walls to measure altitude of land tracts, and cobblers may have used it to describe measuring feet for shoes (Levy & Ronco, 2012).

Benchmarking in U.S. business emerged in the late 1970s. Xerox is generally considered the first corporation to apply benchmarking. Robert Camp (1989), a former Xerox employee, wrote one of the earliest books on benchmarking. Camp described how U.S. businesses took their market superiority for granted and were thus unprepared when higher quality Japanese goods disrupted U.S. markets.

Benchmarking is a specific type of action research, but the process can also be applied during OD intervention evaluations. There are several types of benchmarking (Ellis, 2006), including:

• Competitive: uses performance metrics to assess how well or poorly an organization is performing against direct competitors, such as measuring quality defects between the companies’ products.

• Comparative: focuses on how similar processes are handled by different organiza- tions, such as two organizations’ recruitment and retention activities.

• Collaborative: involves sharing knowledge about particular activity between compa- nies, with the goal of learning.

Almost any issue of interest can be benchmarked, including processes, financial results, investor perspectives, performance, products, strategy, structure, best practices, opera- tions, management practices, and so forth. Benchmarking could be part of the data collec- tion process in OD, an intervention, or the basis of an evaluation. Table 6.1 shows typical benchmarking steps.

Table 6.1: Typical benchmarking process

Benchmarking step Example

1. Identify process, practice, method, or product to benchmark.

Identifying best practices for recruiting and retain- ing a diverse workforce.

2. Identify the industries with similar processes. Finding the companies that are best at retain- ing a diverse workforce, even those in a different industry.

(continued)

bie81554_06_c06_181-216.indd 185 6/30/14 1:00 PM

Section 6.1Defining Evaluation

Benchmarking step Example

3. Identify organization leaders in a target area. Selecting the organizations against which to benchmark.

4. Survey the selected organizations for their measures and practices.

Sending a survey to the target companies asking for information on issues like turnover and hire rates, formal retention programs (e.g., orientation, development), management training and rewards, and so forth.

5. Identify best practices. Analyzing data to identify best practices to imple- ment. Analysis depends on the type of data col- lected, whether it is statistical (quantitative data), such as from a survey of employees on attitudes about diversity, or interpretive (qualitative data), such as from interviews with employees who quit.

6. Implement new and improved practices. Implementing best practices, such as new recruit- ment and retention strategies, affinity groups, or rewards for managers who develop a diverse staff.

Other Purposes of Evaluation Caffarella (1994) and Caffarella and Daffron (2013) identified 12 specific purposes of evalu- ation data. Evaluation helps to

1. adjust the intervention as it is being made in terms of design, delivery, management, and evaluation;

2. keep employees focused on the intervention’s goals and objectives; 3. provide information to inform the continuation of the intervention; 4. identify improvements needed to design and deliver the intervention; 5. assess the intervention’s cost-effectiveness; 6. justify resource allocations; 7. increase application of participants’ learning by building in strategies that help them

transfer learning back to the organization; 8. provide data on the results of the intervention; 9. identify ways to improve future interventions;

10. cancel or change an intervention that is poorly designed or headed for failure; 11. explore why an intervention fails; and 12. provide intervention accountability.

Moreover, during the planning phase, evaluation can help consultants assess needs and make decisions about how best to intervene. The Leadership Academy’s goal was to improve lead- ership, but James and Leah had to assess the content that would be most appropriate for lead- ership in public health. Then, when the participants were selected, they had to make further assessments to ensure the program was relevant to the participants’ particular needs.

Table 6.1: Typical benchmarking process (continued)

bie81554_06_c06_181-216.indd 186 6/30/14 1:00 PM

Section 6.1Defining Evaluation

Evaluation may also help test different theories and models of addressing the problem. In the case of the Leadership Academy, James and Leah based their interventions on theories and models of leadership. They threw out what did not resonate with the participants or work well during sessions and revised the program for the second cohort.

Evaluation also helps monitor how the intervention is going during implementation so it can be adjusted accordingly. Such adjustments occurred throughout the Leadership Academy implementation over the course of a year.

Finally, evaluation helps determine whether the intervention goals were met and what impact the change had on individuals and the organization. Measuring this type of impact may require more longitudinal study than other types of evaluation. The evaluation of impact helps con- sultants decide whether to extend the intervention, change it, or abandon it altogether. The Leadership Academy will be continually reevaluated as new cohorts participate each year.

Clearly, evaluations have the potential to accomplish a variety of goals. Throughout the OD process, it is critical to stay focused on an evaluation’s purpose. Have you experienced any of the evaluation activities discussed here?

Steps in Evaluation

Just as with action research models, so too are there many approaches to undertaking evalua- tion. That is, there are different ways to model the steps in the process. Two are discussed here.

Evaluation Hierarchy Rossi, Lipsey, & Freeman (2004) offered an evaluation hierarchy that recognizes the impor- tance of engaging in evaluation from the beginning of the action research process. That is, evaluation should occur during the initial client contacts, be built into the plan for interven- tion, and be ongoing throughout the implementation, prior to the formal assessment of the intervention’s impact, cost, and efficiency. Doing evaluation is a matter of conducting a mini- action research project.

Caffarella’s Systematic Program Evaluation Caffarella (1994) outlined the steps generally taken during an evaluation. Her steps have been modified to address key OD issues in the following points. Caffarella’s steps are intended to be sequential under ideal conditions, although reality may be quite different. Note that Caffarella has proposed a lot of steps. She has elaborated more on the steps than some other models but still follows an action research process.

bie81554_06_c06_181-216.indd 187 6/30/14 1:00 PM

Section 6.1Defining Evaluation

1. Secure support for the evaluation from stakeholders such as the client and key man- agement. This step should be a provision of the contract as discussed in Chapter 3. It is the process of getting management to commit to the time and resources needed to evaluate the process, as well as being willing to pay attention to the findings.

2. Identify individuals who can be involved in planning and overseeing the evaluation, such as the participants, management, client, and others impacted by the interven- tion. This is usually led by the consultant and client and would involve employees who are engaged in the implementation. It could also involve those affected by the change who did not necessarily participate in it, such as customers or suppliers.

3. Define the evaluation’s purpose and how the results will be used. This step is elabo- rated on in a later section of this chapter. The evaluation’s focus should be deter- mined and then built accordingly. For example, is it aimed at improving a process or judging an outcome? Does it pertain to planning the intervention or the intervention itself ? Is it aimed at assessing adherence to budget or performance outcomes?

4. Specify what will be judged and formulate the evaluation questions. This step is driven by the evaluation’s purpose. If you decide to evaluate how satisfied employ- ees are with a new performance appraisal process, questions should relate to that change and be used to judge whether it was effective and should continue.

5. Determine who will supply the evidence, such as participants, customers, manage- ment, employees, or others affected by the intervention.

6. Specify the evaluation strategy in terms of purpose, outcomes, time line, budget, methods, and so forth.

7. Identify the data collection methods and time line. Data collection was discussed extensively in Chapter 5. The selected methods should best match the evaluation’s purpose.

8. Specify the data analysis procedures and logistics (covered in Chapter 5). However, the analysis is oriented around making decisions and changes to the intervention, not on diagnosing the problem.

9. Define the criteria for judging the intervention. This can be somewhat subjec- tive unless the metrics are defined in advance. For example, if the intervention were aimed at improving employee retention, would a consultant measure simply whether it improved or look for a certain benchmark (such as 10%) to deem it successful?

10. Complete the evaluation, formulate recommendations, and prepare and present the evaluation report. These steps mirror the data analysis steps presented in Chapter 5 and the feedback meeting strategies in Chapter 4.

11. Respond to recommendations for changes as appropriate.

Table 6.2 compares the action research model to Rossi, Lipsey, and Freeman’s and Caffarella and Daffron’s evaluation steps. They vary in terms of detail and number, but essentially follow the three phases of action research.

bie81554_06_c06_181-216.indd 188 6/30/14 1:00 PM

Section 6.1Defining Evaluation

Table 6.2: Comparing the action research model to evaluation models

Action research model Rossi, Lipsey, and Freeman Caffarella and Daffron

Planning 1. Assess intervention cost and efficiency.

2. Assess intervention outcome or impact.

1. Secure support for the evaluation from stakeholders.

2. Identify individuals who can be involved in planning and overseeing the evaluation.

3. Define the evaluation’s purpose and how the results will be used.

4. Specify what will be judged and formulate the evaluation questions.

5. Determine who will supply the evidence. 6. Specify the evaluation strategy in terms

of purpose, outcomes, timeline, budget, methods, and so forth.

7. Identify the data collection methods and time line.

8. Specify the data analysis procedures and logistics.

9. Determine the specific timeline and the budget needed to conduct the evaluation.

Doing 3. Assess intervention implementation.

10. Complete the evaluation, formulate recom- mendations, and prepare and present the evaluation report.

Checking 4. Assess intervention design and theory.

5. Assess need for the intervention.

11. Respond to recommendations for changes as appropriate.

Caffarella and Daffron’s steps are comprehensive, covering the key tasks that must be com- pleted during an intervention’s evaluation. However, it may not always be possible to follow these clearly articulated steps; evaluation can be unpredictable and may present challenges that are often unanticipated. For example, if an implementation has been challenging, a cli- ent may balk at the evaluation out of fear of receiving negative feedback; on the other hand, employees may be reluctant to participate if trust levels are low. Thus, it helps to pay atten- tion to relevant dynamics and expect the unexpected.

Evaluation provides critical information about an intervention’s impact both during and after its implementation. Thus, no matter what model is followed for performing an evaluation, it is essential to begin planning it before the intervention is well underway. A consultant’s job is to ensure that evaluation is integrated into the OD process from start to finish. Unfortunately, evaluation is often overlooked in favor of wanting simply to take action on the problem, and too many consultants consider their work finished once the intervention has occurred. In other cases consultants go about evaluation haphazardly. If they cannot demonstrate that their action was effective, however, they risk undermining their client’s confidence in the OD effort, fail to permanently solve the problem, and put themselves at risk of repeating similar mistakes on future assignments.

bie81554_06_c06_181-216.indd 189 7/9/14 2:00 PM

Section 6.2Types and Categories of Evaluation

6.2 Types and Categories of Evaluation Theorists have proposed different types and categories for evaluation. This section identifies some of these different approaches.

Types of Evaluation

Evaluation can be either formative or summative, depending on the intervention’s goal (Scriven, 1967, 1991a, 1991b). Scriven is considered a leader in evaluation; you can view one of his lectures by visiting the media links provided at the end of this chapter.

Formative Evaluation Making changes to an implementation that is already in progress is called doing a formative evaluation. Formative evaluation is concerned with improving and enhancing the OD process rather than judging its merit. The following types of questions might be asked when conducting a formative evaluation:

• What are the intervention’s strengths and weaknesses?

• How well are employees progressing toward desired outcomes?

• Which employee groups are doing well/not so well?

• What characterizes the implementation problems being experienced?

• What are the intervention’s unintended consequences?

• How are employees responding? What are their likes, dislikes, and desired changes?

worldofvector/iStock/Thinkstock

Formative evaluation involves making assessments and adjustments during the action research process to reach established goals.

Take Away 6.1: Defining Evaluation

• Evaluation is a process of assessing, adjusting, and terminating or recycling the intervention based on data and subsequent decisions.

• The purpose of evaluation is to make data-based decisions about an intervention’s quality, appropriateness, and effectiveness.

bie81554_06_c06_181-216.indd 190 6/30/14 1:00 PM

Section 6.2Types and Categories of Evaluation

• How are the changes being perceived culturally? • How well is the implementation conforming to budget? • What new learning or ideas are emerging?

For example, consider an intervention focused on changing reporting relationships as part of a work redesign in a manufacturing plant. A consultant might discover that some of the new arrangements do not make sense once implemented. These might therefore be modified as the work redesign progresses. Asking questions pertaining to the problems, the employees’ per- spectives, their likes and dislikes, and so forth yields information that helps tweak and improve the process. Formative evaluation is generally ongoing throughout the implementation.

Summative Evaluation Undertaking evaluation at the end of the OD implementation, with the goal of judging whether the change had the intended outcomes and impact, is called summative evaluation. Summa- tive evaluation is also known as outcome or impact evaluation because it allows the interven- tion’s overall effectiveness to be ascertained. A consultant can then decide whether to continue or terminate it (Patton, 1997). The following types of questions might be asked by the consul- tant, management, or an external evaluator when conducting a summative evaluation:

• Did the intervention work? • Did the intervention satisfactorily address the performance gap? • Should the intervention be continued or expanded? • How well did the intervention stick to the budget?

Summative evaluations should follow four steps:

1. Select the criteria of merit—what are the sought metrics? 2. Set standards of performance—what level of resolution is sought? 3. Measure performance—conduct the evaluation. 4. Synthesize results into a judgment of value. (Shadish, Cook, & Leviton, 1991,

pp. 83–94)

Adequate levels of both formative and summative evaluation must be incorporated into the OD process. Failure to conduct formative evaluation leads to missed opportunities to adjust and improve on the implementation as it is in progress. Omitting the summative evaluation means never learning the intervention’s outcomes and impact or lacking adequate data on which to base future decisions.

bie81554_06_c06_181-216.indd 191 6/30/14 1:00 PM

Section 6.2Types and Categories of Evaluation

Cervero’s Evaluation Categories

Cervero (1985) identified seven categories of evaluation for planners of educational pro- grams that have relevance for OD. His list has been adapted for OD interventions in terms of categories of evaluation:

1. Intervention design and implementation. This could be either formative or sum- mative, since the design and intervention are assessed for fit and impact. Imagine implementing a new performance appraisal process. Formative evaluation might involve piloting the evaluation and evaluating how well it worked for both employ- ees and supervisors. The performance appraisal would then be modified and implemented. Summative evaluation in this case might examine whether the new performance appraisal process improved performance, satisfaction, and learning.

2. Employee participation. This type of evaluation assesses employees’ level of involve- ment in the intervention. This could also be formative or summative. In the case of performance evaluation, a consultant might examine the level of involvement and seek feedback from employees. A summative evaluation might evaluate whether the level of employee participation was adequate and whether it yielded positive outcomes.

3. Employee satisfaction. This type of evaluation assesses employees’ level of satis- faction in the intervention. This could also be formative or summative. In the case of performance evaluation, a consultant might examine the level of satisfaction with the new performance appraisal or its implementation process. A summative evaluation might evaluate how satisfied employees are once the new performance appraisal system is in place.

4. Acquisition of new knowledge, skills, and attitudes. This type of evaluation measures learning during and after the intervention and could also be formative or summative. In the case of performance evaluation, a consultant might examine the level of involve- ment and seek feedback from employees. A formative evaluation during the pilot phase might determine that supervisors lack the skills to effectively implement the new process and give the level of feedback desired. It would allow the consultant and client to revise the process and provide adequate training to supervisors. A summative evaluation would assess the level of learning from the new performance appraisal sys- tem. This could take the form of employees improving their performance and supervi- sors showing demonstrated improvement in their ability to give feedback.

5. Application of learning after the intervention. This category is similar to the previous one, but it is summative; it judges how learning was applied after the intervention. A consultant might look for evidence of how supervisors applied what they learned about giving effective performance feedback to other interactions with employees throughout the year.

6. The impact of the intervention on individuals and the organization. This category could be formative or summative. The formative evaluation might look at how the inter- vention impacts organization life via communication, understanding, participation, satisfaction, and so forth. A summative evaluation would look at the overall impact on satisfaction, financial performance, retention, job satisfaction, and so forth.

7. Intervention characteristics associated with outcomes. This type of evaluation attempts to link aspects of the intervention to outcomes and is summative. This can be more difficult to measure if the intervention was complex or had several interventions built into it. A consultant might evaluate how a participative process affected the implementation’s overall success or employee satisfaction.

bie81554_06_c06_181-216.indd 192 6/30/14 1:00 PM

Section 6.3Frameworks of Evaluation

6.3 Frameworks of Evaluation This section profiles some common evaluation frameworks. There is no “best” framework. Rather, you should find what you are comfortable working with and what effectively fits the situation.

Kirkpatrick’s 4-Level Framework

Kirkpatrick’s (2006) 4-level evaluation framework can be formative or summative and is one of the most widely known evaluation typologies that became popular in the 1990s. It was originally created to evaluate training programs, and OD consultants use it to conduct evalua- tion at a range of points over time. The framework classifies an intervention’s outcomes into one of four categories—reaction, learning, behavior, or results (see Table 6.3). An outcome is assigned a category based on how difficult it is to evaluate. For example, the simplest type of outcome to evaluate is participant reaction to the intervention. Thus, this is assigned level 1.

If you have had the opportunity to evaluate organization change efforts, have you experienced any evaluative measures on Cervero’s list? Which ones would be most relevant for the Leader- ship Academy vignette? Can you think of other categories that might be added to Cervero’s list?

Take Away 6.2: Types and Categories of Evaluation

• Evaluation can be formative or summative. Formative evaluation is concerned with improving and enhancing an OD process as it is underway, rather than judging its merit. Summative evaluation occurs after the implementation is complete and ascertains whether the change accomplished the desired outcomes and impact. Both provide valuable ways to assess the intervention before, during, and after it has occurred.

• Cervero’s categories of evaluation show the different approaches to evaluation. It is important to be clear on an evaluation’s purpose at the planning stage. Typical evaluation categories include intervention design and implementation; employee participation and satisfaction; acquisition of new knowledge, skills, and attitudes; application of learning after the intervention; impact of the intervention; and intervention characteristics associated with outcomes.

bie81554_06_c06_181-216.indd 193 6/30/14 1:00 PM

* * * * * * * *

* * * * * *

* *

* * * * * *

* *

* * * *

How Are We Doing?

Section 6.3Frameworks of Evaluation

Table 6.3: Kirkpatrick’s 4-level evaluation framework

Level Focus Examines

1 Reaction Did participants like the intervention?

2 Learning What skills and knowledge did participants gain?

3 Behavior How are participants performing differently?

4 Results How was the bottom line impacted?

Level 1 Evaluation Level 1 measures participant reaction to the intervention. This type of evaluation is some- times referred to as a “smile sheet” because it measures only what participants thought and felt during or immediately after an interven- tion, and in very simple terms—whether they were satisfied, neutral, or dissatisfied. As an example, consider the Leadership Academy vignette. At this level of evaluation, the con- sultants might ask the academy participants questions such as: “How well did you like the session?” “Was the learning environment com- fortable?” “Were the facilitators capable and credible?” and “Did you feel it was time well spent?” This type of evaluation may make facil- itators feel good about introducing an inter- vention, but it does not effectively measure change. Unfortunately, it is the most common form of evaluation employed in organizations.

Level 2 Evaluation Level 2 measures participant learning from an intervention. This level of evaluation assesses whether the intervention helped participants improve or increased their knowledge or skills. At this level, James and Leah might ask the Leadership Academy participants, “What was the key thing you learned from this session?” or “What new skills have you acquired as a result of this experience?” This type of evaluation works best after participants have had a chance to return to their workplace and apply the principles and behaviors they learned (and thus is summative). Participants might also be interviewed or surveyed about learning during the course of the intervention (which would be formative).

Figure 6.1: Reaction evaluation

using a smile sheet

The reaction evaluation sheet shown here is just one way to solicit feedback from an audience.

* * * * * * * *

* * * * * *

* *

* * * * * *

* *

* * * *

How Are We Doing?

bie81554_06_c06_181-216.indd 194 6/30/14 1:00 PM

Section 6.3Frameworks of Evaluation

Level 3 Evaluation Level 3 measures changes in behavior. This level of summative evaluation assesses whether participants are using their new knowledge or skills in their job. At this level James and Leah might ask Leadership Academy participants, their supervisors, or subordinates: “To what extent has the leader’s behavior changed and improved as a result of the Leadership Acad- emy?” or “What is the person doing differently now?” Similar to level 2, this type of evaluation is best done post intervention. It can be accomplished by interviewing, observing, or survey- ing participants and stakeholders affected by the intervention.

Level 4 Evaluation Level 4 measures results for the organization. This level of summative evaluation measures how the intervention affected business performance or contributed to the achievement of organization goals. At this level James and Leah might ask Leadership Academy participants, their supervisors, or subordinates: “How has the organization benefited from the Leader- ship Academy?” “To what degree has employee satisfaction, productivity, or performance improved?” “To what degree has recruitment and retention of employees improved as a result of improved leadership?” “How many promotions have occurred as a result of participating in the academy?” or “How much money has the organization saved due to better leadership decisions?” As these questions indicate, it might be difficult to actually measure and attribute changed leadership to organization results and outcomes.

Kirkpatrick continued to evolve his model and even questioned whether it was a true model or just a guideline. He also expanded his focus to consider an intervention’s cost–benefit ratio and whether it demonstrated a return on investment. Measuring these variables can also present challenges to organizations and OD consultants.

Lawson’s Application of Kirkpatrick’s Framework Building on Kirkpatrick’s framework, Lawson (2006) categorizes variables relating to the what, who, when, how, and why of the framework’s use. Her approach has been adapted for OD and is depicted in Table 6.4.

Tips and Wisdom

If you are facilitating a meeting, workshop, or seminar and want to gauge a participant’s reaction to the event, create an opportunity for them to share feedback. One way to do this is to put a flip chart page with the smiley face symbols shown in Figure 6.1 in the hall outside the session room. Give participants a dot apiece and ask them to place it in the column that best represents how they feel about the session so far. It is important to place the chart out of your eyeshot so people feel comfortable sharing honest feedback. This offers a snapshot of participants’ reactions and allows you to make adjustments in the moment. You can also use Twitter or other social media to solicit this data.

bie81554_06_c06_181-216.indd 195 6/30/14 1:00 PM

Section 6.3Frameworks of Evaluation

Table 6.4: Applying Kirkpatrick’s framework to OD interventions

Level What Who When How Why

1 Reaction: Did they like it?

Participants During or after the intervention

Smile sheet Determine level of participant satisfaction and need to revise intervention if duplicated

2 Learning: What knowledge or skills were retained?

Partici- pants and consultants

During, before, and/ or after the intervention

Pre- and post- tests, skills applications, role plays, case studies, and exercises

Determine whether consul- tant has been effective in imple- menting interven- tion purpose and objectives

3 Behavior: How are participants performing differently?

Participants, supervisors, subordinates, and peers

3 to 6 months after intervention

Surveys, interviews, observation, performance, and appraisal

Determine extent to which partici- pants transferred their learning from the inter- vention to the workplace

4 Results: What is the impact on the bottom line?

Participants and control group

After comple- tion of level 3 assessment

Cost–benefit analysis and tracking opera- tional data

Determine whether the benefits outweigh costs and how the intervention con- tributed to orga- nization goals and strategy

Source: Adapted from Lawson, 2006, p. 256.

Critiques of Kirkpatrick Kirkpatrick’s (1987, 1994, 2006) four levels of criteria have been dominant for decades among evaluators. With popularity, however, comes criticism. First, the model has been critiqued for being primarily focused on postintervention realities; that is, for evaluating what happens after the intervention versus incorporating more formative evaluation into the process. The 4-level framework also does not help evaluators link causal relationships between outcomes and the levels of evaluation. Finally, the framework does not help evaluators determine what changes equate to the different levels of evaluation or how best to measure each level.

Some authors have suggested expanding the reaction level to include assessing participants’ reaction to the intervention techniques and efficiency (Kaufman & Keller, 1994). One might also try splitting the reaction level to include measuring participants’ perceptions of enjoy- ment, usefulness, and the difficulty of the program (Warr & Bunce, 1995). Kaufman and Keller (1994) recommended adding a fifth level to address the societal contribution and outcomes created by the intervention, which is becoming more popular with the higher emphasis on corporate social responsibility and sustainability. Phillips (1996) advocated adding a fifth level that specifically addresses return on investment.

bie81554_06_c06_181-216.indd 196 6/30/14 1:00 PM

Section 6.3Frameworks of Evaluation

Other Frameworks of Evaluation

Although the Kirkpatrick model is one of the dominant evaluation models, it is not necessar- ily the best or most appropriate for every situation. This section briefly profiles some lesser known evaluation models.

Hamblin’s 5-Level Model Similar to Kirkpatrick’s model, this model measures reactions, learning, job behavior, and organizational impacts, as well as a fifth level—the economic outcomes of training. The hier- archy of Hamblin’s (1974) model is more specific than Kirkpatrick’s in that reactions lead to learning, learning leads to behavior changes, and behavior changes have organizational impact. Because of this assertion, Hamblin believed that evaluation at a given level is not meaningful unless the evaluation at the previous level has been performed.

Preskill and Torres’s Evaluative Inquiry Model Preskill and Torres (1999) contributed a model of inquiry to the literature that uses the evalu- ation process as a learning and development opportunity:

Evaluative inquiry is an ongoing process for investigating and understanding critical organization issues. It is an approach to learning that is fully integrated with an organization’s work practices, and as such, it engenders (a) organiza- tion members’ interest and ability in exploring critical issues using evalua- tion logic, (b) organization members’ involvement in evaluative processes, and (c) the personal and professional growth of individuals within the organization. . . . (pp. 1–2)

Evaluative inquiry is the fostering of relationships among organization mem- bers and the diffusion of their learning throughout the organization; it serves as a transfer-of-knowledge process. To that end, evaluative inquiry provides an avenue for individuals’ as well as the organization’s ongoing growth and development. (p. 18)

Their definition emphasizes that evaluation is more than simply reporting survey findings. Rather than being event driven, such as sending a survey to participants after the interven- tion is over, evaluation should be an ongoing part of everyone’s job; that is, a shared learning process. Evaluative inquiry should be focused on

• intervention and organizational processes as well as outcomes; • shared individual, team, and organizational learning; • educating and training organizational practitioners in inquiry skills (action

learning); • collaboration, cooperation, and participation; • establishing linkages between learning and performance; • searching for ways to create greater understanding of the variables that affect orga-

nizational success and failure; and • using a diversity of perspectives to develop understanding about organizational

issues (Preskill & Catsambas, 2006).

bie81554_06_c06_181-216.indd 197 6/30/14 1:00 PM

Section 6.3Frameworks of Evaluation

Preskill and Torres (1999) identified four learning processes—dialogue, reflection, question- ing, and identifying and clarifying values, beliefs, assumptions, and knowledge—that facilitate three phases of evaluative inquiry: focusing the evaluative inquiry, carrying out the inquiry, and applying learning. The phases are depicted in Table 6.5.

Table 6.5: Preskill and Torres’s evaluative inquiry phases

Focusing the evaluative inquiry Carrying out the inquiry Applying learning

• Determine issues and con- cerns for evaluation

• Identify stakeholders • Identify guiding questions for

the evaluation

• Design and implement the evaluation (collect, analyze, interpret data)

• Address evaluative questions

• Identify and select action alternatives

• Develop and implement action plans

• Monitor progress

• Strategies • Focused dialogues • Group model building • Open space technology • Critical incidents • Assumption testing through

questioning

• Strategies • Develop a database for orga-

nization learning • Literature-based discussions • Working session to interpret

survey results (or other data collected)

• Framing findings as lessons learned

• Strategies • Capturing concerns, issues,

and action alternatives • Using technology to facilitate

brainstorming • Developing an action plan • Solving implementation

issues

At each stage of inquiry, the following skills are used: 1. Dialogue 2. Reflection 3. Asking questions 4. Identifying and clarifying values, beliefs, assumptions, and knowledge

Brinkerhoff ’s 6-Stage Model This model defines evaluation as the collection of information to facilitate decision mak- ing (Brinkerhoff, 1989). It requires that consultants articulate how and why each training or development activity is supposed to work, without which comprehensive evaluation is impossible. This model helps to assess whether and how programs benefit the organization; analysis can help trace any failures to 1 or more of the 6 stages of Brinkerhoff ’s model. The stages are:

1. Goal setting (What is the need?): A need, problem, or opportunity worth addressing that could be favorably influenced if someone learned something.

2. Program design (What will work?): A program that teaches the needed topic is cre- ated or, if one already exists, it is located.

3. Program implementation (It is working?): The organization successfully implements the designed program.

4. Immediate outcomes (Did they learn it?): The participants exit the program after successfully acquiring the intended skills, knowledge, or attitudes.

5. Immediate or usage outcomes (Are they keeping and/or using it?): The participants retain and use what they learned.

6. Impacts and worth (Did it make a worthwhile difference?): The organization ben- efits when participants retain and use what they learned.

bie81554_06_c06_181-216.indd 198 6/30/14 1:00 PM

Section 6.3Frameworks of Evaluation

Input, Process, Output, and Outcomes Evaluation This model evaluates training programs at four levels (input, process, output, and outcomes) in terms of their potential contribution to the overall effectiveness of a training program (Bushnell, 1990). It is similar to the systems model introduced in Chapter 2 that considers inputs, throughputs, and outputs in organization systems.

1. Inputs: trainee qualifications, instructor abilities, instructional material, facilities, and budget.

2. Process: value-adding activities such as planning, designing, developing, and deliver- ing the training.

3. Output: trainee reactions, knowledge and skills gained, and improved job performance.

4. Outcomes: profits, customer satisfaction, and productivity.

This model has the following benefits:

1. It can help determine whether training programs are achieving the right purpose. 2. It can help identify the types of changes that could improve course design, content,

and delivery. 3. It can help determine whether students have actually acquired knowledge and skills.

Steps in the evaluation process:

1. Identify evaluation goals: Determine the overall structure of the evaluation effort and establish the parameters that influence later stages.

2. Develop an evaluation design and strategy: Select appropriate measures, develop a data collection strategy, match data types with experimental designs, allocate the data collection resources, and identify appropriate data sources.

3. Select and construct measurement tools: Select or construct tools that best fit the data requirements and meet criteria for reliability and validity. Examples include questionnaires, performance assessments, tests, observation checklists, problem simulations, structured interviews, and performance records.

4. Analyze the data: Tie the results of the data-gathering effort to the evaluation’s origi- nal goals.

5. Make conclusions and recommendations and present the findings.

Take Away 6.3: Frameworks of Evaluation

• Multiple frameworks exist for conducting evaluation. The best known is Kirkpatrick’s 4-level evaluation framework. This model measures reaction, learning, behavior, and results pertaining to the intervention.

• Other models of evaluation include Hamblin’s 5-level model, Preskill and Torres’s evaluative inquiry, Brinkerhoff ’s 6-stage model, and the input, process, output, and outcomes model.

bie81554_06_c06_181-216.indd 199 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

6.4 Planning and Performing the Evaluation Just as with other aspects of the action research process, evaluation requires deliberate plan- ning and buy-in from the client. To plan the evaluation, Cervero (1985) suggests identifying five key factors:

1. What is the purpose of the evaluation? 2. Who needs what information? 3. Are there any practical and ethical constraints? 4. What resources are available? 5. What are the evaluator’s values?

The answers to these five questions offer an evaluation a strong foundation, so it is worth reflecting with the client about them. Once clear on these issues, you can get more specific about evaluation purposes, measurements, information sources, data collection, data analy- sis, feedback, further action, and how to anticipate and manage resistance. These evaluation steps may look familiar, because conducting an evaluation is similar to doing a small-scale action research project. These steps will be explored in the next sections.

Determine the Purpose of the Evaluation

Determining the evaluation’s purpose(s) offers clear focus moving forward. Referring back to Cervero’s categories of evaluation may help pinpoint what is being evaluated. For effective formative evaluation, a consultant should work with the client to determine what needs to be evaluated throughout the action research process, particularly regarding process improve- ment. For example, in the case of the Leadership Academy, formative evaluation consisted of assessing the curriculum for relevance and cost. The examination of a performance appraisal process change earlier in the chapter showed how one might assess employees’ satisfaction with their level of participation in the process or with a pilot phase of the appraisal.

Consultants should also plan the purpose of summative evaluation. Returning to the example of the Leadership Academy, it was essential to find out whether the participants learned and applied the new behaviors and skills, and if so, how their actions impacted their organiza- tions. In the case of the performance appraisal process change, the organization wanted to know if it changed supervisor behavior, improved retention, impacted learning, and increased performance.

Once an evaluation’s purpose has been decided, a consultant can begin to identify what ques- tions to ask the participants. Table 6.6 offers examples of appropriate questions for different evaluation goals. Questions revolve around needs assessment, intervention conceptualiza- tion, intervention delivery, outcomes, and costs.

bie81554_06_c06_181-216.indd 200 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Table 6.6: Typical evaluation questions

Category Examples

Questions about the need for the intervention (planning and needs assessment)

• What is the nature and magnitude of the prob- lem to be addressed?

• What are the characteristics of the individuals, team members, and/or organization?

• What are the needs of the individuals, team members, and/or organization?

• What consulting services are needed? • What is the time frame? • What contracting is necessary?

Questions about the intervention’s conceptualiza- tion or design (planning)

• Who is the client? Who is the target population? • What consulting services should be provided? • What is the best intervention? • What are the best delivery systems for the

intervention? • How can the intervention identify, recruit, and

sustain the intended participants? • How should the intervention be organized? • What resources are necessary and appropriate

for the intervention?

Questions about intervention logistics, delivery, and reach (planning and intervention)

• Are intervention goals being met? • Are the intended interventions being delivered

to the intended individuals, teams, and/or organization?

• Has the intervention missed participants who need it?

• Are sufficient numbers of participants engaged in the intervention?

• Are the participants satisfied with the intervention?

• Are administrative, organizational, and person- nel functions handled well?

Questions about intervention outcomes (evaluation) • Are the outcome goals and objectives being achieved?

• Does the intervention benefit the recipients? • Does the intervention adversely affect the

participants? • Are some participants affected more by the inter-

vention than others? • Did the intervention improve the issue or

problem?

Questions about intervention cost and efficiency (evaluation)

• Are resources used efficiently? • Is the cost reasonable in relation to the magni-

tude of benefits? • Would alternative approaches yield equivalent

benefits at less cost?

Source: Adapted from Rossi, Lipsey, & Freeman, 2004, p. 77.

bie81554_06_c06_181-216.indd 201 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Case Study: Piloting and Evaluating a New Performance Appraisal Process

A paper products manufacturing company begins working with a consultant to improve employee retention, since the company has a significantly higher attrition rate than other companies it benchmarked. One of the interventions selected during the action research process is to overhaul the way supervisors share feedback with employees; the performance appraisal process will become more developmental than punitive, ongoing rather than once a year, with no surprises in the feedback delivered.

Making this change requires developing a new process. A small design team is formed to advise on the new performance appraisal process. It includes the consultant, client, supervisors, and employees. The team designs an intervention that is based on supervisors providing feedback in the moment—that is, when they notice something and want to coach the employee through it. The model also incorporates periodic opportunities for the employee and supervisor to meet, focus on the employee’s developmental plan, and make adjustments as needed. Once the process is developed, the team decides to pilot it with a small department.

Before the pilot begins, the participating employees are briefed on the intervention and confirm their participation. Both the employees and supervisors are trained in the new method, and the supervisors receive additional training on how to provide coaching and give developmental feedback. The design team begins to study the pilot group’s response by questioning whether it is meeting the original need of improving retention and whether the design is best for meeting the needs. To this end, the design team holds informal conversations with the employees and supervisors and asks them questions such as:

• What do you like about this new process? • What don’t you like about this new process? • How do you perceive these changes? • Will this work if we expand it further, or would you suggest changes? • What have you learned in the process?

The informal conversations yield important data that design team members share during a meeting. There is general support for the idea, but the supervisors do not always feel competent to use the new process correctly; they also feel stressed about the time it takes. The employees are unsure what the purpose of the periodic meetings is. Some employees and supervisors are resistant, feeling either distrust toward the process or resignation that things will not change.

The design team decides to adjust the process. It provides more support and training to the supervisors on how to coach and share feedback, with the expectation that it will take less time as they become more comfortable with the process. It also assembles the department and models an ideal periodic meeting to touch base on development. The pilot group continues to work with the process for several more weeks, with ups and downs.

(continued)

bie81554_06_c06_181-216.indd 202 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Prior to rolling out the process to the wider organization, the design team meets with the pilot department to see if additional adjustments are needed. The process works better for most but still has logistical problems that require further change.

Finally, the company is ready to roll out the changes organization-wide. It starts with a communication plan and provide training to all employees. The design team continues to monitor the process over the next year.

After the plan has been in place for a year, the design team comes together and decides to plan and perform an evaluation to assess whether the new performance appraisal process met its intended goals. The steps it follows include:

1. Purpose of the evaluation: To assess whether the new performance appraisal process increased employee retention.

2. Identify appropriate evaluation measures: The design team decided to look at three measures: a. Retention comparing the year before the intervention to the year after it. b. Employee attitudes. c. Supervisor attitudes.

3. Choose and employ data collection methods: This depends on the type of data desired and the question the organization wishes to answer. a. Attrition records to measure the year-to-year comparison of retention rates. b. Survey of employees c. Interviews of supervisors

4. Analyze data and provide feedback: Once the data were collected, the consultant and client made the first analysis and then involved the design team. Once the findings were refined, they shared them during an open meeting with employees. They also shared them with supervisors in a separate meeting to get their feedback and input.

5. Anticipate and manage resistance to the evaluation: Although the team did review worst case scenarios for how the organization or employees might resist, the problems were minimal. For example, complaints were similar to those it heard throughout the pilot process from employees who were skeptical that the new performance appraisal process would work.

Once the analysis is complete, the design team presents its findings to top management. The organization now needs to determine how effective the intervention was and whether it would be wise to invest further in it.

Critical Thinking Questions

1. Given your knowledge of evaluation, what are some steps the design team followed in implementing its evaluation?

2. What steps did the design team miss?

Case Study: Piloting and Evaluating a New Performance Appraisal Process (continued)

bie81554_06_c06_181-216.indd 203 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Identify Appropriate Evaluation Measures

Once an evaluation’s purpose has been determined, actions to measure it should be identified. This book has covered a range of evaluation techniques. The formative measures that the Leadership Academy team members identified allowed them to recruit a small group of top leaders to critique the curriculum. They followed this with a small pilot session to trouble- shoot and revise the curriculum with an actual audience. Participants evaluated the academy throughout the implementation, and adjustments were made accordingly.

Summative measures could have included any of the examples listed in the Kirkpatrick dis- cussion, such as promotions, employee satisfaction, or customer satisfaction. In the case of the Leadership Academy, measures included improved performance, promotions, a leader- ship project, and team satisfaction. Since a main goal was to cultivate leaders from within the organization, measuring the percentage of participants who were promoted from middle management to executive positions was a key metric for evaluating the intervention’s success.

Choose and Employ Data Collection Methods

With the purpose and measures determined, the consultant should identify appropriate sources of information and methods for gathering the information. For example, if you want to measure the results of a customer-service training, you could measure the number of com- plaints, review written complaints, or contact customers. The methods you might use do to this include surveys, documents, or interviews. Table 6.7 offers an overview of data collection methods appropriate for evaluation. The more commonly used methods to collect evaluation data include: archival data, observations, surveys and questionnaires, assessments, inter- views, and focus groups. Each is discussed in detail below. Chapter 4 reviewed methods used to conduct analysis or planning—many of them are similar.

Table 6.7: Evaluation data collection methods

Evaluation method Description

Interviews A conversation with one or more individuals to assess their opin- ions, observations, and beliefs. Questions are usually determined in advance, and the conversation is recorded.

Questionnaires A standardized set of questions intended to assess opinions, observations, and beliefs that can be administered in paper form or electronically.

Direct observation Viewing a task or set of tasks as they are performed and record- ing what is seen.

Tests and simulations Structured situations to assess an individual’s knowledge or proficiency to perform some task or behavior.

Archival performance data Use of existing information, such as files, reports, quality records, performance appraisals, and so forth.

Product reviews Internal or external evaluations of products or services.

(continued)

bie81554_06_c06_181-216.indd 204 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Evaluation method Description

Performance reviews Written assessments of individual performance against an estab- lished criteria.

Records and documents Written materials developed by organizations and communi- ties (performance appraisals, production schedules, financial reports, attendance records, annual reports, company and board minutes, training data, etc.).

Portfolios A purposeful collection of a learner’s work assembled over time that documents events, activities, products, and/or achievements.

Cost–benefit analysis A method for assessing the relationship between the outcomes of an educational program and the costs required to produce them.

Demonstration Exhibiting a specific skill or procedure to show competency.

Pre- and posttests Instruments used to measure knowledge and skills prior to and after the intervention to see if there were changes.

Focus groups Group interviews of approximately 5 to 12 participants to assess opinions, beliefs, and observations. Focus groups require a trained facilitator.

Source: Adapted from Caffarella & Daffron, 2013.

Archival Data Evaluating the degree of change an intervention produced requires establishing a baseline of existing information from employment records, production figures, or quarterly reports. These are referred to as archival data (or documents and records). You are not seeking new data, but using existing data to assess the intervention’s effectiveness. Archival data are eas- ily accessible, typically available for no or minimal cost, and useful for providing historical context or a chronology of events, such as employee satisfaction over time. The Leadership Academy team relied on archival data from performance reviews and employee satisfaction surveys to evaluate impact.

Observation Data Watching the organization engage in its everyday operations involves observation. This type of evaluation is based on detailed descriptions of day-to-day behaviors that cannot be explored by viewing existing archival records. Examples of observation data might include checklists of meeting-leader behaviors completed by one of the team members, call moni- toring forms, listening skills, and body language. Observation did not play an official role in the Leadership Academy evaluation process; however, participants’ supervisors observed the changes they made in their approach to their work and documented these in their perfor- mance reviews. Data collection by observation can range from routine counting of certain occurrences to writing narrative descriptions of what is being observed.

Table 6.7: Evaluation data collection methods (continued)

bie81554_06_c06_181-216.indd 205 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Surveys and Questionnaires Surveys and questionnaires are helpful for measuring the intervention’s effects. They should be completed by respondents with some experience related to the intervention. In the Lead- ership Academy vignette, the consultants used surveys to gather participants’ input on their individual leadership styles during the program. Other examples include end-of-course reac- tion forms or surveys of stakeholders such as customers, employees, or management. Surveys and questionnaires might also be appropriate when evaluators desire new data from multiple individuals who may be dispersed throughout the organization. Surveys and questionnaires are relatively inexpensive and easy to administer, particularly with the use of technology. It is important that these instruments be well constructed; their wording must be unambiguous, and they must be easy to complete.

Paper, Pencil, or Computer-Based Tests Consultants can administer a variety of commercially produced tests to assess the knowledge or skills imparted by an intervention, or they can develop an original test unique to the inter- vention. No matter the type of test employed, the evaluation result is based on the test scores. This type of evaluation works well when trying to determine the quantity and quality of the participants’ education. OD consultants might administer a pretest before an intervention and a posttest afterward, or they might require participants to pass a test to attain a certifi- cate of completion.

Tests should be cautiously designed and prudently administered. First, questions must be written in a way that consistently and accurately measures what was taught. Second, partici- pants may perceive test taking as threatening, especially if the results will be used to make performance appraisal decisions. Therefore, efforts to defuse test apprehension should be built into the process.

This very book uses some of these tools. Teaching you about OD is the intervention, which is executed via concepts presented in book form. Additional interventions take the form of assign- ments and opportunities to engage with other learners. Pre- and posttests check your prior knowledge on the topic and gauge how well you learned the concepts after you engaged them.

Individual and Focus Group Interviews Chapter 4 discussed interviews and focus groups as effective ways of understanding targeted individuals’ or groups’ views, beliefs, or experiences with the issue under investigation. Both approaches depend on developing well-crafted questions that yield useful information. Inter- views and focus groups should be run by an experienced facilitator. These methods yield rich, qualitative information that includes insights about the intervention, critiques, or success stories. Not all participants react well to these data collection methods, however, and may not trust the interviewers or the process; some may not feel comfortable enough to be honest. Participants may also say what they think the facilitator wants to hear.

bie81554_06_c06_181-216.indd 206 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Analyze Data and Provide Feedback

Once data have been collected from an appropriate source and via an appropriate method, they need to be analyzed. Refer to Chapter 5 for a full discussion of how to analyze data. In the Leadership Academy vignette, performance reviews and employee satisfaction data from survey research were analyzed. The team also monitored participants’ leadership projects and promotional advances.

Next, the data analysis should be presented as feedback to key decision makers such as affected employees and management. How to share feedback with a client is covered extensively in Chapter 4, but the same rules apply when sharing evaluation feedback. It is a consultant’s job to determine the feedback meeting’s key purpose and desired outcomes. Does the client need help determining whether to continue the intervention? Modify it? Measure learning or performance? Address unintended consequences of the intervention? Sharing feedback with the client involves determining the focus of the feedback meeting, developing the agenda for feedback, recognizing different types of feedback, presenting feedback effectively, managing the consulting presence during the meeting, addressing confidentiality concerns, and antici- pating defensiveness and resistance.

At any point in the evaluation process, data collection and analysis can prompt the team to decide to change future action. For example, the team might decide to adjust the ongoing pro- cess, continue the process with new interventions, or close the project if the problem is per- manently solved. This is the third step of the evaluation process, defined earlier in the chapter as termination or recycling. It is discussed in detail in the next section.

Anticipate and Manage Resistance to the Evaluation

Sometimes evaluation is resisted by the client, organization, or other stakeholders. Resis- tance and strategies for curbing it were discussed at length in Chapter 5. Resistors may not want to spend more money to learn the results of the intervention. Or the organization may be unwilling to spend the time required to conduct an evaluation and instead want to move on to the next issue. There may be fear about what the evaluation will reveal (perhaps manage- ment failed to implement the changes, or perhaps employee views remain negative). Organi- zation members can also suffer from change fatigue and worry that the evaluation will bring even more change. Of course, such resistance patterns are likely what created problems in the first place, so observing them warrants timely intervention with the client.

Moreover, a consultant should anticipate political issues the evaluation might create. Results of the evaluation can also influence future resource allocations, which could cause trepidation and conflict among organization members. Remaining vigilant as a consultant and working to be authentic and influential is key to navigating the politics of evaluation.

bie81554_06_c06_181-216.indd 207 6/30/14 1:00 PM

Section 6.4Planning and Performing the Evaluation

Some clients may resist doing the evaluation because they are more interested in moving on to the next challenge or opportunity. Or they may not want to subject themselves to poten- tially negative feedback. In this case the consultant should lay out the benefits of measuring results and learning from both positive and negative feedback. Doing so shows good steward- ship of the time and resources committed to the intervention and provides data to support future initiatives. One way to minimize resistance to evaluation is to make sure it has been addressed during contracting, as outlined in Chapter 3.

Even when there is cooperation and investment, evaluation is not easy. Demonstrating impact and results can be challenging for certain interventions such as improving leadership. Linking results to intervention events can also be tricky. Devising appropriate evaluation criteria can also be problematic, especially if intervention outcomes were vague from the initial planning. Finally, the client may balk at making judgments about the intervention.

Take Away 6.4: Planning and Performing the Evaluation

• Planning and performing the evaluation involves several steps, the first of which is determining the evaluation’s purpose. Articulating a clear purpose gives the evaluation focus and helps identify appropriate participants, measures, and methods.

• Identifying appropriate evaluation measures is driven by the evaluation’s purpose. If a consultant aims to measure employee satisfaction after a change in leadership, he or she would likely survey employees to assess their satisfaction with the change.

• Once the evaluation purpose and measures have been chosen, the data collection methods should be determined and carried out. Typical methods include surveys, interviews, focus groups, observations, and documents.

• Once the data are collected, they can be analyzed and fed back to the client and other interested stakeholders. This information is important for making decisions about the continuance of the intervention and future funding.

• It is advisable to anticipate client resistance both to conducting the evaluation and hearing the results. Consultants can write evaluation protocols into their initial contract. When you notice resistance to evaluation, act quickly to defuse the resistance, address the concerns, and help the client use the information most effectively.

Assessment: Testing Your Change Management Skills

This assessment provides a good review of change and some insight into resistance. The web page offers several resources for learning more.

http://www.mindtools.com/pages/article/newPPM_56.htm

bie81554_06_c06_181-216.indd 208 6/30/14 1:00 PM

http://www.mindtools.com/pages/article/newPPM_56.htm

Section 6.5Concluding the Action Research Process

6.5 Concluding the Action Research Process All consulting jobs end. Indeed, your goal as a successful consultant is to become redundant and work yourself out of a job. In our Leadership Academy vignette, Leah terminated her role with the project after 2 years, the first of which focused on planning and the second on imple- mentation. During year 2, she worked with internal consultants who would take over her role leading and facilitating the Leadership Academy in its third year. In this way she fostered a repetition—called a recycling—of the intervention.

Disengagement or Termination

When the client has successfully implemented a change, the OD consultant is no longer needed. At this juncture the client has become self-reliant and can effectively disengage or terminate the consult- ing relationship. Working oneself out of a consulting job may at first seem like a bad idea. On the contrary, smoothly disengaging from a client is how to help clients build capacity and also the way to get repeat business as a consultant. Effectively navigating this stage depends on setting the expectation during contracting, recognizing the appropriate timing, processing any interpersonal issues between the consultant and the client, ensuring that the learning is ongoing, verifying that the client is satisfied, and planning for postconsulting contact.

Contracting About Termination A consultant should start setting expectations about disengagement right from the beginning of the consultancy, during contracting, as discussed in Chapter 3. There are several things that help disen- gagement go smoothly. First, the consultant should work with the client to train others in the organiza- tion to take over the role played by the consultant, as Leah did in the Leadership Academy vignette. A

consultant’s disengagement may be abrupt or more gradual, depending on client needs and resources. If the relationship is expected to be terminated gradually, make sure the client builds the ongoing consulting into the budget.

Ensuring Learning Capacity The action research process focuses on promoting learning and change that helps the client diagnose issues, act on them, and evaluate the results. As emphasized in Chapter 5, change and learning go hand in hand. The action research process helps the client build capacity to

Corbis/SuperStock

The action research cycle is terminated when the implementation has been a success. Ending the consulting relation- ship smoothly and ensuring customer satisfaction helps drive repeat business.

bie81554_06_c06_181-216.indd 209 6/30/14 1:00 PM

Section 6.5Concluding the Action Research Process

solve future problems. When the client has capacity to follow the action research process and continue learning, the client is ready to tackle future challenges without your help.

Recognizing Appropriate Timing It is the consultant’s job to monitor both the client and the change implementation to assess when the organization has the capacity to continue without help. Clients may resist termina- tion because they have become overreliant on the consultant. You can avoid this dependency by striking a collaborative relationship from the beginning. When it is time to terminate, it makes sense to make a grand gesture to signal the relationship has ended. You might want to plan an event with the client, such as presenting a final report, celebrating the key stake- holders, or publishing some type of document that tells the organization’s story. The Leader- ship Academy consultancy culminated in the graduating cohort and the new cohort coming together to celebrate and Leah turning over the management reins to the internal consultants.

Verifying Client Satisfaction We have discussed the importance of being authentic with your client and completing the business of each phase of the action research project, as Block (2011) recommended in his classic consulting text. Those key roles remain relevant right up until the end. That is, a consultant should continue to ask the client questions such as: “Are you getting what you need from me?” “Is this outcome what you expected?” “What are you concerned about in the future?” “Can you maintain this change on your own?”

When you have verified that the client is happy with the OD effort, you can move toward ter- mination. If the client is unhappy, however, work remains.

Planning for Postconsult Contact Although the consulting relationship will end at some point, it is advisable to have a plan for consulting after the intervention has been deemed a success. Clients may run into trouble in the future or need their questions answered. It is thus wise to develop a follow-up and mainte- nance plan with the client that involves periodic checking to make sure the change is on track. Agree on a minimal support maintenance plan such as periodic meetings or reports. Leah, for example, continued to periodically touch base with the Leadership Academy to ensure things were functioning smoothly after her departure.

Although it would be considered a failure if a consultant had to return to solve the problem he or she was initially contracted for, it is likely that the client will face new challenges and seek out help. Ensuring that there is an open communication channel and guidelines for future engagement can put both parties at ease.

Recycling

There are times in the consultancy when termination or disengagement is not a good option for the client. This is true of interventions that are designed to repeat over time. In

bie81554_06_c06_181-216.indd 210 6/30/14 1:00 PM

Section 6.5Concluding the Action Research Process

the Leadership Academy vignette, for example, the project was designed to repeat annually. Although Leah terminated her involvement with the project, she trained internal consultants to carry on her role, effectively repeating or recycling the action research process.

Recycling can also be an option when the client seeks additional changes beyond the change that has already been effectively implemented. For example, consider a company that started providing executive coaching for its emerging leaders. The program was so successful that the company decided to offer training that brought some of the coaching principles to a wider audience.

Recycling can also occur when the intervention was only moderately successful or even failed. An evaluation can usually expose an intervention’s shortcomings and help the organization identify adjustments or new interventions. One example would be an organization that did not follow the action research process and implemented a random intervention that was not clearly linked to the problem, such as requiring employees to attend training unrelated to the organization’s needs. Or the organization might have implemented something similar to the Leadership Academy but failed to prepare upper management to deal with highly enthusi- astic emerging leaders clamoring to make changes that challenge the status quo. In this case a recycled intervention would target upper management members and help them become more equipped to mentor up-and-coming employees.

Regardless of whether the OD intervention and action research process has been terminated or recycled, when your client has been successful at changing and has learned new ways of thinking and behaving, you have completed successful OD. Ultimately, OD seeks to build capacity in individuals and organizations so they can problem solve without your help. That is the mark of an effective action research process.

Take Away 6.5: Concluding the Action Research Process

• The action research process concludes by being terminated or recycled. The process is terminated when the change is successfully implemented. There is no longer a need for a consultant. The client has built capacity to use the action research process on future problems.

• The action research process is recycled when termination is not a good option for the client. For example, there may be a desire to expand or improve the implementation. There may also be a need to continue working with the consultant if the intervention repeats over time. In some cases, however, the intervention has failed and it is time to consider a new approach.

bie81554_06_c06_181-216.indd 211 6/30/14 1:00 PM

Summary and Resources

Chapter Summary • Evaluation is a process of assessing, adjusting, and terminating or recycling the

intervention based on data and subsequence decisions. • The purpose of evaluation is to make data-based decisions about an intervention’s

quality, appropriateness, and effectiveness. • Evaluation can be formative or summative. Formative evaluation is concerned with

improving and enhancing an OD process as it is underway, rather than judging its merit. Summative evaluation occurs after the implementation is complete and ascer- tains whether the change accomplished the desired outcomes and impact. Both pro- vide valuable ways to assess the intervention before, during, and after it has occurred.

• Multiple frameworks for conducting evaluation exist. The best known is Kirkpat- rick’s 4-level evaluation framework. This model measures reaction, learning, behav- ior, and results.

• Other models of evaluation include Hamblin’s 5-level model; Preskill and Torres’s evaluative inquiry; Brinkerhoff ’s 6-stage model; and the input, process, output, and outcomes model.

• Planning and performing the evaluation involves several steps, the first of which is determining the evaluation’s purpose. Articulating a clear purpose gives the evalua- tion focus and helps identify appropriate participants, measures, and methods.

• It is advisable to anticipate client resistance both to conducting the evaluation and hearing the results. Consultants can write evaluation protocols into their initial con- tract. When you notice resistance to evaluation, act quickly to defuse the resistance, address the concerns, and help the client use information most effectively.

• The action research process is recycled when termination is not a good option for the client. For example, there may be a desire to expand or improve the implementa- tion. There may also be a need to continue working with the consultant if the inter- vention repeats over time. In some cases, however, the intervention has failed, and it is time to consider a new approach.

bie81554_06_c06_181-216.indd 212 6/30/14 1:00 PM

Summary and Resources

Think About It! Reflective Exercises to Enhance Your Learning 1. The chapter began with a vignette about a Leadership Academy for a state public

health agency, which featured both formative and summative evaluation. Have you been in a situation where an evaluation occurred? If so, can you recall the different types of evaluation? If you have not experienced a formal evaluation, how might you go about evaluating a change you experienced?

2. Think about a change you have implemented. It could be personal, like changing a habit or starting something new, or professional, like taking on a new responsibil- ity or position or meeting a challenge. Conduct a formative evaluation (focusing on what you did or could have improved on) and a summative evaluation (in which you judge the effectiveness and impact) on the change.

3. Which evaluation framework presented in this chapter was the most appealing to you? Why?

4. Reflect on how you might go about evaluating a recent change in your organization using one of the data collection methods outlined in the chapter.

5. Recall a time you have resisted change, especially organization change. How could a consultant or the organization have helped you become more accepting?

Apply Your Learning: Activities and Experiences to Bring OD to Life 1. Imagine an organization hires you as an external consultant. It needs you to imple-

ment a new recruitment and retention process aimed at hiring a more diverse work- force. How would you go about evaluating whether the change was successful?

a. What is the evaluation’s purpose? b. What steps will you follow to conduct the evaluation? c. What level(s) do you hope to evaluate, as per the Kirkpatrick framework? d. What data collection method will you use?

2. Identify a process, practice, or performance standard you would like to improve and plot how you would go about benchmarking it.

3. Evaluation may be an afterthought in many interventions. How would you ensure evaluation is integrated into a change effort you are involved with or leading? How might you curb resistance?

4. Identify an intervention in which you have participated at work and evaluate it according to Kirkpatrick’s 4-level framework:

a. reaction b. learning c. behavior d. results

bie81554_06_c06_181-216.indd 213 6/30/14 1:00 PM

Summary and Resources

5. Plan an evaluation according to its:

a. purpose b. measures c. information sources and methods d. analysis and feedback e. future action f. political issues

6. If you have ever participated in an OD intervention led by a consultant, identify what types of evaluation were conducted. How well did the consultant do, based on the principles presented in this chapter?

7. Have you experienced a failed OD intervention that had to be recycled? If so, use the information presented in this chapter to diagnose what went wrong.

Additional Resources

Media

Michael Quinn Patton Reflects on Evaluation

https://www.youtube.com/watch?v=7zWHK4Qtvak

Kirkpatrick Model: Should I Always Conduct a Level 1 Evaluation?

https://www.youtube.com/watch?v=dVnBE2W7qAI&list=PL3D286DBB9370267D

Kirkpatrick Model: Monitoring Level 3 to Maximize Results

https://www.youtube.com/watch?v=r-qF4kJrTiI&list=PL3D286DBB9370267D

Web Links

The American Evaluation Association (AEA), an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. http://www.eval.org

Online Evaluation Resource Library, a useful site that collects and makes available evalua- tion plans, instruments, and reports that can be used as examples by principal investigators, project evaluators, and others. http://oerl.sri.com

Centers for Disease Control and Prevention Program Evaluation Resources, which offers a plethora of useful content. http://www.cdc.gov/EVAL/resources/index.htm

bie81554_06_c06_181-216.indd 214 6/30/14 1:00 PM

https://www.youtube.com/watch?v=7zWHK4Qtvak

https://www.youtube.com/watch?v=dVnBE2W7qAI&list=PL3D286DBB9370267D

https://www.youtube.com/watch?v=r-qF4kJrTiI&list=PL3D286DBB9370267D

http://www.eval.org

http://oerl.sri.com

http://www.cdc.gov/EVAL/resources/index.htm

Summary and Resources

Key Terms

adjusting processes The process of chang- ing the OD intervention once assessment data have been collected and analyzed.

archival data Using existing records such as employment records, production figures, or quarterly reports as a data source when collecting evaluation data.

assessing changes The process of gathering and analyzing data related to the learning and change associated with an OD intervention.

behavior Kirkpatrick’s level 3 evaluation, which measures how participants perform differently as a result of the intervention.

benchmarking When an organization com- pares its business practices, processes, and performance standards to other organiza- tions that are best in class.

checking A data-based evaluation to assess whether an intervention had the intended result.

evaluation A data-based checking to assess whether an intervention had the intended result.

formative evaluation Assessments of an intervention before or during its implemen- tation geared toward improving the process.

learning Kirkpatrick’s level 2 evaluation, which measures what skills and knowledge participants gained from the intervention.

observation Watching day-to-day opera- tions to collect evaluation data.

reaction Kirkpatrick’s level 1 evaluation, which measures how well participants liked the intervention.

recycling The process of repeating or revis- ing the action research process when further interventions are desired or the initial inter- vention has failed.

results Kirkpatrick’s level 4 evaluation, which measures how the bottom line was impacted by the intervention.

summative evaluation Assessment that is done once the intervention is completed to judge whether it attained its goals and addressed the problem and to make future decisions about funding and continuance.

surveys and questionnaires Evaluation data collection method that uses instru- ments that participants complete to provide feedback on the intervention.

terminate To disengage from a consulting relationship with an organization at the end of the action research process.

bie81554_06_c06_181-216.indd 215 6/30/14 1:00 PM

bie81554_06_c06_181-216.indd 216 6/30/14 1:00 PM