FINAL PAPER - Creating an Organizational Development Proposal

06CH_Bierema_Organization_2e.pdf

Home >Business & Finance homework help >Operations Management homework help >FINAL PAPER - Creating an Organizational Development Proposal

Action Research: The Checking Phase

DragonImages/iStock/Thinkstock

Learning Outcomes After reading this chapter, you should be able to:

• Describe evaluation according to how it is defined and what steps encompass it.

• Identify the types and categories of evaluation.

• Examine different frameworks of evaluation.

• Determine how to plan and perform an evaluation.

• Explore strategies for concluding the action research process, including terminating the consultant–client relationship or recycling the intervention.

In Chapter 5, we learned about the Public Health Leadership Academy, which was founded by a major university using funds from a federal grant to promote leadership development among public health employees in a southern state. The project involved developing a Leadership Acad- emy for mid-level managers who exhibited potential to advance to higher levels of public health leadership in the state. The intervention was in response to a long-term need based on previous analyses of the state’s public health agency, including succession planning. This need had existed for many years because there were not enough public funds available to provide a comprehensive program. The grant finally created the opportunity to deliver this much-needed program. James (the client) worked with Leah (the external consultant) to plan and implement the program.

James and Leah engaged in action research to collect and analyze data about the needs of the target population (mid-level public health managers) using interviews and surveys to deter- mine the content of the courses that would be offered in the Leadership Academy. The project had a 2-year implementation timeline, with year 1 focused on planning and year 2 devoted to implementation. Evaluation would be ongoing and continue past year 2 with a new cohort start- ing in year 3, staffed by internal consultants.

During the year 1 planning phase, James and Leah were very involved in collecting data to inform the content and process of the Leadership Academy. They continually stopped to reflect on their decisions, plans, and processes and made adjustments to each as the project unfolded. They also piloted the first session among a small group of advisors to the Leadership Academy to make sure their design would resonate with the participants. They made more changes follow- ing the pilot to improve the program.

During year 2, 25 managers chosen for the academy participated in monthly leader- ship development experiences and semi- nars. The Leadership Academy began in September with these 25 managers, who had been competitively selected from across the state. The participants convened at a resort, and the program was kicked off by high-level state public health officials. The first session lasted 3 days, during which time the participants received the results of a leadership styles inventory, listened to innovative lectures and panels on leader- ship, planned an individual leadership proj- ect in their districts, and engaged with each other to develop working relationships. The academy continued meeting monthly for a year and focused on a range of topics related to leadership that were prioritized based on prior data col- lection. The grant provided for an evaluator, so data was collected at each meeting.

The first 2 years of the project involved ongoing assessment of the academy’s plans and imple- mentation, followed by appropriate adjustments. James and Leah included cycles of assessment and adjustment as a regular part of their agenda and conversation.

The evaluator observed all of the sessions and sent out formal evaluations after each monthly session. During the sessions, facilitators regularly asked participants to provide feedback. For example, they were asked to respond to questions like, “How did that exercise work for you?”

PeopleImages/E+/Getty Images Plus The Leadership Academy is off to a lively start.

Section 6.1Defining Evaluation in Action Research

“How are you looking at this now?” and “How could we do this better?” The evaluation data con- tributed to changes to the planned curriculum and program activities. For example, the partici- pants took an inventory to assess leadership style and wanted to spend more time on the topic, so the next month’s agenda was adjusted to accommodate the request. Participants complained that the sequencing of topics was not logical, so the agenda for the second cohort to follow in year 3 of the project was adjusted.

The first cohort graduated at its final session, during which the cohort welcomed the mem- bers of the new cohort. Leah had worked with an internal team of consultants throughout the implementation, and the team was ready to take over the facilitation with the second cohort. Following the event, Leah met with James and the new team to tie up loose ends and make the transition. She met periodically with James during the third year to ensure that the Leadership Academy was running smoothly.

As this vignette illustrates, although checking is the third phase of the action research process, it takes place during the planning and doing phases as well. This chapter focuses on checking, which is a data-based evaluation to assess whether an intervention had the intended result.

6.1 Defining Evaluation in Action Research The model of action research used in this book has three phases: planning, doing, and check- ing. See Table 5.1 in Chapter 5 for a review of each phase.

The final phase of action research, checking, involves three steps. First, the consultant and client gather data about the key changes and learning that have occurred. This step is known as assessing changes. Next, the consultant uses this data to assess if the intended change occurred. Was the change implementation effective? Were the proposed outcomes met? As a result of this assessment, the consultant adjusts the intervention accordingly. This step is known as adjusting processes. The third step is to terminate the OD process or repeat it to correct or expand the intervention (known as recycling). Assessment, adjustment, and terminating or recycling are collectively known as evaluation of the action research process.

Purposes of Evaluation The overall purpose of conducting an eval- uation is to make data-based decisions about the quality, appropriateness, and effectiveness of OD interventions. Evalua- tion helps us determine whether an inter- vention’s intended outcomes were real- ized and assess and adjust the intervention as needed. Evaluation helps ensure accountability and knowledge generation from the intervention.

An evaluation creates criteria of merit, con- structs standards, measures performance, compares performance to standards, and

zimmytws/iStock/Thinkstock Evaluation makes judgments about the effectiveness and impact of OD interventions through the analysis of data such as “employee satisfaction” surveys.

Section 6.1Defining Evaluation in Action Research

synthesizes and integrates data into a judgment of merit or worth (Fournier, 1995). Evalua- tion findings help render judgments, facilitate improvements, or generate knowledge (Patton, 1997). Evaluations used to render judgments focus on accountability for outcomes such as holding management responsible for making changes in leadership. Improvements concen- trate on developmental processes such as creating new learning and growth. Knowledge gen- eration emphasizes academic contributions such as new insights that may change a process.

Establishing a Benchmark To illustrate how evaluation helps OD consultants assess and adjust an intervention, let us consider an organization that has conducted survey research to assess employee satisfaction. The first year creates a benchmark (when an organization compares its business processes, practices, and performance standards to those of other organizations that are considered best in class) that can be used in future evaluations. Further, let us imagine that employee satisfac- tion is at a moderately satisfied level the first time it is measured. When the survey research instrument on employee satisfaction is replicated in future years, the level of satisfaction will be compared with the original baseline to evaluate whether the organization is doing worse, the same, or better than it had originally. The evaluation can help the organization identify key changes and learning that occurred as a result of the intervention. Then the organization can adjust practices accordingly.

The American Productivity and Quality Center developed a benchmarking definition repre- senting consensus among 100 U.S. companies:

Benchmarking is a systematic and continuous measurement process; a pro- cess of continuously measuring and comparing an organization’s business process against business process leaders anywhere in the world to gain infor- mation, which will help the organization take action to improve its perfor- mance. (as cited in Simpson, Kondouli, & Wai, 1999, p. 718)

See Who Invented That? Benchmarking to read about the origins of benchmarking.

Benchmarking is a specific type of action research, but the process can also be applied during OD intervention evaluations. There are several types of benchmarking (Ellis, 2006):

Who Invented That? Benchmarking The exact derivation of the term benchmarking is unknown. It is thought to have possibly origi- nated from using the surface of a workbench in ancient Egypt to mark dimensional measure- ments on an object. Alternatively, surveyors may have used the term to refer to the process of marking cuts into stone walls to measure the altitude of land tracts, and cobblers may have used it to describe measuring feet for shoes (Levy & Ronco, 2012).

Benchmarking in U.S. business emerged in the late 1970s. Xerox is generally considered the first corporation to apply benchmarking. Robert Camp (1989), a former Xerox employee, wrote one of the earliest books on benchmarking. Camp described how U.S. businesses took their market superiority for granted and were thus unprepared when higher-quality Japanese goods disrupted U.S. markets.

Section 6.1Defining Evaluation in Action Research

• Competitive: Uses performance metrics to assess how well or poorly an organization is performing against direct competitors, such as measuring quality defects between the companies’ products

• Comparative: Focuses on how similar processes are handled by different organiza- tions, such as two organizations’ recruitment and retention activities

• Collaborative: Involves sharing knowledge about particular activity between compa- nies, with the goal of learning

Bogan and English (2014) described benchmarking types a bit differently, noting the activity might focus on processes (e.g., order fulfillment or billing processes, similar to comparative), performance (e.g., comparing competitive positions, similar to competitive), and strategy (e.g., identifying winning tactics across an industry, perhaps similar to collaborative). Almost any issue of interest can be benchmarked, including processes, financial results, investor per- spectives, performance, products, strategy, structure, best practices, operations, and manage- ment practices. Benchmarking could be part of the data collection process in OD, an interven- tion, or the basis of an evaluation. Table 6.1 shows typical benchmarking steps.

Table 6.1: Typical benchmarking process

Benchmarking step Example

1. Identify process, practice, method, or product to benchmark.

Identifying best practices for recruiting and retaining a diverse work force

2. Identify the industries with similar processes.

Finding the companies that are best at retaining a diverse work force, even those in a different industry

3. Identify organization leaders in a target area.

Selecting the organizations against which to benchmark

4. Survey the selected organizations for their measures and practices.

Sending a survey to the target companies asking for information on issues such as turnover and hire rates, formal retention programs (e.g., orientation, development), management training, and rewards

5. Identify best practices. Analyzing data to identify best practices to implement. Analysis depends on the type of data collected—that is, whether it is statistical (quantitative data), such as from a survey of employees on attitudes about diversity, or interpretive (qualitative data), such as from inter- views with employees who quit.

6. Implement new and improved practices.

Implementing best practices, such as new recruitment and retention strategies, affinity groups, or rewards for managers who develop a diverse staff

Other Purposes of Evaluation Caffarella (1994) and Caffarella and Daffron (2013) identified 12 specific purposes of evalu- ation data. Evaluation helps to

1. adjust the intervention as it is being made in terms of design, delivery, management, and evaluation;

2. keep employees focused on the intervention’s goals and objectives; 3. provide information to inform the continuation of the intervention;

Section 6.1Defining Evaluation in Action Research

4. identify improvements needed to design and deliver the intervention; 5. assess the intervention’s cost-effectiveness; 6. justify resource allocations; 7. increase application of participants’ learning by building in strategies that help them

transfer learning back to the organization; 8. provide data on the results of the intervention; 9. identify ways to improve future interventions;

10. cancel or change an intervention that is poorly designed or headed for failure; 11. explore why an intervention fails; and 12. provide intervention accountability.

Moreover, during the planning phase, evaluation can help consultants assess needs and make decisions about how best to intervene. The Leadership Academy’s goal was to improve lead- ership, but James and Leah had to assess the content that would be most appropriate for lead- ership in public health. Then, when the participants were selected, they had to make further assessments to ensure the program was relevant to the participants’ particular needs.

Evaluation may also help test different theories and models of addressing the problem. In the case of the Leadership Academy, James and Leah based their interventions on theories and models of leadership. They threw out what did not resonate with the participants or work well during sessions and revised the program for the second cohort.

Evaluation also helps monitor how the intervention is going during implementation so it can be adjusted accordingly. Such adjustments occurred throughout the Leadership Academy implementation over the course of a year.

Finally, evaluation helps determine whether the intervention goals were met and what impact the change had on individuals and the organization. Measuring this type of impact may require more longitudinal study than other types of evaluation. The evaluation of impact helps con- sultants decide whether to extend the intervention, change it, or abandon it altogether. The Leadership Academy will be continually reevaluated as new cohorts participate each year.

Clearly, evaluations have the potential to accomplish a variety of goals. Throughout the OD process, it is critical to stay focused on an evaluation’s purpose. Have you experienced any of the evaluation activities discussed here?

Steps in Evaluation Just as with action research models, so too are there many approaches to undertaking evalu- ation. That is, there are different ways to model the steps in the process. Two are discussed here.

Evaluation Hierarchy Rossi, Lipsey, and Freeman (2004) offered an evaluation hierarchy that recognizes the impor- tance of engaging in evaluation from the beginning of the action research process. That is, evaluation should occur during the initial client contacts, be built into the plan for interven- tion, and be ongoing throughout the implementation, prior to the formal assessment of the

Section 6.1Defining Evaluation in Action Research

intervention’s impact, cost, and efficiency. Doing evaluation is a matter of conducting a mini- action research project.

Caffarella’s Systematic Program Evaluation Caffarella (1994) outlined the steps generally taken during an evaluation. Her steps have been modified to address key OD issues in the following points. Caffarella’s steps are intended to be sequential under ideal conditions, although reality may be quite different. Note that Caffarella has proposed a lot of steps. She has elaborated more on the steps than some other models but still follows an action research process.

1. Secure support for the evaluation from stakeholders such as the client and key management. This step should be a provision of the contract, as discussed in Chap- ter 3. It is the process of getting management to commit to the time and resources needed to evaluate the process, as well as being willing to pay attention to the findings.

2. Identify individuals who can be involved in planning and overseeing the evaluation, such as the participants, management, client, and others affected by the interven- tion. This is usually led by the consultant and client and would involve employees who are engaged in the implementation. It could also involve those affected by the change who did not necessarily participate in it, such as customers or suppliers.

3. Define the evaluation’s purpose and how the results will be used. This step is elaborated on in a later section of this chapter. The evaluation’s focus should be determined and then built accordingly. For example, is it aimed at improving a process or judging an outcome? Does it pertain to planning the intervention or the intervention itself ? Is it aimed at assessing adherence to budget or performance outcomes?

4. Specify what will be judged and formulate the evaluation questions. This step is driven by the evaluation’s purpose. If you decide to evaluate how satisfied employ- ees are with a new performance appraisal process, questions should relate to that change and be used to judge whether it was effective and should continue.

5. Determine who will supply the evidence, such as participants, customers, manage- ment, employees, or others affected by the intervention.

6. Specify the evaluation strategy in terms of purpose, outcomes, timeline, budget, methods, and so forth.

7. Identify the data collection methods and timeline. Data collection was discussed extensively in Chapter 5. The selected methods should match the evaluation’s purpose.

8. Specify the data analysis procedures and logistics (covered in Chapter 5). However, the analysis should be focused on making decisions and changes to the interven- tion, not on diagnosing the problem.

9. Define the criteria for judging the intervention. This can be somewhat subjec- tive unless the metrics are defined in advance. For example, if the intervention were aimed at improving employee retention, would a consultant measure simply whether it improved or look for a certain benchmark (such as 10%) to deem it successful?

10. Complete the evaluation, formulate recommendations, and prepare and present the evaluation report. These steps mirror the data analysis steps presented in Chapter 5 and the feedback meeting strategies in Chapter 4.

Section 6.1Defining Evaluation in Action Research

11. Respond to recommendations for changes as appropriate.

Adapted from Planning Programs for Adult Learners: A Practical Guide for Educators, Trainers, and Staff Developers (pp. 255– 256), by R. S. Caffarella, 1994, San Francisco, CA: Jossey-Bass. © John Wiley & Sons.

Table 6.2 compares the action research model used in this book to Rossi, Lipsey, and Free- man’s and Caffarella and Daffron’s evaluation steps. These models vary in terms of detail and number of steps, but they essentially follow the three phases of action research: planning, doing, and checking. Evaluation is essentially conducting research within an action research process, as shown by these three examples.

Table 6.2: Comparing the action research model to evaluation models

Action research model

Rossi, Lipsey, and Free- man evaluation model Caffarella and Daffron evaluation model

Planning 1. Assess intervention cost and efficiency.

2. Assess intervention outcome or impact.

1. Secure support for the evaluation from stakeholders.

2. Identify individuals who can be involved in planning and overseeing the evaluation.

3. Define the evaluation’s purpose and how the results will be used.

4. Specify what will be judged and formulate the evaluation questions.

5. Determine who will supply the evidence. 6. Specify the evaluation strategy in terms of purpose,

outcomes, timeline, budget, methods, and so forth. 7. Identify the data collection methods and timeline. 8. Specify the data analysis procedures and logistics. 9. Determine the specific timeline and the budget

needed to conduct the evaluation.

Doing 3. Assess intervention implementation.

10. Complete the evaluation, formulate recommendations, and prepare and present the evaluation report.

Checking 4. Assess intervention design and theory.

5. Assess need for the intervention.

11. Respond to recommendations for changes as appropriate.

Caffarella and Daffron’s steps are comprehensive, covering the key tasks that must be com- pleted during an intervention’s evaluation. However, it may not always be possible to follow these clearly articulated steps; evaluation can be unpredictable and may present challenges that are often unanticipated. For example, if an implementation has been challenging, a cli- ent may balk at the evaluation out of fear of receiving negative feedback; on the other hand, employees may be reluctant to participate if trust levels are low. Thus, it helps to pay atten- tion to relevant dynamics and expect the unexpected.

Evaluation provides critical information about an intervention’s impact both during and after its implementation. Thus, no matter what model is followed for performing an evaluation, it is essential to begin planning it before the intervention is well underway. A consultant’s job is

Section 6.2Types and Categories of Evaluation

to ensure that evaluation is integrated into the OD process from start to finish. Unfortunately, evaluation is often overlooked in favor of wanting simply to take action on the problem, and too many consultants consider their work finished once the intervention has occurred. In other cases, consultants go about evaluation haphazardly. If they cannot demonstrate that their action was effective, however, they risk undermining their client’s confidence in the OD effort, fail to permanently solve the problem, and put themselves at risk of repeating similar mistakes on future assignments.

6.2 Types and Categories of Evaluation Theorists have proposed different types and categories for evaluation. This section identifies some of these different approaches.

Types of Evaluation Evaluation can be either formative or summative, depending on the intervention’s goal (Scriven, 1967, 1991a, 1991b). Scriven is considered a leader in evaluation; you can view one of his lectures by visiting the media links provided at the end of this chapter.

Formative Evaluation Making changes to an implementation that is already in progress is called doing a forma- tive evaluation. Formative evaluation is concerned with improving and enhancing the OD process rather than judging its merit. The following types of questions might be asked when conducting a formative evaluation:

• What are the intervention’s strengths and weaknesses? • How well are employees progressing toward desired outcomes? • Which employee groups are doing well/not so well? • What characterizes the implementation problems being experienced? • What are the intervention’s unintended consequences? • How are employees responding? What are their likes, dislikes, and desired changes? • How are the changes being perceived culturally? • How well is the implementation conforming to budget? • What new learning or ideas are emerging?

For example, consider an intervention focused on changing reporting relationships as part of a work redesign in a manufacturing plant. A consultant might discover that some of the new arrangements do not make sense once implemented. These might therefore be modified as

Consider This Think of an evaluation in which you have participated. How well did it follow the plan–do– check steps?

Section 6.2Types and Categories of Evaluation

the work redesign progresses. Asking questions pertaining to the problems, the employees’ perspectives, their likes and dislikes, and so forth yields information that helps tweak and improve the process. Formative evaluation is generally ongoing throughout the implementation.

Summative Evaluation Undertaking evaluation at the end of the OD implementation, with the goal of judging whether the change had the intended out- comes and impact, is called summative evaluation. Summative evaluation is also known as outcome or impact evaluation because it allows the intervention’s overall effectiveness to be ascertained. A consul- tant can then decide whether to continue or terminate it (Patton, 1997). The following types of questions might be asked by the consultant, management, or an external evaluator when conducting a summative evaluation:

• Did the intervention work? • Did the intervention satisfactorily

address the performance gap? • Should the intervention be continued or expanded? • How well did the intervention stick to the budget?

Summative evaluations should follow four steps: 1. Select the criteria of merit—what are the sought metrics? 2. Set standards of performance—what level of resolution is sought? 3. Measure performance—conduct the evaluation. 4. Synthesize results into a judgment of value. (Shadish, Cook, & Leviton, 1991,

pp. 85–94)

Adequate levels of both formative and summative evaluation must be incorporated into the OD process. Failure to conduct formative evaluation leads to missed opportunities to adjust and improve on the implementation as it is in progress. Omitting the summative evaluation means never learning the intervention’s outcomes and impact or lacking adequate data on which to base future decisions.

Consider This What types of formative evaluation have you participated in or observed?

Vgajic/E+/Getty Images Plus An OD consultant must always assess interventions to learn the outcomes and impact, which serve as a foundation for future decisions.

Section 6.2Types and Categories of Evaluation

Cervero’s Evaluation Categories Cervero (1985) identified seven categories of evaluation for planners of educational pro- grams that have relevance for OD. His list has been adapted for OD interventions in terms of categories of evaluation:

1. Intervention design and implementation. This could be either formative or summa- tive, because the design and intervention are assessed for fit and impact. Imagine implementing a new performance appraisal process. Formative evaluation might involve piloting the evaluation and evaluating how well it worked for both employees and supervisors. The performance appraisal would then be modified and imple- mented. Summative evaluation in this case might examine whether the new perfor- mance appraisal process improved performance, satisfaction, and learning.

2. Employee participation. This type of evaluation assesses employees’ level of involve- ment in the intervention. This could also be formative or summative. In the case of performance evaluation, a consultant might examine the level of involvement and seek feedback from employees. A summative evaluation might evaluate whether the level of employee participation was adequate and whether it yielded positive outcomes.

3. Employee satisfaction. This type of evaluation assesses employees’ level of satis- faction in the intervention. This could also be formative or summative. In the case of performance evaluation, a consultant might examine the level of satisfaction with the new performance appraisal or its implementation process. A summative evaluation might evaluate how satisfied employees are once the new performance appraisal system is in place.

4. Acquisition of new knowledge, skills, and attitudes. This type of evaluation measures learning during and after the intervention and could also be formative or summa- tive. In the case of performance evaluation, a consultant might examine the level of involvement and seek feedback from employees. A formative evaluation during the pilot phase might determine that supervisors lack the skills to effectively implement the new process and give the level of feedback desired. It would allow the consul- tant and client to revise the process and provide adequate training to supervisors. A summative evaluation would assess the level of learning from the new performance appraisal system. This could take the form of employees improving their perfor- mance and supervisors showing demonstrated improvement in their ability to give feedback.

5. Application of learning after the intervention. This category is similar to the previous one, but it is summative; it judges how learning was applied after the intervention. A consultant might look for evidence of how supervisors applied what they learned about giving effective performance feedback to other interactions with employees throughout the year.

6. The impact of the intervention on individuals and the organization. This category could be formative or summative. The formative evaluation might look at how the intervention affects organization life via communication, understanding, participa- tion, and satisfaction. A summative evaluation might look at the overall impact on job satisfaction, financial performance, and retention.

7. Intervention characteristics associated with outcomes. This type of evaluation attempts to link aspects of the intervention to outcomes and is summative. This can be more

Section 6.3Frameworks of Evaluation

difficult to measure if the intervention was complex or had several interventions built into it. A consultant might evaluate how a participative process affected the imple- mentation’s overall success or employee satisfaction.

6.3 Frameworks of Evaluation This section profiles some common evaluation frameworks. There is no “best” framework. Rather, you should find what you are comfortable working with and what effectively fits the situation. Further, you should determine the purpose of the evaluation and use a framework that best facilitates it.

Kirkpatrick’s Four-Level Framework Kirkpatrick’s (2006) four-level evaluation framework can be formative or summative and is one of the most widely known evaluation typologies. It became popular in the 1990s, although Kirkpatrick developed it as a doctoral dissertation at the University of Wisconsin in the 1950s. It was originally created to evaluate training programs, and OD consultants use it to conduct evaluation at a range of points over time. The framework classifies an interven- tion’s outcomes into one of four categories—reaction, learning, behavior, or results (see Table 6.3). An outcome is assigned a category based on how difficult it is to evaluate. For example, the simplest type of outcome to evaluate is participant reaction to the intervention. Thus, this is assigned level 1.

Table 6.3: Kirkpatrick’s four-level evaluation framework

Level Focus Examines

1 Reaction Did participants like the intervention?

2 Learning What skills and knowledge did participants gain?

3 Behavior How are participants performing differently?

4 Results How was the bottom line affected?

Consider This If you have had the opportunity to evaluate organization change efforts, have you experi- enced any evaluative measures on Cervero’s list? Which ones would be most relevant for the Leadership Academy vignette? Can you think of other categories that might be added to Cervero’s list?

Section 6.3Frameworks of Evaluation

Level 1: Evaluation Level 1 measures participant reaction to the intervention. This type of evaluation is sometimes referred to as a “smile sheet” (see Figure 6.1 and Tips and Wisdom: Smile Sheet) because it measures only what participants thought and felt during or immediately after an intervention, and in very simple terms—whether they were satisfied, neutral, or dissatisfied. As an example, consider the Leadership Academy vignette. At this level of evaluation, the consultants might ask the academy participants questions such as “How well did you like the session?” “Was the learning environment comfortable?” “Were the facilitators capable and credible?” and “Did you feel it was time well spent?” This type of evaluation may make facilitators feel good about intro- ducing an intervention, but it does not effectively measure change. Unfortunately, it is the most common form of evaluation employed in organizations, because it is the easiest to measure.

Figure 6.1: Reaction evaluation using a smile sheet

The reaction evaluation sheet shown here is just one way to solicit feedback from an audience.

How are we doing?

* * * * * * * *

* * * * * *

* *

* * * * * *

* *

* * * *

Tips and Wisdom: Smile Sheet If you are facilitating a meeting, workshop, or seminar and want to gauge the participants’ reactions to the event, create an opportunity for them to share feedback. One way to do this is to put a flip chart page with the smiley face symbols shown in Figure 6.1 in the hall outside the session room. Give participants a dot apiece and ask them to place it in the column that best represents how they feel about the session so far. It is important to place the chart out of your eyeshot, so people feel comfortable sharing honest feedback. This offers a snapshot of partici- pants’ reactions and allows you to make adjustments in the moment. You can also use Twitter or other social media to solicit this data.

Section 6.3Frameworks of Evaluation

Level 2: Evaluation Level 2 measures participant learning from an intervention. This level of evaluation assesses whether the intervention helped participants improve or increased their knowledge or skills. At this level, James and Leah might ask the Leadership Academy participants, “What was the key thing you learned from this session?” or “What new skills have you acquired as a result of this experience?” This type of evaluation works best after participants have had a chance to return to their workplace and apply the principles and behaviors they learned (and thus is summative). Participants might also be interviewed or surveyed about learning during the course of the intervention (which would be formative).

Level 3: Evaluation Level 3 measures changes in behavior. This level of summative evaluation assesses whether participants are using their new knowledge or skills in their job. At this level, James and Leah might ask Leadership Academy participants, their supervisors, or subordinates, “To what extent has the leader’s behavior changed and improved as a result of the Leadership Acad- emy?” or “What is the person doing differently now?” Similar to level 2, this type of evaluation is best done postintervention. It can be accomplished by interviewing, observing, or survey- ing participants and stakeholders affected by the intervention.

Level 4: Evaluation Level 4 measures results for the organization. This level of summative evaluation measures how the intervention affected business performance or contributed to the achievement of organization goals. At this level, James and Leah might ask Leadership Academy participants, their supervisors, or subordinates, “How has the organization benefited from the Leader- ship Academy?” “To what degree has employee satisfaction, productivity, or performance improved?” “To what degree has recruitment and retention of employees improved as a result of improved leadership?” “How many promotions have occurred as a result of participating in the academy?” or “How much money has the organization saved due to better leadership decisions?” As these questions indicate, it might be difficult to actually measure and attribute changed leadership to organization results and outcomes.

Kirkpatrick continued to evolve his model and even questioned whether it was a true model or just a guideline; his family members have continued developing it (Kirkpatrick & Kirkpat- rick, 2016). He also expanded his focus to consider an intervention’s cost–benefit ratio and whether it demonstrated a return on investment. Measuring these variables can also present challenges to organizations and OD consultants.

Lawson’s Application of Kirkpatrick’s Framework Building on Kirkpatrick’s framework, Lawson (2016) categorized variables relating to the what, who, when, how, and why of the framework’s use. Her approach has been adapted for OD and is depicted in Table 6.4.

Section 6.3Frameworks of Evaluation

Table 6.4: Applying Kirkpatrick’s framework to OD interventions

Level What Who When How Why

1 Reaction: Did they like it?

Participants During or after the intervention

Smile sheet Determine level of participant satisfaction and need to revise intervention if duplicated

2 Learning: What knowledge or skills were retained?

Participants and consultants

During, before, and/ or after the intervention

Pre- and post- tests, skills applications, role plays, case stud- ies, and exercises

Determine whether consul- tant has been effective in imple- menting interven- tion purpose and objectives

3 Behavior: How are participants performing differently?

Participants, supervisors, subordinates, and peers

3 to 6 months after intervention

Surveys, inter- views, observa- tion, perfor- mance, and appraisal

Determine extent to which partici- pants transferred their learning from the inter- vention to the workplace

4 Results: What is the impact on the bottom line?

Participants and control group

After comple- tion of level 3 assessment

Cost–benefit anal- ysis and tracking operational data

Determine whether the benefits outweigh costs and how the intervention con- tributed to orga- nization goals and strategy

Source: Adapted from The Trainer’s Handbook (4th ed., p. 234), by K. Lawson, 2016, Hoboken, NJ: John Wiley & Sons.

Critiques of Kirkpatrick’s Framework Kirkpatrick’s (1994, 2006) four levels of criteria have been dominant for decades among evaluators. With popularity, however, comes criticism. First, the model has been critiqued for being primarily focused on postintervention realities, that is, for evaluating what happens after the intervention versus incorporating more formative evaluation into the process. The four-level framework also does not help evaluators link causal relationships between out- comes and the levels of evaluation. Finally, the framework does not help evaluators determine what changes equate to the different levels of evaluation or how best to measure each level.

Some authors have suggested expanding the reaction level to include assessing participants’ reaction to the intervention techniques and efficiency (Kaufman & Keller, 1994). One might also try splitting the reaction level to include measuring participants’ perceptions of enjoy- ment, usefulness, and the difficulty of the program (Warr & Bunce, 1995). Kaufman and Keller (1994) recommended adding a fifth level to address the societal contribution and outcomes created by the intervention, which is becoming more popular with the increased emphasis

Section 6.3Frameworks of Evaluation

on corporate social responsibility and sustainability. Phillips (1996) advocated adding a fifth level that specifically addresses return on investment.

Other Frameworks of Evaluation Although the Kirkpatrick model is one of the dominant evaluation models, it is not necessarily the best or most appropriate for every situation. In fact, six decades of evaluation research have not yielded a universal evaluation model (Reio, Rocco, Smith, & Chang, 2017). This sec- tion briefly profiles some lesser-known evaluation models.

Hamblin’s Five-Level Model Similar to Kirkpatrick’s model, this model measures reactions, learning, job behavior, and organizational impacts, as well as a fifth level—the economic outcomes of training. The hier- archy of Hamblin’s (1974) model is more specific than Kirkpatrick’s in that reactions lead to learning, learning leads to behavior changes, and behavior changes have organizational impact. Because of this assertion, Hamblin believed that evaluation at a given level is not meaningful unless the evaluation at the previous level has been performed. It is worth noting that a major criticism of the Kirkpatrick model is his assumption that the four levels are caus- ally linked (Reio et al., 2017). That linkage has never been demonstrated, although Hamblin may be trying to make that linkage.

Preskill and Torres’s Evaluative Inquiry Model Preskill and Torres (1999) contributed a model of inquiry to the literature that uses the evalu- ation process as a learning and development opportunity:

Evaluative inquiry is an ongoing process for investigating and understanding critical organization issues. It is an approach to learning that is fully integrated with an organization’s work practices, and as such, it engenders (a) organiza- tion members’ interest and ability in exploring critical issues using evaluation logic, (b) organization members’ involvement in evaluative processes, and (c) the personal and professional growth of individuals within the organization. (pp. 1–2)

Evaluative inquiry is the fostering of relationships among organization mem- bers and the diffusion of their learning throughout the organization; it serves as a transfer-of-knowledge process. To that end, evaluative inquiry provides an avenue for individuals’ as well as the organization’s ongoing growth and development. (p. 18)

Their definition emphasizes that evaluation is more than simply reporting survey findings. Rather than being event driven, such as sending a survey to participants after the interven- tion is over, evaluation should be an ongoing part of everyone’s job, that is, a shared learning process. Evaluative inquiry should be focused on

• intervention and organizational processes as well as outcomes; • shared individual, team, and organizational learning;

Section 6.3Frameworks of Evaluation

• educating and training organizational practitioners in inquiry skills (action learning);

• collaboration, cooperation, and participation; • establishing linkages between learning and performance; • searching for ways to create greater understanding of the variables that affect orga-

nizational success and failure; and • using a diversity of perspectives to develop understanding about organizational

issues (Preskill & Catsambas, 2006).

Preskill and Torres (1999) identified four learning processes—dialogue, reflection, question- ing, and identifying and clarifying values, beliefs, assumptions, and knowledge—that facilitate three phases of evaluative inquiry: focusing the evaluative inquiry, carrying out the inquiry, and applying learning. The phases are depicted in Table 6.5.

Table 6.5: Preskill and Torres’s evaluative inquiry phases

Phase Description Strategies

At each stage of inquiry, the follow- ing skills are used:

1. Dialogue 2. Reflection 3. Asking

questions 4. Identifying

and clarifying values, beliefs, assumptions, and knowledge

Focusing the evaluative inquiry

• Determine issues and concerns for evaluation

• Identify stakeholders • Identify guiding

questions for the evaluation

• Focused dialogues • Group model building • Open space technology • Critical incidents • Assumption testing

through questioning

Carrying out the inquiry

• Design and implement the evaluation (collect, analyze, interpret data)

• Address evaluative questions

• Develop a database for organization learning

• Literature-based discussions

• Working session to interpret survey results (or other data collected)

• Framing findings as lessons learned

Applying learning

• Identify and select action alternatives

• Develop and implement action plans

• Monitor progress

• Capturing concerns, issues, and action alternatives

• Using technology to facilitate brainstorming

• Developing an action plan

• Solving implementation issues

Brinkerhoff ’s Six-Stage Model This model defines evaluation as the collection of information to facilitate decision making (Brinkerhoff, 1989). Brinkerhoff advocated systemic evaluation that considers the change process holistically. It requires that consultants articulate how and why each training or devel- opment activity is supposed to work, without which comprehensive evaluation is impossible. This model helps to assess whether and how programs benefit the organization; analysis can help trace any failures to one or more of the six stages of Brinkerhoff ’s model:

Section 6.3Frameworks of Evaluation

1. Goal setting (What is the need?): A need, problem, or opportunity worth addressing that could be favorably influenced if someone learned something is identified.

2. Program design (What will work?): A program that teaches the needed topic is cre- ated or, if one already exists, it is located.

3. Program implementation (It is working?): The organization successfully implements the designed program.

4. Immediate outcomes (Did they learn it?): The participants exit the program after successfully acquiring the intended skills, knowledge, or attitudes.

5. Immediate or usage outcomes (Are they keeping and/or using it?): The participants retain and use what they learned.

6. Impacts and worth (Did it make a worthwhile difference?): The organization benefits when participants retain and use what they learned.

Brinkerhoff ’s model provides a platform to think about desired impact at the beginning of the change process and takes a more holistic look at the activities and impacts and how they will be communicated.

Input, Process, Output, and Outcomes Evaluation This model evaluates training programs at four levels (input, process, output, and outcomes) in terms of their potential contribution to the overall effectiveness of a training program (Bushnell, 1990). It is similar to the systems model introduced in Chapter 2 that considers inputs, throughputs, and outputs in organization systems:

1. Inputs: trainee qualifications, instructor abilities, instructional material, facilities, and budget

2. Process: value-adding activities such as planning, designing, developing, and deliver- ing the training

3. Output: trainee reactions, knowledge and skills gained, and improved job performance

4. Outcomes: profits, customer satisfaction, and productivity

This model has the following benefits:

1. It can help determine whether training programs are achieving the right purpose. 2. It can help identify the types of changes that could improve course design, content,

and delivery. 3. It can help determine whether students have actually acquired knowledge and skills.

Steps in the evaluation process:

1. Identify evaluation goals: Determine the overall structure of the evaluation effort and establish the parameters that influence later stages.

2. Develop an evaluation design and strategy: Select appropriate measures, develop a data collection strategy, match data types with experimental designs, allocate the data collection resources, and identify appropriate data sources.

3. Select and construct measurement tools: Select or construct tools that best fit the data requirements and meet criteria for reliability and validity. Examples include

Section 6.4Planning and Performing the Evaluation

questionnaires, performance assessments, tests, observation checklists, problem simulations, structured interviews, and performance records.

4. Analyze the data: Tie the results of the data-gathering effort to the evaluation’s original goals.

5. Make conclusions and recommendations and present the findings.

Evaluation: A Simpler Way? Organizations are accountable for showing results, and OD is no exception. Yet, a comprehen- sive, widespread evaluation model does not exist. Paine (2014) asked if evaluation could be simplified and advocated Brinkerhoff and Mooney’s Business Case Success Method (2008), whereby instead of putting a lot of time and resources into conducting evaluation, the ques- tion becomes, “Did this intervention add value?” Paine recommended asking questions to this end such as the following:

1. What results did we expect? 2. What evidence is available to support or refute the results we obtained? 3. What should we explore more in depth? 4. What additional compelling evidence is available? 5. How can we communicate the results in an understandable way to the organization?

The Brinkerhoff and Mooney model is based on taking a systemic approach that attempts to understand how the change was applied on the job, what the results were of the learning or change, and how the change contributed to organization goals. Paine added a final variable to the model: How did the benefits compare with the costs?

6.4 Planning and Performing the Evaluation Just as with other aspects of the action research process, evaluation requires deliberate plan- ning and buy-in from the client. To plan the evaluation, Cervero (1985) suggested identifying five key factors:

1. What is the purpose of the evaluation? 2. Who needs what information? 3. Are there any practical and ethical constraints? 4. What resources are available? 5. What are the evaluator’s values?

Consider This After reading these evaluation models, you might think they look familiar. Indeed, they follow a mini action research cycle of planning, doing, and checking. If you were planning an evalua- tion, which approach would you use and why?

Section 6.4Planning and Performing the Evaluation

The answers to these five questions offer an evaluation a strong foundation, so it is worth reflecting with the client about them. Once clear on these issues, you can get more specific about evaluation purposes, measurements, information sources, data collection, data analy- sis, feedback, further action, and how to anticipate and manage resistance. These evaluation steps may look familiar, because conducting an evaluation is similar to doing a small-scale action research project. These steps, illustrated in Figure 6.2, will be explored in the next sections.

Figure 6.2: Steps in the checking phase

This figure outlines the steps of a typical evaluation process. Notice that it looks very similar to the action research process of plan, do, check.

Determine evaluation purpose

Conduct summative evaluation

Develop evaluation

plan

Choose and employ data collection

methods

Analyze data and provide

feedback

Identify appropriate evaluation measures

Consider This Apply Cervero’s five questions to an evaluation you have completed or anticipate doing in the future. How have the questions helped you reflect on a past evaluation or plan for a future one?

Section 6.4Planning and Performing the Evaluation

Determine the Purpose of the Evaluation Determining the evaluation’s purpose(s) offers clear focus moving forward. Referring back to Cervero’s categories of evaluation may help pinpoint what is being evaluated. For effective formative evaluation, a consultant should work with the client to determine what needs to be evaluated throughout the action research process, particularly regarding process improve- ment. For example, in the case of the Leadership Academy, formative evaluation consisted of assessing the curriculum for relevance and cost. The examination of a performance appraisal process change earlier in the chapter showed how one might assess employees’ satisfaction with their level of participation in the process or with a pilot phase of the appraisal.

Consultants should also plan the purpose of summative evaluation. Returning to the example of the Leadership Academy, it was essential to find out whether the participants learned and applied the new behaviors and skills, and if so, how their actions affected their organiza- tions. In the case of the performance appraisal process change, the organization wanted to know if it changed supervisor behavior, improved retention, affected learning, and increased performance.

Once an evaluation’s purpose has been decided, a consultant can begin to identify what ques- tions to ask the participants. Table 6.6 offers examples of appropriate questions for different evaluation goals. Questions revolve around needs assessment, intervention conceptualiza- tion, intervention delivery, outcomes, and costs.

Table 6.6: Typical evaluation questions

Questions about the need for the interven- tion (planning and needs assessment)

• What is the nature and magnitude of the problem to be addressed? • What are the characteristics of the individuals/team members/

organization? • What are the needs of the individuals/team members/organization? • What consulting services are needed? • What is the time frame? • What contracting is necessary?

Questions about the intervention’s concep- tualization or design (planning)

• Who is the client? Target population? • What consulting services should be provided? • What is the best intervention?

What are the best delivery systems for the intervention? • How can the intervention identify, recruit, and sustain the intended

participants? • How should the intervention be organized? • What resources are necessary and appropriate for the intervention?

Questions about inter- vention logistics, deliv- ery, and reach (plan- ning and intervention)

• Are intervention goals being met? • Are the intended interventions being delivered to the intended

individuals/teams/organization? • Has the intervention missed participants who need it? • Are sufficient numbers of participants engaged in the intervention? • Are the participants satisfied with the intervention? • Are administrative, organizational, and personnel functions handled well?

(continued on next page)

Section 6.4Planning and Performing the Evaluation

Table 6.6: Typical evaluation questions (continued)

Questions about intervention outcomes (evaluation)

• Are the outcome goals and objectives being achieved? • Does the intervention have beneficial effects on the recipients? • Does the intervention have adverse side effects on the participants? • Are some participants affected more by the intervention than others? • Did the intervention improve the issue/problem?

Questions about intervention cost and efficiency (evaluation)

• Are resources used efficiently? • Is the cost reasonable in relation to the magnitude of benefits? • Would alternative approaches yield equivalent benefits at less cost?

Adapted from Evaluation: A Systematic Approach (7th ed., p. 77), by P. H. Rossi, M. W. Lipsey, and H. E. Freeman, 2004, Thousand Oaks, CA: Sage.

See Case Study: Piloting and Evaluating a New Performance Appraisal Process.

Case Study: Piloting and Evaluating a New Performance Appraisal Process A paper products manufacturing company begins working with a consultant to improve employee retention, because the company has a significantly higher attrition rate compared with other companies it benchmarked. One of the interventions selected during the action research process is to overhaul the way supervisors share feedback with employees; the per- formance appraisal process will become more developmental than punitive, ongoing rather than once a year, with no surprises in the feedback delivered.

Making this change requires developing a new process. A small design team is formed to advise on the new performance appraisal process. It includes the consultant, client, supervisors, and employees. The team designs an intervention that is based on supervisors providing feedback in the moment—that is, when they notice something and want to coach the employee through it. The model also incorporates periodic opportunities for the employee and supervisor to meet, focus on the employee’s developmental plan, and make adjustments as needed. Once the process is developed, the team decides to pilot it with a small department.

Before the pilot begins, the participating employees are briefed on the intervention and con- firm their participation. Both the employees and supervisors are trained in the new method, and the supervisors receive additional training on how to provide coaching and give develop- mental feedback. The design team begins to study the pilot group’s response by questioning whether it is meeting the original need of improving retention and whether the design is best for meeting the needs. To this end, the design team holds informal conversations with the employees and supervisors and asks them questions such as these:

• What do you like about this new process? • What don’t you like about this new process? • How do you perceive these changes? • Will this work if we expand it further, or would you suggest changes? • What have you learned in the process?

The informal conversations yield important data that design team members share during a meeting. There is general support for the idea, but the supervisors do not always feel com- petent in using the new process correctly; they also feel stressed about the time it takes. The

(continued on next page)

Case Study: Piloting and Evaluating a New Performance Appraisal Process (continued) employees are unsure of what the purpose of the periodic meetings is. Some employees and supervisors are resistant, feeling either distrust toward the process or resignation that things will not change.

The design team decides to adjust the process. It provides more support and training to the supervisors on how to coach and share feedback, with the expectation that it will take less time as they become more comfortable with the process. It also assembles the department and models an ideal periodic meeting to touch base on development. The pilot group continues to work with the process for several more weeks, with ups and downs.

Prior to rolling out the process to the wider organi- zation, the design team meets with the pilot depart- ment to see if additional adjustments are needed. The process works better for most but still has logis- tical problems that require further change.

Finally, the company is ready to roll out the changes organization-wide. It starts with a communication plan and provides training to all employees. The design team continues to monitor the process over the next year.

After the plan has been in place for a year, the design team comes together and decides to plan and per- form an evaluation to assess whether the new per- formance appraisal process met its intended goals. The evaluation follows these steps:

1. Describe the purpose of the evaluation. The design team determines that the evalua- tion’s purpose is to assess whether the new performance appraisal process increased employee retention.

2. Identify appropriate evaluation measures. The design team decides to look at three measures: a. Retention comparing the year before the intervention with the year after it b. Employee attitudes c. Supervisor attitudes

3. Choose and employ data collection methods. This depends on the type of data desired and the question the organization wishes to answer. The design team chooses three methods: a. Attrition records to measure the year-to-year comparison of retention rates b. Survey of employees c. Interviews of supervisors

4. Analyze data and provide feedback. Once the data is collected, the consultant and client make the first analysis and then involve the design team. Once the findings are refined, they share them during an open meeting with employees. They also share them with supervisors in a separate meeting to get their feedback and input.

(continued on next page)

FS-Stock/iStock/Getty Images Plus The design team asks employees how they perceive the changes in the communication with supervisors.

Section 6.4Planning and Performing the Evaluation

Table 6.6: Typical evaluation questions (continued)

Questions about intervention outcomes (evaluation)

Questions about intervention cost and efficiency (evaluation)

• Are resources used efficiently? • Is the cost reasonable in relation to the magnitude of benefits? • Would alternative approaches yield equivalent benefits at less cost?

Adapted from Evaluation: A Systematic Approach (7th ed., p. 77), by P. H. Rossi, M. W. Lipsey, and H. E. Freeman, 2004, Thousand Oaks, CA: Sage.

See Case Study: Piloting and Evaluating a New Performance Appraisal Process.

(continued on next page)

1. Describe the purpose of the evaluation. The design team determines that the evalua- tion’s purpose is to assess whether the new performance appraisal process increased employee retention.

(continued on next page)

FS-Stock/iStock/Getty Images Plus The design team asks employees how they perceive the changes in the communication with supervisors.

Section 6.4Planning and Performing the Evaluation

Identify Appropriate Evaluation Measures Once an evaluation’s purpose has been determined, actions to measure it should be identified. This book has covered a range of evaluation techniques. The formative measures that the Leadership Academy team members identified allowed them to recruit a small group of top leaders to critique the curriculum. They followed this with a small pilot session to trouble- shoot and revise the curriculum with an actual audience. Participants evaluated the academy throughout the implementation, and adjustments were made accordingly.

Summative measures could have included any of the examples listed in the Kirkpatrick dis- cussion, such as promotions, employee satisfaction, or customer satisfaction. In the case of the Leadership Academy, measures included improved performance, promotions, a leader- ship project, and team satisfaction. Because a main goal was to cultivate leaders from within the organization, measuring the percentage of participants who were promoted from middle management to executive positions was a key metric for evaluating the intervention’s success.

Choose and Employ Data Collection Methods With the purpose and measures determined, the consultant should identify appropriate sources of information and methods for gathering the information. For example, if you want to measure the results of a customer-service training, you could measure the number of com- plaints, review written complaints, or contact customers. The methods you might use to do this include surveys, documents, or interviews. Table 6.7 offers an overview of data collection methods appropriate for evaluation. The more commonly used methods to collect evalua- tion data include archival data, observations, surveys and questionnaires, assessments, inter- views, and focus groups. Chapter 4 reviewed methods used to conduct analysis or planning— many of them are similar.

Case Study: Piloting and Evaluating a New Performance Appraisal Process (continued) 5. Anticipate and manage resistance to the evaluation. Although the team reviewed worst-

case scenarios for how the organization or employees might resist, the problems are minimal. For example, complaints are similar to those heard throughout the pilot pro- cess from employees who were skeptical that the new performance appraisal process would work.

Once the analysis is complete, the design team presents its findings to top management. The organization now needs to determine how effective the intervention was and whether it would be wise to invest further in it.

Critical Thinking Questions

1. Given your knowledge of evaluation, what are some steps the design team followed in implementing its evaluation?

2. What steps did the design team miss?

Section 6.4Planning and Performing the Evaluation

Table 6.7: Evaluation data collection methods

Evaluation method Description

Interview A conversation with one or more individuals to assess their opinions, observations, and beliefs. Questions are usually determined in advance, and the conversation is recorded.

Questionnaire A standardized set of questions intended to assess opinions, observations, and beliefs that can be administered in paper form or electronically

Direct observation Viewing a task or set of tasks as they are performed and recording what is seen

Tests and simulations Structured situations to assess an individual’s knowledge or proficiency to perform some task or behavior

Archival performance data Use of existing information, such as files, reports, quality records, or perfor- mance appraisals

Product reviews Internal or external evaluations of products or services

Performance reviews Written assessments of individual performance against established criteria

Records and documents Written materials developed by organizations and communities (perfor- mance appraisals, production schedules, financial reports, attendance records, annual reports, company and board minutes, training data, etc.)

Portfolio A purposeful collection of a learner’s work assembled over time that docu- ments events, activities, products, and/or achievements

Cost–benefit analysis A method for assessing the relationship between the outcomes of an educa- tional program and the costs required to produce them

Demonstration Exhibiting a specific skill or procedure to show competency

Pre- and posttests Instruments used to measure knowledge and skills prior to and after the intervention to see if there were changes

Focus groups Group interviews of approximately five to 12 participants to assess opin- ions, beliefs, and observations. Focus groups require a trained facilitator.

Source: Adapted from Planning Programs for Adult Learners: A Practical Guide (3rd ed.), by R. S. Caffarella and S. R. Daffron, 2013, San Francisco, CA: Jossey-Bass.

Archival Data Evaluating the degree of change an intervention produced requires establishing a baseline of existing information from employment records, production figures, or quarterly reports. This information is referred to as archival data (or documents and records). You are not seeking new data but are using existing data to assess the intervention’s effectiveness. Archival data is easily accessible, typically available for no or minimal cost, and useful for providing historical context or a chronology of events, such as employee satisfaction over time. The Leadership Academy team relied on archival data from performance reviews and employee satisfaction surveys to evaluate impact.

Section 6.4Planning and Performing the Evaluation

Observation Data Watching the organization engage in its everyday operations involves observation. Obser- vation is a type of evaluation based on detailed descriptions of day-to-day behaviors that cannot be explored by viewing existing archival records. Examples of observation data might include checklists of meeting-leader behaviors completed by one of the team members, call monitoring forms, listening skills, and body language. Observation did not play an official role in the Leadership Academy evaluation process; however, participants’ supervisors observed the changes they made in their approach to their work and documented these in their per- formance reviews. Data collection by observation can range from routine counting of certain occurrences to writing narrative descriptions of what is being observed.

Surveys and Questionnaires Surveys and questionnaires are helpful evaluation data collections for measuring the inter- vention’s effects. They should be completed by respondents with some experience related to the intervention. In the Leadership Academy vignette, the consultants used surveys to gather participants’ input on their individual leadership styles during the program. Other examples include end-of-course reaction forms or surveys of stakeholders such as customers, employ- ees, or management. Surveys and questionnaires might also be appropriate when evaluators desire new data from multiple individuals who may be dispersed throughout the organiza- tion. Surveys and questionnaires are relatively inexpensive and easy to administer, particu- larly with the use of technology. It is important that these instruments be well constructed; their wording must be unambiguous, and they must be easy to complete.

Paper, Pencil, or Computer-Based Tests Consultants can administer a variety of commercially produced tests to assess the knowledge or skills imparted by an intervention, or they can develop an original test unique to the inter- vention. No matter the type of test employed, the evaluation result is based on the test scores. This type of evaluation works well when trying to determine the quantity and quality of the participants’ education. OD consultants might administer a pretest before an intervention and a posttest afterward, or they might require participants to pass a test to attain a certifi- cate of completion.

Tests should be cautiously designed and prudently administered. First, questions must be written in a way that consistently and accurately measures what was taught. Second, partici- pants may perceive test taking as threatening, especially if the results will be used to make performance appraisal decisions. Therefore, efforts to defuse test apprehension should be built into the process.

This very book uses some of these tools. Teaching you about OD is the intervention, which is executed via concepts presented in book form. Additional interventions take the form of assignments and opportunities to engage with other learners. Pre- and posttests check your prior knowledge on the topic and gauge how well you learned the concepts after you engaged with them.

Section 6.4Planning and Performing the Evaluation

Individual and Focus Group Interviews Chapter 4 discussed interviews and focus groups as effective ways of understanding targeted individuals’ or groups’ views, beliefs, or experiences with the issue under investigation. Both approaches depend on developing well-crafted questions that yield useful information. Inter- views and focus groups should be run by an experienced facilitator. These methods yield rich, qualitative information that includes insights about the intervention, critiques, or success stories. Not all participants react well to these data collection methods, however, and some may not trust the interviewers or the process; they may not feel comfortable enough to be honest. Participants may also say what they think the facilitator wants to hear.

Analyze Data and Provide Feedback Once data has been collected from an appropriate source and via an appropriate method, it needs to be analyzed. Refer to Chapter 5 for a full discussion of how to analyze data. In the Leadership Academy vignette, performance reviews and employee satisfaction data from sur- vey research were analyzed. The team also monitored participants’ leadership projects and promotional advances.

Next, the data analysis should be presented as feedback to key decision makers such as affected employees and management. How to share feedback with a client is covered extensively in Chapter 4, and the same rules apply when sharing evaluation feedback. It is a consultant’s job to determine the feedback meeting’s key purpose and desired outcomes. Does the client need help determining whether to continue the intervention? Modify it? Measure learning or performance? Address unintended consequences of the intervention? Sharing feedback with the client involves determining the focus of the feedback meeting, developing the agenda for feedback, recognizing different types of feedback, presenting feedback effectively, managing the consulting presence during the meeting, addressing confidentiality concerns, and antici- pating defensiveness and resistance.

At any point in the evaluation process, data collection and analysis can prompt the team to decide to change future action. For example, the team might decide to adjust the ongoing pro- cess, continue the process with new interventions, or close the project if the problem is per- manently solved. This is the third step of the evaluation process, defined earlier in the chapter as termination or recycling. It is discussed in detail in the next section.

Anticipate and Manage Resistance to the Evaluation Sometimes, evaluation is resisted by the client, organization, or other stakeholders. Resis- tance and strategies for curbing it were discussed at length in Chapter 5. Resistors may not want to spend more money to learn the results of the intervention. Or the organization may be unwilling to spend the time required to conduct an evaluation and instead want to move on to the next issue. There may be fear about what the evaluation will reveal (perhaps manage- ment failed to implement the changes, or perhaps employee views remain negative). Organi- zation members can also suffer from change fatigue and worry that the evaluation will bring even more change. Of course, such resistance patterns are likely what created problems in the first place, so observing them warrants timely intervention with the client.

Section 6.5Concluding the Action Research Process

Moreover, a consultant should anticipate political issues the evaluation might create. Results of the evaluation can also influence future resource allocations, which could cause trepidation and conflict among organization members. Remaining vigilant as a consultant and working to be authentic and influential is key to navigating the politics of evaluation. See Assessment: Testing Your Change Management Skills.

Some clients may resist doing the evaluation because they are more interested in moving on to the next challenge or opportunity. Or they may not want to subject themselves to poten- tially negative feedback. In this case, the consultant should lay out the benefits of measuring results and learning from both positive and negative feedback. Doing so shows good steward- ship of the time and resources committed to the intervention and provides data to support future initiatives. One way to minimize resistance to evaluation is to make sure it has been addressed during contracting, as outlined in Chapter 3.

Even when there is cooperation and investment, evaluation is not easy. Demonstrating impact and results can be challenging for certain interventions such as improving leadership. Link- ing results to intervention events can also be tricky. Devising appropriate evaluation criteria can be problematic, especially if intervention outcomes were vague from the initial planning. Finally, the client may balk at making judgments about the intervention.

6.5 Concluding the Action Research Process All consulting jobs end. Indeed, your goal as a successful consultant is to become redundant and work yourself out of a job. In our Leadership Academy vignette, Leah terminated her role with the project after 2 years, the first of which focused on planning and the second on imple- mentation. During year 2, she worked with internal consultants who would take over her role leading and facilitating the Leadership Academy in its third year. In this way, she fostered a repetition—called a recycling—of the intervention.

Disengagement or Termination When the client has successfully implemented a change, the OD consultant is no longer needed. At this juncture, the client has become self-reliant and can effectively disengage or terminate the consulting relationship. Working oneself out of a consulting job may at first seem like a bad idea. On the contrary, smoothly disengaging from a client is how to help clients build capacity and also the way to get repeat business as a consultant. Effectively navigating this

Assessment: Testing Your Change Management Skills Visit the following link to take an assessment that provides a good review of change and some insight into resistance. The website offers several resources for learning more about manage- ment skills.

http://www.mindtools.com/pages/article/newPPM_56.htm

Section 6.5Concluding the Action Research Process

stage depends on setting the expectation during contracting, recognizing the appro- priate timing, processing any interpersonal issues between the consultant and the cli- ent, ensuring that the learning is ongoing, verifying that the client is satisfied, and planning for post-consulting contact.

Contracting About Termination A consultant should start setting expecta- tions about disengagement right from the beginning of the consultancy, during con- tracting, as discussed in Chapter 3. There are several things that help disengagement go smoothly. First, the consultant should work with the client to train others in the organization to take over the role played by the consultant, as Leah did in the Leadership Academy vignette. A consultant’s disengagement may be abrupt or more gradual, depending on client needs and resources. If the relationship is expected to be terminated gradually, make sure the client builds the ongoing consulting into the budget.

Ensuring Learning Capacity The action research process focuses on promoting learning and change that helps the client diagnose issues, act on them, and evaluate the results. As emphasized in Chapter 5, change and learning go hand in hand. The action research process helps the client build capacity to solve future problems. When the client has the capacity to follow the action research process and continue learning, the client is ready to tackle future challenges without your help.

Recognizing Appropriate Timing It is the consultant’s job to monitor both the client and the change implementation to assess when the organization has the capacity to continue without help. Clients may resist termina- tion because they have become over-reliant on the consultant. You can avoid this dependency by striking a collaborative relationship from the beginning. When it is time to terminate, it makes sense to make a grand gesture to signal the relationship has ended. You might want to plan an event with the client, such as presenting a final report, celebrating the key stake- holders, or publishing some type of document that tells the organization’s story. The Leader- ship Academy consultancy culminated in the graduating cohort and the new cohort coming together to celebrate and Leah turning over the management reins to the internal consultants.

Verifying Client Satisfaction We have discussed the importance of being authentic with your client and completing the business of each phase of the action research project, as Block (2011) recommended in his classic consulting text. Those key roles remain relevant right up until the end. That is, a consul- tant should continue to ask the client questions such as “Are you getting what you need from

Skynesher/E+/Getty Images Plus The action research cycle is terminated when the implementation has been a success. Ending the consulting relationship smoothly and ensuring customer satisfaction helps drive repeat business.

Section 6.5Concluding the Action Research Process

me?” “Is this outcome what you expected?” “What are you concerned about in the future?” “Can you maintain this change on your own?”

When you have verified that the client is happy with the OD effort, you can move toward ter- mination. If the client is unhappy, however, work remains.

Planning for Post-Consult Contact Although the consulting relationship will end at some point, it is advisable to have a plan for consulting after the intervention has been deemed a success. Clients may run into trouble in the future or need their questions answered. It is thus wise to develop a follow-up and mainte- nance plan with the client that involves periodic checking to make sure the change is on track. Agree on a minimal support maintenance plan such as periodic meetings or reports. Leah, for example, continued to periodically touch base with the Leadership Academy to ensure things were functioning smoothly after her departure.

Although it would be considered a failure if a consultant had to return to solve the problem he or she was initially contracted for, it is likely that the client will face new challenges and seek out help. Ensuring that there is an open communication channel and guidelines for future engagement can put both parties at ease.

Recycling There are times in the consultancy when termination or disengagement is not a good option for the client. This is true of interventions that are designed to repeat over time. In the Lead- ership Academy vignette, for example, the project was designed to repeat annually. Although Leah terminated her involvement with the project, she trained internal consultants to carry on her role, effectively repeating or recycling the action research process.

Recycling can also be an option when the client seeks additional changes beyond the change that has already been effectively implemented. For example, consider a company that started providing executive coaching for its emerging leaders. The program was so successful that the company decided to offer training that brought some of the coaching principles to a wider audience.

Recycling can also occur when the intervention was only moderately successful or even failed. An evaluation can usually expose an intervention’s shortcomings and help the organization identify adjustments or new interventions. One example would be an organization that did not follow the action research process and implemented a random intervention that was not clearly linked to the problem, such as requiring employees to attend training unrelated to the organization’s needs. Or the organization might have implemented something similar to the Leadership Academy but failed to prepare upper management to deal with highly enthusias- tic emerging leaders clamoring to make changes that challenge the status quo. In this case, a recycled intervention would target upper management members and help them become more equipped to mentor up-and-coming employees.

Regardless of whether the OD intervention and action research process has been terminated or recycled, when your client has been successful at changing and has learned new ways

Summary and Resources

of thinking and behaving, you have completed successful OD. Ultimately, OD seeks to build capacity in individuals and organizations so they can problem solve without your help. That is the mark of an effective action research process.

Summary and Resources Chapter Summary

• Evaluation is a process of assessing, adjusting, and terminating or recycling the intervention based on data and subsequent decisions.

• The purpose of evaluation is to make data-based decisions about an intervention’s quality, appropriateness, and effectiveness.

• Evaluation can be formative or summative. Formative evaluation is concerned with improving and enhancing an OD process as it is underway, rather than judging its merit. Summative evaluation occurs after the implementation is complete and ascertains whether the change accomplished the desired outcomes and impact. Both provide valuable ways to assess the intervention before, during, and after it has occurred.

• Cervero’s categories of evaluation show the different approaches to evaluation. It is important to be clear on an evaluation’s purpose at the planning stage. Typical evaluation categories include intervention design and implementation; employee participation and satisfaction; acquisition of new knowledge, skills, and attitudes; application of learning after the intervention; impact of the intervention; and inter- vention characteristics associated with outcomes.

• Multiple frameworks for conducting evaluation exist. The best known is Kirkpat- rick’s four-level evaluation framework. This model measures reaction, learning, behavior, and results.

• Other models of evaluation include Hamblin’s five-level model; Preskill and Torres’s evaluative inquiry; Brinkerhoff ’s six-stage model; and the input, process, output, and outcomes model.

• Planning and performing the evaluation involves several steps, the first of which is determining the evaluation’s purpose. Articulating a clear purpose gives the evalua- tion focus and helps identify appropriate participants, measures, and methods.

• Identifying appropriate evaluation measures is driven by the evaluation’s purpose. If a consultant aims to measure employee satisfaction after a change in leadership, he or she would likely survey employees to assess their satisfaction with the change.

• Once the evaluation purpose and measures have been chosen, the data collection methods should be determined and carried out. Typical methods include surveys, interviews, focus groups, observations, and documents.

• Once the data is collected, it can be analyzed and fed back to the client and other interested stakeholders. This information is important for making decisions about the continuance of the intervention and future funding.

• It is advisable to anticipate client resistance both to conducting the evaluation and to hearing the results. Consultants can write evaluation protocols into their initial con- tract. When you notice resistance to evaluation, act quickly to defuse the resistance, address the concerns, and help the client use information most effectively.

• The action research process concludes by being terminated or recycled. The process is terminated when the change is successfully implemented. There is no longer a

Summary and Resources

need for a consultant. The client has built capacity to use the action research process on future problems.

• The action research process is recycled when termination is not a good option for the client. For example, there may be a desire to expand or improve the implementa- tion. There may also be a need to continue working with the consultant if the inter- vention repeats over time. In some cases, however, the intervention has failed, and it is time to consider a new approach.

Think About It! Reflective Exercises to Enhance Your Learning

1. The chapter began with a vignette about a leadership academy for a state public health agency, which featured both formative and summative evaluation. Have you been in a situation where an evaluation occurred? If so, can you recall the different types of evaluation? If you have not experienced a formal evaluation, how might you go about evaluating a change you experienced?

2. Think about a change you have implemented. It could be personal, like changing a habit or starting something new, or professional, like taking on a new responsibility or position or meeting a challenge. Conduct a formative evaluation (focusing on what you did or could have improved on) and a summative evaluation (in which you judge the effectiveness and impact) on the change.

3. Which evaluation framework presented in this chapter was the most appealing to you? Why?

4. Reflect on how you might go about evaluating a recent change in your organization using one of the data collection methods outlined in the chapter.

5. Recall a time you have resisted change, especially organization change. How could a consultant or the organization have helped you become more accepting?

Apply Your Learning: Activities and Experiences to Bring OD to Life

1. Imagine an organization hires you as an external consultant. It needs you to imple- ment a new recruitment and retention process aimed at hiring a more diverse work force. How would you go about evaluating whether the change was successful? a. What is the evaluation’s purpose? b. What steps will you follow to conduct the evaluation? c. What level(s) do you hope to evaluate, as per the Kirkpatrick framework? d. What data collection method will you use?

2. Identify a process, practice, or performance standard you would like to improve and plot how you would benchmark it.

3. Evaluation may be an afterthought in many interventions. How would you ensure evaluation is integrated into a change effort you are involved with or leading? How might you curb resistance?

4. Identify an intervention in which you have participated at work and evaluate it according to Kirkpatrick’s four-level framework: a. reaction b. learning c. behavior d. results

5. Plan an evaluation according to its a. purpose b. measures

Summary and Resources

c. information sources and methods d. analysis and feedback e. future action f. political issues

6. If you have ever participated in an OD intervention led by a consultant, identify what types of evaluation were conducted. How well did the consultant do, based on the principles presented in this chapter?

7. Have you experienced a failed OD intervention that had to be recycled? If so, use the information presented in this chapter to diagnose what went wrong.

Additional Resources Media

• Michael Quinn Patton Evaluation Videos https://www.youtube.com/results?search_query=michael+patton+quin+evaulation

• Kirkpatrick Model: Should I Always Conduct a Level 1 Evaluation? https://www.youtube.com/watch?v=dVnBE2W7qAI&list=PL3D286DBB9370267D

• Kirkpatrick Model: Monitoring Level 3 to Maximize Results https://www.youtube.com/watch?v=r-qF4kJrTiI&list=PL3D286DBB9370267D

Web Links

• The American Evaluation Association (AEA), an international professional associa- tion of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation:

http://www.eval.org

• The Online Evaluation Resource Library, a useful site that collects and makes avail- able evaluation plans, instruments, and reports that can be used as examples by principal investigators, project evaluators, and others:

http://oerl.sri.com

• The Centers for Disease Control and Prevention Program Evaluation Resources site, which offers a plethora of useful content on conducting evaluations:

http://www.cdc.gov/EVAL/resources/index.htm

Key Terms adjusting processes The process of chang- ing the OD intervention once the assess- ment data has been collected and analyzed.

archival data Existing records such as employment records, production figures, or quarterly reports. Used as data sources when collecting evaluation data.

assessing changes The process of gath- ering and analyzing data related to the learning and change associated with an OD intervention.

behavior Kirkpatrick’s level 3 evaluation, which measures how participants perform differently as a result of the intervention.

Summary and Resources

benchmark When an organization com- pares its business practices, processes, and performance standards with other organi- zations that are best in class.

checking A data-based evaluation to assess whether an intervention had the intended result.

evaluation A data-based checking process (assessment, adjustment, terminating, or recycling) to assess whether an interven- tion had the intended result.

formative evaluation Assessments of an intervention before or during its imple- mentation geared toward improving the process.

learning Kirkpatrick’s level 2 evaluation, which measures what skills and knowledge participants gained from the intervention.

observation Type of evaluation based on detailed descriptions of day-to-day operations.

reaction Kirkpatrick’s level 1 evaluation, which measures how well participants liked the intervention.

recycling The process of repeating or revising the action research process when further interventions are desired or the initial intervention has failed.

results Kirkpatrick’s level 4 evaluation, which measures how the bottom line was affected by the intervention.

summative evaluation Assessment that is done once the intervention is completed to judge whether it attained its goals and addressed the problem and to make future decisions about funding and continuance.

surveys and questionnaires Evaluation data collection method that uses instru- ments that participants complete to provide feedback on the intervention.

terminate To disengage from a consulting relationship with an organization at the end of the action research process.