Organization Development

By Steve H. Cady and Joo-Hyung Kim

Unlike other capital investments, the value of learning appreciates, rather than

depreciates.—Christopher Lee

The evaluation of OD interventions has never been more promising than in this era with the advancements of such tools as data analytics; yet it remains challeng- ing for organizations to utilize a proper evaluation strategy. In this article, we begin by exploring the paradox of competing demands and then introduce a pragmatic model for selecting the optimal evaluation strategy by considering both the scope and rigor of analysis and leveraging the levels of evaluation from the field of train- ing and development. We conclude with reflections on evaluation from the perspec- tive of Gestalt OD Theory and collective consciousness.

In today’s world of artificial intel- ligence, quantum computing, and data analytics, we are on the cusp of demon- strating the central importance of OD inter- ventions to the health and vitality of the systems we lead, consult, study, educate, and serve (Shah, Irani, & Sharif, 2017; Boje & Henderson, 2014). These advancements have enabled us to examine more readily how OD interventions positively impact the organization across multiple contexts and levels of analysis. For example, we are now capable of directly linking change processes to organizational performance (Pettigrew, Woodman, & Cameron, 2016). Further, this possibility does not stop with proving the importance of OD interventions. Evalua- tion tools enable us to learn from and tease

out the nuances, and more specifically learn from interventions in order to improve them for future applications.

These conditions above have set the stage for an evolutionary leap in the field of OD (Kleiner, 2015; Laloux, 2014; Lawler & Worley, 2011). On a national level, the US department of labor predicts that behavioral science fields such as OD will be serving the fastest growing occupa- tions (Career Trends, 2016). Likewise, the federal government has been supportive of the utility of behavioral sciences, as can be seen from the following statement: “Where federal policies have been designed to reflect behavioral science insights, they have substantially improved outcomes for the individuals, families, communities, and businesses those policies serve” (Presiden- tial Executive Order, 2016, p.1).

While we are on the cusp of a new era, there are still challenges to be resolved. A successful evaluation of OD interventions is one of the difficulties many organiza- tional practitioners and leaders confront.

The Paradox of Competing Demands

Assessing the efficacy of OD interventions can be elusive, because it is dependent on the required accuracy of analysis ( Terborg, Howard, & Maxwell, 1980; Butler, Scott, & Edwards, 2003). The greater the required accuracy, the more rigorous and therefore more costly the design. It is this tension that describes the paradox of competing demands (Cady & Milz, 2015; Cady, Auger, & Foxon, 2010), which partly arises from

What We Can Learn from Evaluating OD Interventions The Paradox of Competing Demands

“In today’s world of artificial intelligence, quantum computing, and data analytics, we are on the cusp of demonstrating the central importance of OD interventions to the health and vitality of the systems we lead, consult, study, educate, and serve.”

50 OD PRACTITIONER Vol. 49 No. 1 2017

the reality that you “can’t have your cake and eat it too.” The competing demands present a polarity of the purpose (i.e. prove and improve), and the cost (i.e. time and money).

Purpose: Prove and Improve

Proving Questions

❑ How many people are expected to be impacted by the intervention?

❑ Who is accountable for ensuring the intervention is implemented?

❑ Who else has the legal and govern- mental oversight?

❑ What were the political implications for the OD group and champions of the intervention?

The first dimension of prove and improve relates to the intended purpose of the inter- vention. OD professionals find themselves defending interventions, proving that the interventions had the intended impact and added value to the bottom line. The need for proof is about accountability for results, while making the business case to the skeptics. Because OD is considered to be based on the softer behavioral sciences, the role of evaluation can be undervalued and oversimplified. For example, it is easier to measure one’s blood pressure (hard science) than one’s attitude (soft science). Or, it is easier to measure cost savings from “right sizing” an organization, than it is

to measure cost savings from a training initiative.

All the while, a core value of OD is to evaluate for the sake of learning, improv- ing the intervention and leveraging its strengths. In other words, the second aim is to evaluate in order to improve the inter- vention and the organization for the future. This future focus is based on the desire to understand how interventions worked, identify the relatively important elements of the intervention, advance theories, and create more robust interventions.

Improving Questions

❑ How can the results inform us about future applications?

❑ Will the intervention be used in the future and, if so, how much?

❑ What are the uses for and scal- ability of the intervention for other settings?

❑ What is the expected impact on the wellbeing and culture of the organization?

In some cases, an accurate evaluation is necessary to determine whether the inter- vention should be continued or scaled for other settings (Kirkpatrick, 1998; Nielsen & Abildgaard, 2013). However, perform- ing rigorous high quality evaluations of all interventions is not always practical, though it might appear ideal. It is a particu- larly relevant challenge when considering the competing paradox of time and money (Cady et al, 2010).

Cost: Time and Money Conducting evaluations can be costly in terms of both time and money for all stakeholders involved. If either is too high, it is likely that the key decision makers will decide against conducting an evaluation altogether.

Time Questions

❑ What are the required details for the design, data collection, and reporting?

❑ What are the demands by leader- ship on the evaluation?

❑ Who needs to be included in and informed of the evaluation?

❑ How many competing interventions are currently being implemented?

In terms of time, leaders will need to pull people away from their typical tasks for evaluations, such as being interviewed, filling out surveys, and providing perfor- mance data. This opportunity cost is not always seen as a good use of time. Too often, surveys are filled out and go into the proverbial “black box,” never to be mentioned again. This lack of closing the loop is one of the biggest hindrances to getting reasonable response rates. Further, it erodes the trust of those impacted by the intervention, not to mention the loss of quality information, undermining the future applications of the intervention.

Money may be another major determi- nant of an evaluation strategy, especially when outcomes are difficult to measure. It is possible that the evaluation can be the largest item in the budget for an inter- vention. Therefore, if the financial cost associated with an evaluation is beyond expectation, it may lead the organization to decide against conducting any evalua- tion. Evaluation competencies are another consideration. As you move to higher levels of evaluation, the required sophistication often surpasses the expertise of those responsible for the intervention (Cady & Milz, 2015; Cady et al, 2010), requiring additional inputs such as training and out sourced professionals.

It is possible that the evaluation can be the largest item in the budget for an intervention. Therefore, if the financial cost associated with an evaluation is beyond expectation, it may lead the organization to decide against conducting any evaluation. Evaluation competencies are another consideration. As you move to higher levels of evaluation, the required sophistication often surpasses the expertise of those responsible for the intervention, requiring additional inputs such as training and outsourced professionals.

51What We Can Learn from Evaluating OD Interventions: The Paradox of Competing Demands

Money Questions

❑ What are the intervention-related fees and staffing costs?

❑ How much are the opportunity costs from doing or not doing the evaluation?

❑ What are the technology, soft- ware, materials, and logistical requirements?

❑ How much of the intervention’s budget has been allocated for the evaluation?

You Can’t Have Your Cake, and Eat It Too The competing demand comes from the desire to prove the effect of an interven- tion and improve it while minimizing the associated costs. Although a focus on the proving and improving of an intervention may result in better outcomes, it will cost more in terms of time and money (see Figure 1). In this case, how do you find the optimal point that balances the polarity of the benefits and costs?

You can choose a costly evaluation that bears important implications for the

organization or community. If there is a long-term implementation plan for the intervention, the intended impact is likely vital to the future; further, demonstrat- ing results in an objective fashion may be necessary for continued funding. However, there are also cases where a costly evalua- tion is not plausible or desirable.

There may be no time or budget allo- cated to the evaluation because evaluations are often reactionary and not given enough planning. For example, the whole evalu- ation process might begin with a leader simply saying, “Did that program work . . . can you confirm that it was worth our time and money . . . are we better off . . . we are not done, right . . . oh, by the way, can you provide an update focused on these ques- tions at our meeting next week?” Moreover, some believe that funds would be better spent on additional interventions or even argue that evaluations are just a bureau- cratic mechanism for show. The perception is that the outcomes of the process are so obvious that it is not necessary to conduct an evaluation (Cady & Milz, 2015; Cady et al, 2010). In this case, you can choose a less rigorous approach that will not only

give you some general guidance for the intervention but also help persuade the skeptics about the need for an evaluation.

Selecting an Evaluation Strategy

Evaluations can take place at a single level or across multiple levels, and this is often determined by the tradeoff of the compet- ing demands. This multi-level nature of evaluations has been originally adopted in the field of training and development, based on two prominent models: Kirk- patrick’s Levels of Evaluation (1998) and Phillips’ Return on Investment (1996). As shown in Table 1, there are five levels

Table 1: Levels of Evaluation with Low and High Rigor Options

Level Focus High Rigor Example Low Rigor Example

I Attitude: How satisfied are people?

A validated survey with established reliability measures and limited open-ended questions for qualitative analysis.

Three questions on a flip chart with a couple of them scaled and a third open-ended question.

II Knowledge: What do people know?

Administer a test that addresses key concepts, information, and policies related to the initiative.

Ask questions in an open forum about the key aspects of the initiative and make notes of what you hear.

III Behavior: What are people doing?

Conduct behavioral observations of people; identify the frequencies of specific desired behaviors related to the initiative.

Administer a short questionnaire asking people to self-report their actions (verbal assessment also acceptable).

IV Impact: What has been achieved?

Compare specific key performance indicators before, during, and after the initiative has been implemented.

Gather reports from a short survey, emails, and meetings with examples of what has been achieved.

V Return: How much money has been generated versus spent?

Conduct a formal ROI analysis with key personnel reporting on verified costs (e.g., waste, errors, and cycle time) and the financial gains from the intervention.

Calculate a verifiable ROI. Provide documented examples of best practices.

Conduct a pseudo ROI analysis with key personnel reporting on estimated costs (e.g., waste, errors, and cycle time) and gains from the intervention. Calculate a SWAG (serious wild ass guess) ROI with anecdotal evidence.

Time & Money

P ro

ve &

Im p

ro ve

Figure 1. The Paradox of Competing Demands in Evaluating OD Interventions

OD PRACTITIONER Vol. 49 No. 1 201752

of evaluation that progress from assess- ing attitudes at Level I up to assessing the Return on Investment at Level V. Kirk- patrick’s Evaluation Model was originally introduced in a trade journal, and was later published in his book titled Evaluat- ing Training Programs (1975). Then, J.J. Phillips’ (1997) built on the Kirkpatrick’s model by adding a fifth level: the Return on Investment (ROI) framework. These two perspectives have been integrated and applied to a variety of change initiatives (Phillips, Phillips, & Ray, 2015; Russ-Eft et al., 2008), and we recommend that OD professionals include these approaches in their toolkit for the evaluation and inter- vention design process. The two frame- works will provide you with the common language that is often used throughout organizations among leaders and manag- ers alike.

In Figure 2, we propose that you begin your evaluation design process with iden- tifying the competing demand of purpose, namely the desire to prove and improve the intervention.

Then, ascertain the availability of time and money to implement the evalua- tion. Figure 2 provides the options you can choose from with regard to the amount of rigor. For example, if you find yourself in need of both proving and improving the intervention, then it is recommended that you use all five levels of evaluation. If you have unlimited time and money, you are in a position to provide the highest level of rigor in your assessment. However, as we have discussed, this is typically not pos- sible. While the tools for data analytics are available, the staffing and related analytical costs are still considerably high (BI-Survey. com, 2015; Microstrategy, 2014).

Rigor influences how formal and planned an evaluation is, thus impacting the validity and reliability of the results. In addition, the scope of the evaluation can change from examining the participants’ experiences to verifying the intervention’s impact on the organization. Next, we will walk through some scenarios using Table 1 as a guide to help you decide the level of evaluation. Then, Figure 2 provides further examples on the ways to conduct less rigor- ous evaluations when there are limitations on time and money.

Let’s start with a scenario in which you want to both prove and improve a team-level mediation intervention. When conducting an evaluation with high rigor, the scope is the most invasive and extreme; it involves higher costs with a formal sur-

vey to assess reactions (Level 1), a test on the mediation model being used (Level 2), and the monitoring of the mediation practices utilized (Level 3). The evaluation also requires the evidence for supporting high-leverage improvements while also proving the degree of impact, such as the

frequency of mediations (Level 4), and a well- documented ROI (Level 5). These measures attempt to answer the following questions, for example: » How effective was the team mediation

intervention? » How can it be more effective? » What difference has it made? » What errors have been averted? » How much cost has been saved? » What revenues have been improved?

All five levels of evaluation are to be conducted in a formal and valid fashion by trained research professionals.

Now, let’s consider a situation where you find yourself in need of prov- ing whether the intervention worked as intended; however, you do not want to improve the intervention for the future. While the evaluation aims at assessing the changes in operations and behaviors, the focus is on the direct impact of the inter- vention. In the case of the team mediation intervention, this might involve assessing the mediation practices utilized (Level 3) and the number of mediations conducted or the amount of time to resolution (Level 4). These measures would be taken before, during, and after the intervention implementation.

The lowest level of evaluation, before there is no evaluation required at all, is the scenario where you do not want to prove whether the intervention worked, yet you need to improve it for future applications. In this evaluation strategy, you would con- duct Level 1, 2, and 3 evaluations. Here you

Ti m

e &

M o

n ey

Hi

ye s

yes

yes

no

no

no

Prove?

Improve?

All Levels (1–5)

Middle Levels (3–4)

Low Levels (1–3)

No Eval Needed

Improve?

Lo

Figure 2. Evaluation Strategy Decision Tree

When conducting an evaluation with high rigor, the scope is the most invasive and extreme; it involves higher costs with a formal survey to assess reactions (Level 1), a test on the mediation model being used (Level 2), and the monitoring of the mediation practices utilized (Level 3). The evaluation also requires the evidence for supporting high-leverage improvements while also proving the degree of impact, such as the frequency of mediations (Level 4), and a well-documented ROI (Level 5).

53What We Can Learn from Evaluating OD Interventions: The Paradox of Competing Demands

examine the specific intervention’s impact on individuals’ perceptions and behaviors. For example, you might assess reactions to the team mediation (Level 1) and check if the mediation model has been properly understood (Level 2). You might use three questions on a flip chart to assess the sat- isfaction level, and ask them to write down the components of the mediation to check the learning outcomes.

Within all of these strategies, you may need to conduct a less rigorous evaluation providing a rough understanding of how to improve the intervention by gathering

information with limited resources in a shorter time period. For example, you may walk around the organization taking note of changes, conduct an informal poll during a meeting, and seek personal inputs about the perceived impact of the inter- vention (Table 1). Leaders may be asked to estimate the approximate cost savings and revenue improvements, thereby estimating a pseudo-ROI.

Summary and Reflections

The process of evaluating interventions is about determining whether the path you are on has been the most effective strategy efficiently executed for the system you are in. In turn, this will influence how you move into the future. The fundamental premise of a good evaluation, as consistent with the principle of the Gestalt Theory of OD, is self-awareness on a large-scale. Self-awareness serves as the starting point of the whole energizing action cycle in the Gestalt Group Cycle (Zinker, 1980); to be able to figure out the future path, people

and the whole systems must be aware of themselves. Likewise, as in the notion of collective consciousness, group aware- ness can act as a unifying force for taking ownership in determining how to move forward together (Tsoukalas, 2007).

The models presented here provide a framework for considering the competing demands and the trade-offs to be made. Developing an effective evaluation strategy is subtle, and grey areas are manifold. For example, imagine you are working on the decision tree model (Figure 2) and it sug- gests that an informal SWAG (serious wild

ass guess) evaluation is the best option with Levels 4 and 5, because time and money are limited. Is this “bad”? We say “no.” The reason is two-fold. First, getting leaders to slow down and think through the questions necessary to evaluate an OD intervention is important. For example, a venture capitalist conducts valuations using internal rates of return (IRR), which are based on critical assumptions. The value of these invest- ments often dwarfs the costs of most OD interventions; and, it is interesting to note that IRR calculations are made with some SWAG. Similarly, engaging leaders in the analysis of the intervention’s impact is about creating conscious and more mind- ful decisions on the ways to improve the organization. Second, if the leaders paying for the intervention get feedback from cred- ible peers that the intervention resulted in a meaningful return on the investment, they will deem the feedback a valid data point. While this belief is a perception, it is helpful to know key stakeholders’ collective reality in order to figure out the next steps for organizational change efforts. In short,

evaluations do not happen in a perfect world; there are always compromises that must be made.

The bottom line is this: if you can achieve the intended purpose of the evalua- tion, then you have found the balance with the paradox of competing demands. You may find that your desired evaluation strat- egy differs significantly from those you are serving. As a result, go back to the drawing board and rethink what is most important to you, which may require renegotiating expectations with key leaders and other stakeholders.

In closing, the evaluation of OD interventions supports the development of a learning organization, and this is where a consultant as an intervener can play an important role in helping the organization in service or disservice of itself. In provid- ing the system with a better understanding of itself through rigorous evaluations, the system will change its thinking paradigm and move toward a more intentional way of self-organizing (Nevis, 2013). When the system begins to change its thinking para- digm, the learning has begun and “action learning” occurs. This shift in thinking and learning is what will lead to new orga- nizational habits that result in long-term sustainability.

References

BI-Survey.com (2015). How companies benefit from big data. Retrieved from http://bi-survey.com/big-data-benefits

Boje, D.M., & Henderson, T.L. (Eds.). (2014), Being quantum: Ontological storytelling in the age of antenarrative. Newcastle upon Tyne, UK: Cambridge Scholars Publishing.

Butler, J., Scott, F., & Edwards, J. (2003). Evaluating organizational change: The role of ontology and epistemology. Journal of Critical Postmodern Organiza- tion Science, 2(2), 55–67.

Cady, S. H., Auger, J., & Foxon, M. (2010). Situation evaluation. In W. Rothwell, J. M. Stavros, & R. I. Sullivan (Eds.), Prac- ticing organization development: A guide for leading change (3rd ed., pp. 363–376).

The bottom line is this: if you can achieve the intended purpose of the evaluation, then you have found the balance with the paradox of competing demands. You may find that your desired evaluation strategy differs significantly from those you are serving. As a result, go back to the drawing board and rethink what is most important to you, which may require renegotiating expectations with key leaders and other stakeholders.

OD PRACTITIONER Vol. 49 No. 1 201754

San Francisco, CA: Jossey-Bass Publishers.

Cady, S. H., & Milz, S. A. (2015). Evaluating organizational transformation. In W. Rothwell, J. M. Stavros, & R. L. Sullivan (Eds.), Practicing organization develop- ment: Leading transformation and change (4th ed., pp. 195–210). San Francisco, CA: Jossey-Bass Publishers.

Career Trends (2016, July 12). Retrieved from: http://psychology.about.com/od/ psychologycareerprofiles/p/iopsychcareers. htm https://blog.dol.gov/2015/03/15/ the-10-fastest-growing-jobs/

Kirkpatrick, D. (1998). Evaluating training programs: The four levels. San Francisco, CA: Berrett-Koehler Publishers.

Kleiner, A. (2015, Spring). Ellen Langer on the Value of Mindfulness in Business, s+b, Research on Teal-like attributes: wholeness and control over context. Retrieved from http://www.strategy- business.com/article/00344?gko=10921

Phillips, J. J. (1997). Handbook of train- ing evaluation and measurement methods (3rd ed.). Boston, MA: Butterworth-Heinemann.

Phillips, P. P., Phillips, J. J., & Ray, R. L. (2015). Measuring the success of leader- ship development: A step-by-step guide for measuring impact and calculating ROI. Alexandria, VA: ATD Press.

Laloux, F. (2014). Reinventing organizations: A guide to creating organizations inspired by the next stage in human consciousness. Brussels, BE: Nelson Parker.

Lawler III, E. E., & Worley, C. G. (2011). Management reset: Organizing for sus- tainable effectiveness. Hoboken, NJ: John Wiley & Sons.

Microstrategy (2014). Summary results from the BI survey 13. Retrieved from https://www.microstrategy.com/Strategy/ media/downloads/about-us/microstrategy- bi-survey-13.pdf

Nevis, E. (2013). Organizational consult- ing: A Gestalt approach. Abington, UK: Taylor & Francis.

Nielsen, K., & Abildgaard, J. S. (2013). Organizational interventions: a research-based framework for the evaluation of both process and effects. Work & Stress, 27(3), 278–297.

Pettigrew, A.M., Woodman, R.W., & Cameron, K.S. (2016). Studying organizational change and develop- ment: Challenges for future research. Academy of Management Journal, 44(4), 697–713.

Presidential Executive Order (2016, August 1). Retrieved from: https://www. whitehouse.gov/the-press-office/2015/ 09/15/executive-order-using-behavioral- science-insights-better-serve-american

Russ-Eft, D., Bober, M. J., De La Teja, I., Foxon, M., & Koszalka, T. A. (2008). Evaluator competencies: Standards for the practice of evaluation in organizations (Vol. 22). Hoboken, NJ: John Wiley & Sons.

Shah, N., Irani, Z., & Sharif, A.M. (2017). Big data in an HR context: Explor- ing organizational change readiness, employee attitudes and behaviors. Jour- nal of Business Research. 70(1), 366–378.

Terborg, J.R., Howard, G.S., & Maxwell, S.E. (1980). Evaluating planned organi- zational change: A method for assess- ing alpha, beta, and gamma change. Academy of Management Review, 5(1), 109–121.

Tsoukalas, I. (2007). Exploring the micro- foundations of group consciousness. Culture and Psychology, 13(1), 39–81.

Zinker, J. C. (1980). The developmental process of a Gestalt therapy group. In B. Feder & R. Ronall (Eds.), Beyond the Hot Seat—Gestalt Approaches to Groups (pp. 55–77). New York, NY: Brunner/ Mazel.

Interested in learning more about the differences OD makes? See the call

for articles on page 5.

Steve Cady, PhD, is a professor, author, speaker, and consultant. He is a graduate faculty mem- ber in the Bowling Green State University Master of Organiza- tion Development Program. Cady is the co author of The Change Handbook, author of Stepping Stones to Success and PBS DVD titled Life Inspired. He is also the Chief Editor and Curator for the BKpedia Change & Innovation Collection. He can be reached at steve@stevecady.com.

Joo-Hyung Kim is an MBA can- didate at Bowling Green State University. She earned her bachelor’s degree in Business Administration from Korea University. Her research interests include judgment and decision making, emotion, and applicant attraction. She can be reached at jookim@bgsu.edu.

55What We Can Learn from Evaluating OD Interventions: The Paradox of Competing Demands

Copyright of OD Practitioner is the property of Organization Development Network and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.