4 pg report - 4 question 1 pg each

Evaluation_Practice_for_Collaborative_Growth_.pdf

Home >Business & Finance homework help >Accounting homework help >4 pg report - 4 question 1 pg each

P A R T O N E

Prepare

Program evaluation is as much a process as a way to make decisions about a program’s need, value, worth, or fidelity. Therefore, an adequate background and preparation for conducting an evaluation of a program is essential. The time and energy you invest in preparing for a program’s evaluation will go a long way toward its success and utility to stakeholders. Chapters One through Four equip you with the mindset, knowledge, and skills you will need to ade- quately prepare for a program evaluation. These preparations include thinking like an evaluator; understanding your role in an evaluation; conducting the evaluation using high standards of professional, inclusive, and ethical prac- tice; and communicating and negotiating with stakeholders to determine the evaluation’s purpose, use, and questions. This part of the process provides you with the clues and information you need to select an evaluation approach. There are multiple approaches to evaluation, and it is important that you un- derstand their philosophical underpinnings and theoretical foundations in order to select one or more of them to guide the evaluation appropriately. You also will learn that some programs are not necessarily ready to be evaluated, so you will acquire the background needed to determine whether to proceed with a program evaluation. You also will be given guidance on preparing an evaluation proposal so that the agreement among you and the evaluation’s stakeholders is very clear. Collectively, these four chapters will prepare you for the next phase of the process, which is to determine the details of the evaluation’s design.

Bakken, Lori L.. <i>Evaluation Practice for Collaborative Growth : A Guide to Program Evaluation with Stakeholders and Communities</i>, Oxford University Press, Incorporated, 2018. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/usf/detail.action?docID=5400963. Created from usf on 2019-10-03 16:13:23.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

C H A P T E R 1

Thinking Like an Evaluator

In today’s world of accountability, it is becoming increasing important for social science practitioners to evaluate the programs and interventions they develop and implement. Moreover, government agencies and foundations that typically fund these programs are increasingly requesting that social science professionals demonstrate their programs’ impacts on and values to communities through evidence- based practices. An evaluation skillset, there- fore, is essential for practitioners in service- related fields, such as education, nonprofit management, social work, or public health (Davis, 2006).

Although external evaluators often are called upon to perform this service, social science practitioners must develop requisite knowledge and skills to en- gage in evaluation practice. By doing so, the internal capacity of their organ- izations and respective fields for evaluation is enhanced (Stevenson, Florin, Mills, & Andrade, 2002). Active participation in evaluation activities increases a sense of ownership, which promotes greater use of evaluation results in the decision- making and program implementation processes; thereby, facilitating successful outcomes (Hoole & Patterson, 2008; Mercier, 1997). Consequently, practitioners who incorporate evaluative thinking into daily professional practice experience and acknowledge the benefits of evaluation (Taut, 2007). By building the capacity of practitioners to conduct evaluations, sustainable practices of informed decision- making and action planning are created, which, in turn, foster high- quality, effective public services (Preskill & Boyle, 2008).

With that background, let us take a moment to engage your evaluative thinking skills. Suppose you and a group of friends were given two types of freshly baked chocolate chip cookies and asked to pick the one you liked most. What criteria would you use to select your favorite cookie? Do you prefer crispy cookies or soft ones? Do you like them loaded with chocolate chips? Do you prefer a nice, rich buttery flavor over a less rich cookie? Do

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 4 ] Part One: Prepare

you prefer chocolate chunks over small chocolate chips? How might you nego- tiate differences of opinion among your friends? How would you assess each cookie so your personal biases don’t interfere with your judgment about the best cookie? What evidence would you collect and how would you collect it to determine which cookie is the best? As we go about our daily lives, we are con- stantly using evaluative thinking to assess various things and phenomenon that we encounter in our daily world. Cookies are just one example. We use these skills when we purchase groceries, buy a new car, determine whether we need or want to learn a new skill, select a life partner, and so on. We apply these same thinking patterns when we evaluate programs. It’s just that programs and interventions in the social sciences are a more complex, because they involve people and communities.

This book will expand your evaluative thinking skills and guide you through a four- step evaluation process. It is designed to provide you with a fundamental understanding of evaluation approaches, methods, tools, and practices so that you will become more proficient at evaluating your own programs and interventions. This chapter begins by introducing you to a few basic evaluation concepts using an example from everyday life. I will then use these same concepts to shift your thinking about program evaluation from an isolated activity to an activity that is intricately intertwined with program planning.

PURCHASING A CAR: AN EXAMPLE OF EVALUATION IN EVERYDAY LIFE

When was the last time you purchased a car? What did you do before you de- cided which car to buy? What type of information did you use to decide among several makes and models of cars? Why did you purchase the type of vehicle you chose? As you think about the answers to these questions, I envision a process not unlike my own. It goes something like this.

I decide that I like the nine- year- old mini sport utility vehicle (SUV) that I currently own, but it is time that I purchase a new one. As models change over time, I also decide that I want to explore similar vehicles made by other manufacturers. I begin my thoughts with a few ideas about the car’s features. I like the fact that SUV’s sit higher than regular cars and have more cargo space. I also like the way they can accommodate my recreational gear (e.g., skis, bikes, kayaks). My current SUV has a standard transmission of which I am growing weary, so I decide I also would like an automatic transmission. My current SUV is dark green, which shows dirt, so I would like to get a lighter- colored car, preferably beige. I also am getting older, and I like the standard features that come with many midsized cars— air conditioning, power door locks, CD/ MP3 player, etc.— so I keep these features on my wish list.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

T h i n k i n g L i k e a n e va L u a T o r [ 5 ]

At this point, I have given some thought to the type and features of the vehicle I am looking for, so I go online to compare models that have these features. While online, I also look at prices, maintenance costs, and features that are standard on each make and model. I also like my cars to have a “small” feel because I am a petite woman, so I look at the overall dimensions of each vehicle. “Perks,” such as built- in cargo racks, also are factored into my com- parison because these features can be expensive and make a difference in the overall purchase price.

After comparing models and narrowing my selection to two vehicles, I de- cide to test drive each of them. During my test drive, I transcend steep hills, navigate tight corners, and accelerate quickly to interstate speeds. I notice that one of the models feels a bit underpowered, but it has that smaller feel for which I am looking. It is also a bit less expensive than the more powerful model, so I consult a few of my friends and acquaintances who own that model to get their opinions on the model’s performance. After pondering the infor- mation, I decide to begin my negotiations with a salesperson. Let’s pause now and think about the process thus far and what I have done.

I determined the criteria for my vehicle purchase when I decided that I wanted a beige SUV with an automatic transmission and cargo capacity to accommodate my recreational activities. Each of these features (i.e., beige color, automatic transmission, and cargo capacity) is a criterion that ultimately would help me to compare models and make a decision, or judgment, about the car. When I did the online comparison, I collected evidence to facilitate my decision and acquired information about each vehicle’s cost, maintenance, and features. I collected additional evidence about the vehicle engine power when I did the test drive and consulted the opinions of friends and acquaintances. In summary, I collected evidence and compared it to my criteria to make a judgment about the vehicle I in- tended to purchase.

Evidence, criteria, and judgment are the basic elements of every evalua- tion, and you use these concepts every day as you go about your lives. You are evaluators when you purchase fruits and vegetables in the grocery store, select a physician to oversee your health care, meet a new neighbor, or taste a friend’s chocolate chip cookies just out of the oven. If you do it every day, why does it seem so foreign and difficult when, in your professional roles, you are asked to evaluate the programs you or others develop? The answer to this question is multifaceted. First, a social science practitioner’s language, knowl- edge, and skills for evaluation practice typically have not been developed. Second, as an interdisciplinary field, the evaluation field has grown tremen- dously in recent years and in doing so, it has adapted highly rigorous and so- phisticated approaches, methods, and tools. Although this book will not cover the evaluation field in any depth (other textbooks do that very well), it will provide you with an overview of program evaluation, so you can incorporate

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 6 ] Part One: Prepare

some evaluation practices in your programming efforts and build your per- sonal capacity to work with professional evaluators.

MAJOR COMPONENTS OF AN EVALUATION PROCESS

Over the years, the evaluation field has expanded and become increasingly sophisticated as the questions and problems being addressed by programs and interventions have become more challenging and expansive in scope. The approaches, methods, and tools that evaluators use also have expanded as the field shifted from objective- based approaches to those of complex systems. This book takes you through a four- part process of preparing for, designing, conducting, and reporting an evaluation (Figure 1.1).

Within this process are multiple tasks and activities that will require your attention. The preparation and design for an evaluation are the most time- consuming parts of the process, but they are critical to an evaluation’s success. In order to set the stage for subsequent chapters and provide you with an overview of program evaluation, in the remainder of this chapter, I briefly describe the basic activities and tasks that are part of an evaluation.

Prepare

The first step in preparing for any evaluation is to be certain you have the requisite background to plan and conduct the evaluation. This background includes knowledge of the professional and ethical standards that guide

Report

Conduct

Design

Prepare

Figure 1.1. Four- step Evaluation Process.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

T h i n k i n g L i k e a n e va L u a T o r [ 7 ]

evaluation practice, competencies needed to conduct an evaluation, and skills to build and maintain partnerships and collaborations throughout the eval- uation process. It is also important to know about the various perspectives and approaches that guide an evaluation’s design. Evaluation approaches are driven by the philosophies or world views that both you and your stakeholders bring to the process. Will your approach be based on program goals and learning objectives? Will your approach be theory driven? Will it advocate par- ticipation of all relevant stakeholders? Does the approach need to be designed in a way that will aid decision- making? The answers to these questions will be- come apparent as you engage with key stakeholders to gain an understanding of the evaluation’s context, purpose, and questions.

This part of the preparation process requires conversations with sev- eral individuals or groups of people who have a stake in the program or its evaluation (i.e., stakeholders). An examination of a program’s context also necessitates visits to locations and communities in which the program is pro- vided. Therefore, the preparation needed to design and conduct an evalua- tion is a time- intensive process that requires strong listening skills, excellent communication, abilities to successfully negotiate, keen observation, and pa- tience. Often, the language and processes familiar to evaluators are unfamiliar and may feel intimidating to a program’s stakeholders; therefore, it is best to avoid jargon and explain concepts in ways that facilitate understanding. For example, evaluation concepts such as, desired or expected “outcomes”, are more easily sought with questions such as, “What change do you expect or is likely to happen as a result of your program?” It often takes a series of several conversations with key stakeholders to acquire the information necessary to evaluate a program. A worksheet I sometimes use to guide these conversations is presented in Box 1.1.

Once acquired, this information should be conveyed back to stakeholders in order to reach a mutual understanding about how a program is to be evaluated. Other considerations that are important in the preparation pro- cess are a clear understanding of everyone’s role in the evaluation, cultural norms and practices, protection of individuals and groups who will participate in the evaluation study, and adherence to evaluation principles and standards (American Evaluation Association, 2004; Yarbrough, Shulha, Hopson, & Caruthers, 2011). It is often helpful to conduct an evaluability assessment (Wholey, 2004) to be certain that critical elements of the preparation process are discussed.

Your early conversations with stakeholders should attempt to ascertain the evaluation’s purpose, use, goals, and guiding questions. The purpose can be determined with the questions, “Why would you like to do this evaluation?” By asking the question, “How will you use the findings from the evaluation?” you can establish stakeholders’ intended use for the evaluation’s results. Evaluation questions can be determined with the question, “What would you

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 8 ] Part One: Prepare

like to learn from this evaluation?” or “What questions do you have about your program?”

Often, program directors have specific goals that they would like to achieve through an evaluation, such as building staff capacity or becoming more com- petitive for grants, so it is helpful to know these as well. They also may need an evaluation to inform new programming efforts. Needs assessments (Queeney, 1995; Wambeam, 2015) coupled with environmental scans (Simerly & Associates, 1987) are useful for addressing this purpose.

Box 1.1. PROGRAM LOGIC QUESTION GUIDE

E V A L U A T I O N - R E L A T E D Q U E S T I O N S

1. What program are you evaluating? 2. Why are you evaluating this program? 3. How will you use findings from this evaluation? 4. What goals are you trying to achieve by evaluating this program? 5. Who cares about the evaluation’s findings?

P R O G R A M - R E L A T E D Q U E S T I O N S

1. In what ways do you hope participants will change as a result of this program? Immediately (i.e., knowledge, attitudes, and skills)? In one to three years (i.e., application of knowledge and skills or behav- ioral changes)?

2. In what ways (expected or unexpected) do you hope programming efforts will change your organization? Your community? The folks you serve?

3. How will your program achieve the immediate changes stated in Question #1?

4. What evidence will you have to demonstrate that the program was implemented as planned?

5. What resources (community, human, organizational, financial, etc.) are available for programming efforts that lead to the desired changes (refers to Questions #1 and #2)?

6. What are some of the stakeholder assumptions that may influ- ence the program’s design, implementation, or goals? Can these assumptions be validated through scholarship or other “credible” sources?

7. Should planners and evaluators consider other potential changes to program participants, the organization, or community that may be suggested by scholarship, professional groups, or others and have not been considered in answers to Questions #1– 5?

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

T h i n k i n g L i k e a n e va L u a T o r [ 9 ]

In practice, stakeholders often have difficulty articulating or conveying the questions they would like answered by an evaluation. It is through your preparatory conversations with them that you acquire information that will allow you to pose some initial evaluation questions that are refined in the de- sign process. When you have a thorough understanding of the program’s and evaluation’s context and purpose, as well as stakeholders and their questions, you can select an evaluation approach or combination of approaches that will guide the evaluation’s design and answer the evaluation questions.

Design

Your chosen approach for the evaluation will guide you toward specific details of the evaluation design. These design elements are determined through a process that engages as many stakeholders as possible to refine and prioritize the initial evaluation questions that guide the evaluation’s design. I would like to pause for a moment to emphasize my point that an evaluation’s purpose, use, and questions determine an evaluation’s design. Once the questions are clearly determined, you can articulate a general study design. Questions that begin with “How . . . ?” “Why . . . ?” or “In what ways . . . ?” are typically process- related and signal qual- itative designs. Questions that imply comparisons or program effectiveness, such as “Is my program successful?” or “Does my intervention reduce parental stress levels?” typically call for quantitative designs. Questions related to the fidelity of the program often suggest a qualitative approach. Common to our age of accountability are questions such as, “What are the long- term outcomes or impact of the program?” These questions often require more complex quan- titative or mixed methods designs. This is not an exhaustive list of evaluation questions you will encounter in your experiences as an evaluator. Evaluation questions vary by stakeholders and context. Some basic designs often can be applied to answer them, but the context of the program will undoubtedly make each design unique and possibly more complex. Other design elements include instruments used to collect data, determining from whom or what source data will be collected, plans for tracking and storing electronic data, and finally, an- alytical plans that describe data analysis in detail. The details of these study designs and data collection tools and strategies are covered in Part II.

Conduct

When you have a clear plan for the study design, data collection, and analysis you can begin the evaluation. It is important that you have a well- thought- through plan before you conduct your evaluation study; otherwise, your evaluation may be fraught with unnecessary challenges and problems. Good

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 10 ] Part One: Prepare

planning is essential to avoid the pitfalls brought on by factors such as unnec- essary or insufficient data collection, unclear expectations, lack of sufficient resources, and other issues that may rob you of valuable time and support. For example, you may have difficulty recruiting participants for an evaluation study or obtaining their consent, so it is important to mentally walk through your evaluation plan in order to be prepared for any setbacks that may be encountered along the way. Not all pitfalls can be anticipated, so an evaluator must remain open and flexible to changing the initial plan if necessary. Your or your team’s job is to ensure that all information collected is complete and ac- curate (Yarbrough et al., 2011). Evaluation participants should be clear about the purpose of the study and how you will use their information. They also should be informed that their participation is voluntary, and that they can withdraw from the evaluation at any time. When appropriate, participants should provide written or oral consent to participate in a study (see Chapter 2 for more information). Once information is collected from study participants, you can analyze data according to plan. Because your analyses often reveal unexpected findings, it may be necessary to perform post- hoc analysis or in- terview additional participants.

Report

Reporting the evaluation’s findings marks the final step in an evaluation process. This is a great time to get stakeholders more heavily involved. They are essential for interpreting data and presenting it in a form that will be readily understood and used. The reporting step is also a time when conflicts may arise between you and the program’s stakeholders who may have been surprised by an unanticipated negative finding and wish to withhold the information from written or oral reports. When this happens, it is impor- tant that you remind stakeholders about the standards for evaluation and the importance of integrity. It is also helpful to emphasize the evaluation as a way to learn about a program and improve its performance and quality. Finally, when a written report is prepared, be sure to include an executive summary that highlights the evaluation questions and the answers gleaned from the evaluation. Chapter 10 will provide you with more details on how to organize and present an evaluation’s findings to various stakeholder groups.

The final chapter of this book brings the evaluation process full circle by preparing you for some of the challenges you may be faced with as you are asked to evaluate programs in the context of complex social or political environments. Chapter 11 covers emergent trends in evaluation that include systems thinking and approaches to evaluation (e.g., developmental evalua- tion), contribution models, and social justice and inclusion.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

T h i n k i n g L i k e a n e va L u a T o r [ 11 ]

PROGRAM PLANNING AND EVALUATION

Evaluation provokes anxiety among programmers and other stakeholders for a variety of reasons. These reasons include lack of knowledge, skills, and lan- guage regarding evaluation; one’s inability to transfer that knowledge when obtained; lack of confidence to apply the skills needed to do an evaluation; an apparent lack of time to get the job done; inappropriate planning; fear of being held accountable or losing a program; and the overwhelmingly complex nature of some programs. These are very good reasons to get anxious about evaluation, but many of them can be avoided with advance planning, appro- priate knowledge and skills, support from and coaching by an experienced evaluator, and strong interpersonal skills (King & Stevahn, 2013).

This section stresses the importance of planning an evaluation as part of a program planning process. Why is it important to plan an evaluation as part of a program planning process instead of waiting until the program has been implemented? Let’s suppose I wanted to have a barbeque for 30 to 40 friends and family members. I would need to purchase sufficient groceries to feed them or hire a caterer. I also would need a backyard space and seating to accommo- date them. Additionally, I would want to purchase beverages— summers are hot— and be sure that I had utensils, plates, cups, and condiments. If I de- cided to make the food myself, I also would have to allow adequate time to prepare and store the food in advance. What if it rains? I should probably have a backup plan that includes shelter. Evaluation planning is very similar in that you need to be sure you have the resources and skills to conduct an evaluation, given its scope. You also will need a backup plan or flexibility when things don’t go as expected, such as insufficient or skewed participation, missing survey responses, or delayed meetings with stakeholders. Your goal is to con- duct an evaluation in a timely manner so that you can deliver its findings in an appropriate time frame. You also want to be sure that the evaluation findings actually deliver answers to the evaluation questions. Without some careful planning, it is easy to get off target or expand the evaluation’s scope in a way that originally was not intended.

Several program planning models include evaluation as part of the overall planning process (Caffarella, 2002; Donaldson & Kozoll, 1999; Sork, 2000). As a point of reference, I would like to propose one particular model for helping you to think about evaluation on the “front end” of program design, develop- ment, and implementation. Rosemary Caffarella in her book titled, Planning Programs for Adult Learners (Caffarella, 2002), discusses the Interactive Model of Program Planning as one that reflects the typical nonlinear process by which most programs are planned. Central to this model are 12 components of a pro- gram planning process that can begin with any one of these components and often changes in contour, content, and size (p. 21). Moreover, planners often multitask and perform many of the model’s components simultaneously. One

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 12 ] Part One: Prepare

of these components is “formulating evaluation plans.” The important point to my emphasis on program planning as an interactive model of multiple components, including evaluation, is that evaluation planning is part of a pro- gram planning process. An evaluation should not be planned the day before the program is implemented or when all the participants have gone home. I cannot emphasize this point enough, so I will say it again. Evaluation pla- nning should be done as part of a program planning process. Like all rules, this rule has some exceptions, which I will address in the last chapter of this book when I talk about developmental evaluation and systems approaches.

SUMMARY

This chapter has engaged your evaluative thinking, introduced you to some fundamental evaluation concepts, and provided you with an overview of the process for conducting an evaluation. The remaining chapters of this book will assume that you are planning your evaluation as part of a program pla- nning process and emphasize only components of an evaluation plan that are imbedded in that program plan. To begin this journey, Chapter 2 will describe the requisite knowledge and skills for a successful program evaluation.

REFERENCES

American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved from http:// www.eval.org/ p/ cm/ ld/ fid=51

Caffarella, R. S. (2002). Planning programs for adult learners: A practical guide for educators, trainers, and staff directors (2nd ed.). San Francisco, CA: Jossey- Bass Publishers.

Davis, M. (2006). Teaching practical public health evaluation methods. American Journal of Evaluation, 27(2), 247– 256.

Donaldson, J. F., & Kozoll, C. E. (1999). Collaborative program planning: Principles, practices, and strategies. Malabar, FL: Krieger Publishing Company.

Hoole, E., & Patterson, T. (2008). Voices from the field: Evaluation as part of a learning culture. In J. Carman & K. Fredericks (Eds.), Nonprofits and evaluations: New directions for evaluation, 119, 93– 113.

King, J. A., & Stevahn, L. (2013). Interactive evaluation practice: Mastering the interper- sonal dynamics of program evaluation. Thousand Oaks, CA: Sage Publications, Inc.

Mercier, C. (1997). Participants in stakeholder- based evaluation: A case study. Evaluation and Program Planning, 20, 467– 475.

Preskill, H., & Boyle, S. (2008). A conceptual model of evaluation capacity building: A multidisciplinary perspective. American Journal of Evaluation, 29(4), 443– 459.

Queeney, D. S. (1995). Assessing needs in continuing education. San Francisco, CA: Jossey- Bass Publishers.

Simerly, R. G., & Associates. (1987). Strategic planning and leadership in continuing edu- cation. San Francisco, CA: Jossey- Bass Publishers.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.eval.org/p/cm/ld/fid=51

T h i n k i n g L i k e a n e va L u a T o r [ 13 ]

Sork, T. J. (2000). Planning educational programs. In E. R. Hayes & A. L. Wilson (Eds.), Handbook of adult and continuing education (pp. 171– 190). San Francisco, CA: Jossey- Bass Publishers.

Stevenson, J. F., Florin, P., Mill, D. S., & Andrade, M. (2002). Building evaluation ca- pacity in human service organisations: A case study. Evaluation and Program Planning, 25, 233– 243.

Taut, S. (2007). Defining evaluation capacity building: Utility considerations [Letter to the editor]. American Journal of Evaluation, 28(1), 120.

Wambeam, R. A. (2015). The community needs assessment workbook. Chicago, IL: Lyceum Books, Inc.

Wholey, J. S. (2004). Evaluability assessment. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 33– 62). San Francisco, CA: Jossey- Bass Publishers.

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The pro- gram evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

C H A P T E R 2

Acquiring Requisite Knowledge and Skills

Chapter 2 describes a fundamental skill set needed to practice evalua-tion ethically and responsibly. I begin by describing the various roles of an evaluator in planning, implementing, and reporting an evaluation. These roles have implications for the ways you will conduct an evaluation and use the various tools and skills needed to function in a professional manner. Evaluation standards and principles are described that will guide your professional work and practice, regardless of your role in an evaluation. These standards also help to guide you through difficult challenges when negotiating with stakeholders who may not agree with your approaches or the findings of an evaluation. As a vital part of those standards, evaluators are expected to uphold ethical practices and protect those involved in the evaluation or program being evaluated. Competencies for ethical practice are a single part of a set of competencies that are necessary to practice ef- fectively and efficiently. Effective communication and negotiation skills are another essential component of an evaluator’s competencies and are used often in the course of an evaluation. These skills are most effective when used with integrity, honesty, and from a perspective of reciprocal learning that helps to model and build capacity among the organizations we work with and serve.

ROLES OF EVALUATORS

Evaluators can occupy a variety of roles in planning, conducting, and re- porting an evaluation. They often are brought into an organization or

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 15 ]

community setting as an external and objective “eye” for a program. For ex- ample, a funder who requests an external evaluator to oversee the collective evaluation activities of several grantees. An organization also might em- ploy an external evaluator to evaluate an existing program or help to design an evaluation for a program in the planning stage. Metaevaluators are ex- ternal evaluators who evaluate the evaluation activities of a program or set of programs. Typically, this role is used for accreditation or reaccreditation of existing programs. External evaluators also may help to build evalua- tion capacity within an organization or oversee a set of evaluation activities conducted by the organization’s staff. Evaluators can serve as coaches for staff who have acquired some evaluation skills and need support as they further develop their skills. Program staff can serve as evaluators of their own programs, too. In this role, staff are internal evaluators. Many of you likely consider yourselves to be in this role. Sometimes, staff are internal to an organization and asked to evaluate programs that are not of their own development. They, too, are referred to as internal evaluators. In partic- ipatory evaluation practice, also known as collaborative inquiry (Cousins & Whitmore, 1998), evaluators engage key stakeholders to the extent pos- sible and throughout the evaluation planning, implementation, and re- porting processes. When doing so, an evaluator’s role is typically facilitative in nature.

As facilitators, evaluators bring a structure for engaging clients and stakeholders, grounded in their evaluation methodology (e.g., planning meetings or conducting interviews, focus groups, and surveys); contribute to the potential effectiveness of teams by offering them accurate and reli- able data that are easy to understand and use in making decisions and improving programs and organizational performance; and remain inde- pendent, aiming to include different perspectives in the evaluation without taking sides (Catsambas, 2016, p. 20). According to Catsambas (2016), “good facilitation of the evaluation process increases client engagement and buy- in, fosters a safe place for honesty, and creates a neutral space for airing different views with an impartial and independent evaluator” (p. 20). “Good facilitation” is not easy, however, especially when group tensions and disagreements are high, and we are trying to facilitate transformative change (Mertens, 2009). These situations require you to be self- reflective, self- knowing, effectively communicative, and culturally competent. Self- understanding and awareness, in addition to facilitation skills, allow you to confront differing viewpoints, un- silence marginalized voices, and re- main neutral as you manage difficult interpersonal and group dynamics in evaluation. You, therefore, must be a confident individual who lives up to professional standards of practice and exercises effective interpersonal skills.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 16 ] Part One: Prepare

EVALUATION STANDARDS

Over the years, practitioners and researchers in the evaluation field have de- veloped important standards for professional and ethical evaluation practice. The Program Evaluation Standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011), now in its third edition, were created by the Joint Committee on Standards for Educational Evaluation and are applied broadly across the field. These standards are grouped into five major categories that provide a founda- tion for professional evaluation practice; namely, utility, feasibility, propriety, accuracy, and accountability.

The utility standards stress the importance of an evaluator’s credibility and skills to involve stakeholders throughout the evaluation process as they clarify and negotiate the purpose(s) of an evaluation, understand and make explicit stakeholder values, provide useful and relevant information, provide timely and accurately communicated reports, and maintain a neutral posturing during the conduct of an evaluation. Above all, this set of standards empha- size the importance of the evaluation’s utility in providing information that can benefit its stakeholders. The feasibility standards stress the importance of assuring that sufficient resources are available to conduct an evaluation in a timely manner, managing the evaluation in ways that will maintain its progress and provide appropriate feedback to stakeholders, and carefully considering the political and cultural context of the evaluation. The propriety standards speak to the importance of inclusive participation, formal agreements, respect for human rights, clarity and fairness in addressing the stakeholders’ needs and purpose for the evaluation, and transparency throughout the evaluation’s conduct. This set of standards also addresses conflicts of interest and the im- portance of conducting an evaluation in a fiscally responsible manner. The ac- curacy standards emphasize the need and importance for valid and reliable information; sound design and analysis; precise data collection and tracking; clear and explicit reasoning, communication, and reporting; and interpreta- tion of findings relative to the context of the evaluation. Finally, the account- ability standards are intended to assure the overall quality of an evaluation through precise and detailed documentation and internal and external eval- uation of an evaluator’s work (i.e., metaevaluation). This last standard often is addressed through committees formed by an evaluator and stakeholders to monitor the evaluation and assure that these standards are continually maintained. It is also not uncommon for external evaluators to be requested by funders as a part of grant awards.

The evaluation field also is guided by the American Evaluation Association’s Guiding Principles for Evaluators (American Evaluation Association, 2004), which support ethical evaluation practice. The first principle, systematic in- quiry, highlights the importance of conducting systematic, data- based inquiries in order to ensure the accuracy and credibility of information

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 17 ]

obtained from sources. This principle also highlights the importance of clearly and correctly communicating information in ways that are understandable to the evaluation’s stakeholders. The second principle, competence, refers to an evaluator’s competence to perform an evaluation in ways that will address the information needs of its stakeholders. Competent evaluators possess the knowledge, skills, and abilities to conduct an evaluation and refrain from conducting them (or recruit additional experts) when their professional limits are exceeded. Competent evaluators are also sensitive to and respect differences reflected by cultural backgrounds, racial and ethnic heritage, sexual orientation, gender, and religious and other cultural beliefs and values. Culturally responsive “evaluators should also display honesty and integrity, [the third principle], in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.” (American Evaluation Association, 2011). The fourth principle, respect for people, emphasizes an evaluator’s “re- spect [for] the security, dignity and self- worth of respondents, program participants, clients, and other evaluation stakeholders.” This means that during the conduct and reporting of an evaluation, an evaluator avoids phys- ical or emotional harm to a person or the organizations and groups which they represent. Consistent with this principle is the final principle, responsibilities for general and public welfare, for which “evaluators articulate and take into account the diversity of general and public interests and values that may be re- lated to the evaluation.” As you may have observed, the American Evaluation Association’s five Guiding Principles are very consistent with the Program Evaluation Standards developed by the Joint Commission.

If you work globally, you also should be aware of evaluation standards applied in other countries. Similar to the Program Evaluation Standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011), the African Evaluation Guidelines— Standards and Norms (African Evaluation Association, 2006) more strongly emphasize community participation and ownership. The Canadians also have adopted a set of standards and evaluator competencies, which are described in Competencies for Canadian Evaluation Practice (Canadian Evaluation Society, 2010).

Why are these standards important to you? From an outsider’s perspective (outside the evaluation profession), these professional standards are impor- tant for many reasons. First, they help you to understand the lens that guides an evaluator’s work and thinking. Second, the standards also provide a set of guidelines for hiring an evaluator who possesses the appropriate knowledge, skills, and experiences to conduct an evaluation for you. From an insider’s perspective, these standards provide guidance as you increasingly acquire knowledge, skills, and experience for conducting your own evaluations or contributing to an evaluation in a substantial way. For a leader of an evalua- tion, the standards are useful for managing conflict and upholding standards of professional and ethical practice.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 18 ] Part One: Prepare

ETHICS AND HUMAN SUBJECTS PROTECTIONS

In years past, some educators and other professionals presumed that evaluations were “exempt” from federal guidelines for ethical practice and the protection of human subjects that were commonly known among researchers. As a result, some institutions, researchers, and program evaluators expe- rienced adverse consequences for not adhering to these practice standards (DuBois, 2002; Tomkowiak & Gunderson, 2004). Evaluators always should uphold standards for ethical practice and human subject protections and, when possible, gain an institutional review board’s (IRB) approval or exemp- tion for any evaluation study that is planned (James Bell Associates, 2008; Morris, 2008; Rice, 2008). Evaluators and researchers are responsible for protecting humans involved, or potentially involved, in research, including evaluation studies. The role of an IRB in the United States is to oversee re- search and implement the federal regulations for the protection of research subjects (i.e., 45 CFR 46 or the “Common Rule”). These regulations are guided by three ethical principles: beneficence, respect for persons, and justice (National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1979). These principles provide guidance for assuring that the benefits of research exceed the risks associated with partic- ipation, participants consent to participate in the research and can withdraw at any time, and the burdens of research are not experienced by any specific class or group of people.

Human Subjects Protections (HSP) programs expand beyond IRBs and in- clude additional institutional oversight committees that serve nine primary functions: (1) minimize risks to research participants, (2) determine if a study’s benefit- risk ratio is reasonable, (3) make sure that subject selection is fair, (4) there is adequate monitoring of the research, (5) informed consent has been obtained from research participants, (6) participants’ privacy and confidentiality are maintained, (7) a researcher’s conflicts of interest are made explicit, (8) peoples’ vulnerabilities as research participants are addressed, and (9) researchers have participated in and are current in HSP training (Ross, Loup, Nelson, Botkin, Kost, Smith, & Gehlert, 2010). This means that, as an evaluator, you will need to provide the appropriate documentation for an IRB and other ethical oversight committees to make these determinations. Some studies are deemed “exempt” from ongoing IRB oversight, so once your study is deemed exempt, further oversight is not necessary, unless your study plan changes in a way that may increase risks to participants, alter the purpose of the study or data collection process, or contribute to generalizable knowledge. If your evaluation study does require IRB oversight, however, you will be ex- pected to communicate any changes in your study plan (including personnel changes and changes to any forms or documentation used in the research), to maintain meticulous and accurate study records, and to provide annual

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 19 ]

reports. You are also responsible for reporting any adverse events involving study participants and keeping records in secure locations (electronic and physical spaces).

You may be thinking, “What if my organization does not have an IRB or I want to do independent consulting?” Your first step would be to determine whether a proposed evaluation is “non- research” or involves “non- human subjects.” If you do not generalize the evaluation’s findings nor publish the findings from the evaluation study, your work likely qualifies as “non- research,” and you do not need to have your evaluation plan reviewed by an IRB. Quality- assurance and quality- improvement projects, for example, typ- ically do not contribute to generalizable knowledge. If the evaluation study does not involve humans, then it likely can be deemed “non- human.” There are, however, exceptions to these qualifiers, so an evaluator cannot make a determination of “non- research” or “non- human.” You, therefore, have three options: (1) start an IRB at your institution (requires federal approval); (2) use an external, commercial IRB (which will likely charge a fee); or (3) partner with an institution that has an IRB and is willing to serve as the IRB of record for an evaluation study (Rice, 2008).

In participatory and community- based evaluation, the potential burdens and harm to people expand beyond individual research participants. Ross, Loup, Nelson, Botkin, Kost, Smith, & Gehlert (2010) describe poten- tial risks to well- being and agency associated with individuals, groups, and communities when research involves participation to the extent that partners and participants are engaged in the research process (Figure 2.1).

At each level, participants are subject to risks associated with the research process (e.g., physical and psychosocial risks of interactions), outcomes (e.g., influence of findings on group structure or function), and individual, group, or community agency (e.g., undermining personal autonomy or the group’s moral and sociopolitical authority). Therefore, the authors have recommended

Individual

Group

Community

Figure 2.1. Three levels of risk in participatory evaluation.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 20 ] Part One: Prepare

several points to consider when reviewing this type of research or evaluation study. Among them are questions such as, “If there are plans to collaborate with community partners, have the community risks been identified and has a management plan been developed?” (p. 39); “What are the expected benefits to the individual, to the individual by association with the group or to the group as a whole?” (p. 40); and “Who will collect data, where will it be stored, and who will have access to stored data, including individual contact infor- mation?” (p. 42). These are helpful questions to use as guides when designing evaluation studies that use participatory approaches, so that all levels of human subject protections are considered in the inquiry process.

EVALUATION COMPETENCIES

As stated in evaluation standards and guiding principles, evaluators should possess the appropriate competencies to perform and manage an evaluation. What are those competencies? An original set of essential competencies for program evaluators was developed by Jean King and colleagues (King, Stevahn, Ghere, & Minnema, 2001), revised (Stevahn, King, Ghere, Minnema, 2005) and subsequently validated by Wilcox (2012). The revised set of 61 competencies includes the knowledge, skills, and dispositions that program evaluators need for effective professional practice. These competencies are organized into six categories that represent the major activities of an evaluator: (1) professional practice, (2) systematic inquiry, (3) situational analysis, (4) project management, (5) reflective practice, and (6) interpersonal competence (pp. 48– 52). Three of these general categories or domains (i.e., situational analysis, systematic inquiry, and project man- agement) have been deemed especially important for assessment of eval- uation competence among Cooperative Extension academic professionals (Rodgers, Hillake, Haas, & Peters, 2012) who typically work closely with their local communities. The improvement of an organization’s evaluation competency is critical to capacity building; therefore, Preskill and Boyle (2008) articulated the knowledge, skills, and attitudes needed to support capacity- building efforts. Many of these objectives for evaluation capacity building reflect the competencies described by Stevahn and colleagues (2005). Subsequently, Taylor- Ritzler and colleagues (Taylor- Ritzler, Suarez- Balcazar, Garcia- Iriarte, Henry, & Balcazar, 2013), developed a model and corresponding instrument for assessing evaluation capacity building among staff of nonprofit organizations. The Evaluation Capacity Assessment Instrument includes 14 general items (knowledge and skills) for assessing evaluation competence. Again, these items parallel those described by Stevahn and colleagues (Stevahn, King, Ghere, Minnema, 2005), but they

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 21 ]

are limited and lack the specificity of the full set of competencies described in other sources (Canadian Evaluation Society, 2010; ibstpi, 2006; Stevahn, King, Ghere, & Minnema, 2005). More recently, King and Stevahn (2015) cited the necessity for and challenges to updating evaluator competencies to include those that have evolved throughout the last decade and are rele- vant to: cultural competence, international settings, systems thinking and complexity, and developing technologies. What does it take to acquire this skill set?

Dillman (2012) studied the relationship between evaluation competencies and educational experiences and found that both coursework and field ex- perience were necessary to support competency development. Traditionally, evaluators have acquired their education and training within other disciplines, such as education, sociology, public health, or international development. More recently, certificate and graduate programs have been developed, in addition to multiple specialized workshops, for evaluation training and ed- ucation. The American Evaluation Association, a professional organization cur- rently boasting over 7000 members, is a good place to start when looking for evaluation training. Other sources for evaluation training and resources include the Kellogg Foundation, United Way, The Evaluators’ Institute (Claremont Graduate University; http:// tei.cgu.edu/ about/ about- tei/ ), the Minnesota Evaluation Studies Institute (http:// www.cehd.umn.edu/ OLPD/ MESI/ ), the Innovation Network, the National Science Foundation, among other state and national organizations. Why is it important to be a competent evaluator and how do competencies get used?

Evaluation competencies are useful in five major ways (Perrin, 2005). Most importantly, they can be used for self- assessment and monitoring and improving your own knowledge, skills, and attitudes about evaluation. Second, they are useful for assessing gaps in knowledge, skills, and attitudes and for designing or participating in education and training programs to fill those gaps. Third, they foster the development of the profession; and fourth, they provide information and support advocacy about the skills required for competent evaluations and professional practice. They also provide a useful set of standards for selecting, monitoring, and managing evaluators who are hired as consultants and coaches. From my experience, one of the most useful sets of competencies one can possess as an evaluator are skills for effective communication and negotiation. Many of the other important skills you will need for evaluation will be taught in forthcoming chapters. Unfortunately, effective communication skills are not given ade- quate attention in many training or education programs, and they can make or break a successful evaluation effort. In the next section, I will give special attention to the communication and negotiation competencies needed for evaluation.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://tei.cgu.edu/about/about-tei/

http://www.cehd.umn.edu/OLPD/MESI/

[ 22 ] Part One: Prepare

COMMUNICATION AND NEGOTIATION

With the evolution of systems approaches, such as developmental evaluation and collective impact, which involve people representing multiple and diverse sectors and cultural groups that are more prominent in the field, it has become increasingly important for evaluators to be competent communicators and negotiators. So much so that in 2013, Jean King and Laurie Stevahn wrote a book titled, Interactive Evaluation Practice, which emphasized this topic. What are these skill sets?

To begin, clear and direct communication among all stakeholders in- volved in an evaluation are imperative for fostering good relationships and positive experiences in evaluation. Evaluators must be attentive and open listeners, able to convey a clear understanding and interpretation of infor- mation, be aware of cultural norms and values, be open to learning, and have a clear understanding of one’s own biases when communicating with others. Furthermore, evaluators should strive to promote positive relations, shared understandings, and a trustful environment (King & Stevahn, 2013). To do this, evaluators must work with great integrity and be able to manage conflict in a timely manner. Some level of conflict will be likely in almost any evalu- ation but will be reduced if effective communication skills and strategies are routinely used and reinforced.

Several strategies can be used to promote and facilitate effective communi- cations among those involved in an evaluation. It is often helpful to establish communication “ground rules” early in an evaluation partnership. The skills and strategies listed in Box 2.1 are those that I have found most helpful in my work.

Most often, these strategies will help to maintain a smooth and productive evaluation process; however, disagreements and conflicts are not uncommon, and they should be anticipated and expected. How can you manage conflict?

King and Stevahn (2013) provide an excellent chapter on Managing Conflict Constructively in Evaluation Settings (pp. 166– 194), which I encourage you to read. In the next few paragraphs, I will highlight some of the major points from that chapter. The first thing to do is recognize that there is a conflict and manage it with intention and poise. Conflicts are recognized as “incom- patible activities that block, delay or prevent the accomplishment of goals” (p. 167). If left unattended, they can lead to many hard feelings, possibly damaged reputations, and failure to make progress or complete the work. People respond to conflict in multiple ways, including attempts at forcing opinions or ideas on others, withdrawing all together, smoothing over the conflict, reaching a compromise, and working together to solve a problem (King & Stevahn, 2013). The most effective means of resolving conflict is through mutual problem solving in which everyone gains from the solution. King and Stevahn (2013) recommend 13 different strategies to foster positive

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 23 ]

interdependence and provide a list of skills and steps needed for successful co- operative negotiations (pp. 173– 180) as summarized in Box 2.2.

Recently, I was facilitating a meeting of a new partnership, which had the goal of designing and evaluating a novel new program. During our first and daylong meeting, we decided to implement this program within a year’s time. When I left the meeting that day, I felt some reservations about proceeding too quickly and I also sensed some partnering members to be a little disap- pointed in where we ended our conversation that day. At our next meeting, I decided I would facilitate a check- in session following an overview from our last meeting. To do so, I designed a graphic- friendly questionnaire that asked three questions: (1) What are your overall feelings about our partnership

Box 2.1. INTERPERSONAL SKILLS AND STRATEGIES THAT ARE HELPFUL IN EVALUATION PRACTICE

• Listen attentively without interrupting the person who is speaking. • State back what you hear to gain clarity and understanding. • Ask questions when you don’t understand something you hear. • Encourage people to keep an open mind, by posing questions that

suggest opposing viewpoints or present alternative ideas. • Use techniques, such as round- robin check- ins, to provide

opportunities to express opinions, feelings, and ideas. • Confront issues by addressing them immediately and with face- to-

face communication. Avoid e- mail. • Watch body language and make your observations explicit in order

to provide people with opportunities to express opinions and ideas. • Avoid accusations and blame. Listen to all perspectives before

drawing any conclusions. • When giving feedback, “sandwich” negative feedback between posi-

tive feedback. • Present alternative opinions and ideas by stating them in the ab-

sence of personal identifiers. For example, “That’s one way to think about it. Could it be thought about in this way?”

• Communicate in a timely manner and use written documentation to gain and maintain clarity.

• If your efforts to communicate effectively fail, seek the help of experts or someone outside the relationship who can serve as a mediator.

• Manage your own emotions by giving yourself downtime or breaks to put angry feelings in perspective or to seek outside opinions and perspectives.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 24 ] Part One: Prepare

today? (very apprehensive, cautiously optimistic, optimistic, very excited); (2) Does our plan for [action] seem realistic given what you know so far? (not sure, no, maybe, yes); and (3) What three things need to happen today to over- come any apprehensive feelings or move forward with our plans? Much to my surprise, most members of the partnership felt cautiously optimistic about moving forward. When explored further, the sense was that the group was moving a bit too fast, given a variety of circumstances. One partnering or- ganization was in the midst of a reorganization that involved a budget cut. Another partner was looking to retirement with great uncertainty about her position’s future— a position that was key to our program’s success. What surprised me was how open our dialogue became and how well it provided a reality check that clarified next steps. This conversation went a long way toward moving forward with integrity and establishing open communication among partners. In essence, we collectively engaged in a process of exploring each person’s experience of our collaboration by expressing our feelings, thoughts, wants, and observations in an open forum (Bushe, 2009). This exer- cise was effective because it acknowledged everyone’s (as opposed to specific individuals’) experiences in real time.

BUILDING PARTNERSHIPS AND CAPACITY FOR EVALUATION

Partnerships and capacity building efforts are necessary to keep up with the growing demands for skilled evaluators in this age of accountability. By building evaluation capacity, organizations are better equipped to improve

Box 2.2. SKILLS AND STEPS FOR SUCCESSFUL COOPERATIVE NEGOTIATIONS IN EVALUATION PRACTICE (KING & STEVAHN, 2013)

• Express cooperative intentions • Mutually define the conflict • State wants • Express feelings • Explain underlying reasons/ interests • Reverse perspective to communicate understanding • Generate multiple integrative solutions that attend to all interests • Reach agreement on a mutual solution • Process the effectiveness of the problem solving

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 25 ]

their programming efforts. For evaluation capacity building (ECB) efforts to be successful, however, organizations and key stakeholders must be empowered to use evaluation for learning and program or process improvement.

Evaluation capacity building efforts can be challenging, however. Labin and colleagues (2012) reported a research synthesis of the evaluation ca- pacity building literature and found that formal training and hands- on experiences are necessary to change knowledge and behaviors in evaluation. Furthermore, the majority of capacity building efforts involve collaborating partners and participatory evaluation approaches, which are necessary to be adaptive and attend to cultural needs. Many organizations are not accus- tomed to being involved in evaluation processes, such as data collection, that are an integral part of participatory approaches to evaluation. So, evaluators are challenged by the difficulties of engaging partners throughout the evalu- ation process (Preskill, 2014). Resources, especially those related to technical assistance, are necessary to support ongoing data collection and analysis that are critical for process monitoring and ongoing feedback. It is imperative that these efforts be supported by an organization that has supportive leadership and a culture of learning and improvement. This means that an organization supports a culture in which it learns from its failures as well as its successes. It also means that blame is shared and not pointed and that efforts are in place to foster a culture of openness, integrity, and trust. This type of culture is not easy to build or maintain, but it can be done with effective leadership and a consistent emphasis on values and teamwork. As Hoole and Patterson (2008) point out,

The commitment of organizational leadership is critical in transforming the role of evaluation from one of basic reporting and accountability to a true process of continuous organizational learning. This transformation requires leadership commitment to development of an evaluative learning culture and development of an infrastructure to support it. (p. 111)

However, the critical role of a supportive and knowledgeable leader in ca- pacity building efforts remains one of the least understood and studied (Preskill, 2014).

Preskill (2014) highlights two additional challenges for those who desire to build evaluation capacity in organizations or communities. We need to acquire a better understanding of how knowledge, skills, and attitudes are transferred to sustained evaluation practice. This can be accomplished with expanded efforts to evaluate the effects and impacts of our capacity building work (Preskill, 2014). An example for some ways to evaluate ECB efforts can be found in Bakken, Núñez and Couture, 2014, as well as in King and Stevahn, 2013. From my experiences as an evaluator, there is nothing more enjoyable than partnering with a learning organization to do an evaluation.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 26 ] Part One: Prepare

These organizations typically are primed for ongoing assessment and evalu- ation because their learning is typically data driven. This tradition typically is grounded in disciplines other than program evaluation, such as quality improvement, human resources, or human factors engineering, so it is im- perative that you align your program evaluation efforts with these ongoing activities. When entering a community consisting of multiple organizations, however, your role in building evaluation capacity can be more challenging because many organizations do not view themselves as continuing learning organizations. In these circumstances, knowledge of and skills for inclusive practice that builds and maintains effective partnerships for collaborative in- quiry are useful and necessary.

EFFECTIVE PARTNERSHIPS AND COLLABORATIONS

An evaluator’s ability to initiate and maintain effective partnerships and collaborations is crucial to evaluation practice. Today’s complex evaluations typically require an array of expert skills that go beyond those of any single evaluator. Furthermore, relationships with stakeholders who are viewed as partners or collaborators in a participatory evaluation are essential to its success. Therefore, I would like to begin this section by citing one of my fa- vorite sources for defining and clarifying the differences among terms such as partnership, cooperation, and collaboration, to name a few. Drawing from collaborative theory, Gadja (2004, pp. 67– 70) described five principles that define effective collaborations (Box 2.3).

These principles are important because they help to define the differences among cooperation, coordination, collaboration, and coadunation— each re- flecting a different level of integration along a continuum (Gadja, 2004). For those of you who typically become involved with collaboratives (e.g., coalitions) at various stages in their development, these distinctions are helpful for under- standing and facilitating interrelationships among stakeholders as they work toward collaboration and effective participatory evaluation practice (Cousins, Whitmore, & Shulha, 2012). To collaborate effectively, “each person [should] feel equally responsible for the success of their joint project or process . . . ; [therefore], attempts to define who has the ‘right experience’ reduce collabo- ration” (Bushe, 2009, p. 19). So, it is important that open communication and dialogue be facilitated and advocated in participatory evaluation practice.

INCLUSIVE PRACTICE IN EVALUATION

Evaluators continually work with people who have a variety of cultural backgrounds. By culture, I mean cultural experiences derived through gender,

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 27 ]

race, ethnicity, age, sexual orientation, socioeconomic status, different work environments, family backgrounds, and so on. We are all different and some- times those differences create tensions among us. It is, therefore, important to have a skill set that helps to both acknowledge difference and reduce tension when it occurs. Where do you begin? First, it is imperative that you understand your own cultural norms, biases, and values and realize that they are different from everyone else’s. Moreover, you must work “inside- out” to “cultivate empa- thetic perspective taking [that] acknowledges and regularly polishes the lenses and filters that frame your perceptions and meaning- making reflections and interpretations— discover what they illuminate and, even more important, what they obscure or ignore” (Symonette, 2004, p. 97). Through understanding our own cultural identity and bias, we are better able to recognize the differences among us and open our minds and hearts to other perspectives. Second, you must learn what values, experiences, and ways of knowing the world are held by those with whom you work. When you are open to learning and knowing other cultures, you can begin to engage effectively with others and value their unique contributions. Third, you must look at strengths in others that can ben- efit your partnerships and evaluations. Doing so acknowledges everyone and can contribute something special to your work together.

Methods of Appreciate Inquiry can be particularly helpful (Coghlan, Preskill, & Catsambas, 2003; Preskill & Catsambas, 2006). For example, years

Box 2.3. FIVE PRINCIPLES THAT DEFINE EFFECTIVE COLLABORATIONS (GADJA, 2004)

1. Collaboration is an imperative when dealing with complex social issues and community needs;

2. Collaboration is known by many names including, but not limited to, joint ventures, partnerships, coalitions, collaboratives, alliances, consortiums, task forces, and so on (p. 68);

3. Collaboration is a journey not a destination of efforts that “fall across a continuum of low to high integration. The level of integra- tion is determined by the intensity of the alliance’s process, struc- ture, and purpose” (p. 68);

4. With collaboration, the personal is as important as the procedure or process of evaluation because unhealthy interpersonal connections and lack of trust can thwart the group’s efforts.

5. Collaboration develops in stages as group members get to know one another and determine their collaborative vision, mission, and agenda.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 28 ] Part One: Prepare

ago I supervised a woman who was extremely detail oriented— an orientation that I do not have and must force myself into when needed. Our conversations initially drove me nuts as I had to hear every little detail about her work when I just wanted to “cut to the chase” and get the simplified version. I came to learn that her detail orientation was a great asset and complement to my big picture thinking. I grew to rely on her to keep track of details that seemed overwhelming to me but absolutely necessary for the function of our pro- gramming efforts. I bring this up because almost everyone has a quality like my detail- oriented colleague that can be an asset to an evaluation partnership. Capitalize on their talents, whether it be someone who likes rigid structure, someone who speaks another language (broadly defined), someone who likes to keep detailed notes, someone who likes to track progress, someone who shares experiences with program participants, someone who likes to work with numbers, or someone who likes to socialize— you can find a role for them that will make a valuable contribution to an evaluation.

When you are entering culturally unfamiliar settings, it is imperative that you develop relationships and do a thorough cultural analysis with a learning mindset (Dweck, 2006) before you begin any evaluation. This process can take from just a few weeks to one or two years depending on the evaluation and the culture in which you find yourself. Learn who has power, how things get done, how fast work gets done, who might have to give approval, what stakeholders should be involved, how contacts are made, how people communicate, who communicates most effectively, what different forms of communication mean and so on. BE A LEARNER— just as a toddler would explore his new world. Keep your eyes and ears open and learn.

In the context of understanding a community’s needs, I recommend that you begin with a set of five questions described by Wambeam (Wambeam, 2015, pp. 11– 12) to define your community (Box 2.4).

These questions provide a foundation for gaining a collective understanding of a community, which defines the context, or in some cases the “unit of anal- ysis,” for your evaluation. As you can see by the questions, they help to inform your understanding of the cultural context for your evaluation; however, this step is only the beginning. As evaluators we must be inclusive and perform our practice with cultural competence.

What does it mean to be a culturally competent and inclusive evaluator? In 2005, AEA’s membership convened a task force of evaluation and cultural competence experts to tackle this question and issue a statement, which was approved in 2011. The statement’s summary reads in part,

Culture is the shared experiences of people, including their languages, values, customs, beliefs, and mores. It also includes worldviews, ways of knowing, and ways of communicating. Culturally significant factors encompass, but are not

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 29 ]

limited to, race/ ethnicity, religion, spirituality, social class, caste, language, lineage, disability, sexual orientation, age, gender, geographic region, and so- cioeconomic circumstances. It is important to note that while these factors in- clude culture they are not fixed and can change over time. (American Evaluation Association, 2011)

This statement’s summary also provides three main reasons why cultural com- petence is important in evaluation.

First, the evaluation team is ethically responsible to be culturally competent in order to produce work that is honest, accurate, respectful of stakeholders, and considerate of the general public welfare. Second, cultural competence supports validity by insuring that diverse voices and perspectives are honestly and fairly represented, which in turn, helps to make valid inferences and interpretations. Third, evaluation is steeped in theories that are, themselves shaped by cultural values and perspectives; therefore, it is important to scrutinize theories in order to understand how they describe societal issues, and how to address them.

In Box 2.5, I summarize five essential practices for cultural competence in evaluation as they are listed in AEA’s summary statement (2011). Detailed descriptions of each practice can be found at AEA’s web site, www.eval.org.

Box 2.4. FIVE QUESTIONS FOR UNDERSTANDING A COMMUNITY’S NEEDS (WAMBEAM, 2015, PP. 11– 12)

1. Is your community geographic, organizational, or cultural? If your community is geographic, define the boundaries of your community’s location and describe its size.

2. Describe your community. What is its culture like? Is it rural or urban? What kind of climate does it experience? What values, attitudes, and beliefs are most important? What kinds of jobs do people have? Are there dominant religions?

3. Document your community demographics, including the total pop- ulation, median age, percentage of the population in various ethnic groups, percentage of the population with specific levels of educa- tion, and the median household income. Other demographics you may wish to include are gender, spoken languages, marital status, and other culturally relevant demographics.

4. In sum, what makes your community unique? 5. Where did you find the information to answer the above questions?

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.eval.org

[ 30 ] Part One: Prepare

By adhering to these practice guidelines, we reap the rewards of learning from and building and sustaining relationships across multiple cultural groups. By knowing ourselves and others in this way, our lives and careers be- come enriched with meaning and joy.

SUMMARY

This chapter described the essential competencies and interpersonal skills needed for evaluators and the standards of professional evaluation practice. It also highlighted the necessity and importance of protecting human subjects as subjects of and participants in evaluations. I emphasized the need to under- stand culture and be a self- reflective and culturally competent evaluator. These competencies and skills help to foster and facilitate positive relationships with stakeholders and clients that support successful evaluation practice. In the next chapter, you will learn about the major approaches to evaluation and how stakeholders may influence the approach you choose to evaluate a program.

REFERENCES

American Evaluation Association. (2004). Guiding Principles for Evaluators. Retrieved from http:// www.eval.org/ p/ cm/ ld/ fid=51

American Evaluation Association. (2011). American Evaluation Association Public Statement on Cultural Competence in Evaluation, Summary. Fairhaven, MA: Author. Retrieved from http:// www.eval.org/ d/ do/ 155.

African Evaluation Association. (2006). African Evaluation Guidelines— Standards and Norms. Retrieved from http:// www.ader- evaluare.ro/ docs/ African%20 Evaluation%20Association.pdf

Bakken, L. L., Núñez, J., & Couture, C. (2014). A course model for building evalua- tion capacity through a university- community partnership. American Journal of Evaluation, 35(4), 579– 593.

Box 2.5. AEA’S FIVE ESSENTIAL PRACTICES FOR CULTURAL COMPETENCE IN EVALUATION

1. Acknowledge the complexity of cultural identity. 2. Recognize the dynamics of power. 3. Recognize and eliminate bias in social relations. 4. Employ culturally congruent epistemologies, theories, and methods. 5. Continue self‐assessments.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.eval.org/p/cm/ld/fid=51

http://www.eval.org/d/do/155

http://www.ader-evaluare.ro/docs/African%20Evaluation%20Association.pdf

a c q u i r i n g r e q u i s i T e k n o w L e d g e a n d s k i L L s [ 31 ]

James Bell Associates. (2008). Evaluation Brief: Understanding the IRB. Arlington, VA. January 2008. Retrieved from https:// www.betterevaluation.org/ sites/ default/ files/ understanding%20the%20irb.pdf

Bushe, G. R. (2009). Learning from collective experience: A different view of organiza- tional learning. Organizational Development Practitioner, 41(3), 19– 23.

Canadian Evaluation Society. (2010). Competencies for Canadian Evaluation Practice. Retrieved from https:// evaluationcanada.ca/ txt/ 2_ competencies_ cdn_ evalua- tion_ practice.pdf

Catsambas, T. T. (2016). Facilitating evaluation to lead meaningful change. In R. S. Fierro, A. Schwartz, & D. H. Smart (Eds.), Evaluation and Facilitation, New Direction for Evaluation, 149, 19– 29.

Coghlan, A. T., Preskill, H., Catsambas, T. T. (2003). An overview of appreciative in- quiry in evaluation. In H. Preskill & A. T. Coghlan (Eds.), New Directions for Evaluation, 100, 5– 22.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. In E. Whitmore (Ed.), New Directions for Evaluation, 80, 5– 23.

Cousins, J. B., Whitmore, E., & Shulha, L. (2012). Arguments for a common set of principles for collaborative inquiry in evaluation. American Journal of Evaluation, 34(1), 7– 22.

Dillman, L. M. (2012). Evaluator skill acquisition: Linking educational experiences to competencies. American Journal of Evaluation, 34(2), 270– 285.

DuBois, J. M. (2002). When is informed consent appropriate in educational research? IRB: Ethics & Human Research, 24(1), 1– 8.

Dweck, C. (2006). Mindset: The new psychology of success. New York: Random House, Inc. Gadja R. (2004). Utilizing collaboration theory to evaluate strategic alliances. American

Journal of Evaluation, 25(1), 65– 77. Hoole, E., & Patterson, T. E. (2008). Voices from the field: Evaluation as part of a

learning culture. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and evalu- ation. New Directions for Evaluation, 119, 93– 113.

International Board of Standards for Training, Performance and Instruction (ibstpi). (2006). Evaluator Competencies. Retrieved from http:// ibstpi.org/ evaluator- competencies/

King, J. A., & Stevahn, L. (2013). Interactive evaluation practice: Mastering the interper- sonal dynamics of program evaluation. Thousand Oaks, CA: Sage Publications.

King, J. A., & Stevahn, L. (2015). Competencies for program evaluators in light of adapt ive action: What? So what? Now what? In J. W. Altschuld & M. Engle (Eds.), Accreditation, certification, and credentialing: Relevant convers for U.S. evaluators. New Directions for Evaluation, 145, 21– 37.

King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of es- sential evaluator competencies. American Journal of Evaluation, 22(2), 229– 247.

Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A re- search synthesis of evaluation capacity building literature. American Journal of Evaluation, 33(3), 307– 338. doi: 10.1177/ 1098214011434608.

Mertens, D. (2009). Transformative research and evaluation. New York, NY: Guilford Press. Morris, M. (2008). Evaluation ethics for best practice: Cases and commentaries. New York,

NY: The Guilford Press. National Commission for the Protection of Human Subjects of Biomedical and

Behavioral Research. (1979). The Belmont Report. Retrieved from the U.S. Department of Health and Human Services, Office for Human Research Protections website: https:// www.hhs.gov/ ohrp/ regulations- and- policy/ belmont- report/ index.html

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

https://www.betterevaluation.org/sites/default/files/understanding%20the%20irb.pdf

https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf

https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html

[ 32 ] Part One: Prepare

Perrin, B. (2005). How can information about the competencies required for evalua- tion be useful? The Canadian Journal of Program Evaluation, 20(2), 169– 188.

Preskill, H. (2014). Now for the hard stuff: Next steps in ECB research and practice. American Journal of Evaluation, 35(1), 116– 119. doi:10.1177/ 1098214013499439/

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443– 459. doi:10.1177/ 1098214008324182.

Preskill, H., & Catsambas, T. T. (2006). Reframing Evaluation through Appreciative Inquiry. Thousand Oaks, CA: Sage Publications, Inc.

Rice, T. W. (2008). How to do human- subjects research if you do not have an Institutional Review Board. Respiratory Care, 53(10), 1362– 1367.

Rodgers, M. S., Hillake, B. D., Haas, B. E., & Peters, C. (2012). Taxonomy for assessing evaluation competencies in extension. Journal of Extension, 50(4). Retrieved from https:// joe.org/ joe/ 2012august/ a2.php

Ross, L. F., Loup, A., Nelson, R. M., Botkin, J. R., Kost, R., Smith, G. R., & Gehlert, S. (2010). Nine key functions for a human subjects protection program for community- engaged research: Points to consider. Journal of Empirical Research on Human Research Ethics, 5(1), 33– 47.

Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43– 59.

Symonette, H. (2004). Walking pathways toward becoming a culturally competent evaluator: Boundaries, borderlands, and border crossings. In M. Thompson- Robinson, R. Hopson, & S. SenGupta (Eds.), In Search of Cultural Competence in Evaluation: Toward Principles and Practices. New Directions for Evaluation, Summer 2004(102), 95– 109. doi: 10.1002/ ev.118

Taylor- Ritzler, T., Suarez- Balcazar, Y., Garcia- Iriarte, E., Henry, D. B., & Balcazar, F. E. (2013). Understanding and measuring evaluation capacity: A model and instru- ment validation study. American Journal of Evaluation, 34(2), 190– 206.

Tomkowiak, J. M., & Gunderson, A. J. (2004). To IRB or Not to IRB? Academic Medicine, 79(7), 628– 632.

Wambeam, R. A. (2015). The community needs assessment workbook. Chicago, IL: Lyceum Books, Inc.

Wilcox, Y. (2012). An initial study to develop instruments and validate the Essential Competencies for Program Evaluators (ECPE) [doctoral dissertation]. Retrieved from the University of Minnesota. (http:// conservancy.umn.edu/ handle/ 11299/ 132042)

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The Program Evaluation Standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

https://joe.org/joe/2012august/a2.php

http://conservancy.umn.edu/handle/11299/132042

C H A P T E R 3

Choosing an Evaluation Approach

Program evaluation designs are rooted in an array of philosophical perspectives and approaches. These approaches have evolved over time and are grouped in various ways depending on the source that describes them. For any given evaluation, it is common for an evaluator to use more than one approach. What are common approaches to evaluation in today’s world of ac- countability? This chapter will begin with a discussion of the philosophical perspectives that ground each set of approaches, provide an historical over- view of the evolution of evaluation approaches, and describe five groups of approaches commonly used today.

PHILOSOPHICAL PERSPECTIVES THAT INFLUENCE EVALUATION

Four fundamental philosophical perspectives (i.e., world views or lenses) influ- ence the evaluation approaches described in this chapter. These perspectives orient the role of the evaluator in conducting an evaluation, the degree to which stakeholders are involved, the methodologies selected, and the level of objectivity or subjectivity of the evaluation’s approach and findings. Authors who write about evaluation perspectives organize them in various ways. For the purpose of this section, I will use the philosophical distinctions de- fined and described by Fitzpatrick, Sanders, and Worthen (2011) in Program Evaluation: Alternative Approaches and Practical Guidelines and incorporate concepts from the ways other authors have described or organized them as a way to tie them together.

The aim of positivist and postpositivist approaches is to be as objective as possible in the conduct of an evaluation; therefore, the desirable role for an

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 34 ] Part One: Prepare

evaluator is to be positioned “outside” the evaluation so that the evaluation is less subject to personal bias. For this reason, too, stakeholder involve- ment in the evaluation’s plan, conduct, or interpretation is typically limited in order to seek objective truth. Quantitative analytical methods typically are used because this perspective values the “more credible” evidence pro- vided by these methodologies (see Chapter 5 for further discussions of cred- ible evidence). In contrast, humanist perspectives highly value the ideas and opinions of stakeholders; therefore, any evidence provided by stakeholders is considered credible and useful when planning, conducting, or reporting an evaluation. Furthermore, the role of an evaluator is to engage with stakeholders to the extent that the stakeholders “own” the evaluation and its findings. Although positivists would consider this perspective and its accompanying approaches to be highly subjective, the humanist would argue that truth is best known through those who experience it. Constructivists acknowledge the important role of context in evaluation and the influence it has on knowledge construction. Context influences stakeholders and evaluators and the power relationships among them. Therefore, an evalu- ator might engage with stakeholders to varying degrees depending on power dynamics and their corresponding influences. Transformative approaches to evaluation emphasize political, social, and economic factors that shape re- ality. Therefore, stakeholder empowerment and social justice are emphasized (Mertens, 2009; Mertens & Wilson, 2012). Stakeholders’ knowledge and experiences are highly valued along with the contextual influences that may have marginalized specific persons or groups.

These various perspectives are important because they have shaped evalua- tion approaches both historically and practically. As you read through the var- ious approaches to evaluation, be mindful of the philosophical perspectives that form their foundations. Can you think of specific contexts in which certain approaches may be more acceptable or useful than others? Which approaches might be most useful with oppressed cultural groups? Which approaches might be most useful when working with government agencies or policymakers? How do your own values and beliefs align with these approaches?

EXPERTISE- ORIENTED APPROACHES

As their name implies, expertise- oriented approaches rely on experts to set the standards or criteria for which a program or individuals are to be judged. The goal of these approaches is to provide professional judgments of quality. Can you think of any examples that meet this definition within your own pro- fessional groups? Examples from practice include but are not limited to blue ribbon panels, accrediting organizations, and groups such as search and screen

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 35 ]

committees, dissertation committees, or judges at county fairs. Accreditation groups are numerous and highly respected in various professional fields. Some common examples that may be familiar to you are:

The Joint Commission that provides accreditation and certification to more than 19,000 healthcare organizations and programs in the United States (http:// www.jointcommission.org);

The Leadership in Energy and Environmental Design (LEED) Certification and Accreditation for organizations and individuals that comply with a set of standards to protect human health and our environment (http:// www.usgbc. org/ leed);

The Council on Social Work Education’s (CSWE) Commission on Accreditation (COA) “develops accreditation standards that define competent preparation [for social work] and ensures that social work programs meet them” (http:// www. cswe.org/ accreditation.aspx);

The Council for the Accreditation of Educator Preparation (CAEP) that creates standards for educators and accredits programs throughout the country (http:// caepnet.org/ ); and

The Council for Higher Education Accreditation (CHEA).

CONSUMER- ORIENTED APPROACHES

When I think of consumer- oriented approaches, I think of the few times I have walked into a local dairy center and been invited to taste- test a new product. I am directed to a private booth where I find a scoring sheet and one to three food samples. Other, more common, examples include the occasional marketing surveys I am asked to complete when I make a store or online pur- chase. The defining feature of these evaluation approaches is their emphasis on judging the quality of products or services to aid purchasing decisions. In this case, evaluation criteria are established by the producer and used as a way to market products. Common data- collection tools are checklists or rating forms.

PROGRAM- ORIENTED APPROACHES

For educators, social workers, healthcare professionals, community developers, and other service providers, program- oriented approaches to eval- uation are the most common approaches practiced in today’s accountability- oriented society. These approaches are driven by program theory primarily derived through stakeholders’ ideas and experiences and knowledge reported in academic literature. The goal of these approaches is to determine the extent

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.jointcommission.org

http://www.usgbc.org/leed

http://www.cswe.org/accreditation.aspx

http://caepnet.org/

[ 36 ] Part One: Prepare

to which program goals (i.e., desired outcomes) and objectives are achieved. Over the years, there has been an increasing emphasis on programs’ impacts on communities or organizations as decisions are made by governing bodies to determine a program’s public worth or value. Evaluation criteria, there- fore, may be established collectively by involving representatives of various stakeholder groups. Program- oriented approaches are highly endorsed among local, state, and federal funding agencies and nonprofit foundations that re- quire evaluations of their grantees, so they tend to dominate many evaluations done today.

There are three major types of program- oriented evaluation approaches: (1) objectives- based, (2) discrepancy- based, and (3) theoretical. Objectives- based program evaluation (Tyler, 1942) is common among educators who typically want to understand the extent to which learners have achieved a program’s or curriculum’s desired learning objectives or outcomes. The nursing and nutri- tion education fields use this approach extensively to evaluate learning and assess professional performance. The major steps in an objectives- based pro- gram evaluation (Fitzpatrick, Sanders, & Worthen, 2011, p. 155) are listed in Box 3.1. In many fields (e.g., nursing, medicine, and K– 18 education), it is common practice to embed an objectives- based approach into an evaluation of an educational program.

A more recent emergence related to objectives- based evaluation is the assessment and evaluation of competence. King and Stevahn (2015) make an important distinction between competence and competencies.

Competence is the ‘habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection in daily

Box 3.1. MAJOR STEPS IN AN OBJECTIVES- BASED PROGRAM EVALUATION

1. Establish broad program goals or objectives 2. Classify the goals or objectives 3. Define objectives in behavioral terms 4. Select a set of learning activities that create situations in which

learning objectives can be achieved 5. Develop and/ or select measurement techniques to assess the

achievement of learning objectives 6. Collect performance data 7. Compare performance data with stated objectives

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 37 ]

practice for the benefit of the individual and community being served’ (Epstein & Hundert, 2002, p. 226). By contrast, the requirements for competencies are knowledge, skills, and attitudes that are applied and observable (Schoonover Associates, 2003), specific practices related to particular knowledge, skills or dispositions (p. 23).

Therefore, competency- based assessment and evaluation take objectives- based evaluation one step further by focusing the assessment on the application and use of knowledge, skills, and attitudes. Competency- based assessment in program evaluation is the cornerstone of medical education (Carraccio et al., 2016; ten Cate & Billett, 2014), and it is becoming increasingly popular in higher education (Lacey & Murray, 2015). Objectives- and competency- based approaches to assessment and program evaluation provide a means to main- tain professional standards and monitor performance. Used as an assessment approach, however, competencies can create a mindset to learn and teach “to the test,” so facts and memorization tend to be emphasized versus the higher- order thinking and learning required of application (Dweck, 2007).

Discrepancy- based program evaluation (Provus, 1971) is an extension of objectives- based program approaches that rely on a set of standards by which to judge a program’s performance. This type of evaluation takes the form of a con- tinuous information management process from which performance data is used to improve, maintain, or terminate all or part of a program. Steps for conducting this type of evaluation are to determine the inputs, processes, and outcomes that define a program; identify discrepancies between program design and implemen- tation; collect data on participants’ abilities to achieve “enabling” objectives; and determine whether program objectives have been achieved (Fitzpatrick, Sanders, & Worthen, 2011). A cost- benefit analysis may be added to determine whether a program’s benefits to learners or a community exceed the costs associated with implementing or maintaining a program. Although not as common as in years past, it remains a useful approach for program evaluations.

Theory- based evaluations (Chen, 1990; 2005; Weiss, 1997) are the most common form of program- oriented approaches used today. They are extremely popular among funders who typically require them in grant applications. This requirement often can create anxiety for a program planner or coordinator with limited experience in designing or conducting an evaluation and is one of the reasons why program evaluators are frequently consulted. So, what is a theory- based or rather, theory- driven evaluation? Theory- driven evaluations capitalize on the features of objectives- based and discrepancy- based evaluations by combining program goals, objectives, and desired outcomes to create a theory about how a program is perceived to work. Dating back to Carol Weiss’s work in 1997, program theory helps to focus an evaluation and its associated questions, and guides the evaluation’s overall design.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 38 ] Part One: Prepare

A logic model is a technique and tool for visually conceptualizing a theory about how a program affects desired changes. In simple terms, a logic model illustrates the relationship between action (what you do) and change (what you get) (Knowlton & Phillips, 2009). The basic elements of a logic model, as il- lustrated in Figure 3.1, include inputs (e.g., people, funding, or space) for a program, the program’s activities (e.g., participant recruitment, a workshop, or informational brochure), immediate outputs (e.g., immediate knowledge or skills) from the program’s activities, and the outcomes or changes (e.g., knowledge application or use) expected as a result of the program’s activi- ties. Program impacts, or the broad effects of a program on organizations and communities, have become an increasingly important component of program theory.

By incorporating impacts into logic models, program stakeholders can account for ways they expect a program and its associated outcomes to con- tribute or provide value to communities. For example, if I teach someone how to search, apply, and interview for a job (activities and outputs), then they apply those skills by applying and interviewing for jobs more often (short- term outcomes), and then they become employed (medium- term out- come). This effort will eventually lead to higher employment in a community (impact).

When developing program theory, it is important to identify the assumptions that influence the theory. Explicit assumptions can help to identify biases, limitations, overlooked components, or excluded stakeholders.

As you view the basic components of the logic model depicted in Figure 3.2, you will notice the arrows that connect each of its components. These arrows signify the relationships between the theory’s components. All too often, in- sufficient attention is given to the various relationships between or among the program’s activities and anticipated changes. The result is that critical components are omitted, components are identified that have no logical

Input Activities Outputs Outcomes Impact

Figure 3.1. Five elements of a basic logic model.

Instructors, space,

participants Workshop

Knowledge & Skills for applying for

jobs

Increased Job

applications and

interviews

Increased employment

rates

Figure 3.2. Example of logic model for employment.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 39 ]

connection, or the model is oversimplified because the program has more than one “unit” of change.

As you read the extant literature on program theory, you may notice that “impact” is conceptualized in two main ways. Scholars such as Knowlton and Phillips (2009) and Preskill and Russ- Eft (2005) view impacts as the end point of a logical process; whereas, others (e.g., Bjurulf, Vedung, & Larsson, 2012) define impact as a type of evaluation design in which outcomes are assessed in relation to a comparison or counterfactual group. In my opinion, the latter definition is really only a direct assessment of a program’s outcomes that is based on causal static models and does not provide evidence of the relationship of outcomes to impacts as previously defined. To truly demon- strate an impact, one must be able to provide evidence that a relationship exists between a program’s outcomes and broader organizational or commu- nity change. This point is echoed in my forthcoming discussions of impact and will become clearer in Chapter 6, when I discuss the various study designs commonly used for program evaluations.

As typically illustrated, program theories are conveyed to summarize the results of a program’s implementation. In other words, logic models are used to plan a summative evaluation. Indeed, many of you may be inter- ested in assessing only the summative outcomes of a program. Outcomes can also be used, however, as information to improve a program and its associ- ated outcomes by considering them as input in an ongoing process of change (Chen, 1996). Figure 3.3 provides a simple illustration of a program viewed as part of a program improvement cycle, or formative evaluation process. My point in illustrating this form of a logic model is twofold: (1) Program theories can take on multiple forms and (2) evaluations can be conceptualized as part of a program improvement process, or formative evaluation.

Why is it important to create a program’s theory of change? Often, you may have a desire or be expected to determine the effectiveness of a pro- gram to create specific outcomes or an impact on individuals, organizations, or communities. Without having some idea or vision of how that happens,

Input

OutputsOutcomes

Impact Activities

Figure 3.3. Logic model illustrated as a theory for formative evaluation.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 40 ] Part One: Prepare

programming efforts become a “hit or miss” game. In other words, the assessed outcomes may not have any association with the activities that are part of the program being evaluated. Moreover, in the absence of critical thought or di- alogue with stakeholders about the broad effects of a program, evaluations often are limited to participants’ immediate gains in specific knowledge and skills (as in objectives- based evaluations). As a result, longer- term changes (e.g., participants’ actions or behavioral changes) that stem from program- ming efforts can be overlooked.

Another point to keep in mind is that in order to evaluate a program’s effec- tiveness to achieve a desired change, an evaluator must rule out other factors that might contribute to that change. To do this, theory- driven approaches that consider potential mediators and moderators of change are needed (Chen, 2005). To truly understand whether a program has caused a desired change or effect (i.e., efficacy evaluation), very controlled conditions are nec- essary. However, as practitioners in social science fields, we know that highly controlled conditions are typically impossible to achieve because we work with humans who aren’t very controllable. Chen (2005), therefore, advocates effec- tiveness evaluations, which encourage us to account for factors outside the pro- gram that could be influencing the desired change(s) (outcomes or impacts). By doing so, we can evaluate with greater certainty a program’s contributions to those desired changes.

Program theory and its associated logic models have been criticized for their linear reasoning and oversimplification because programs don’t operate in vacuums (Weiss, 1997). Group dynamics influence learning experiences, en- vironmental factors influence program implementation, and social conditions influence program effects. When used in practice, components of logic models also may be misidentified or the relationships among them may not be clear and well- defined, or they may be missing. Consequently, data collected for evaluation becomes disassociated buckets of information that may have lim- ited utility in making inferences about a program’s effectiveness or need for improvement. More recently, logic models have been expanded to include action theory in order to reduce program planners’ and directors’ inclinations to “box” each element void of their relationships or restrict theories and cor- responding assessments to program outputs and short- term outcomes that have no actionable effect (Funnell & Rogers, 2011, pp. 199– 240). “In simple terms, the theory of action is about what the program does or expects to do in order to activate the change theory” (p. 199). The action elements of a robust program theory include the:

Desired attributes of intended outcomes; attention to unintended outcomes; Program features and external factors that affect outcomes; and What the program does to address key program and external factors. (p. 200)

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 41 ]

As a result, program theories often become less linear and more robust as they attempt to include more action- oriented components. These robust pro- gram theories are typically more complex and may require sophisticated study designs and analytical methods to evaluate them. Nonetheless, logic models can be useful tools for defining and establishing clear relationships among program elements and can provide useful frameworks for designing an evalu- ation plan when constructed and used appropriately.

It is important to understand that theory- based evaluation approaches and their accompanying conceptual models (e.g., logic models) do not by them- selves constitute an evaluation plan. They simply help to visualize and focus program- related factors that are critical for designing an evaluation. Program theories also may include some different elements depending on their in- tended purposes and underlying theoretical perspectives of change (e.g., empowerment, diffusion, socioecological, or network theories). Variations of them exist with names such a “logframes” or “outcome maps” (Funnell & Rogers, 2011). Regardless of the approach or specific model, the steps listed in Table 3.1 are recommended by Donaldson (2007) when one works with stakeholders to develop program theory (Fitzpatrick, Sanders, & Worthen, 2011, p. 163). I have adapted these steps by adding an additional and optional step to conduct a logic analysis as a reminder to consider best practices, re- search, and expert knowledge (including stakeholders’) when developing or analyzing a program theory.

A logic analysis (Brousselle & Champagne, 2011) offers a process for de- termining the accuracy of a program’s theory of change. It is guided by three steps that evaluate the plausibility or reasonableness of a program’s theory based on criteria established through a literature review and expert knowl- edge. In the first step of a logic analysis, a logic model is created or obtained through a situational analysis to understand a program’s current or active theory of change. In the second step of the logic analysis, knowledge of best practices and approaches for achieving the desired outcomes and impacts are obtained through field experts and a targeted literature review. I would like to

Table 3.1. S T E P S F OR DE V E L OP I N G A PR O G R A M T H E ORY

STEP 1. Engage relevant stakeholders

STEP 2. Develop a first draft of a program theory

STEP 3. Present the draft to stakeholders for further discussion, reaction, and input

STEP 4. Communicate these findings to key stakeholders and revise the program theory as

needed

STEP 5. Probe arrows for model specificity

STEP 6. Finalize program impact theory

STEP 7. Conduct logic analysis (optional)

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 42 ] Part One: Prepare

note that in participatory evaluation, key stakeholders are considered experts. When experts’ information is compared with the logic model developed in Step One, new information is obtained that can be used to improve program- ming and evaluation efforts.

As an example, I and colleagues at an academic medical center used logic analysis to expand an educational series’ program theory to a systems’ theory for improved patient care. When our team began to learn about how the program’s key stakeholders envisioned a theory for how the program impacted patient care, we quickly learned that many of the desired short- and long- term changes they expected were not reflected in an existing logic model or being assessed. Moreover, the stakeholders identified key people and relationships to other units within the academic medical center that highlighted the organization’s emphasis on continual learning, quality improvement, and improved patient care within a complex medical system. By using logic anal- ysis to incorporate local expertise about the healthcare center and litera- ture on organizational learning, systems change, and similar programming efforts, we were able to illustrate a more robust systems theory for how the educational series contributed to patient care (Bakken, Olson, Ross, Turco, Jackson, & Murphy, 2015). This effort provided a foundation for designing a three- phase evaluation study and targeting our data collection efforts, so we could determine how and, ultimately, the extent to which, the educational series influenced healthcare providers’ practices toward improving patient outcomes.

DECISION- ORIENTED APPROACHES

Decision- oriented approaches build on program- oriented approaches by using information about outcomes, impacts, or costs to inform or make decisions about a program. Evaluators tend to work closely with managers or other administrators in decision- making roles when planning and conducting these types of evaluations. A common example includes a funder who may use eval- uation findings to discontinue or provide continued funding for a program. It is also not uncommon for an administrator to ask, “What is my return on investment for that program?” Consequently, an evaluator’s knowledge of the analyses used to determine a program’s cost to benefit ratio, cost- effectiveness, or return on investment are particularly useful. The decision- making criteria in this case often are determined by directors, administrators, governing boards, or others who have a financial stake in the program, but who often don’t design or implement a program.

Two models have been described to guide evaluations directed at making decisions. The Context, Input, Process, Product (CIPP) Model (Stufflebeam, 1983, 2013) was developed to support programming decisions for

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 43 ]

accountability, dissemination, and understanding. It has been described by some evaluators as “four evaluations in one,” because each component is meant to address a question or questions typically raised at various points in the evaluation process. To understand the context of a program, evaluators typically ask, “What are the needs and problems of those being served?” The an- swer to this question guides planning decisions and leads to the next question related to program input: “In what ways should their needs be addressed?” Once these questions are answered and the program is designed and implemented, a process evaluation is needed to answer the question, “Was the program implemented as planned?” When the program’s products or outputs are deter- mined, the final questions can be evaluated. “Should the program be revised, ex- panded, or discontinued?” As you carefully critique each step in the CIPP Model and think about my earlier descriptions of other approaches, can you identify some of the major assumptions and limitations of this model?

The UCLA Model (Alkin, 1991) expands the CIPP Model by accounting for not only the program planning and implementation process, but by consid- ering the purpose(s) of an evaluation aimed at decision- making. For example, a program’s purpose might be to determine whether or not to implement a program. Another purpose might be to improve or terminate a program. Yet another purpose of an evaluation might be to determine whether a program meets accreditation criteria.

Decision- oriented approaches can create tension among staff and administrators. To ease this tension, it is helpful to combine these and other program- oriented approaches with approaches advocating participation. In the sections that follow, an evaluator’s role begins to shift when he or she utilizes approaches that emphasize stakeholders as important and valued experts and contributors to an evaluation.

PARTICIPANT- ORIENTED APPROACHES

As their categorization implies, these approaches advocate stakeholder in- clusion and empowerment throughout the program planning, implementa- tion, and evaluation process. Today, the goal of these approaches centers on social justice by incorporating multiple realities of key stakeholder groups. “The term participatory, however, is used quite differently by different people. For some it implies a practical approach to broadening decision making and problem solving through systematic inquiry; for others, reallocating power in the production of knowledge and promoting social change are the root issues” (Cousins & Whitmore, 1998, p. 5). Participatory approaches tend to have greater emphasis on the formative process of a program’s evolution and typi- cally strive to build an organization’s or community members’ capacity for eval- uation through participation and learning. Using participatory approaches, the

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 44 ] Part One: Prepare

evaluator serves the roles of coach, teacher, and facilitator and works collabo- ratively with a variety of stakeholders to determine the questions and criteria for an evaluation.

Participatory evaluation is divided into two general approaches based on goals, purposes, and historical and ideological roots— namely, Practical Participatory Evaluation and Transformative Participatory Evaluation (Cousins & Whitmore, 1998). Practical approaches emphasize utilization; whereas, transformative approaches emphasize “principles and actions in order to democratize social change” (p. 7). Practical approaches include respon- sive (Stake, 1973), utilization- focused (Patton, 1997, 2004), developmental (Patton, 1994, 2012), and naturalistic (Guba & Lincoln, 1985, 1989) evalu- ation. Transformative approaches include empowerment (Fetterman, 1994; Fetterman & Wandersman, 2005), deliberative democratic (House & Howe, 1999, 2000), and transformative evaluation (Mertens, 2009).

Responsive evaluation carved the pathway for participatory approaches by emphasizing flexible and changing methods and approaches; recognizing multiple realities and the value of pluralism; and valuing local knowledge, theories, and program “particulars” (Fitzpatrick, Sanders, & Worthen, 2011, p. 193). An evaluator’s goal in a responsive evaluation is to be responsive to the needs and desires of the clients requesting an evaluation. Similarly, Utilization- focused evaluation (Patton, 1997, 2004, 2012) expands this approach by focusing on the people (i.e., users) who care about the results of an evaluation. The intent of these approaches is to involve key users of the evaluation in the conduct of the evaluation study, interpretation of its results, and decisions about a program. Naturalistic evaluation (Guba & Lincoln, 1985, 1989) focuses on the evolution of an evaluation through its primary stakeholders. This type of participatory approach may or may not di- rectly involve stakeholders in the conduct of an evaluation, but an evaluator does seek the input and ideas of key stakeholders throughout the evaluation process. When an evaluator actively involves key stakeholders as designers, data collectors, and results interpreters throughout the evaluation process, an evaluation becomes fully participatory. In practice, full stakeholder par- ticipation is not always possible for reasons often related to their availa- bility or relationships of power or privilege within or across organizations. Participation also may be limited by lack of perceived skills to conduct eval- uation, restricted timelines, or available staffing. Therefore, participatory evaluations vary by the amount of evaluator control over the process, breadth of stakeholders involved in the evaluation, and the stages of an evaluation in which stakeholders are involved (Chouinard, 2013; Cousins, Whitmore, & Shulha, 2012).

Four key principles guide transformative evaluation approaches: (1) cul- tural respect, (2) promotion of social justice, (3) furtherance of human rights, and (4) addressing inequalities (Mertens, 2009; Mertens & Wilson, 2012).

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 45 ]

When executing these principles, an evaluator’s goal is to engage and involve marginalized groups; therefore, an evaluator’s cultural competence is absolutely necessary for recognizing “multiple social identities and group memberships (e.g., gender, income or educational level, race/ ethnicity, disability); and [en- gaging in] transparent discussions of relevant issues of power and privilege” (Mertens & Wilson, 2012, p. 165). Transformative approaches to evaluation include, but are not limited to, deliberative democratic evaluation (House & Howe, 1999, 2000), empowerment evaluation (Fetterman, 1994; Fetterman & Wandersman, 2005), those grounded in critical race theory (Mertens & Wilson, 2012), indigenous evaluation (LaFrance & Nichols, 2010), culturally responsive evaluation (Hood, 2000), transformative participatory evaluation (Mertens, 2009), and those approaches targeting specific circumstances and cultures (e.g., disability, feminism, race, or ethnicity). All of these approaches seek to engage stakeholders in activities or action steps that improve their lives and create a more just society. In doing so, these approaches utilize collabo- rative forms of inquiry and “grass roots” participation that require time, pa- tience, and resources to build and maintain strong relationships and efforts that lead to social change. An important distinction among these approaches is the level of evaluator control over the evaluation and stakeholder partici- pation. In empowerment evaluation, for example, evaluators relinquish their control to stakeholders. In contrast, in other approaches to transformative evaluation (e.g., transformative participatory), evaluators retain some con- trol of the evaluation in order to exercise their professional responsibilities and avoid “chaos” (Cousins, Whitmore, & Shulha, 2012, pp. 14– 15). Because these approaches differ in practice but are built from a set of core values and goals, Cousins, Whitmore, and Shulha (2012) argue that forms of collaborative inquiry be driven by principles (shown in Box 3.2) rather than be described

Box 3.2. PRINCIPLES TO GUIDE COLLABORATIVE APPROACHES TO EVALUATION (SHULHA, WHITMORE, COUSINS, GILBERT, & HUDIB, 2016)

Clarify motivation for collaboration Foster meaningful relationships Develop a shared understanding of the program Promote appropriate participatory processes Monitor and respond to resource availability Monitor evaluation progress and quality Promote evaluation thinking Follow through to realize use

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 46 ] Part One: Prepare

as “compartmentalized,” “discrete genres,” or approaches to evaluation. These principles emphasize collaboration, shared understanding, evaluative thinking and capacity building, and use of findings.

SYSTEMS APPROACHES

Because evaluators increasingly are asked to evaluate “wicked” social problems involving programs within interdependent systems of people, organizations, and communities, systems thinking and approaches to evaluation are gaining popularity in the evaluation field. This topic will be discussed in greater detail in the final chapter of this book, so this section is limited to a brief introduction of systems approaches in the evaluation field. Grounded in systems theories, such as systems dynamics, complexity theory, soft and critical systems, network theory, and learning systems, among others, these approaches to evaluation require an understanding of concepts unfamiliar to many social scientists (Hargreaves & Podems, 2012). Fundamentally, these approaches embrace three interactive concepts: boundaries (or distinctions among them), interrelationships, and perspectives (Cabrara & Cabrara, 2015; Eoyang & Holladay, 2013; Williams & Hummelbrunner, 2011). To date, these approaches have been described as an evolution of program theory (Funnell & Rogers, 2011), as a way to re- frame outcomes evaluation through realism (Pawson & Tilley, 1997), and as a utilization- focused developmental evaluation process (Patton, 2012). The dialogue among most scholars has thus far been limited to theoretical or conceptual descriptions, so information is limited about how to effectively engage in the practice of systems evaluation. As the field rapidly grows in this direction, however, more complex and practical approaches to evalua- tion are likely to emerge.

APPROACHES AND THE EVALUATION TREE

Alkin and Christie (2004) summarized evaluation approaches in what they called the evaluation theory tree (Figure 3.4). This metaphorical tree aligns the philosophical paradigms with the different foci of each set of approaches (i.e., methods, use, values, social justice). The Pragmatist branch aligns best with humanism philosophy and the importance of stakeholder involvement in the evaluation process.

In 2004, Michael Patton questioned the tree metaphor and suggested that a river and its associated branches might be a better metaphor for evaluation paradigms and approaches (Patton, 2004). A river metaphor not only would

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 47 ]

capture the various “branches” of evaluation, but it would illustrate flows be- tween branches and their potential blending as well as the complexity of problems that are common in evaluation practice (Mertens, 2012). These met- aphorical illustrations provide helpful ways to organize and distinguish the various approaches to evaluation that guide our selection of an appropriate approach or approaches for an evaluation design. You should be aware, how- ever, that these metaphorical descriptions, or any of the approaches described in this chapter, do not necessarily account for approaches that are field specific. Examples include the RE- AIM model in public health (Glasgow, Vogt, & Boles, 1999), The Roberta Straessle Abruzzese (RSA) model in nursing (Abruzzese, 1996; DeSilets, 2010), and the Snyder model in agriculture (Dart, Petheram, & Straw, 1998, p. 81). I mention them because, if you work with multiple disciplines or across disciplines, it is important to know and understand the philosophical foundations, methods, assumptions, and approaches of these dis- cipline- specific models.

MATCHING APPROACHES WITH EVALUATION QUESTIONS

As mentioned earlier, a single approach to an evaluation is typically not ad- equate for addressing the variety of questions posed by stakeholders at any given time. Common evaluation questions include, but are not limited to, those listed in Table 3.2. Table 3.2 also indicates the approach or approaches that an evaluator likely would use to answer commonly posed evaluation questions. It should be recognized, however, that this table provides only a guideline. The selection of an evaluation approach or approaches will depend not only on the evaluation questions, but on the context, power dynamics, and political dynamics of those involved.

Pragmatist

Postpositivist Methods

Social Accountability, fiscal control, and social inquiry

Use Values

Constructivist

Transformative Social Justice

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Table 3.2. E VA L UAT ION A P PR OAC H E S T H AT C A N B E U S E D TO A N S W E R C OM M ON E VA L UAT ION Q U E S T ION S

Expertise Program Decision Participatory Systems

Did that professional meet the

accreditation standards?

Is my program effective or

successful?

X X X X

Is the program achieving its

intended goals or outcomes?

X X

Are participants learning as a

result of the program?

X X X

Why don’t participants apply

what we teach them/ what

they learn?

X X X

Does variable x improve

or decline following the

program?

What outcomes result from my

program?

What are the impacts of my

program on organization x or

community y?

X X X X

What programs should we

create to tackle issue x?

X X

Should we continue program x? X

What does program x cost and

is it worth the expense?

How or to what extent is my

program being implemented?

Which programs should we

continue, and which should

we eliminate?

What contributions is my

program making to outcome

x in my organization?

How and to what extent does

my program impact x in my

local community?

X X X

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 49 ]

EXAMPLES OF INTEGRATED APPROACHES TO EVALUATION

As mentioned at the beginning of this chapter, evaluators often draw from multiple approaches when evaluating programs in practice (Bledscoe & Graham, 2005). This section offers an example of how multiple approaches become embedded in an evaluation design.

A few years ago, and on short notice, I was asked to evaluate a leadership training program for emerging leaders in agricultural business. The pro- gram participants were selected by their superiors to attend this leadership training program. Over the course of a year, the participants met approxi- mately once per month for two days. Summers were used to practice and apply skills acquired through the leadership program. For each two- day session, a specific content area was defined, and learning objectives were articulated. Additionally, the program’s goals were to develop organizational leadership capacity so there was an interest in building the skills of new employees. The program’s directors were accountable to a state- level funder, so the evalua- tion had to demonstrate its value and effectiveness in training new leaders in agriculture.

Because I was under a short timeline to plan and begin this evaluation, I drew from the leadership literature to derive a theoretical foundation for the program that articulated some of the outcomes desired by the program’s directors and by participants’ superiors. This theory provided a set of variables that could be assessed to determine whether participants’ knowledge and skills were being effectively applied. Interestingly (or maybe not, given the short timeline), a logic model was never prepared for this evaluation; how- ever, this evaluation was clearly informed by theory so that links could be made between the immediate program outputs (in the form of achieved learning objectives) and outcomes identified by both the theory and those desired by the program directors. In this evaluation, I drew from objectives- based approaches to evaluate each session’s immediate learning outcomes; I drew from theoretical approaches to connect learning outcomes to behavioral changes that could be observed and recorded by the participants’ supervisors and colleagues, and I used the supervisors and colleagues to collect data about the program’s participants, thus, drawing from participatory approaches.

This is just one example of how various approaches to evaluation may be used to respond to a program director’s goals and need for information. More often than not, in practice you will rely upon multiple approaches to evalua- tion, especially as your evaluations become more complicated or complex (e.g., systems- level evaluations). I, therefore, encourage you to think about the var- ious approaches to evaluation as part of your toolbox and to use them as ap- propriate to the context, purpose, and questions for the evaluation.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 50 ] Part One: Prepare

SUMMARY

In summary, an evaluator has a large and varied toolbox of approaches to guide an evaluation’s design. A single approach is seldom used in evaluation practice. Instead, an evaluator draws from more than one of these approaches to customize and adapt the evaluation to each context and set of questions posed by a client. In other words, one size does not fit all, and a skillful and knowledgeable evaluator will draw from these multiple approaches to focus and guide an evaluation. In the next chapter, I will discuss how you select and refine your approach based on the purpose of the evaluation and stakeholders’ values and needs for information.

REFERENCES

Abruzzese, R. S. (1996). Nursing staff development: Strategies for success (2nd ed.). St. Louis: Mosby Year Book.

Alkin, M. C. (1991). Evaluation theory development II. In M. McLaughlin & D. Phillips (Eds.), Evaluation and education at quarter century (pp. 91– 112). Chicago, IL: University of Chicago Press.

Alkin, M. C., & Christie, C. A. (2004). An evaluation theory tree. In M. Alkin (Ed.), Evaluation roots (pp. 12– 66). Thousand Oaks, CA: Sage Publications

Bakken, L.L., Olson, C. A., Ross, J. M., Turco, M. G., Jackson, L. M., Murphy, W. (2015). MM&I Phase I Evaluation Report. Hanover, NH: Department of Medicine, Dartmouth- Hitchcock Medical Center.

Bjurulf, S., Vedung, E., & Larsson, C. G. (2012). A triangulation approach to impact evaluation. Evaluation, 19(1), 56– 73.

Bledscoe, K. L., & Graham, J. A. (2005). The use of multiple evaluation approaches in program evaluation. American Journal of Evaluation, 26(3), 302– 319.

Brousselle, A., & Champagne, F. (2011). Program theory evaluation: Logic analysis. Evaluation and Program Planning, 34, 69– 78.

Cabrara, D., & Cabrara, L. (2015). Systems thinking made simple: New hope for solving wicked problems. New York, NY: Odyssean Press.

Carraccio, C., Englander, R., Van Melle, E., ten Cate, O., Lockyer, J., Chan, M- K., Frank, J. R., & Snell, L. S. (2016). Advancing Competency- Based Medical Education: A Charter for Clinician- Educators. Academic Medicine, 91(5), 645– 649.

ten Cate, O., & Billett, S. (2014). Competency- based medical education: Origins, perspectives and potentialities. Medical Education, 48, 325– 332.

Chen, H- T. (1990). Theory- driven evaluations. Newbury Park, CA: Sage Publications, Inc. Chen, H- T. (1996). A comprehensive typology for program evaluation. Evaluation

Practice, 17(2), 121– 130. Chen, H- T. (2005). Practical program evaluation: Assessing and improving planning, imple-

mentation, and effectiveness. Thousand Oaks, CA: Sage Publications. Chouinard, J. A. (2013). The case for participatory evaluation in an era of accounta-

bility. American Journal of Evaluation, 34(2), 237– 253. Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. In

E. Whitmore (Ed.), Understanding and practicing participatory evaluation. New Directions in Evaluation, 80, 3– 23.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

c h o o s i n g a n e va L u a T i o n a p p r o a c h [ 51 ]

Cousins, J. B., Whitmore, E., & Shulha, L. (2012). Arguments for a common set of principles for collaborative inquiry in evaluation. American Journal of Evaluation, 34(1), 7– 22.

Dart, J., Petheram, R. J., & Straw W. (1998). Review of evaluation in agricultural exten- sion (Publication No. 98/ 136). Barton, Australia: Rural Industries Research and Development Corporation.

DeSilets, L. D. (2010). Another look at evaluation models. Journal of Continuing Education in Nursing, 41(1), 12– 13.

Donaldson, S. I. (2007). Program theory- driven evaluation science. Strategies and applications. New York, NY: Lawrence Erlbaum Associates.

Dweck, C. S. (2007). Mindset: The new psychology of success. New York, NY: Ballantine Books.

Epstein, R. M., & Hundert, E. M. (2002). Defining and assessing professional compe- tence, JAMA, 287(2), 226– 235.

Eoyang, G. H., & Holladay, R. J. (2013). Adaptive action: Leveraging uncertainty in your organization. Stanford, CA: Stanford Business Books.

Fetterman, D. M. (1994). Empowerment evaluation. Evaluation Practice, 15(1), 1– 15. Fetterman, D. M., & Wandersman, A. (Eds.). (2005). Empowerment evaluation principles

in practice. New York, NY: The Guilford Press. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative

approaches and practical guidelines (4th ed.). Upper Saddle River, NJ: Pearson Education.

Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco, CA: Jossey- Bass Publishers.

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE- AIM framework. American Journal of Public Health, 89(9), 1322– 1327.

Guba, E., & Lincoln, Y. S. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publications.

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage Publications.

Hargreaves, M. B., & Podems, D. (2012). Advancing systems thinking in evaluation: A review of four publications. American Journal of Evaluation, 33(3), 462– 470.

Hood, S. (2000). Commentary on deliberative democratic evaluation. In K. E. Ryan & L. DeStefano (Eds.), Evaluation as a Democratic Process: Promoting Inclusion, Dialogue, and Deliberation. New Directions for Evaluation, 85, 77– 83.

House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage Publications.

House, E. R., & Howe, K. R. (2000). Deliberative democratic evaluation. In K. E. Ryan & L. DeStefano (Eds.). Evaluation as a democratic process: Promoting inclusion, dia- logue, and deliberation. New Directions in Evaluation, 85, 3– 12.

Lacey, A., & Murray, C. (2015). Rethinking the regulatory environment of competency- based education. Washington, DC: American Enterprise Institute. Retrieved from the American Enterprise Institute website: http:// www.aei.org/ pub- lication/ rethinking- the- regulator y - environment- of- competenc y - based- education/ .

LaFrance, J., & Nichols, R. (2010). Reframing evaluation: Defining an indigenous eval- uation framework. The Canadian Journal of Program Evaluation, 23(2), 13– 31.

King, J., & Stevahn, L. (2015). Competencies for program evaluators in light of adapt- ive action: What? So what? Now what? In J. W. Altschuld & M. Engle (Eds.),

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.aei.org/publication/rethinking-the-regulatory-environment-of-competency-based-education/

[ 52 ] Part One: Prepare

Accreditation, certification, and credentialing: Relevant concerns for U.S. evaluators. New Directions for Evaluation, 145, 21– 37.

Knowlton, L. W., & Phillips, C. C. (2009). The logic model guidebook: Better strategies for great results. Thousand Oaks, CA: Sage Publications.

Mertens, D. M. (2009). Transformative research and evaluation. New York, NY: The Guilford Press.

Mertens, D. M., & Wilson, A. T. (2012). Program evaluation theory and practice: A compre- hensive guide. New York, NY: The Guilford Press.

Patton, M. Q. (1997). Utilization- focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage Publications.

Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), 311– 319. Patton, M. Q. (2004). The roots of utilization- focused evaluation. In M. Alkin (Ed.),

Evaluation roots (pp. 276– 292). Thousand Oaks, CA: Sage Publications. Patton, M. Q. (2012). Developmental evaluation: Applying complexity concepts to enhance

innovation and use. New York, NY: The Guilford Press. Pawson, R., & Tilley, N. (1997). Realistic evaluation. Thousand Oaks, CA: Sage

Publications. Preskill, H., & Russ- Eft, D. (2005). Building evaluation capacity: 72 activities for teaching

and training. Thousand Oaks, CA: Sage Publications. Provus, M. M. (1971). Discrepancy evaluation. Berkeley, CA: McCutchan. Schoonover Associates. (2003). FAQ: Competency model building. Retrieved from http://

www.schoonover.com/ schoonover- faqs.asp. Shulha, L. M., Whitmore, E., Cousins, J. B., Gilbert, N., & Hudib, H. (2016).

Introducing evidence- based principles to guide collaborative approaches to eval- uation: Results of an empirical process. American Journal of Evaluation, 37(2), 193– 215.

Stake, R. E. (1973). Program evaluation, particularly responsive evaluation. Keynote address at the conference “New Trends in Evaluation.” Institute of Education, University of Goteborg, Sweden, October 1973. In G. F. Madaus, M. S. Scriven, & D. L. Stufflebeam (Eds.), Evaluation models: Viewpoints on educational and human services evaluation. (1983, pp. 287– 288). Boston, MA: Kluwer- Nijhoff Publishing.

Stufflebeam, D. L. (1983). The CIPP model for program evaluation. In G. F. Madaus, M. S. Scriven, & D. L. Stufflebeam (Eds.), Evaluation modes: Viewpoints on educa- tional and human services evaluation (pp. 117– 141). Boston, MA: Kluwer- Nijhoff Publishing.

Stufflebeam, D. L. (2013). The CIPP evaluation model: Status, origin, development, use and theory. In M. C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists’ views and influences (2nd ed., pp. 243– 260). Thousand Oaks, CA: Sage Publications.

Tyler, R. W. (1942). General statement on evaluation. Journal of Educational Research, 35, 492– 501.

Weiss, C. H. (1997). How can theory- based evaluation make greater headway? Evaluation Review, 21(4), 501– 524.

Williams, B., & Hummelbrunner, R. (2011). Systems concepts in action; A practitioner’s toolkit. Stanford, CA: Stanford University Press.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.schoonover.com/schoonover-faqs.asp

C H A P T E R 4

Planning a Program Evaluation

In this chapter, I describe an evaluation plan as a major component of a program plan and discuss why it is important that it be linked to program elements. To begin, it is important to understand the broad context of a pro- gram. This understanding is important for determining what is valued by the organization(s) or communities that are planning or conducting the program, for whom the evaluation is important, whom the program influences, why it is being planned or implemented the way it is, what traditions ground the program, and who will be supportive and involved (or not) in the program pla- nning or evaluation process (i.e., stakeholders).

Common to both program and evaluation planning and implementation is the involvement of stakeholders. Stakeholder involvement is important for increasing the relevance of a program for its participants, gaining adminis- trative buy- in for successful implementation and financial support, executing a program in a manner that aligns with staff roles and responsibilities, and communicating with and engaging personnel who are critical to the design, implementation, and evaluation of a program.

Stakeholders are key to providing various perspectives that are knowl- edgeable of or can inform a program’s purpose, goals, and activities. Moreover, knowledge of these program components (i.e., purpose, goals, and activities) is necessary for helping to focus and align an evaluation plan with a program plan. A theory about how a program affects changes in learners, organizations, or communities helps to situate the program within a defined context, estab- lish relationships among program components, and give clarity to a program’s elements for evaluation and assessment.

Evaluation questions are the primary anchor for an evaluation plan and re- flect the various program elements and processes described in a theory. These questions are best determined by engaging stakeholders in a dialogue that

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 54 ] Part One: Prepare

draws from their various perspectives and addresses questions and needs of importance to them. By engaging stakeholders in discussions about the pro- gram and any questions associated with it, you can begin to focus the evalua- tion and establish its purpose, goals, scope, and use. Once these main elements are clearly established, you can design an evaluation and draft a proposal or contract to describe a plan that will clarify and guide your work.

UNDERSTANDING THE EVALUATION’S CONTEXT

Before you begin any program evaluation, it is necessary to have a thor- ough understanding of the evaluation’s context. I suggest that you begin by having conversations with one or two of the program’s key stakeholders. These stakeholders are typically program directors, program coordinators, administrators, or individuals requesting the evaluation. A context anal- ysis probes for information about the program and its context to help you make choices and decisions about the approaches and methods for a pro- gram evaluation. In essence, it helps evaluators answer the question, “What evaluation approach provides the highest quality and most actionable evi- dence in which contexts?” (Rog, 2012, p. 26). A context analysis also helps you to understand the culture in which the evaluation will be conducted, so that it is sensitive to issues such as political, organizational, racial, ethnic, and community dynamics, among others (Fitzpatrick, 2012). Rog (2012) recommends that five “contexts” be considered when performing a con- text analysis: problem, program/ intervention, decision- making, evalua- tion, and the broader context of the environment. Within these five areas, she suggests that seven dimensions be probed: physical, organizational, social, cultural, traditional, historical, and political. Within each of these dimensions and contexts, a series of questions could be asked to gain an in- depth understanding of the context surrounding a program evaluation. The questions would vary according to whether the context analysis is being done during the planning, implementation, or decision- making/ use phase of an evaluation (Conner, Fitzpatrick, & Rog, 2012). Table 4.1 includes some examples of questions from my own experiences that I might pose within the contexts and dimensions suggested by Rog (2012) and at various phases of an evaluation.

A context analysis, therefore, requires careful and attentive listening skills to understand various factors that will influence the evaluation. Because I ad- vocate participatory approaches to evaluation, I typically involve multiple stakeholders when I perform an informal or formal context analysis. This pro- cess can typically take a few days to two to three months or more and often continues as an evaluation is planned, implemented, and used. Consequently,

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Table 4.1. Q U E S T ION M AT R I X F OR A C ON T E X T A N A LY S I S

Problem Program Decision- making Evaluation Broader Environment

Physical Is the physical location

influencing the problem or

reason for the evaluation?

Where is the program being held? Why is

it held at that location?

Who made the decision to hold

the program at its location

and why?

Are there sufficient resources

(space, personnel) to support

the evaluation?

Is this the ideal location for the

program for the targeted

participants?

Organizational What stakeholders determined

the problem or need for the

program?

What organizations are involved

in designing, planning, and

implementing the program? What

resources does each organization bring

to the program?

Who are the organization’s key

decision- makers?

Are staff training in evaluation?

What staff might be helpful in

performing the evaluation, and

to what extent can or would

they be willing to engage?

How are the organizations

situated in the community?

Are they viewed favorably? To

which organizations are they

linked?

Social What social issue or problem is

necessitating the program/

intervention? What are

the cultural considerations

surrounding the issue?

Is the program conceptualized in a

way that is sensitive to cultural

backgrounds and needs?

Who are the decision- makers?

How will their decisions

influence the social issue?

What assumptions are they

making about the issue?

Who benefits from the program

and how? How will the

information be used by key

stakeholders?

To what extent does the

community support efforts to

resolve this social issue? Who

are advocates and champions

from the community?

Culture Does the problem or issue

involve marginalized groups?

Does program planning involve

stakeholders representing a variety

of cultural groups, especially those

targeted for the program?

Are stakeholders from various

cultural groups involved in

the decision- making?

Is the evaluation inclusive of

stakeholders from important

cultural groups?

Who holds power within the

broader community and how

does this influence various

cultural groups?

Traditional In what ways might cultural

traditions be influencing the

way the problem is perceived?

Are cultural traditions creating barriers

or facilitators to program- related

change?

How or in what ways do

traditions influence program

decision- making?

How will cultural or community

traditions influence the

evaluation approach or ways of

data collection?

To what extent are community

traditions shaping the

program or effecting attitudes

about intended changes?

Historical How long has the organization

or community been trying to

tackle the issue?

How as the program evolved over time?

What factors influenced program

development?

Over time, how have

decisions been made in the

organization or community?

Has the program been evaluated

before? What was learned and

how was the information used?

To what extent have different

programs been implemented

to tackle the issue? What’s

been successful?

Political In what ways have political

agendas helped to shape the

problem?

Who or what group determined the

program was needed and how was it

determined?

Who holds decision- making

power for the program or

organization?

To what extent will political

agendas try to influence the

evaluation or its findings?

Whose needs are being served

in the community and who

decides?

B akken, Lori L.. <

i> E

valuation P ractice for C

ollaborative G row

th : A G

uide to P rogram

E valuation w

ith S takeholders and

C om

m unities<

/i> , O

xford U niversity P

ress, Incorporated, 2018. P roQ

uest E book C

entral, http://ebookcentral.proquest.com /lib/usf/detail.action?docID

= 5400963.

C reated from

usf on 2019-10-03 16:13:23.

[ 56 ] Part One: Prepare

I typically begin these conversations by asking two to three key stakeholders the following questions:

Why do you want this program evaluation? What do you hope to learn from the program evaluation? How do you intend to use the results of this program evaluation? Who cares about the evaluation’s findings? Who supports or does not support this evaluation? What is the budget and time frame for the evaluation? Tell me about the program being evaluated.

Who is involved? Where, when, and how frequently does it occur? Who benefits from the program? Who is excluded from the program and why? Why is the program being implemented? Whose needs does it serve? Is the program evaluated or was any evaluation done to inform pro-

gram planning? What are the intended outcomes for the program? Who values this program and why? Who has power and control over programming decisions?

After gaining this information, I reach out to some of the people mentioned in my discussions with these few key stakeholders and pose the same questions. In other words, through these key stakeholders, I identify some additional stakeholders and ask them the same questions in an iterative process. By doing so, I gain a variety of perspectives and identify the best approaches for the program evaluation. I am also careful to note my observations related to political influences, people who are referred to as champions, the interactions of coworkers, the physical space and location of the program, and other con- textual factors. In the next section, I will talk about additional ways to iden- tify key stakeholders and how to engage them in the evaluation’s design and planning.

IDENTIFYING AND ENGAGING STAKEHOLDERS

Stakeholders are the individuals or groups of people who have a vested in- terest or “stake” in a program and its evaluation. Common examples include participants, program planners and coordinators, instructional designers, program administrators or directors, funders, board members, members of the community, and other program beneficiaries. In early phases of a pro- gram and evaluation design, it is important to determine a program’s key stakeholders and explicitly identify their “stake” in the program. Williams and

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 57 ]

Hummelbrunner (2011, p. 110) recommend that four key questions be asked to guide a stakeholder analysis:

Who is affected by the strategy, that is, the program? Who has an interest in it? Who can affect its adoption, execution, or implementation? Who cares about it?

By clearly articulating the stakeholders’ interests and stake in the program, you begin to understand the program’s context and can narrow the focus and scope of an evaluation. Shown in Figure 4.1 is a checklist adopted from Fitzpatrick, Sanders, and Worthen (2011, p. 289) that is useful for determining key stakeholders in program planning and evaluation. As you consider the list of entities along the left- hand column, you may wish to swap out the word “evaluation” for “program,” and vice versa, in order to gain a comprehensive list of the program evaluation’s stakeholders. Once identified, stakeholders’ perspectives during the planning process are critical for engaging their buy- in and facilitating or providing resources for the programming and evaluation efforts.

Often, you must begin your conversations with known stakeholders, such as program directors, to determine the full scope of individuals or groups who have a stake in the program. While doing so, you can gain their impressions of the program, what they expect from it, what questions they may have about the program, and how they perceive it working to impact change. As you speak to them, you also will learn about the political, economic, and physical context of the program. This information is helpful in understanding how these con- textual dynamics may influence your programming efforts.

Stakeholders can be engaged in several ways. One- on- one interviews or discussions with them can reveal personal biases that may be critical for negotiating some of the more political challenges of program delivery or evaluation and for determining where, with whom, or when obstacles might be confronted in the planning process. When conducted with those who are in positions to provide resources for the program, interviews pro- vide opportunities to discuss the specific resources that will be needed for the evaluation. Small group meetings or focus groups with specific groups of stakeholders can provide an efficient means of collecting the perspectives of those most affected by the program, such as potential program participants or community groups. You also will want to explore stakeholders’ opinions about evaluation and how they may or may not support efforts or provide resources to evaluate the program. Surveys of various stakeholder groups also may help to gain their perspectives and provide insights into the pla- nning process. Knowing these contextual elements of a program is impor- tant for understanding the stakeholders’ values and beliefs about evaluation

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Figure 4.1. Checklist of Potential Stakeholders and Audiences. Reproduced by permis- sion from Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Program evalua- tion: Alternative approaches and practical guidelines (4th ed.) (Upper Saddle River, NJ: Pearson Education, Inc., 2011).

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 59 ]

and for selecting an evaluation approach that will likely gain their support. Stakeholders also can be helpful in the implementation of a program and its evaluation. They can serve as advocates, group facilitators, instructors, data collectors, reporters, and disseminators. I encourage you to think about ways to involve stakeholders and engage them in a program’s planning, implemen- tation, and evaluation as much as possible.

A PROGRAM’S PURPOSE, GOALS, AND ACTIVITIES

As you analyze context, you will want to gain an understanding of the program’s purpose, goals, and activities. A program’s purpose establishes the reason(s) for developing and implementing it. Purpose statements often re- flect a community problem, organizational need, or gap in learners’ knowl- edge, skills, or attitudes that the program is trying to address. For example, if a program’s goal or aim is to reduce a high teen pregnancy rate in a community (i.e., an impact), it might be designed to target middle- school youth with a purpose to empower youth to make wise decisions about contraceptive use. A program’s objectives “provide clear statements of the anticipated results to be achieved” through the program (Caffarella, 2002, p. 156). Therefore, program objectives can be reframed as the intended outcomes (i.e., resulting changes) desired by its stakeholders. Extending the previous example, the objectives of the program might be to (1) engage youth as peer- educators to encourage and practice open communication among participants, (2) create a climate that is sensitive to and respectful toward cultural values about con- traception, and (3) increase the number/ proportion of youth who use contra- ceptive devices and/ or practices by 20% in the year following the program. Notice how the first two objectives reflect the educational process in contrast to the third objective, which represents an expected outcome. Embedded in these statements are clues about the type of evaluation design that is needed. Program objectives also become a guide for defining a set of learning or behav- ioral objectives and designing activities that will help to achieve them.

By definition, programs are a set of activities that promote or facilitate learning and behavioral change among participants. For example, “engaging youth as peer- educators” and “creating a sensitive and respectful climate” as reflected in the program’s objectives, are examples of program strategies or ac- tivities that are intended to facilitate the intended process outcomes of “youth who openly communicate about contraception” and “youth who acknowledge and respect their peers’ cultural values toward contraception.”

When designing educational programs, learning objectives are used to guide the selection and design of learning activities and the specific con- tent that each activity will contain. Learning objectives are defined as the knowledge, skills, and attitudes that participants develop as the result of

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 60 ] Part One: Prepare

engaging in a program’s learning activities. In evaluation terms, learning objectives can be defined as a set of intended short- term outcomes. Outcomes are changes that occur as the result of a set of activities (Knowlton & Phillips, 2009). Personally, I prefer to think of learning objectives as evidence that the program’s activities have been conducted (i.e., outputs), rather than as outcomes of program activities, because participants must apply and use what they learn in order for change to occur. Defined this way, learning outcomes are the higher order changes (e.g., transfer of knowledge and behav- ioral changes) that are expected as the result of participating in a program’s activities. If successfully achieved, most of a program’s objectives should be accomplished.

To illustrate this point, I will elaborate on the aforementioned hypothet- ical program by using the second program objective, “create a climate that is sensitive to and respectful toward cultural values about contraception,” as a specific example. From a program designer’s perspective, I would want to achieve this objective by acknowledging different views toward contraception and procreation among different racial, ethnic, gender, and religious cultural groups. I also might want to acknowledge various perspectives about the co- existence of procreation and contraception and engage youth participants in discussions about how this could occur. To achieve this program objective, then, instructors or facilitators will . . .

Describe various cultural views toward procreation and conception, Engage youth in discussions about how their backgrounds influence their views on contraception and procreation, and Challenge youth to think about how they would handle cultural differences when confronted with contraceptive choices.

These sub objectives or instructional objectives help to define learning activities (e.g., discussion) and content (contraceptive values and beliefs among var- ious cultural groups) for this program. From a learner’s perspective, I would expect that learners would come away with knowledge of how their own cul- tural backgrounds influence their and others’ beliefs about contraception, re- spectful attitudes toward others’ desires for contraception, and the skills to communicate openly and respectfully with sexual partners about their desire and preferences for contraception. So, learning objectives for this program would be articulated as follows:

At the end of this program, learners will . . . Explain how contraception and procreation differ among racial, ethnic, gender, and religious groups, Describe how cultural differences might influence contraceptive preferences or choices, and

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 61 ]

From a scenario, explain how they would handle cultural differences when confronted with contraceptive choices.

Notice how these and the previous lists of objectives (i.e., instructor and learner) were derived through different perspectives or orientations. These orientations represent two perspectives toward achieving a program’s objectives. Both perspectives are important and both of them align with the program objective to “create a climate that is sensitive to and respectful to- ward cultural values about contraception.” In evaluation terms, all three perspectives can be associated with the primary unit of analysis— the pro- gram. The instructor and learners are sources of information for evaluating the program. Typically, the instructor’s objectives are not tracked in a program evaluation. One exception might be a study of a program’s fidelity or imple- mentation. In this case, an evaluator would assess the extent to which the instructor is achieving his or her own objectives in addition to those of the learner and overall program. Notice, too, that I have defined sets of outcomes (program and learning) that can be assessed in an evaluation of a program’s effectiveness. However, the evaluation questions posed by stakeholders may not suggest an evaluation of this type; therefore, it is necessary to determine the stakeholders’ questions and needs for information before designing an evaluation plan. Before doing so, it can be helpful to engage key stakeholders in a process of describing a theory for how they think the program will work to produce the desired changes or effects. The next session revisits and describes the utility of program theory, how it is developed, and how it has been ex- panded to a systems level in recent years.

USING THEORY TO FOCUS AN EVALUATION

As described in Chapter 3, program theories provide conceptual models or frameworks that describe how the elements of a program relate to one an- other to produce desired changes in learners, organizations, or communities. Program theory has dominated evaluation practice in recent years; however, collective impact models and systems theory is beginning to permeate the evaluation field as demands for information become broader, more sophisti- cated, and highly complex. I will begin this section with a description of pro- gram theory and how it is used in practice; then, I will introduce and describe collective impact, briefly describe systems theory, and provide examples of situations that suggest how they can be applied and used.

Program Theory and Logic Models. Program theory is a useful tool for de- fining and establishing factors and processes that may be assessed in your efforts to evaluate a program. Simply put, a program theory describes the associations between what we do as part of a program and what we get as the

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 62 ] Part One: Prepare

result of doing it (Knowlton & Phillips, 2009). Theories described in the aca- demic literature may be used as a basis for developing a program’s theory or incorporated into a theory that is created in collaboration with stakeholders. For example, when I and my colleagues studied clinical research self- efficacy, we used social cognitive career theory (Lent, Brown, & Hackett, 1994) to in- form our work and the development and evaluation of an educational inter- vention (Bakken, Byars- Winston, Gundermann, Ward, Slattery, King, Scott, & Taylor, 2010). Program theories also can be developed inductively through qualitative studies of existing programs in order to determine a theory about what is actually occurring versus what was planned or intended.

Multiple models can be used as tools to develop a program’s theory (Funnell & Rogers, 2011). Common to program evaluation and used across many disciplines are logic models (discussed in Chapter 3). Because logic models are commonly used across disciplines as a tool for devel- oping program theory, their application and use are elaborated upon in this chapter.

To specify some of the long- term changes expected from a program, it is helpful to ask the question, “If you achieve the program’s goals and program participants learn, what do you expect will happen to individuals, within or- ganizations, or in communities?” This question is one that is typically referred to by evaluators and researchers as the “So what?” question. This question is very important because it forces us to think beyond the immediate or short- term outcomes of a program to the longer- term and broader outcomes and impacts on those whom a program serves. This is why efforts to develop a program theory are useful and important.

When working with stakeholders to develop logic models, it is often helpful to begin with “the end in mind” and work backward to establish a rationale for how a program will work to produce the anticipated or desired changes. This backward process often calls into question the original activities specified for the program because it initiates a discussion about what a program’s stakeholders should be doing to achieve the desired change (i.e., “get”). Logic models are also helpful for reflecting on intended program outcomes and the actual or possible unintended outcomes that may occur as part of the program’s implementation.

To critique a program’s theory, whether developed using a logic model or other model, it is also helpful and important to examine some of the assumptions that underlie that theory. By examining the underlying assumptions of a program theory, potential roadblocks (e.g., a targeted participant’s ability to travel to the program’s location or planned activities that misalign with cultural norms) to the program’s success at achieving the desired outcomes are revealed. Once established, a program’s logic model provides a useful tool for aiding discussions with stakeholders about the pur- pose of an evaluation, what questions come to mind about the program, and

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 63 ]

how findings of an evaluation will be used. These discussions lay the ground- work for developing an evaluation plan.

Collective Impact. Frequently, a program is one of several programs aimed at achieving an impact within an organization or community. Often, these impacts are defined by an organization’s strategic plan. For example, a group of graduate programs aimed at preparing innovative practitioners and researchers who will improve the efficiency and cost- effectiveness of health- care delivery and management. So, the goal or expected outcome of a graduate program preparing practitioners to work in public health settings (Program A1) might be that practitioners are able to analyze the effectiveness of ex- isting community programs to provide cost- effective health care. On the other hand, researchers educated in this graduate program (Program A2) might be prepared to develop and study more efficient and cost- effective models for providing basic health care services (e.g., prenatal care or immunizations). Notice that each program prepares students toward different outcomes, but both outcomes support the impact that graduate education is trying to achieve for this organization. More broadly, this organization might also have a quality improvement office that is working independently to change the way physicians practice so that they provide more cost- effective health care (Program B). Again, the desired impact is the same, but the programs, strategies, or mechanisms to create this impact are different and have different goals or intended outcomes. In a community setting, programs to inform the general public about ways to manage their own health in order to reduce healthcare spending as an outcome toward cost- effective health care may be in existence (Program C). All these programs have the desired impact of re- ducing the costs of health care in the community— some more directly than others, but each program uses different strategies for creating outcomes that will contribute to the overall desired impact. If this is the case in your settings, I encourage you to consider using systems- oriented approaches or a collective theory of change for the organization’s or community’s set of programs.

To successfully achieve a collective impact, five key conditions are nec- essary: (1) a shared vison for change (i.e., common agenda), (2) shared measurement systems, (3) mutually reinforcing activities, (4) continuous com- munication, and (5) a backbone support organization (Kania & Kramer, 2011). Two of these conditions (shared vision/ agenda and mutually reinforcing activ- ities) resemble those common to the “do → get” logic of program theory and one of them (shared measurement systems) can be directly tied to evaluation. By applying modeling approaches commonly used to develop program theory, one could envision a theory of collective impact (a desired “get”) to consist of multiple programs or mutually reinforcing activities (the “do”) designed to achieve a shared vision (presumably synonymous with a desired collective impact) in several ways. In this section, I offer three linear ways to theorize collective impact as a baseline.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 64 ] Part One: Prepare

One way to conceptualize a theory of collective impact is to think about how the goals/ outcomes achieved by each autonomously functioning program contribute to the desired collective impact (Figure 4.2). Viewed this way, the activities used by each program are independent, and they lead to independ- ently determined outcomes toward achieving the collective impact and shared vision of change.

In an alternative model, each program contributes separately to achieving a common or shared set of goals aimed at making an impact, as illustrated in Figure 4.3. Viewed in this way, each program’s individual set of activities would need to be mutually reinforcing and produce outcomes that serve as benchmarks toward achieving a collective impact. Funders typically use this type of model when evaluating the efforts of multiple grantees toward a collective change.

A third way to conceptualize a collective impact model is to envision each program as an activity among a set of activities aimed at achieving a collective

Program A1

Outcome A1

Program A2

Outcome A2

Impact Program

B Outcome

Program C

Outcome C

Figure 4.2. Independent Model of Collective Impact.

Impact

Outcome #1

Outcome #2

Outcome #3

Outcome #4

Program A1

Program A2

Program B

Program C

Figure 4.3. Shared Outcomes Model of Collective Impact.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 65 ]

set of outcomes toward a specific impact (Figure 4.4). This model views programs and outcomes with the least amount of independence and requires the most amount of collaborative effort and collective visioning among stakeholders to create the desired impact.

In all three models, a shared measurement system, supportive backbone organization, and continuous communication are key to effectively producing a collective change. As an example of a collective model of change, Woodhouse and colleagues (Woodhouse et al., 2013) illustrate a model for a Comprehensive Asthma Management Program and describe the utility of interfacing this model with the Plan- Do- Study- Act cycle for improving the quality of this pro- gram. These efforts require a significant amount of commitment to ongoing communication and collaboration. In practice, the more integrated and col- laborative the group’s efforts, the more likely they are to achieve a collective impact successfully (Kramer, Parkhurst, & Vaidyanathan, 2009). In my ex- perience, however, problems occur when groups within a collective model have different visions of achieving collective impact, or their programs are at different stages of the development or implementation process.

The three hypothetical frameworks for collective impact illustrated in this chapter reflect highly structured and linear models for a system of activities or programs oriented toward changes in society. Systems theorists would argue that these models are oversimplified because programs and their as- sociated outcomes and impacts rarely occur in ways they are conceptualized (Friedman, 2001; McDonald, 2015). In other words, programs often are not implemented as planned. Unanticipated outcomes occur. Program outcomes are produced at different times and in different ways depending on the organ- ization that implements them. Program theories omit critical elements that influence programs’ effects and so on. In other words, programs often don’t happen in the way we theorize them. Traditionally, evaluators have managed

Program A1

Program A2

Program B

Program C

Outcome #1

Outcome #2

Outcome #3

Outcome #4

Impact

Figure 4.4. Integrated Model of Collective Impact.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 66 ] Part One: Prepare

this level of uncertainty and complexity in programming efforts by carefully monitoring a program’s development and implementation through mixed methods approaches and procedures such as that provided through logic anal- ysis (Brousselle & Champagne, 2011).

Pawson and Tilley (1997) proposed that evaluators abandon causal linear models for evaluating outcomes and impact in favor of models and evalua- tion approaches based on scientific realism. “To be realistic is to acknowledge that there is no universal ‘logic of evaluation’, no absolute ‘science of val- uing’, no general ‘warranty for decision- making’ applicable to all judgements” (p. xiii). This approach requires a shift of thinking from outcomes that can be attributed to programs to outcomes for which programs have contributed. Therefore, outcomes are evaluated by understanding the mechanisms and context from which they are derived. These evaluations typically require both qualitative and quantitative forms of data collection and their associated ana- lytical methods. In the final chapter of this book, I will describe realistic eval- uation in greater detail and discuss how it can provide one way of evaluating systems- level change.

EVALUABILITY ASSESSMENT

Assuming that a theory or conceptual framework has been determined with key stakeholders, it is time to take a step back and ask yourself whether or not the program is ready to be evaluated. Not all programs are ready for eval- uation nor is it necessary to evaluate all programs all the time. Fitzpatrick, Sanders, and Worthen (2011, p. 269) describe four conditions necessary for meaningful evaluation:

Program goals and priority information needs are well defined (including agree- ment on performance criteria), Program objectives are plausible, Relevant performance data can be obtained at reasonable cost, and Intended users of the evaluation have agreed on how they will use the informa- tion. (Wholey, 2004, p. 34)

These criteria can be summarized with four short words: necessity, plausi- bility, feasibility, and utility. To determine a program’s readiness for evalua- tion, an evaluability assessment (Wholey, 1987, 2004) can be performed using the following major steps listed in Box 4.1 (Fitzpatrick, Sanders, & Worthen, 2011, p. 269):

Figure 4.5 provides a useful 10-step checklist for guiding this process and a determining a program’s evaluation readiness.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 67 ]

Once you determine that a program is ready to be evaluated, you can begin to develop an evaluation plan. This plan begins by determining the purpose and use for the program evaluation.

AN EVALUATION’S PURPOSE AND USE

It is essential that you gain clarity about the purpose for evaluating a pro- gram or set of programs. Is it to learn about ways to improve a program? Do stakeholders want to know if a program was implemented as planned? Do they want to know how effective a program was at facilitating learning and behavioral change? Do they want to know what impact a program has in an organization or community? Do stakeholders want to know if a program is effective at facilitating a change in learners, organizations, or communities before making a decision about whether the program should continue? An evaluation’s purpose is different from the purpose of a program and should be clearly understood before an evaluation is planned or implemented. Using my earlier example of the youth contraception program, the program’s purpose was to empower youth to make wise decisions about contraceptive use, but the purpose of an evaluation would likely be to determine whether the program is effectively contributing to the impact of reducing a high teen pregnancy rate in the community. A purpose statement for an evaluation communicates the reason why the evaluation is being conducted, and it should be stated in a clearly articulated manner. If an evaluation does not have a clear purpose, it should not be conducted.

At the same time, you should be clear about how stakeholders will use the findings of an evaluation study once it is completed. Do they wish to use the findings to make decisions about a program’s continuance? Do they wish to

Box 4.1. STEPS IN DETERMINING A PROGRAM’S READINESS FOR EVALUATION

STEP 1. Clarify the intended program model or theory. STEP 2. Examine the program in implementation to determine whether

it matches the program model and could, conceivably, achieve the program goals and objectives.

STEP 3. Explore various evaluation approaches to determine the degree to which they meet stakeholders’ information needs and are fea- sible to implement.

STEP 4. Agree on evaluation priorities and intended uses of the study.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 68 ] Part One: Prepare

Figure 4.5. Checklist for determining a program’s readiness for evaluation. Reproduced by permission of Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen, Program evaluation: Alternative approaches and practical guidelines (Upper Saddle River, NJ: Pearson Education, 2011).

use the findings to support additional or future funding? Would stakeholders like to use the evaluation’s findings to make judgments about the best way to implement a program? Do they want to use the evaluation’s findings to demonstrate a program’s success to a governing board or other administrative

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 69 ]

group? These and other reasons for doing evaluation and using its findings are important and should be explicit among a program’s stakeholders. Therefore, I encourage you to always write a statement that clearly conveys an evaluation’s purpose and how its findings will be used. Here’s an example:

The purpose of this evaluation is to determine whether the teen contraception pro- gram is effective at reducing the community’s teen pregnancy rate over the next five years. The results will be used to inform future program funding and continuance.

An evaluation can serve more than one purpose, so you may have more than one intended use for the evaluation’s findings. Multiple purposes tend to expand the scope of an evaluation, so be sure that adequate resources and time are provided to meet those expanded demands. Stakeholders who are in- sistent on broadening the scope of a program evaluation are great targets for invitations to be involved in participant recruitment, data collection, and re- porting and disseminating the evaluation’s findings. Some stakeholders may volunteer their time to be involved in later phases of the evaluation simply because they are interested in them. Others may volunteer because they wish to influence the findings or the ways findings are reported. Again, be cautious and rely on the Program Evaluation Standards (Yarbrough et al., 2011) and AEA’s Guiding Principles for Evaluators (American Evaluation Association, 2004) to support ethical and fair practices. I don’t wish to paint a negative image of stakeholders. Most of them are very helpful, supportive, and cooper- ative. When you least expect it, however, you may be confronted with at least one of these situations, and these tips are my effort to prepare you for these instances should they occur.

Another critical component of an evaluation plan is the evaluation questions. The next section will discuss why they are so important and why they are nec- essary to maintain a focus and scope for the evaluation. The purpose, use, and questions for an evaluation provide an anchor for other components of an evaluation plan (Figure 4.6). These elements are crucial for establishing the focus and direction for an evaluation.

EVALUATION QUESTIONS

After the evaluation’s purpose, goals, and use are clearly defined and articu- lated, it is time to refine your list of stakeholders’ evaluation questions and work with them to prioritize them so that a limited number of questions can be answered through your program evaluation. When working with stakeholders and clients, their evaluation questions are typically vague and often posed in a manner that cannot be immediately operationalized for anal- ysis. For example, a program director or administrator might wish to ask, “Is

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 70 ] Part One: Prepare

the program successful?” To determine whether a program is successful, it is important to engage them in a conversation about what it means for the pro- gram to be successful. What does a successful program look like? By doing so, you establish a set of criteria from which to evaluate the program’s success. Another question commonly asked in these days of accountability is, “How can I measure the outcomes of my program?” Your next question is, “What outcomes do you anticipate for your program?” This type of question typically stimulates a series of conversations as you attempt to determine the logical process by which the program produces desired outcomes. Some stakeholders may confuse outcomes with other program elements, such as activities or outputs. Often, this is evidenced through program evaluation forms in which the items referred to as learning outcomes are expressed as activities. My point in describing these various scenarios is that evaluation questions al- most always come in a way that needs to be clarified and operationalized for an evaluation.

Once clarified, the evaluation questions can be prioritized with stakeholders. Fitzpatrick, Sanders, & Worthen (2011) provide an ex- cellent matrix for prioritizing evaluation questions when working with stakeholders (Figure 4.7). Briefly, the parameters or criteria used to pri- oritize evaluation questions include the feasibility of time and resources needed to answer the question, the extent to which information gleaned by answering the question will be of major interest to stakeholders and help to influence decisions or events, and whether the question’s answers will provide information that has sustainable value. Again, evaluation questions are the primary anchor for an evaluation. Once established and prioritized, they determine the focus, scope, and information collected for the evaluation.

Findings & Dissemination

Evaluation purpose, use and questions

Data Analysis

Data Collection Methods

Approach

Figure 4.6. Figure illustrating how the evaluation purpose, use, and questions anchor an evaluation’s approach, data collection methods, data analysis, findings, and dissemination.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 71 ]

Often, stakeholders will be tempted by interesting and emergent infor- mation that threatens the focus or scope of the evaluation during its imple- mentation. If this information is unrelated or unhelpful for answering an evaluation question, remind the stakeholder(s) of the evaluation questions and discourage them from changing course. An evaluation gone off- course will expend valuable time, resources, and energy. One of the best ways to avoid or manage such an occurrence, is to develop a formal or informal written evaluation plan that is shared with all key stakeholders before an evaluation begins. Written evaluation plans are not only helpful for maintaining the scope and focus of the evaluation, but help to clarify and codify roles, responsibilities, expectations, processes, and procedures prior to the evaluation’s commencement.

EVALUATION PROPOSALS AND CONTRACTS

Another helpful step in planning for an evaluation is to prepare a written proposal, memo of understanding, or contract that describes as many of its components as possible. Early drafts of this document help to clarify each component of an evaluation plan and facilitate clear communication and un- derstanding among the evaluation’s key stakeholders, which is the reason I discuss proposals and contracts in this chapter. Their details vary depending on the complexity, scope, and context of the evaluation. Stufflebeam (1999) provides a comprehensive list of items to include in an evaluation contract

Figure 4.7. Matrix for Ranking or Selecting Evaluation Questions. Reproduced by per- mission from Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen, Program eval- uation: Alternative approaches and practical guidelines (Upper Saddle River, NJ: Pearson Education, 2011).

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 72 ] Part One: Prepare

or plan, which I use as a guide or checklist when preparing formal contracts. I also use it to select the most helpful and relevant items for a less formal eval- uation plan. Minimally, I recommend that all evaluation plans contain the eight elements listed in Box 4.2. Although I have discussed only three of those elements, I present them here as a way of emphasizing your goals in creating a clear evaluation plan that both you and the program’s stakeholders are comfort- able with implementing.

O’Sullivan (2004) recommends that an evaluation proposal or plan also include a statement describing the qualifications of personnel who will con- duct the study and the organizational resources that will be available to them. Often, I will refer to the Program Evaluation Standards in my plan, so it is clear that I am upholding professional standards of practice. If you do this, I recom- mend including a brief copy of the standards in the appendices of your plan.

When incorporating evaluation plans within program grant proposals, it is important that an evaluation plan be aligned with the goals and purpose of the grant in addition to that of the program you are evaluating (which should also align with them). It is also important that the evaluation plan include all components required in the request for application or proposal and include

Box 4.2. EIGHT ELEMENTS TO INCLUDE IN AN EVALUATION PLAN

1. A description of the object (i.e., the program) of the evaluation and its context

2. Statements specifying the evaluation’s purpose, goals, and use 3. A brief list of evaluation questions in prioritized order 4. Evidence and indicators for data collection 5. Data collection methods and tools 6. An analytical plan 7. Processes and procedures for reporting and disseminating findings 8. A management plan

a. Roles and responsibilities of the evaluation team (which may in- clude key stakeholders)

b. Protection of participants as humans involved in an evaluation study

c. A section on data ownership and who holds primary decision- making power over the evaluation

d. A timeline with anticipated dates for deliverables (e.g., data summaries or interim reports)

e. A budget

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 73 ]

linkages between program outcomes and the goals or impact a funder is trying to achieve. Typically, I include the funder’s desired goals, outcomes, or impacts as a component of a program theory or logic model in the proposed plan. These days, funders are very familiar with the language of an evalu- ator, but it is helpful to use language from the Request for Application (RFA) or Proposal (RFP) and clarify these terms in the plan’s narrative by using parentheses, italics, or other ways of denoting their connections. For ex- ample, if a desired goal is to increase access to food for those in need and an outcome is defined as increased access to or placement of grocery or thrift stores in specific community locations, you might write the following in a grant’s narrative:

“Increase the proportion of grocery or thrift stores located in impoverished neighborhoods of Clark City from 20% to 80% in five years (outcome)”

You might also follow it with a statement such as the following in order to link it to the funder’s (a state agency) goals:

“This program will contribute to the state’s goal to reduce food insecurity by 10% in five years by increasing nearby food sources in impoverished neighborhoods.”

The logic model, then, would look similar to the one shown in Figure 4.8 (in very simplified form). With a visual representation of the program’s activities and outcomes in relation to the funder’s goals and objectives, the funder is better able to understand the potential contribution of the proposed program to the intended use of and benefits from the funding.

Funders also typically look for a clear connection between program outcomes and their indicators and measures. These connections can be clearly identified by using matrices, such as the one shown in Table 4.2.

Inputs

• Investment Capital • Locations

Program Activities

• Construct grocery stores in 20 inpoverished neighborhoods

Program Outputs

• Grocery stores are constructed and stocked • Grocery stores hire employees

Program Outcomes

• 80% of impoverished neighborhoods have grocery stores • Increased number of people who travel less than one mile to get groceries

Funder Goals

• Level of food insecurity in state is reduced by 10% in five years

Figure 4.8. Example of a logic model for a program targeting an impact to reduce food inse- curity by 10% in five years.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

[ 74 ] Part One: Prepare

In this example, you will notice that the program’s outcomes also serve as indicators toward meeting the funder’s goal to reduce food insecurity by 10% in five years.

Often, program planners do not incorporate sufficient funds in a grant’s budget to adequately cover the evaluation that they desire. As a rule of thumb, an adequate program evaluation typically requires about 10% of the total budget for a program. Some funders will even request that a specific proportion of the budget include funds to support an evaluation. When planning and preparing a grant application to support a program, I strongly urge you to develop an eval- uation plan (with or without an external evaluator) early in the process so that a reasonable budget for an evaluation can be incorporated into the proposal.

SUMMARY

In this chapter, I described the basic elements of a program design and plan that contains the elements of an evaluation plan. This chapter marks the major preparation and planning activities that lay the ground work for an evaluation’s design. By now, you should have a clear idea about the program’s context and its stakeholders, the stakeholder’s questions and vision or theory about how the program works to bring about change, and an idea of what approach(es) you will use for the evaluation. The next five chapters will walk you through some of the choices you will make in designing the evaluation, including the methods you will use to collect and analyze in- formation. As you may have gleaned by now, an evaluation plan generally

Table 4.2. M AT R I X S HO W I N G T H E R E L AT ION S H I P S OF I N DIC ATOR S , DATA S O U R C E S / M E T HO D S TO PR O G R A M O U TC OM E S

Program Outcome Evidence/ Indicator of Change Data Source and Methods

80% of impoverished

neighborhoods have

grocery stores

• Pre/ post mapping shows an

increase in the number of dots

that represent grocery stores

Neighborhood maps of

grocery stores

• Dots appear in impoverished

neighbors where previous “dots”

did not exist.

Increased number of

people who travel less

than one mile to get

groceries

• Pre/ post survey of the number

of miles people travel to get

groceries and the locations of

those facilities relative to a

person’s residence

Survey of residents

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

p L a n n i n g a p r o g r a m e va L u a T i o n [ 75 ]

comprises at least half of a thorough program plan. Therefore, it takes time and resources to develop it. Typically, an evaluation plan’s data collection and analysis methods are somewhat vague when I share early drafts of an evaluation plan with key stakeholders. As discussions about the plan ensue with stakeholders, the plan is further refined to include specific elements of the evaluation’s design. The next two chapters (Part II) cover key elements of an evaluation’s design. Chapter 5 will cover several design options and guide you through their selection based on the evaluation’s purpose and questions. Chapter 6 will provide guidance for choosing a sample, sampling methods, and data sources.

REFERENCES

American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved from http:// www.eval.org/ p/ cm/ ld/ fid=51

Bakken, L. L, Byars- Winston, A., Gundermann, D. M., Ward, E. C., Slattery, A., King, A., . . . Taylor, R. E. (2010). Effects of an educational intervention on female bi- omedical scientists’ research self- efficacy. Advances in Health Sciences Education, 15, 167– 183.

Brousselle, A., & Champagne, F. (2011). Program theory evaluation: Logic analysis. Evaluation and Program Planning, 34, 69– 78.

Caffarella, R. S. (2002). Planning Programs for Adult Learners (2nd ed.). San Francisco: Jossey- Bass.

Conner, R. F., Fitzpatrick, J. L., & Rog, D. J. (2012). A first step forward: Context assessment. In D. J. Rog, J. L. Fitzpatrick, & R. F. Conner (Eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135, 89– 105.

Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San- Francisco: Jossey- Bass Publishers.

Fitzpatrick, J. L. (2012). An introduction to context and its role in evaluation practice. In D. J. Rog, J. L. Fitzpatrick, & R. F. Conner (Eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135, 7– 24.

Fitzpatrick, J. L., Sanders J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, NJ: Pearson Education.

Friedman, V. J. (2001). Designed blindness: An action science perspective on program theory evaluation. America Journal of Evaluation, 22(2), 161– 181.

Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey- Bass.

Kania, J., & Kramer, M. (2011). Collective impact. Stanford Social Innovation Review, Winter, 35– 41. Retrieved from www.ssireview.org.

Knowlton, L. W., & Phillips, C. C. (2009). The logic model guidebook: Better strategies for great results. Thousand Oaks, CA: Sage Publications, Inc.

Kramer, M., Parkhurst, M., & Vaidyanathan, L. (2009). Breakthroughs in shared meas- urement and social impact, 1– 52. Retrieved from www.fsg- impact.org/ ideas/ item/ breakthroughs_ in_ measurement.html.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.eval.org/p/cm/ld/fid=51

http://www.ssireview.org

http://www.fsg-impact.org/ideas/item/breakthroughs_in_measurement.html

[ 76 ] Part One: Prepare

Lent, R., Brown, S., & Hackett, G. (1994). Toward a unifying social cognitive theory of career and academic interest, choice and performance. Journal of Vocational Behavior, 45, 79– 122.

McDonald, N. (2015). The evaluation of change. Cognition, Technology and Work, 17, 193– 206. doi: 10.1007/ s10111- 014- 0296- 9.

O’Sullivan, R. G. (2004). Designing collaborative evaluations. In R. G. O’Sullivan (Ed.), Practicing evaluation: A collaborative approach (pp. 53– 74). Thousand Oaks, CA: Sage Publications.

Pawson, R., & Tilley, N. (1997). Realistic evaluation. Thousand Oaks, CA: Sage. Rog, D. J. (2012). When background becomes foreground: Toward context- sensitive

evaluation practice. In D. J. Rog, J. L. Fitzpatrick, & R. F. Conner (Eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135, 25– 40.

Stufflebeam, D. L. (1999). Evaluation contracts checklist. Available online at: http:// www.wmich.edu/ sites/ default/ files/ attachments/ u350/ 2014/ contracts.pdf.

Wholey, J. S. (1987). Evaluability assessment: Developing program theory. In L. Bickman (Ed.), Using program theory in evaluation. New Directions for Evaluation, 33, 77– 92.

Wholey, J. S. (2004). Evaluability assessment. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed.). San Francisco: Jossey Bass Publishers.

Williams, B., & Hummelbrunner, R. (2011). Systems concepts in action: A practitioner’s toolkit. Stanford, CA: Stanford University Press.

Woodhouse, L. D., Toal, R., Nguyen, T., Keene, D., Gunn, L., Kellum, A., . . . Livingood, W. C. (2013). A merged model of quality improvement and evaluation: Maximizing Return on Investment. Health Promotion Practice, 14(6), 885– 892.

C op

yr ig

ht ©

2 01

8. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

http://www.wmich.edu/sites/default/files/attachments/u350/2014/contracts.pdf