Program Theory and Logic Model

chapter3.pdf

Home >Social Science homework help >Program Theory and Logic Model

THE PROCESS

CENTERS FOR DISEASE CONTROL AND PREVENTION

The previous two chapters presented the rational of how case- and program-level evaluations (internal and external) help us to become more accountable

to society. As you know, our programs are extremely complex and dynamic organizations that have numer- ous outside pressures to attend to, as well as concentrat- ing on their own internal struggles—all at the same time providing efficient and effective services to clients.

Not only do program evaluations (i.e., need, pro- cess, outcome, efficiency) bring us a step closer to accountability; they also help line-level workers and evaluators alike learn about our clients’ life experi- ences, witness client suffering, observe client progress and regress, and feel the public’s pressure to produce totally unrealistic “magnificent and instant positive change” with extremely limited resources.

Integrating evaluation activities into our program’s service delivery system, therefore, presents an immense opportunity for us to learn more about social problems, the people they affect, and how our interventions actu- ally work. For organizational learning to occur, however, there must be an opportunity for continuous, meaning- ful, and useful evaluative feedback. And this feedback must make sense to all of our stakeholder groups. All levels of staff within a program have an influence on the program’s growth and development, so they all must be involved in the “evaluative processes” as well. Thus we now turn our attention to the evaluative process.

THE PROCESS

What’s this “evaluative process,” you ask? The answer is simple. It’s a tried-and-true method that contains

six general steps as presented in Figure 3.1. As with the previous editions of this book, the steps and all related text have been adopted and modified from the Centers for Disease Control and Prevention (CDC; 1999a, 1999b, 1999c, 2005, 2006, 2010, 2011, 2013); Milstein, Wetterhall, and CDC Evaluation Working Group (2000), and Yarbrough, Shulha, Hopson, and Caruthers (2011).

It’s our opinion that the CDC’s evaluative frame- work that we use to describe the program evaluation process is the “gold standard” of all the evaluation frameworks that exist today. Hopefully you will agree after reading this chapter.

The steps are all interdependent on one another and, more often than not, are executed in a nonlinear sequence. An order exists, however, for fulfilling each step—earlier steps provide the foundation for sub- sequent steps. Now that we know there are six steps when evaluating social work programs we imme- diately turn our attention to the first step: engaging stakeholders in the evaluative process.

STEP 1: ENGAGE STAKEHOLDERS

For all four types of evaluations mentioned in the pre- vious chapter and presented in depth in Part III of this book, the evaluation cycle begins by engaging all of our stakeholder groups. As we know by now, almost all social work evaluations involve partnerships with and among its stakeholders; therefore, any evaluation of a program requires considering the value systems of the various stakeholder groups. As you know from the previous two chapters, they must be totally engaged in

3Excellence is a continuous process and not an accident.

~ A. P. J. Abdul Kalam C h a p t e r

Grinnell, R. M., Gabor, P. A., & Unrau, Y. A. (2015). Program evaluation for social workers : Foundations of evidence-based programs. Oxford University Press, Incorporated. Created from capella on 2023-01-30 18:36:21.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

52 Part I: Preparing for Evaluations

the evaluation of your program in order to ensure that their perspectives are understood, appreciated, and, more important, heard. We simply cannot overem- phasize this point enough—if you don’t include your stakeholders in an evaluation it will fail. Guaranteed.

Stakeholders are people or organizations invested in your program, interested in the results of your evaluation, and/or with a stake in what will be done with the results of your evaluation. Representing their needs and interests throughout the process is funda- mental to doing a good program evaluation.

When stakeholders are not engaged, your evalu- ation findings can easily be ignored, criticized, or resisted because your evaluation doesn’t address your stakeholders’ individual evaluation questions or val- ues. After becoming involved, stakeholders can easily help to execute the other five steps. Identifying and engaging three stakeholder groups are critical to your evaluation:

Group 1: Those involved in your program’s operations, such as sponsors, collaborators, coalition partners, funding officials, administrators, executive

directors, supervisors, managers, line-level social workers, and support staff.

Group 2: Those served or affected by your program, such as clients, family members, neighborhood organizations, academic institutions, elected officials, advocacy groups, professional associations, skeptics, opponents, and personnel at related or competing social service programs.

Group 3: Primary users of your evaluation’s results, such as the specific persons in a position to do and/or decide something regarding the findings that were derived from your evaluation.

Clearly, the three categories are not mutually exclusive; in particular, the primary users of evalu- ation findings are often members of the other two groups. For example, your program’s executive direc- tor (Group 1) could also be involved in an advocacy organization or coalition (Group 2) in addition to being the main person who would utilize your evalu- ation’s recommendations (Group 3).

Why Stakeholders Are Important to an Evaluation

Stakeholders can help (or hinder) an evaluation before it’s even conducted, while it’s being con- ducted, and after the results are collected and ready for use. Because so many of our social service efforts are complex and because our programs may be sev- eral layers removed from frontline implementation, stakeholders take on a particular importance in ensuring meaningful evaluation questions are identi- fied and your evaluation results will be used to make a difference.

Stakeholders are much more likely to support your evaluation and act on the results and recom- mendations if they are involved in the evaluation pro- cess. Conversely, without stakeholder support, your evaluation may be ignored, criticized, resisted, or even sabotaged.

You need to identify those stakeholders who mat- ter the most by giving priority to those stakeholders who:

Step 1 Engage

Stakeholders

Step 2 Describe the

Program

Evaluation Standards Utility

Feasibility Propriety Accuracy

Step 6 Ensure Use & Share Lessons

Learned

Step 5 Justify

Conclusions

Step 4 Gather Credible

Data

Step 3 Focus the Evaluation

Figure 3.1: The program evaluation process.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 53

IN A NUTSHELL

Description of the Program Evaluation Process (Narrative for Figure 3.1)

Steps (Figure 3.1) Description of Steps

Step 1 Engage

stakeholders

Evaluation stakeholders are people or organizations that are invested in your program, are

interested in the results of the evaluation, and/or have a stake in what will be done with evaluation

results. Representing their needs and interests throughout the process is fundamental to a good

program evaluation.

Step 2 Describe

the program

A comprehensive program description clarifies the need for your program, the activities you are

undertaking to address this need, and the program’s intended outcomes. This can help you when

it is time to focus your evaluation on a limited set of questions of central importance. Note that

in this step you are describing the program and not the evaluation. Various tools (e.g., logic and

impact models) will be introduced to help you depict your program and the anticipated outcomes.

Such models can help stakeholders reach a shared understanding of the program.

Step 3 Focus the

evaluation design

Focusing the evaluation involves determining the most important evaluation questions and the

most appropriate design for an evaluation, given time and resource constraints. An entire program

does not need to be evaluated all at once. Rather, the “right” focus for an evaluation will depend

on what questions are being asked, who is asking them, and what will be done with the resulting

information.

Step 4 Gather

credible data

Once you have described the program and focused the evaluation, the next task is to gather data

to answer the evaluation questions. Data gathering should include consideration of the following:

indicators, sources of methods of data collection, quality, quantity, and logistics.

Step 5 Justify

conclusions

When agencies, communities, and other stakeholders agree that evaluation findings are justified,

they will be more inclined to take action on the evaluation results. Conclusions become justified

when analyzed and synthesized data is interpreted through the ‘prism’ of values that stakeholders

bring, and then judged accordingly. This step encompasses analyzing the data you have collected,

making observations and/or recommendations about the program based on the analysis, and

justifying the evaluation findings by comparing the data against stakeholder values that have

been identified in advance.

Step 6 Ensure use

and share lessons learned

The purpose(s) you identified early in the evaluation process should guide the use of evaluation

results (e.g., demonstrating effectiveness of the program, modifying program planning,

accountability). To help ensure that your evaluation results are used by key stakeholders, it’s

important to consider the timing, format, and key audiences for sharing information about the

evaluation process and your findings.

3.1

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

54 Part I: Preparing for Evaluations

• Can increase the credibility of your efforts or the evaluation process itself

• Are responsible for day-to-day implementation of the activities that are part of your social work program

• Will advocate for (or authorize changes to) your program that the evaluation may recommend

• Will fund or authorize the continuation or expansion of your program

In addition you must include those who partici- pate in your program and are affected by your pro- gram and/or its evaluation, such as:

• Program line-level staff, supervisors, managers, and administrative support

• Local, state, and regional coalitions interested in the social problem you are trying to solve

• Local grantees of your funds • Local and national advocacy partners • Other funding agencies, such as national and

state governments • State education agencies, schools, and other

educational groups • Universities, colleges, community colleges,

and other educational institutions • Local government, state legislators, and state

governors • Privately owned businesses and business

associations • Health-care systems and the medical

community • Religious organizations • Community organizations • Private citizens • Program critics • Representatives of populations

disproportionately affected by the social problem you are trying to solve. This should include current clients and perhaps clients who have “graduated” from your program.

• Law enforcement representatives

The Role of Stakeholders in an Evaluation

Stakeholder perspectives should influence every step of your evaluation. Stakeholder input in Step 2:

Describe the Program, ensures a clear and consen- sual understanding of your program’s activities and outcomes. This is an important backdrop for even more valuable stakeholder input in Step 3: Focus the Evaluation to ensure that the key questions of most importance are included.

Stakeholders may also have insights or preferences on the most effective and appropriate ways to collect data from target respondents. In Step 5: Justify Conclusions, the perspectives and values that stakeholders bring to your project are explicitly acknowledged and honored in making judgments about the data gathered.

The product of this step is a list of stakeholders to engage and a rationale for their involvement.

Finally, the considerable time and effort you spent in engaging and building consensus among stake- holders pays off in the last step, Step 6: Ensure Use and Lessons Learned, because stakeholder engage- ment has created a market for the evaluation’s results, or findings.

Stakeholders can be involved in your evalu- ation at various levels. For example, you may want to include coalition members on an evaluation team and engage them in developing relevant evaluation questions, data collection procedures, and data anal- yses. Or consider ways to assess your partners’ needs and interests in the evaluation, and develop means of keeping them informed of its progress and inte- grating their ideas into evaluation activities. Again, stakeholders are more likely to support your evalu- ation and act on its results and recommendations if they are involved in the evaluation process from the get-go.

Involve Critics as Well

Have you ever heard the phrase, “keep your friends close and your enemies closer?” Well, this slo- gan aptly applies to the evaluation process as well. It’s very important for you to engage your program’s crit- ics in your evaluation. Critics will help you to identify issues around your program’s strategies and evalua- tion data that could be attacked or discredited, thus helping you strengthen the evaluation process. This

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 55

information might also help you and others under- stand the opposition’s rationale and will help you engage potential agents of change within the opposi- tion. However, use caution: it’s important to under- stand the motives of the opposition before engaging them in any meaningful way.

The emphasis on engaging stakeholders mirrors the increasing prominence of participatory models or “action” research in the research/evaluation commu- nity. A participatory approach combines systematic inquiry with the collaboration of diverse stakehold- ers to meet specific needs and to contend with broad issues of equity and justice.

STEP 2: DESCRIBE THE PROGRAM

Writing a good description of your program sets the frame of reference for all subsequent decisions in the evaluation process. Your description enables comparisons with similar programs and facilitates attempts to connect your program’s components to its intended outcomes. Moreover, your stakeholders might have differing ideas regarding your program’s overall goal and objectives. Evaluations done without agreement on your program’s description will be of limited use.

Using a Logic Model to Describe Your Program

Your evaluation plan must include a logic model for your program as a whole. When developing your evaluation plan it’s important to develop a logic model that specifically describes what your propose to evaluate. Simply put, the product of this step is a logic model of what is being evaluated, which must be accompanied by a text-based description.

Let’s use a quick example of what we mean by a text-based description. Figure 3.1 presents the six steps of doing a program evaluation and the following illustration, “In a Nutshell 3.1,” provides a text-based description of each step. One shows (i.e., Figure 3.1) and the other describes (In a Nutshell 3.1).

We strongly encourage you to develop a text-based description to accompany your logic model. This description must explain what you are

proposing to evaluate and how this contributes to accomplishing your program’s intended outcomes. This section should also describe important program features of what is being evaluated, such as the con- text in which your program operates, the character- istics of the population your program is intended to reach, and its stage of development (e.g., a pilot activ- ity versus an activity that has been in place for a num- ber of years).

The product of this step is creation of a logic model accompanied by a text-based description.

Such descriptions are invaluable, not only for your own records but also for others who might be interested in implementing activities similar to those contained in your program. With a clear description of the activity and context in which your program resides, other social service programs will be better able to determine how likely it is that the evaluation results you obtained relate to what they would see if they chose to implement these same activities in their programs. Chapter 8 describes how to construct logic models in depth. Without a doubt, constructing logic models causes social work students a great deal of anxiety. It’s hard to do, as it makes one think in a logi- cal and consistent manner.

Logic models are nothing more than simple tools that help people physically see the interrelations among the various components of your program. They are concept maps with narrative depictions of programs in that they visually describe the logic of how your program is supposed to work.

Figure 3.2 presents the basic five elements of the standard run-of-the-mill logic model broken down into the work you plan to do (i.e., numbers 1 and 2) and the intended results that you expect to see from your work (i.e., numbers 3–5). Using Figure 3.2 as a guide, Figure 3.3 describes how to read a logic model (W. K. Kellogg Foundation, 2004).

In sum, a logic model is a pictorial diagram that shows the relationship among your program’s compo- nents. It provides your program staff, collaborators, stakeholders, and evaluators with a picture of your

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

56 Part I: Preparing for Evaluations

program, how it operates, and how it’s intended to accomplish your program’s objectives.

By discussing the logic model with different stakeholder groups, you can share your understand- ing of the relationships among the resources you have to operate your program, the activities you plan do, and the changes or results you wish to achieve from your activities. The CDC (2006) provides nine steps that you can follow when developing your logic model:

Step 1: Establish a logic model work group. Your evaluation work group can be composed of program staff, collaborators, evaluators, and other stakeholders. Identify areas where each stakeholder is needed and contact them to discuss their potential interest in participating in the

discussion and any questions or concerns they have about your program.

Step 2: Convene the work group to discuss the purpose and steps for constructing your logic model. Review and summarize relevant literature, planning documents, reports, and data sources that will help explain your program’s purposes, activities, and intended outcomes.

Step 3: Provide an overview of the general logic modeling process. Review the definitions of terms, outline the overall steps to construct or revise a logic model, choose the type of logic model that best fits your program needs, review your goals and objectives (if they already exist), or reach consensus on program goals and

Resources/ Inputs

1 2 3 4 5

Activities Outputs Outcomes

Your Intended ResultsYour Planned Work

Impact

Figure 3.2: The basic logic model.

Resources/ Inputs

1 2 3 4 5

Activities Outputs Outcomes

Your Intended ResultsYour Planned Work

Impact

Certain resources are

needed to operate your

program

If you have access to

resources, then you can use them to accomplish your planned

activities

If you accomplish your planned

activities, then you will

hopefully deliver the amount of product and/or

service that you intended

If you accomplish your planned

activities to the extent you

intended, then your participants

will benefit in certain ways

If these benefits to

participants are achieved, then

certain changes in organizations,

communities, or systems might be

expected to occur

Figure 3.3: How to read a logic model.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 57

IN A NUTSHELL

Step 1 Engaging Stakeholders

When a variety of stakeholders are involved in evaluation planning from the outset you can (a) plan and conduct

evaluations that more closely fit your collective needs, (b) have greater buy-in for the use of your evaluation’s results, and

Purpose

Fostering input, participation, and power-sharing among those persons who have an investment in the

conduct of your evaluation and its findings; it’s especially important to engage primary users of your

evaluation’s findings.

Role

Helps increase chances that your evaluation will be useful; can improve your evaluation’s credibility,

clarify roles and responsibilities, enhance cultural competence, help protect evaluation participants,

and avoid real or perceived conflicts of interests.

Activities

Consulting insiders (e.g., leaders, staff, clients, and program funding sources) and outsiders (e.g.,

skeptics); taking special effort to promote the inclusion of less powerful groups or individuals;

coordinating stakeholder input throughout the process of your evaluation’s design, operation, and use;

avoiding excessive stakeholder identification, which might prevent the progress of your evaluation.

It’s time to engage a group of stakeholders to help you create your evaluation plan. The planning team for your evaluation

should include individuals who are interested in—and perhaps affected by—the specific evaluation to be carried out.

There are three major categories of evaluation stakeholders to consider (Russ-Eft & Preskill, 2009, pp. 141–143):

• Primary stakeholders. Individuals who are involved in program operations and who have the ability to use evaluation

findings to alter the course of a program. Examples of primary stakeholders include program staff and managers, as well

as funders.

• Secondary stakeholders. Individuals who are served by the program and therefore are likely to be affected by any

changes made as a result of the evaluation findings. Examples include program participants (e.g., workshop or training

attendees) or others who are directly reached by your program.

• Tertiary stakeholders. Individuals who are not directly affected by programmatic changes that might result from the

evaluation but who are generally interested in the results. Examples include legislators and other state social service

programs.

A final set of stakeholders—often overlooked but important to engage—are program critics. These are individuals or groups

that may oppose your program based on differing values about how to create change, what changes are necessary, or how

best to utilize limited resources. Engaging opponents of the program in your evaluation can strengthen the credibility of

your results and potentially reduce or mitigate some of the opposition.

Multiple stakeholder perspectives can contribute to rich and comprehensive descriptions of what’s being evaluated

while also facilitating a well-balanced and useful evaluation. Your stakeholders may also be engaged in carrying out your

evaluation or in implementing its recommendations.

3.2

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

58 Part I: Preparing for Evaluations

subsequently outline the objectives in support of each goal.

Step 4: Decide whether you will use the “if-then” method, reverse logic method, or both to construct the logic model. If you have a clear picture of what your inputs and activities will be, you will want to use the “if-then” approach, in which you construct your logic model from left to right, starting with the process components and working toward the outcomes.

The “reverse logic” approach can be used to work from the right to the left of the logic model, starting with the goal and working backward through the process components. If outputs are predetermined, you can start from the middle and branch out in both directions (an approach that combines the previous two methods).

Step 5: Brainstorm ideas for each logic model column. After brainstorming is complete, arrange these items into groups such as professional development, collaborations, and so on. Check that each activity logically links to one or more outputs and each output links to one or more outcomes.

Step 6: Determine how to show your program’s accomplishments and select indicators to measure your outputs and short-term outcomes. The question number for each associated indicator should be placed under the output or short-term outcome that it measures.

Step 7: Perform checks to assure links across logic model columns. You should be able to read the logic model from both left to right and right to left, ensuring that a logical sequence exists between all of the items in each column. It’s often helpful to color-code specific sections of your logic model to illustrate which sections logically follow one another.

Step 8: Ensure that your logic model represents your program but does not provide unnecessary detail. Review the

items placed under the headings and subheadings of the logic model, and then decide whether the level of detail is appropriate. The work group should reach consensus in fine-tuning the logic model by asking: What items in the logic model can be combined, grouped together, or eliminated?

Step 9: Revise and update your logic model periodically to reflect program changes. Changes in your logic model may be needed to reflect new or revised program- matic activities or interventions or to account for a change in a new intervention or new evaluation findings.

Concept Maps

Logic models are nothing more than concept maps. Concept mapping is a tool that can be used to visually illustrate key elements of either the program’s design or aspects of the evaluation plan. Concept mapping is a technique that is used to display infor- mation visually. Surely you have heard the expression “a picture is worth a thousand words.” Concept map- ping makes a complicated thing simple. As Albert Einstein said, “If you can’t explain it simply, you don’t understand it well enough,” and “If I can’t see it, I can’t understand it.” And this is the guy who came up with E = mc2

Concept mapping facilitates communication through pictures; as such, it reduces the amount of text reading that would otherwise be needed in a planning process. Specifically, it’s used to dia- gram concepts and the relationships between them. Concept maps can illustrate simple or complex ideas. For example, Figure 7.6 in Chapter 7 shows a simple concept map illustrating the relationship of the goal of an agency to the goals of three programs housed within the agency.

A more complex concept map is shown in Figure 3.4, which offers a visual illustration of a client-centered program design for a family and com- munity support program. The illustration shows the relationship between the family and community sup- port components of the program, which share both

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 59

office space and program objectives. Figure 3.4 also features the program’s goal and details various activi- ties that workers engage in. Indeed, Figure 3.4 high- lights many key program design concepts that are discuss thoroughly in Chapter 7.

Another example of a concept map is shown in Figure 3.5. Rather than diagramming the relationship between program design concepts (as shown in Figure 3.4), the concept map featured in Figure 3.5 shows the fit of evaluation as a key phase of program operations in both components of the program. Furthermore, the picture reveals that the two program components (fam- ily support and community support) will have separate evaluations but the results of both will be considered together when shared with the community.

Communication tools. Concept maps are communica- tion tools. Thus they can have the effect of answering evaluation questions about a group’s thinking or gen- erating new questions that aim for fuller understand- ing. It’s important to understand that the concept

maps featured in Figures 3.4 and 3.5 present only two of many possible representations. In viewing the two illustrations, perhaps you had ideas about how the program design or the evaluation plan could be illus- trated differently.

It may be that your idea is to add concepts not fea- tured, such as identifying priority evaluation questions or specific measurement instruments. On the other hand, it may be your opinion that Figure 3.4 could be simplified by deleting parts of the illustration such as the program goal statement. Perhaps you see the rela- tionships between concepts differently and would pre- fer to see the concept shapes in another arrangement.

Evaluation planning tools. Concepts maps are also planning tools. To be useful as a planning tool, the exercise of building concept maps should involve representatives of key stakeholder groups. Bringing different stakeholders—especially those with diver- gent views—together to build one concept map can generate rich discussion. Because communication

Friendly office space in highly visible part of neighborhood

• Reduce child abuse reports • Increase support to parents • Increase community efficacy

*drop-in support * meeting space *volunteer hub

*emergency provisions *telephone, fax, Internet

*referral information *program office

Program Objectives

• Peer support hotline to families with children under 18 years old

• Short-term support counseling • Peer support and support groups • Emergency goods and services • Liaising with courts, schools, and

other service providers • Employment training for community

members in volunteer roles

Activities

Program Goal: To enhance quality of life for families that are living in the Edison neighborhood where such problems as poverty, substance abuse, mental illness, and domestic violence put children at risk for abuse and neglect. By improving the

capacity of families and the community they live in, the program aims to build a safe neighborhood that values child well-being.

Family and Community Support Program

High Risk DCFS Neighborhood

Family Support Program Center

• Community outreach and awareness on child safety and well-being

• Time-limited improvement projects within community

• Recruit and sustain community board members and volunteers

• Resource and fund-raising efforts

Activities

Community Support

Figure 3.4: Concept map of a client-centered program design.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

60 Part I: Preparing for Evaluations

can result in intense and impassioned discussions as stakeholders promote different points of view, it’s wise to have a skilled facilitator to accomplish the task.

Once concept maps are created they can be used as visual reminders throughout the planning and evaluation processes. The visual illustrations can function as literal maps that chart future discussion and planning decisions. As such, they should be easily accessible or displayed in clear sight of those working on the program and evaluation plans.

For example, suppose that stakeholders of the family and community support programs wind up spending 40 minutes of a 60-minute meeting in a heated debate about the type of activities that workers are expected to perform in the family support compo- nent of the program. It would be possible, and perhaps strategic, for a workgroup member to mention this fact, point to Figure 3.5, and add the suggestion that the group needs to wrap up discussion about family support to ensure that discussion about the commu- nity support component of the program does not get ignored.

STEP 3: FOCUS THE EVALUATION

After completing Steps 1 and 2, you and your stake- holders should have a clear understanding of your pro- gram and have reached a consensus on its description. Now your evaluation team needs to focus the evaluative efforts. This includes determining the most meaning- ful evaluation questions to ask and the most appropriate evaluation design to implement that would produce the most valid and reliable data that will be used to answer the questions.

Focusing your evaluation assumes that your entire program does not need to be evaluated at any specific point in time. Rather, the precise evaluation design to use entirely depends on what questions are being asked, who is asking the questions, and what will be done with the results.

Since resources for evaluation are always limited, we provide a series of decision criteria to help you determine the best evaluation focus at any point in time. These cri- teria are inspired by two of the four CDC’s evaluation standards that are discussed in the following chapter:

Family and Community Support Program Evaluation Plan

Community Board

Phase 4 Evaluate

Phase 2 Define solutions

(consider linkages with

other providers)

Phase 1 Determine family

needs Family Support

Component

Phase 4 Evaluate

Phase 3 Implement family

solution

Phase 5 Educate

community about program plans

and accomplishments

Phase 1 Determine community

needs

Phase 2 Define time-limited

community improvement

projects

Community Support

Component

Phase 3 Implement community

improvement projects

Figure 3.5: Concept map of an evaluation plan.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 61

• Utility (who will use the results and what information will be most useful to them)

• Feasibility (how much time and resources are available for the evaluation)

The logic model developed in the previous step, Step 2: Describing Your Program, sets the stage for determining the best evaluation focus. The approach to focusing an evaluation in the CDC Evaluation Framework differs slightly from traditional evalua- tion approaches. Rather than a summative evaluation conducted when your program had run its course and asking “Did your program work?”, the CDC framework views evaluation as an ongoing activity over the life of a program that asks,” Is your program working?”

A description of formative and summative evalu- ations is presented in Box 2.1. This may be an excellent time to revisit it before reading further.

In short, your social service program will always be ready for some kind of an evaluation. Because your logic model displays your program from inputs through activities/outputs through to the sequence of outcomes from short term to most distal, it can guide a discussion of what you can expect to achieve at a given point in the life of your program.

Should you focus your evaluative efforts on dis- tal outcomes or only on short- or midterm ones? Conversely, does a process evaluation make the most sense right now?

Types of Evaluations

Many different questions can be part of a program evaluation, depending on how long your program has been in existence, who is asking the question, and why the evaluation information is needed. As we know from the previous chapter, there are four types of evaluations: need, process, outcome, and efficiency. This section ignores needs assessments for the moment and concentrates on questions that the remaining three types of evaluations can answer for a program that is already in existence:

• Process evaluations • Outcome/efficiency evaluations

Process Evaluations

As we know, process evaluations—sometimes referred to as implementation evaluations—document whether a program has been implemented as intended and the reasons why or why not. In process evalua- tions you might examine what activities are taking place, who is conducting the activities, who is reached through the activities, and whether sufficient inputs have been allocated or mobilized. How to do process evaluations is discussed in depth in Chapter 11.

The products of this step include a final set of evaluation questions and the evaluation design that will be used to answer the questions.

Process evaluations are important to help distin- guish the causes of poor program performance—was your program a bad idea in the first place, or was it a good idea that could not reach the standard for imple- mentation that you previously set? In all cases, process evaluations measure whether your actual program’s performance was faithful to your initial plan. Such measurements might include contrasting actual and planned performance along all or some of the following:

• The locale where your services or program are provided (e.g., rural, urban);

• The number of people receiving your services; • The economic status and racial/ethnic

background of people receiving your services; • The quality of your services; • The actual activities that occur while the your

services are being delivered; • The amount of money the evaluation going to

cost; • The direct and in-kind funding for your

services; • The staffing for your services or programs; • The number of your activities and meetings;

and • The number of training sessions conducted.

When evaluation resources are limited—as they usually are—only the most important issues of imple- mentation can be included. The following are some

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

62 Part I: Preparing for Evaluations

IN A NUTSHELL

Step 2: Describing Your Program

Program descriptions set the frame of reference for all subsequent decisions in an evaluation. The description enables

comparisons with similar programs and facilitates attempts to connect program components to their effects. Moreover,

stakeholders might have differing ideas regarding your program’s goal and objectives.

Evaluations done without agreement on the program definition are likely to be of limited use. Sometimes, negotiating

with stakeholders to formulate a clear and logical description will bring benefits before data are available to evaluate your

program’s effectiveness.

Content areas to include in a program description are presented below.

Purpose

Scrutinizing the features of the program being evaluated, including its purpose and place

in the larger social service delivery context. Description includes information regarding the

way your program was intended to function and the way that it actually was implemented.

Also includes features of your program’s context that are likely to influence conclusions

regarding your program.

Role

Improves evaluation’s fairness and accuracy; permits a balanced assessment of strengths

and weaknesses; and helps stakeholders understand how program features fit together

and relate to a larger context.

Activities

Characterizing the need (or set of needs) addressed by your program; listing specific

expectations as goals, objectives, and criteria for success; clarifying why program

activities are believed to lead to expected changes; drawing an explicit logic model to

illustrate relationships between program elements and expected changes; assessing

your program’s maturity or stage of development; analyzing the context within which

your program operates; considering how your program is linked to other ongoing efforts;

avoiding creation of an overly precise description for a program that is under development.

Suggested content

areas to address

when describing your

program

Questions to ask and answer about each content area when describing your program are

listed below. Your main goal is to end up with a logic model that clearly paints an accurate

picture of what your program is all about.

Need What problem or opportunity does your program address? Who experiences it?

Context

What is the operating environment around your program? How might environmental

influences such as history, geography, politics, social and economic conditions, secular

trends, or efforts of related or competing organizations affect your program and its

eventual evaluation?

Stage of development

How mature is your program? Is your program mainly engaged in planning, implementation,

or effects? Is your program the only game in town, or are there similar programs in your

immediate area?

3.3

continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 63

“usual suspects” that compromise a program’s imple- mentation and might be considered for inclusion in a process evaluation:

• Transfers of accountability: When a program’s activities cannot produce the intended outcomes unless some other person or organization takes appropriate action, there is a transfer of accountability.

• Dosage: The intended outcomes of a program’s activities (e.g., training, case management, counseling) may presume a threshold level of participation or exposure to the intervention.

• Access: When intended outcomes require not only an increase in consumer demand but also an increase in supply of services to meet it, then the process evaluation might include measures of access.

• Staff competency: The intended outcomes may presume well-designed program activities delivered by staff that are not only technically competent but also matched appropriately to the target audience. Measures of the match of staff and target audience might be included in the process evaluation

Outcome/Effectiveness Evaluations

Outcome evaluations assess progress on the sequence of outcomes your program is to address.

Programs often describe this sequence using terms like “short-term,” “intermediate,” and “long-term out- comes,” or “proximal” (close to the intervention) or “dis- tal” (distant from the intervention). How to do outcome evaluations is discussed in depth in Chapter 12.

Depending on the stage of development of your program and the purpose of the evaluation, outcome evaluations may include any or all of the outcomes in the sequence, including:

• Changes in client’s attitudes, behaviors, feelings, cognitions, and beliefs;

• Changes in risk or protective behaviors; • Changes in the environment, including

public and private policies, formal and informal enforcement of regulations, and inf luence of social norms and other societal forces; and

• Changes in trends in morbidity and mortality.

While process and outcome evaluations are the most common of all four types of evaluations, there are several other types of evaluation questions that can be central to a specific program evaluation. These include the following:

• Efficiency: Are your program’s activities being produced with minimal use of resources such as budget and staff time? What is the volume of outputs produced by

Resources

(will go in logic model)

What assets are available to conduct your program’s activities, such as time, talent,

technology, information, and money?

Activities

(will go in logic model) What steps, strategies, or actions does your program take to effect change?

Expected effects

(will go in logic model)

What changes resulting from your program are anticipated? What must your program

accomplish to be considered successful?

Logic model

What is the hypothesized sequence of events for bringing about change? How do your

program’s elements (i.e., resources, activities, expected effects) connect with one another

to form a plausible picture of how your program is supposed to work? Logic models are

discussed in Chapter 8.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

64 Part I: Preparing for Evaluations

the resources devoted to your program? (This topic is covered in Chapter 13.)

• Cost-effectiveness: Does the value or benefit of your program’s outcomes exceed the cost of producing them? (This topic is covered in Chapter 13.)

• Cause-effect: Can the your program’s outcomes be related to your program, as opposed to other things that are going on at the same time? (This topic is covered in Tool E.)

All of these types of evaluation questions relate to a part, but not all, of your logic model. Figures 3.6a and 3.6b show where in the logic model each type of evalu- ation focuses. As can be seen in Figure 3.6a, process evaluations would focus on the inputs, activities, and outputs and would not be concerned with outcomes/ effectiveness. Effectiveness evaluations would do the opposite—focusing on some or all outcome boxes but not necessarily on the activities that produced them.

As can be seen in Figure 3.6b, efficiency evalua- tions care about the arrows linking inputs to activi- ties/outputs—how much output is produced for a given level of inputs/resources. Cause-effect focuses on the arrows between specific activities/outputs and specific outcomes—whether progress on the outcome is related to the specific activity/output.

Determining the Focus of an Evaluation

Determining the “correct” evaluation focus is solely determined on a case-by-case basis. Several

guidelines inspired by the utility and feasibility evalu- ation standards (discussed in the following chapter) can help you determine the best focus.

Utility Considerations

1. What is the purpose of your evaluation? “Purpose” refers to the general intent of your evaluation. A clear purpose serves as the basis for your evaluation questions, evaluation design, and data collection methods. Some common purposes are:

• Gain new knowledge about your program’s activities;

• Improve or fine-tune an existing program’s operations (e.g., program processes or strategies);

• Determine the effects of your program by providing data concerning your program’s contributions to its long-term goal; and

• Affect your program’s participants by acting as a catalyst for self-directed change (e.g., teaching).

2. Who will use the results from your evalua- tion? Users are the individuals or organiza- tions that will utilize your evaluation findings. The users will likely have been identified during Step 1 in the process of engaging stake- holders. In this step you needed to secure their input in the selection of evaluation questions

CASUAL ATTRIBUTION

EFFICIENCY

INPUTS ACTIVITIES OUTPUTS SHORT-TERM EFFECTS/

OUTCOMES

INTERMEDIATE EFFECTS/

OUTOCMES

LONG-TERM EFFECTS/

OUTCOMES

Figure 3.6b: Using logic models to determine types of possible evaluations.

INPUTS ACTIVITIES

PROCESS/IMPLEMENTATION OUTCOME/EFFECTIVENESS

OUTPUTS SHORT-TERM EFFECTS/

OUTCOMES

INTERMEDIATE EFFECTS/

OUTOCMES

LONG-TERM EFFECTS/

OUTCOMES

Figure 3.6a: Using logic models to determine types of possible evaluations.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 65

and the evaluation design that would gather data to answer the questions. As you know by now, support from the intended users will increase the likelihood that your evaluation results will be used for program improvement.

3. How will the users actually use the evaluation results? Many insights on use will have been identified in Step 1. Data collected may have varying uses, which should be described in detail when design- ing your evaluation. Some examples of uses of evaluation findings are as follows:

• To document the level of success in achieving your program’s objectives;

• To identify areas of your program that need improvement;

• To decide on how to allocate resources; • To mobilize community support; • To redistribute or expand the locations

where your program or intervention, is being carried out;

• To improve the content of your program’s materials;

• To focus your program’s resources on a specific client population; and

• To solicit more funds or additional partners.

4. What do other key stakeholders need from your evaluation? Of course the most important stakeholders are those who request or who will use the results from your evaluation. Never- theless, in Step 1, you may also have identified stakeholders who, while not using the findings of the current evaluation, have key questions that may need to be addressed in your evalua- tion to keep them engaged. For example, a par- ticular stakeholder may always be concerned about costs, disparities, or cause-effect issues. If so, you may need to add those questions when deciding on an evaluation design.

Feasibility Considerations

The first four questions will help you to identify the most useful focus of your evaluation, but you must

also determine whether it’s a realistic and feasible one. Questions 5 through 7 provide a reality check on your desired focus:

5. What is the stage of development of your program? During Step 2 you identified your program’s stage of development. There are roughly three stages in program development—planning, implementation, and maintenance—that suggest different focuses. In the planning stage, a truly for- mative evaluation—who is your target cli- entele, how do you reach them, how much will it cost—may be the most appropriate focus.

An evaluation that included program outcomes would make little sense at this stage. Conversely, an evaluation of a pro- gram in a maintenance stage would need to include some measurement of progress on developing program outcomes, even if it also included questions about its imple- mentation.

6. How intensive is your program? As you know, some social work programs are wide-ranging and multifaceted. Others may use only one approach to address a large problem. Some programs provide extensive exposure (“dose”) of a program, while oth- ers involve participants quickly and superfi- cially. Simple or superficial programs, while potentially useful, cannot realistically be expected to make significant contributions to distal outcomes of a larger program, even when they are fully operational.

7. What are relevant resource and logistical considerations? Resources and logistics may influence decisions about your evaluation’s focus. Some outcomes are quicker, easier, and cheaper to measure, while others may not be measurable at all. These facts may tilt the decision about the focus of your evaluation toward some outcomes as opposed to others.

Early identification of inconsistencies between utility and feasibility is an impor- tant part of the evaluation focus step. But

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

66 Part I: Preparing for Evaluations

we must also ensure a “meeting of the minds” on what is a realistic focus for a specific program evaluation at a specific point in time.

Narrowing Down Evaluation Questions

As should be evident by now, social work programs are complex entities. In turn, any evaluation within them can also be multifaceted and can easily go in many different directions. For example, a program evaluation can produce data to answer many different questions, such as, “Is a program needed in the first place?” (Chapter 10); “What exactly is my program?” (Chapter 11); “Is my program effective?” (Chapter 12); and “Is my program efficient?” (Chapter 13)?

The list of possible evaluation questions is limit- less, but program resources—human and fiscal—are not. As such, an essential planning task of any evalua- tion is to decide on a reasonable number of questions that will be the main focus of your evaluation. The W. K. Kellogg Foundation (1998) provides four tips for developing evaluation questions:

Tip 1: Ask yourself and evaluation team members why you are asking the questions you are asking and what you might be missing.

Tip 2: Different stakeholders will have different questions. Don’t rely on one or two people (external evaluator or funder)

to determine questions. Seek input from as many perspectives as possible to get a full picture before deciding on what questions to answer.

Tip 3: There are many important questions to address. Stay focused on the primary purpose for your evaluation activities at a certain point in time and then work to prioritize which are the critical questions to address. Because your evaluation will become an ongoing part of project management and delivery, you can and should revisit your evaluation questions and revise them to meet your current needs.

Tip 4: Examine the values embedded in the questions you are asking. Whose values are they? How do other stakeholders, particularly evaluation participants, think and feel about this set of values? Are there different or better questions the evalua- tion team members and other stakeholders could build consensus around?

Sources for Questions

By focusing a program evaluation around clearly defined questions, evaluation activities can be kept manageable, economical, and efficient. All too often stakeholders identify more interests than any single evaluation can reasonably manage.

A multitude of stakeholder-related sources can be utilized to generate a list of potential evalu- ation questions. The W. K. Kellogg Foundation (1998) lists nine stakeholder-related sources for our consideration:

Source 1: Program Director: Directors are usually invaluable sources of information because they are likely to have the “big picture” of the project.

Source 2: Program Staff/Volunteers: Staff members and volunteers may suggest unique evaluation questions because they are involved in the day-to-day operations of the program and have an inside perspective of the organization.

Evaluation questions

Chapter 10

Is a program needed?

Chapter 11

What exactly is my

program anyway?

Chapter 12

Is my program

effective?

Chapter 13

Is my program

efficient?

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 67

Source 3: Program Clientele: Participants/ consumers offer crucial perspectives for the evaluation team because they are directly affected by the program’s services. They have insights into the program that no other stakeholder is likely to have.

Source 4: Board of Directors/Advisory Boards/Other Project Leadership: These groups often have a stake in the program and may identify issues they want addressed in the evaluation process. They may request that certain questions be answered to help them make decisions.

Source 5: Community Leaders: Community leaders in business, social services, and government can speak to issues underlying the conditions of the target population. Because of their extensive involvement in the community, they often are invaluable sources of information.

Source 6: Collaborating Organizations: Organizations and agencies that are collaborating with the program should always be involved in formulating evaluation questions.

Source 7: Program Proposal and Other Documents: The program proposal, funder correspondence, program objectives and activities, minutes of board and advisory group meetings, and other documents may be used to formulate relevant evaluation questions.

Source 8: Content-Relevant Literature and Expert Consultants: Relevant literature and discussion with other professionals in the field can be potential sources of information, and of possible questions, for evaluation teams.

Source 9: Similar Programs/Projects: Evaluation questions can also be obtained from executive directors and staff of other programs, especially when their programs are similar to yours.

Techniques to Focus Questions

Figure 3.7 shows a simple survey that we used to aid us in an evaluation planning session within a rural literacy program. The 24 questions shown in Figure 3.7 are only a sample of those generated by the program’s stakeholders, which included representation from the program’s steering committee, administration, and workers, as well as other professionals and local citi- zens; a total of 20 stakeholders participated in the plan- ning process. The complete brainstorm list (not shown) included more than 80 questions—far too many to focus the program’s evaluation, which had a modest budget.

The simple survey shown in Figure 3.7 was created to gather stakeholder input that would help identify priority questions of interest. The questions listed were created by the program’s stakeholders. Thus the survey itself also had the added benefit of showing stakehold- ers that their ideas were both valued and were being put to good use in planning the program’s evaluation strategy.

Evaluations that are not sufficiently focused gener- ally result in large and unwieldy data collection efforts. Unfortunately, when mass quantities of data are col- lected without a forward thinking plan—linking the data collected to the evaluation questions to be answered—the data may be compromised by poor reliability and validity. On the other hand, evaluation data derived from carefully focused questions make it much easier to maintain the integrity of the data col- lection process and produce credible results.

Focusing an evaluation does not imply that only one part or aspect of a program or service will be of interest. In fact, there are usually a number of different interests that can be accommodated within a single evaluation. Figure 3.7, for example, suggests that, depending on the stakeholders’ ratings, the literacy program’s evalua- tion could end up focusing on questions related to cli- ent characteristics (Questions 1–10), program services (Questions 11–18), or client outcomes (Questions 19–24), or a combination of all three.

Focusing evaluation questions means that pro- gram interests are first identified and the evaluation’s activities are then organized around those interests. Thus there can be multiple points of focus within a

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

68 Part I: Preparing for Evaluations

Figure 3.7: Example of a simple survey determining the priority of the evaluation questions that were selected for the final evaluation.

Evaluation Question Priority Survey

Instructions: (1) Rate each question by circling one number using the scale to the right of each question. (2) Feel free to

add questions that you consider to be a priority for evaluation.

Client Characteristic Questions: Definitely Keep Deserves Consideration Throw Out

1. Who referred family to the program? 1 2 3

2. How many children in the family? 1 2 3

3. How old is each family member? 1 2 3

4. How long has the family lived in the community? 1 2 3

5. What is the family structure? 1 2 3

6. Does the family live in town or rural? 1 2 3

7. Does the family access other community services? 1 2 3

8. What languages are spoken in the home? 1 2 3

9. What are the education levels of parents? 1 2 3

10. Does family have (or want) a library card? 1 2 3

Program Service Questions:

11. How many visits were made to the family? 1 2 3

12. How long was each visit? 1 2 3

13. How many scheduled visits were missed? Why? 1 2 3

14. How many times was family not ready for the visit? 1 2 3

15. Did family readiness improve over time? 1 2 3

16. How satisfied were parents with program? 1 2 3

17. How satisfied was family with the worker? 1 2 3

18. What was easiest/most difficult for you in the

program? 1 2 3

continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 69

single evaluation, but it’s important that these be clearly identified and planned from the beginning.

The focal questions selected for a program’s evalu- ation need not remain static. Questions may be added or deleted as circumstances and experiences dictate. In other words, a specific set of questions may guide the focus of an evaluation for a limited period of time.

STEP 4: GATHERING CREDIBLE DATA

In this step you will work with your stakeholders to identify the data collection methods and sources you will use to answer your evaluation questions. You will need to review your data collection plan in light of the work you did in your evaluation planning process:

• Are there new data sources you may want to incorporate?

• Do your methods meet your stakeholders’ needs for information?

• Do you need to adjust your data collection timeline?

• Are there measures you might standardize across evaluations?

For new evaluative efforts, you may want to build in a pilot test or more small-scale data collection efforts before conducting a more intensive effort. As you develop your data collection approach, it’s critical

to keep in mind why you are collecting the data and how you will use them. Being explicit about the use of data before they are collected helps you to conserve resources and reduces respondent burden.

The products of this step include data collection methods and indicators that will be used to answer your evaluation questions.

Your stakeholders may also help identify indi- cators that could be used to judge your program’s success. Let’s say you have chosen to evaluate a rela- tively new educationally oriented type of intervention designed to educate line-level social workers within your community about how the Affordable Care Act (aka, Obamacare) will affect their clientele. You want to know, for example, to what extent your intended tar- get audience is attending (item 1 below) and complet- ing the training, or your intervention (item 2 below) broken down by the type of practitioner they are (item 3 below). Your stakeholders decide that training atten- dance logs will be maintained and recommend includ- ing the following three specific indicators:

1. Attendance 2. Proportion of attendees who complete the

training 3. Type of social work practitioner (commu-

nity organizers, group workers, school social workers, medical social workers,

Client Outcome Questions:

19. Do clients show change after the program? 1 2 3

20. Do children’s literacy skills improve? 1 2 3

21. Do reading behaviors change? 1 2 3

22. Were the parents’ expectations of program met? 1 2 3

23. What is the support worker’s evaluation of services? 1 2 3

24. Has enjoyment for reading increased? 1 2 3

Figure 3.7: (continued)

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

70 Part I: Preparing for Evaluations

IN A NUTSHELL 3.4 Step 3:

Focusing Your Evaluation Design

The direction and process of your evaluation must be focused to assess issues of greatest concern to stakeholders while

using time and resources as efficiently as possible. Not all design options are equally well suited to meeting the information

needs of your stakeholders.

After data collection begins, changing procedures might be difficult or impossible, even if better methods become obvious.

A thorough plan anticipates intended uses and creates an evaluation strategy with the greatest chance of being useful,

feasible, ethical, and accurate.

Content areas to include when focusing your evaluation design are presented below.

Purpose

Planning in advance where your evaluation is headed and what steps will be taken; process is

iterative (i.e., it continues until a focused approach is found to answer evaluation questions

with methods that stakeholders agree will be useful, feasible, ethical, and accurate); evaluation

questions and methods might be adjusted to achieve an optimal match that facilitates use by

primary users.

Role

Provides investment in quality; increases the chances that your evaluation will succeed by

identifying procedures that are practical, politically viable, and cost-effective; failure to plan

thoroughly can be self-defeating, leading to an evaluation that might become impractical

or useless; when stakeholders agree on a design focus, it’s used throughout the evaluation

process to keep your project on track.

Activities

Meeting with stakeholders to clarify the real intent or purpose of your evaluation; learning

which persons are in a position to actually use the findings, then orienting the plan to meet their

needs; understanding how your evaluation results are to be used; writing explicit evaluation

questions to be answered; describing practical methods for sampling, data collection,

data analysis, interpretation, and judgment; preparing a written protocol or agreement

that summarizes your evaluation procedures, with clear roles and responsibilities for all

stakeholders; revising parts or all of your evaluation plan when critical circumstances change.

Suggested

content areas to

address when

focusing your

evaluation design

Questions to ask and answer about each content area when focusing your evaluation design

are listed below. Your main goal is to end up with an evaluation design that is useful, feasible,

ethical, and accurate.

Purpose What is the intent or motive for conducting your evaluation (i.e., to gain insight, change practice,

assess effects, or affect participants)?

Users

Who are the specific persons that will receive your evaluation findings or benefit from being

part of your evaluation? How will each user apply the information or experiences generated

from your evaluation?

Uses How will each user apply the information or experiences generated from your evaluation?

continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 71

DHS workers, child protection workers, and so on)

Learn more about how to collect credible data to answer your evaluation questions in Tool H in the Evaluation Toolkit.

You can see from this list of indicators that it will be important to have a question on the attendance sheet that asks attendees what type of social work practitioner they are (item 3). Had you not discussed the indicators that will be used to determine the “suc- cess” of your intervention, it’s possible this important question would have been left off the attendance log.

STEP 5: JUSTIFYING YOUR CONCLUSIONS

Planning for data analyses and interpretation of the data prior to conducting your evaluation is impor- tant to ensure that you collect the “right” data to fully answer your evaluation questions. Think ahead to how you will analyze the data you collect, what methods you will use, and who will be involved in interpreting the results.

Part of this process is to establish standards of performance against which you can compare the indicators you identified earlier. You may be familiar

with “performance benchmarks,” which are one type of standard. In this example, a benchmark for the indicator “proportion of attendees who complete training” may be “more than 60% of attendees com- pleted the training.” Standards often include compar- isons over time or with an alternative approach (e.g., no action or a different intervention). It’s important to note that the standards established by you and your stakeholders do not have to be quantitative in nature.

The products of this step include a set of performance standards and a plan for synthesizing and interpreting evaluation findings.

Regardless of whether your “indicators” are quali- tative or quantitative in nature, it’s important to dis- cuss with evaluation stakeholders what will be viewed as a positive finding. The standards you select should be clearly documented in the individual evaluation plan.

Make sure to allow time for synthesis and inter- pretation in your individual evaluation plan. At the completion of your evaluation, you will want to be able to answer such questions as:

• Overall, how well does what is being evaluated perform with respect to the standards established in the evaluation plan?

• Are there changes that may need to be made as a result of your evaluation’s findings?

Questions

What questions should your evaluation answer? What boundaries will be established to create

a viable focus for your evaluation? What unit of analysis is appropriate (e.g., a system of related

programs, a single program, a project within a program, a subcomponent or process within a

project)?

Methods

What procedures will provide the appropriate information to address stakeholders’ questions

(i.e., what research designs and data collection procedures best match the primary users,

uses, and questions)? Is it possible to mix methods to overcome the limitations of any single

approach?

Agreements

How will your evaluation plan be implemented within available resources? What roles and

responsibilities have the stakeholders accepted? What safeguards are in place to ensure that

standards are met, especially those for protecting human subjects?

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

IN A NUTSHELL 3.5 Step 4:

Gathering Credible Data

Persons involved in an evaluation should strive to collect data that will convey a well-rounded picture of your program and

be seen as credible by your evaluation’s primary users. Data should be perceived by your stakeholders as believable and

relevant for answering their questions. Such decisions depend on the evaluation questions being posed and the motives for

asking them. Having credible data strengthens evaluation judgments and the recommendations that follow from them.

Although all types of data have limitations, an evaluation’s overall credibility can be improved by using multiple procedures

for gathering, analyzing, and interpreting data. When stakeholders are involved in defining and gathering data that they find

credible, they will be more likely to accept your evaluation’s conclusions and to act on its recommendations.

The following aspects of data gathering typically affect perceptions of credibility.

Purpose

Compiling data that stakeholders perceive as trustworthy and relevant for answering their

questions. Such data can be experimental or observational, qualitative or quantitative, or it

can include a mixture of methods. Adequate data might be available and easily accessed,

or it might need to be defined and new data collected. Whether a body of data are credible

to stakeholders might depend on such factors as how the questions were posed, data

sources, conditions of data collection, reliability of the measurement procedures, validity

of interpretations, and quality control procedures.

Role

Enhances the evaluation’s utility and accuracy; guides the scope and selection of data and

gives priority to the most defensible data sources; promotes the collection of valid, reliable,

and systematic data that are the foundation of any effective evaluation.

Activities

Choosing indicators that meaningfully address evaluation questions; describing fully the

attributes of data sources and the rationale for their selection; establishing clear procedures and

training staff to collect high-quality data; monitoring periodically the quality of data obtained

and taking practical steps to improve their quality; estimating in advance the amount of data

required or establishing criteria for deciding when to stop collecting data in situations where an

iterative or evolving process is used; safeguarding the confidentiality of data and data sources.

Suggested content

areas to address when

collecting credible data

Questions to ask and answer about each content area in relation to collecting credible data

are listed below. You main goal is to collect valid and reliable relevant data.

Indicators

How will general concepts regarding the program, its context, and its expected effects

be translated into specific measures that can be interpreted? Will the chosen indicators

provide systematic data that are valid and reliable for the intended uses?

Sources

What sources (i.e., persons, documents, observations) will be accessed to gather data?

What will be done to integrate multiple sources, especially those that provide data in

narrative form and those that are numeric?

Quality Are the data trustworthy (i.e., reliable, valid, and informative for the intended uses)?

Quantity What amount of data are sufficient? What level of confidence or precision is possible? Is

there adequate power to detect effects? Is the respondent burden reasonable?

Logistics What techniques, timing, and physical infrastructure will be used for gathering and handling data?

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 73

STEP 6: ENSURING USAGE AND SHARING LESSONS LEARNED

As we have seen, you can promote the use of your evaluation findings by the actions you take throughout your evaluation’s planning process. Building a com- mitment to using evaluation results both internally and with your stakeholders is extremely important. Sharing what you have learned will also add to our knowledge base about what interventions work with specific clientele.

The product of this step includes a communication and reporting plan for your evaluation.

Thinking about the use of your evaluation findings does not need to wait until your evaluation is completed and results are ready to be disseminated. Think early and often about how and at what points you can (and need to) make use of your evaluation’s results. Pilot-test results can be used to improve program processes. Baseline results can help to better target your interven- tion. Preliminary findings can help you to refine your data collection strategies in future rounds. Build in time to your schedule to ensure your evaluation’s findings are actually used. For example, will you have enough time after your results are finalized to develop an action plan for program improvement?

Dissemination of results and communication about lessons learned should not be an afterthought. To increase the likelihood that intended audiences will use your evaluation findings for program improve- ment, it’s important to think through how and with whom you will communicate as you plan and imple- ment each evaluation, as well as after the evaluation has been completed. Your strategy should consider the purpose, audience, format, frequency, and timing of each communication (Russ-Eft & Preskill, 2009).

As you develop your dissemination plan, keep in mind the following:

• Consider what information you want to communicate. What action do you hope each of your audiences will take based on the information you provide? Are you just keeping them informed, or do you want them to act in some way? Tailor your communication plan accordingly.

• Your audience will likely vary greatly across evaluations and also may change as an evaluation progresses. Think broadly about who to include in communication. For instance, at various points in time you may want to include executive directors, program managers, supervisors, individuals participating in planning the evaluation, legislators or funders, or individuals affected by your program.

• Formats can be formal or informal and may include a mix of e-mail correspondence, newsletters, written reports, working sessions, briefings, and presentations. Formats may differ by audience and may also differ over time for the same audience as information needs change.

• Consider your communication strategies when estimating the resources that will be required to carry out your evaluation. If your evaluation resources are limited, we recommend giving the greatest consideration to the information needs of the primary evaluation stakeholders (those who have the ability to use your evaluation’s findings).

SUMMARY

This chapter presented a discussion on how the CDC’s six-step evaluation process unfolds and stressed how our stakeholders need to be involved in every aspect of our evaluation. The next chapter discusses how we, as professional social workers, must follow strict pro- fessional standards when evaluating our programs, taking into account the contents of the first three chapters of this book.

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

74 Part I: Preparing for Evaluations

IN A NUTSHELL 3.6 Step 5:

Justifying Your Conclusions

The conclusions that you draw from your evaluation conclusions are only justified when they are directly linked to the data

you gathered. They will be judged against agreed-on values or standards set by your stakeholders. Stakeholders must

agree that your conclusions are justified before they will use the results from your evaluation with any confidence.

Purpose

Making claims regarding your program that are warranted on the basis of data that have

been compared against pertinent and defensible ideas of merit, value, or significance (i.e.,

against standards of values); conclusions are justified when they are linked to the data

gathered and consistent with the agreed-on values or standards of stakeholders.

Role

Reinforces conclusions central to the evaluation’s utility and accuracy; involves values

clarification, qualitative and quantitative data analysis and synthesis, systematic

interpretation, and appropriate comparison against relevant standards for judgment.

Activities

Using appropriate methods of analysis and synthesis to summarize findings; interpreting

the significance of results for deciding what the findings mean; making judgments

according to clearly stated values that classify a result (e.g., as positive or negative

and high or low); considering alternative ways to compare results (e.g., compared with

program objectives, a comparison group, national norms, past performance, or needs);

generating alternative explanations for findings and indicating why these explanations

should be discounted; recommending actions or decisions that are consistent with the

conclusions; and limiting conclusions to situations, time periods, persons, contexts, and

purposes for which the findings are applicable.

Suggested content

areas to address

when justifying your

conclusions

Questions to ask and answer about each content area when in comes to justifying your

conclusions are listed below. Your main goal is to end up with conclusions that are based

on solid reliable and valid data that your stakeholders will appreciate.

Standards

Which stakeholder values provide the basis for forming the judgments? What type or level

of performance must be reached for your program to be considered successful? to be

unsuccessful?

Analysis and synthesis What procedures will you use to examine and summarize your evaluation’s findings?

Interpretation What do your findings mean (i.e., what is their practical significance)?

Judgment What claims concerning your program’s merit, worth, or significance are justified based on

the available data (evidence) and the selected standards?

Recommendations

What actions should be considered resulting from your evaluation? (Note: Making

recommendations is distinct from forming judgments and presumes a thorough

understanding of the context in which programmatic decisions will be made.)

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 75

IN A NUTSHELL 3.7 Step 6:

Ensuring Usage and Sharing Lessons Learned

Assuming that the lessons you learned in the course of your evaluation will automatically translate into informed decision-making

and appropriate action would be naive. Deliberate effort is needed on your part to ensure that your evaluation processes and

findings are used and disseminated appropriately. Preparing for use involves strategic thinking and continued vigilance, both of

which begin in the earliest stages of stakeholder engagement and continue throughout the entire evaluation process.

Purpose

Ensuring that stakeholders are aware of the evaluation procedures and findings; the findings

are considered in decisions or actions that affect your program (i.e., findings use); those who

participated in the evaluation process have had a beneficial experience (i.e., process use).

Role

Ensures that evaluation achieves its primary purpose—being useful; however, several

factors might influence the degree of use, including evaluator credibility, report clarity,

report timeliness and dissemination, disclosure of findings, impartial reporting, and

changes in your program or organizational context.

Activities

Designing the evaluation to achieve intended use by intended users; preparing stakeholders

for eventual use by rehearsing throughout the project how different kinds of conclusions

would affect program operations; providing continuous feedback to stakeholders regarding

interim findings, provisional interpretations, and decisions to be made that might affect

likelihood of use; scheduling follow-up meetings with intended users to facilitate the transfer

of evaluation conclusions into appropriate actions or decisions; disseminating both the

procedures used and the lessons learned from the evaluation to stakeholders using tailored

communications strategies that meet their particular needs.

Suggested content

areas to address when

ensuring usage and

sharing lessons learned

Questions to ask and answer about each content area when it comes to ensuring usage of your

findings and sharing the lessons you learned. Your main goal is to be sure your findings are

utilized in addition to sharing with others what lessons you learned from your evaluation.

Design Is your evaluation organized from the start to achieve the intended uses by your primary

stakeholder groups?

Preparation Have you taken steps to rehearse the eventual use of your evaluation findings? How have

your stakeholder groups been prepared to translate new knowledge into appropriate action?

Feedback What communication will occur among parties to the evaluation? Is there an atmosphere of

trust among stakeholders?

Follow-up

How will the technical and emotional needs of users be supported? What will prevent lessons

learned from becoming lost or ignored in the process of making complex or politically

sensitive decisions? What safeguards are in place for preventing misuse of the evaluation?

Dissemination

How will the procedures or the lessons learned from your evaluation be communicated to

your relevant stakeholders in a timely, unbiased, and consistent fashion? How will your

reports be tailored to your various stakeholder groups?

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

76 Part I: Preparing for Evaluations

Study Questions Chapter 3 The goal of this chapter is to provide you with a beginning knowledge base for you to feel comfortable in answering the

following questions. AFTER you have read the chapter, indicate how comfortable you feel you are in answering each

question on a 5-point scale where

Very

uncomfortable

Somewhat

uncomfortable

Neutral

Somewhat

comfortable

Very

comfortable

If you rated any question between 1–3, reread the section of the chapter where the information for the question is found. If

you still feel that you’re uncomfortable in answering the question then talk with your instructor and/or your classmates for

more clarification.

Questions Degree of comfort? (Circle one number)

Without peeking at Figure 3.1, list the six steps that you would have to go through in

doing an evaluation. Then describe each step in relation to your field placement (or

work setting) to illustrate your main points.

1 2 3 4 5

List the main stakeholder groups that you would need to formulate for your

evaluation. Then describe the role that each stakeholder group would have in

relation to your field placement (or work setting) to illustrate your main points.

1 2 3 4 5

In your own words, describe the purpose of a logic model when describing your

program (Step 2). Then describe how it would be used in relation to your field

placement (or work setting) to illustrate your main points.

1 2 3 4 5

List the five elements of a logic model and describe each element in detail. Then

construct a logic model in relation to your field placement (or work setting) to

illustrate your main points.

1 2 3 4 5

5 In reference to logic models, what are “if-then” statements? Make an “if-then” statement in

relation to your field placement (or work setting) to illustrate your main points. 1 2 3 4 5

What are concept maps? How are they used when doing an evaluation? Provide

specific social work examples from your field placement (or work setting) to

illustrate your main points.

1 2 3 4 5

7 What are the differences between a formative and a summative evaluation?

Describe how your field placement (or work setting) could use both of them. 1 2 3 4 5

When focusing an evaluation you must be concerned with two of CDC’s standards:

utility and feasibility. List the four questions that you will need to ask and answer

under the utility standard and the three questions under the feasibility standard.

Then describe the two evaluation standards in relation to your field placement (or

work setting) to illustrate your main points.

1 2 3 4 5

continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

Chapter 3: The Process 77

List and describe the four main types of evaluation questions that an evaluation

can answer. Then describe each question in relation to your field placement (or work

setting) to illustrate your main points.

1 2 3 4 5

10 What four chapters in this book describe how to do needs assessments, process

evaluations, outcome evaluations, and efficiency evaluations? 1 2 3 4 5

In reference to formulating evaluation questions, list four tips that you can use to

make the task easier. Then describe each tip in relation to your field placement (or

work setting) to illustrate your main points.

1 2 3 4 5

In reference to formulating evaluation questions, list the nine stakeholder groups

(sources) that you can use to make the task easier. Then describe how you can use each

source in relation to your field placement (or work setting) to illustrate your main points.

1 2 3 4 5

13 Describe how you will work with stakeholders to describe your program (Step 2).

Use your field placement (or work setting) to illustrate your main points. 1 2 3 4 5

14 Describe how you will work with stakeholders to focus your evaluation (Step 3). Use

your field placement (or work setting) to illustrate your main points. 1 2 3 4 5

15 Describe how you will work with stakeholders to gather credible data (Step 4). Use

your field placement (or work setting) to illustrate your main points. 1 2 3 4 5

Describe how you will work with stakeholders to justify your conclusions from an

evaluation (Step 5). Use your field placement (or work setting) to illustrate your

main points.

1 2 3 4 5

17 Describe how you will work with stakeholders to ensure that your evaluation’s findings are

used (Step 6). Use your field placement (or work setting) to illustrate your main points. 1 2 3 4 5

Discuss how you would engage “stakeholders” for a program evaluation. Then

discuss how you would engage client systems within your field placement setting.

Notice any differences between the two? If so, what are they? Provide specific

social work examples throughout your discussion.

1 2 3 4 5

Discuss in detail how you would describe a program before it’s evaluated. Then

discuss in detail how you assess your client systems psychosocial environments

before you intervene. Notice any differences between the two? If so, what are they?

Provide specific social work examples throughout your discussion.

1 2 3 4 5

Discuss in detail how you would focus an evaluation. Then discuss how you would

narrow down a client’s presenting problem area so it can become more specific and

manageable. Notice any differences between the two? If so, what are they? Provide

specific social work examples throughout your discussion.

1 2 3 4 5

Study Questions for Chapter 3 Continued

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

78 Part I: Preparing for Evaluations

Chapter 3 Assessing Your Self-Efficacy

AFTER you have read this chapter AND have completed all of the study questions, indicate how knowledgeable you feel you

are for each of the following concepts on a 5-point scale where

Not knowledgeable

at all

Somewhat

unknowledgeable

Neutral

Somewhat

knowledgeable

Very

knowledgeable

Concepts Knowledge Level? (Circle one number)

1 Listing the six steps (in the order they are presented in this book) of doing an

evaluation 1 2 3 4 5

2 Describing in detail each one of the six step of the evaluation process 1 2 3 4 5

3 Utilizing stakeholders to help you describe your program (Step 2) 1 2 3 4 5

4 Utilizing stakeholders to help you focus your evaluation (Step 3) 1 2 3 4 5

5 Utilizing stakeholders to help you gather credible data for your evaluation (Step 4) 1 2 3 4 5

6 Utilizing stakeholders to help you to justify your conclusions from your evaluation

(Step 5) 1 2 3 4 5

7 Utilizing stakeholders to help you to ensure that the findings from your evaluation

will be used (Step 6) 1 2 3 4 5

8 Constructing logic models 1 2 3 4 5

9 Constructing “if-then” statements for logic models 1 2 3 4 5

10 Developing concept maps 1 2 3 4 5

Add up your scores (minimum = 10, maximum = 50) Your total score =

A 48–50 = Professional evaluator in the making

A− 45–47 = Senior evaluator

B+ 43–44 = Junior evaluator

B 40–42 = Assistant evaluator

B− 10–39 = Reread the chapter and redo the study questions

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .

CHAPTER OUTLINE

THE FOUR STANDARDS Utility

Feasibility

Propriety

Accuracy

STANDARDS VERSUS POLITICS When Standards Are Not Followed

SUMMARY

C op

yr ig

ht ©

2 01

5. O

xf or

d U

ni ve

rs ity

P re

ss , I

nc or

po ra

te d.

A ll

rig ht

s re

se rv

ed .