Mentoring in workplace- assignment

3

1 The Essence of Program Theory

A N APPLE A DAY KEEPS the doctor away—or does it? Thinking about how we would fi nd out if this is true and how we might use those

fi ndings shows the value of program theory. In this chapter, we set out the

key ideas in program theory and show how program theory can be used to

learn from success, failure, and mixed results to improve planning, manage-

ment, evaluation, and evidence-based policy.

EVALUATION WITHOUT PROGRAM THEORY

Let us imagine that we have implemented a program based on the broad

policy objective of an apple a day in order to keep the doctor away. This pro-

gram, which we dubbed An Apple a Day, involves distributing seven apples a

week to each participant. A representation of this program without program

theory would simply show the program followed by the intended outcome

of improved health (Figure 1.1).

CH001.indd 3CH001.indd 3 1/5/11 12:49:09 PM1/5/11 12:49:09 PM

Funnell, S.C. & Rogers, P.J., 2011, Purposeful program theory : effective use of theories of change and logic models, Jossey-Bass, San Francisco, CA

P u r p o s e f u l P r o g r a m T h e o r y4

Figure 1.1 An Evaluation of An Apple a Day Without Program Theory

This is what is often referred to as a black box evaluation: one that

describes an evaluation that analyzes what goes in and what comes out with-

out information about how things are processed in between.

PROGRAM OUTCOMES

Improved health

ORIGINS OF “BLACK BOX”

Different sources have been suggested for the term black box. The current

Wikipedia entry for black box traces the term, when used for fl ight data recorders,

to World War II Royal Air Force terminology, when prototypes of new electronic

devices were installed in airplanes in metal boxes, painted black to avoid refl ections

and therefore referred to as black boxes.

Former electronics buff turned evaluator Bob Briggs, on the American

Evaluation Association’s discussion list EVALTALK (Briggs, 1998), reminisced how

electronics manufacturers would often cover components with opaque material to

prevent consumers from “opening the black box” to see how it worked (and assem-

bling their own version more cheaply). The parallel with evidence-based practices

is useful: program theory aims to help policymakers and practitioners “open up the

box” of successful programs to understand how it works rather than having to buy

the whole package and plug it in.

However, as the evaluator and author Michael Quinn Patton (1998) pointed out

in the same EVALTALK thread, the term can be seen as inappropriate: “Most uses

of ’black box’ or ’black box design’ carry a negative connotation. The association of

’black’ with negativity is what can be experienced as offensive, or at least insensi-

tive” (Patton, 1998). He suggested using instead terms such as empty box, magic

box, or mystery box designs to describe evaluations without program theory.

CH001.indd 4CH001.indd 4 1/5/11 12:49:09 PM1/5/11 12:49:09 PM

T h e E s s e n c e o f P r o g r a m T h e o r y 5

It can be diffi cult to interpret results from an evaluation that has no

program theory. For an intervention that involves a discrete product for indi-

viduals, an experimental or quasi-experimental design might be appropriate

for the evaluation. We will assume that people have been assigned to either a

treatment group, who received the program, or to a control group, who went

onto a waiting list to receive the program later if the evaluation shows it is

effective. “Keeping the doctor away” has been operationalized as “maintain-

ing or achieving good physical health.” Data collection has been carefully

designed to avoid measurement failure of outcome variables, with adequate

sample size, appropriate measures of health, and systems in place to avoid

accidental or deliberate data corruption.

Despite careful evaluation, it can be impossible to interpret evaluation

results correctly in the absence of program theory. If the program failed to

achieve signifi cant differences in health outcomes between the groups (apple

versus no apple), it might seem that the policy does not work—but it might

also be that it has not been implemented properly. Maybe the apples were

delivered but not eaten, or maybe they were too small, or too unripe, or too

overripe to work as expected. Although the evaluation might include some

measures of the quality and extent of implementation, it can be hard to

know what aspects should be included unless there is a program theory.

An evaluation using program theory would identify how we understand

this program works and what intermediate outcomes need to be achieved for

the program to work. This allows us to distinguish between implementation

failure (not done right) and theory failure (done right but still did not work).

Without program theory, it is impossible to know if we have measured the

right aspects of implementation quality and quantity.

If the results showed that the program seemed to have succeeded, as the

treatment group had signifi cantly better outcomes than the no-treatment

group, we might also have trouble using these results more broadly. If we

do not know what elements of the policy are important, we can only copy it

exactly for fear of missing something essential. It does not provide any guid-

ance for adapting the policy for other settings.

Finally, if we had mixed results, where the policy worked on only some

sites or for some people, we might not even notice them if we were looking

CH001.indd 5CH001.indd 5 1/5/11 12:49:09 PM1/5/11 12:49:09 PM

P u r p o s e f u l P r o g r a m T h e o r y6

only at the average effect. If we did see differential effects in different contexts

(for example, for men compared to women, or in urban areas rather than

rural areas), an evaluation without program theory leaves us in the position

of having to do simple pattern matching (for example, using the policy for

the groups or sites where it has been shown to work) but with little ability to

generalize to other contexts.

EVALUATION WITH PROGRAM THEORY

If we used a program theory approach, we would try to understand the

causal processes that occur between delivering apples and improved health.

We might start by unpacking the box to show the important intermediate

outcome that people actually eat the apples. The logic model diagrams in

Figure 1.2 show this: one in the form of a pipeline model and one as an out-

comes chain. The pipeline logic model represents the program in terms of

inputs, processes, outputs, and outcomes. The outcomes chain model shows

a series of results at different stages along a causal chain.

Although these look like many logic models that are used regularly in eval-

uation, they are not much of a theory; rather, they are more like a two-step

Pipeline model version

INPUTS Apples

People in poor health

PROCESSES Apples

delivered

OUTPUT Apples eaten

OUTCOMES Improved

health

Outcomes chain version

Apples eaten

Improved health

Figure 1.2 Simple Pipeline and Outcomes Chain Logic Models

CH001.indd 6CH001.indd 6 1/5/11 12:49:10 PM1/5/11 12:49:10 PM

T h e E s s e n c e o f P r o g r a m T h e o r y 7

process, as Mark Lipsey and John Pollard (1989) called it, that identifi es an

intermediate variable without really explaining how it works. These models

make it clear that eating the apples is understood to be part of the causal

chain (rather than some other variable, such as social interaction with the

apple deliverer or physical exercise from playing with the apples). But they

do not explain how delivering apples leads to people eating apples or how

eating apples improves health.

A plausible explanation would be that delivering apples increases the

availability of fresh fruit, which leads to the apples being eaten, which

increases the amount of vitamin C in the diet, which improves the physi-

cal health of participants. This is only one possible explanation, of course.

Figure 1.3 shows this explanation as both a pipeline logic model and an

outcomes chain.

The diagrams in Figure 1.3 represent a program theory that articulates

the causal mechanisms involved in producing the two changes (changed

behavior and changed health status). The fi rst change relates to participants’

willingness to act in the way the program intended and the second to the

impacts of their actions. For many programs, it can be helpful to articulate

both types of changes in the program theory.

Pipeline model version

Outcomes chain version

INPUTS Apples

People in poor health

PROCESSES Apples

delivered

OUTPUT Improved access to fresh fruit

Apples eaten

SHORT-TERM OUTCOMES

Improved level of

vitamin C

OUTCOMES Improved

health

Improved access to fresh fruit

Apples eaten

Improved level of

vitamin C

Improved health

Figure 1.3 A Logic Model Showing a Simple Program Theory for An Apple a Day

Based on Improved Vitamin Intake

CH001.indd 7CH001.indd 7 1/5/11 12:49:10 PM1/5/11 12:49:10 PM

P u r p o s e f u l P r o g r a m T h e o r y8

Learning from Failure

An evaluation based on this program theory would collect data about

changes in access to fresh fruit, apple eating behavior, and nutritional status,

as well as overall health. If the intended outcomes have not been achieved,

we could work through the causal chain to see where it has broken down. If

the apples were not even delivered, there is obvious implementation failure;

if they were delivered but not eaten, then our theory of how to engage peo-

ple in changing their behavior seems not to work. Similarly, if the expected

health benefi ts had not been achieved, we would start by seeing if it was

because the apples had not been eaten. If the apples had been delivered and

had been eaten but without producing health improvements, then we have

a problem with the theory of change that underpins the program. Based on

these results, one option would be to reject the theory and look at other ways

of improving health. Another would be to look at dosage: maybe vitamin C

levels increased, but not enough to make a difference.

Learning from Partial Success

Developing a program theory also helps clarify differential effects, learning

from those participants for whom the program was effective. The simple

program theory is based on the assumption that the apples are both neces-

sary and suffi cient—that is, the apples will lead to good health in all circum-

stances and without contributions from other factors. Developing a more

complicated logic model would focus on the differential effects we might

expect for different types of participants, and we would collect and analyze

data to examine these. Disaggregating the data would investigate whether the

theory works in some contexts but not in others.

This review might show that the program works only for certain types

of participants—for example, those who are affected by diseases related to

inadequate nutrition. For people affected by infectious diseases, apples by

themselves might not be enough to improve health. Based on these results, we

might target the program to people most likely to benefi t: those with nutrition-

related diseases. Given the importance of the interaction between the interven-

tion and the characteristics of clients, it would be helpful to revise the theory of

change and its logic model to show this complicated causal path.

CH001.indd 8CH001.indd 8 1/5/11 12:49:11 PM1/5/11 12:49:11 PM

T h e E s s e n c e o f P r o g r a m T h e o r y 9

If the program works for some groups but not for others or at some

sites but not others, it is important to try to understand why by identifying

possible explanations and then checking these out empirically. For example,

if the program worked for men but not for women, it might be because

of differences in labor force patterns which affected access to fresh fruit or to

differences in nutritional needs related to pregnancy. Finding exceptions

to the pattern (the men who did not improve and the women who did)

would provide more evidence to test these emerging program theories.

Learning from Success

Program theory has another benefi t when an evaluation fi nds that something

works: it helps in adapting the intervention to new situations. To be useful

for evidence-based policy and practice, a program theory evaluation needs to

identify the causal mechanism by which it works and determine whether this

is different for different people and in different implementation contexts.

To explore this use, imagine that the evaluation has found that the pro-

gram theory works: people are healthier when they eat an apple a day. Now

the job is to implement a new program based on this evidence. In this

case, the goal is not to understand failure but to understand success. Apples

might produce these effects through quite different theories of change, which

would lead us to quite different intervention theories and different program

activities to suit the context. We would immediately have many questions

about the statement. Does it work for everyone? Does it have to be a particu-

lar variety of apple (Granny Smiths? crab apples?), or does it apply to all vari-

eties? What if apples are not available? Can we substitute other fruit, or apple

juice, or vegetables? Would red onions work as well as red apples? An evalua-

tion without program theory would reveal only that it works, with no guid-

ance for how to translate the fi ndings to a particular situation. Without this

guidance, we can only blindly copy everything. With this guidance, we can

understand how we might adapt it and still achieve the intended results.

We previously sketched out a program theory with a theory of change of

providing a good source of vitamins in diets that are otherwise defi cient. To

test this out if we were implementing it would require data about people’s

nutritional status through either direct measures or relevant indicators so we

CH001.indd 9CH001.indd 9 1/5/11 12:49:11 PM1/5/11 12:49:11 PM

P u r p o s e f u l P r o g r a m T h e o r y1 0

could see if there was any change and also to identify the people we would

expect to get the most benefi t from the program. We would want to check

that they actually ate the apples. And we would want to rule out alternative

explanations by fi nding out if there had been other changes in their diets

that might have contributed to changes in their nutrition. If this is the case,

then other types of fruit are likely to be equally effective. In a country where

apples are hard to obtain or expensive, distribution or subsidization of local

fruit is likely to be an effective program, at least for people at risk of nutri-

tional defi ciency, if it is implemented well.

But maybe this is not how it works at all. Maybe it is not about the fl esh

or juice of the apples but their skin. The skin of apples contains a plant-based

chemical called quercetin. Some research studies have suggested quercetin

may help to prevent cancer, heart disease, and infl ammation of the prostate.

An evaluation would look at the intake of quercetin from various sources

and outcomes in terms of these specifi c diseases, focusing on outcomes for

people at risk of these diseases. If apples were not available, another source

of quercetin could be used. Red onions, a rich source of quercetin, might

be an effective substitute—an adaptation of the program that would not be

immediately obvious if we were thinking only about fruit.

Another possible explanation focuses on apples as a substitute for high-

calorie, low-nutrition snacks. Perhaps apples improve health by helping to

reduce obesity as people stop eating potato chips and doughnuts and choose

apples instead. An evaluation of this possibility would look at what people

were eating in addition to apples and whether there had been a decline in

their consumption of junk food. It also might measure short-term outcomes

such as body mass index (BMI) and percentage fat, which have been linked to

subsequent longer-term health outcomes. The evaluation would have to take

into account criticisms that have been made of BMI as an indicator and pre-

dictor of health. Making other low-calorie snacks such as carrots and celery

readily available might be equally effective. Figure 1.4 shows how these three

different change theories might plausibly explain why the policy works.

Other possible explanations, involving different theories of change,

would lead to different critical features in implementation that should be

ensured. For example, if health improvements came about through increased

CH001.indd 10CH001.indd 10 1/5/11 12:49:11 PM1/5/11 12:49:11 PM

T h e E s s e n c e o f P r o g r a m T h e o r y 1 1

fiber consumption, eating the whole apple, not just drinking the juice,

would be important. Once the plausible theories have been identifi ed, they

can be used to guide data collection and analysis of an evaluation. They can

also be used to synthesize data from previous evaluations and research (we

discuss this in Chapter Four).

Success in terms of achieving intended results might not mean success

in terms of the theory. Another possible pattern of results is that the health

outcomes have been achieved but not the intermediate results of changes

in vitamin C. This would suggest that something other than the interven-

tion had caused the health improvements or that a quite different theory of

change was operating that did not involve vitamin C. Results like this would

indicate theory failure despite success in terms of results.

INPUTS Apples

People in poor health

INTERMEDIATE RESULT Apples eaten

INTERMEDIATE RESULT Improved nutritional

status

FINAL RESULT Improved

health

INPUTS Apples

(could use oranges)

People with vitamin C

deficiencies

INTERMEDIATE RESULT Apples eaten

INTERMEDIATE RESULT

Adequate levels of vitamin C

FINAL RESULT Reduced incidence of scurvy

INPUTS Apples

(could use red onions)

People at risk of cancer,

heart disease

INTERMEDIATE RESULT Apples eaten

INTERMEDIATE RESULT

Increased levels of quercetin

FINAL RESULT Reduced incidence of heart disease, prostate

inflammation

INPUTS Apples

(could use carrot sticks) Obese and overweight

people

INTERMEDIATE RESULT Apples eaten

INTERMEDIATE RESULT

Decreased consumption of

junk food

FINAL RESULT Reduced incidence of obesity

and associated conditions

Figure 1.4 Logic Models Showing Different Possible Causal Mechanisms Involved

in Eating an Apple a Day

CH001.indd 11CH001.indd 11 1/5/11 12:49:11 PM1/5/11 12:49:11 PM

P u r p o s e f u l P r o g r a m T h e o r y1 2

Learning from “An Apple a Day”

Speculating on different possible causal mechanisms enables us to develop an

evaluation that will collect and analyze data to be able to understand to what

extent, for whom, and why an intervention does or does not work. (Chapter

Fourteen describes how to use program theory to guide evaluation design.)

Although a single evaluation is limited in its scope, program theory makes it

easier to combine evidence from a number of studies. Table 1.1 summarizes

how an evaluation informed by program theory can distinguish among dif-

ferent types of success and failure.

The apple a day example shows the importance of developing program

theory that identifi es the causal mechanism that is understood to be involved

in producing the intended outcomes. This can help to produce more useful

evaluations and better evidence for policy.

Table 1.1 Using Program Theory to Interpret Evaluation Findings

Apples Delivered

Apples Eaten

Vitamin C Levels Raised

Health Outcomes Improved Interpretation

✗ ✗ ✗ ✗ Implementation failure ✓ ✗ ✗ ✗ Engagement or adherence failure

(fi rst causal link) ✓ ✓ ✗ ✗ Theory failure (early causal link) ✓ ✓ ✓ ✗ Theory failure (later causal link) ✓ ✓ ✓ ✓ Consistent with theory ✓ ✓ ✓/✗ ✓/✗ Partial theory failure (works in some

contexts) ✓ ✓ ✗ ✓ Theory failure (different causal path)

CH001.indd 12CH001.indd 12 1/5/11 12:49:12 PM1/5/11 12:49:12 PM

T h e E s s e n c e o f P r o g r a m T h e o r y 1 3

SUMMARY

This chapter has used a hypothetical example to explore how articulating a

program theory—an explicit statement of how change will occur and how

an intervention will produce these causal processes—can make evaluations

more useful. Throughout the rest of the book, we use examples from actual

evaluations to show how to develop, represent, and use program theory

for evaluation and other purposes.

EXERCISES

1. If a social marketing campaign was used instead of direct delivery of

apples for the Apple a Day program, what would implementation

failure look like? What would theory failure look like? What would

partial theory failure look like, where it works only in particular con-

texts?

2. Consider a policy that aims to increase student performance by

increasing teachers’ salaries. What might be some alternative causal

mechanisms that would produce the intended outcomes?

CH001.indd 13CH001.indd 13 1/5/11 12:49:12 PM1/5/11 12:49:12 PM