Summer end 1

209

CHAPTER SEVEN

Working with Stakeholders

Establishing the Context and the Evaluand

You have already read about a wide variety of evaluands that refl ect many disciplines and issues, such as programs to provide youth mentoring, address homelessness and unemployment, provide effective mental health services, increase literacy skills, provide safe housing, improve schools, and prevent the spread of HIV/AIDS. An evaluand may seem pretty clear in the published version of an evaluation; however, this clarity generally comes from many hours of discussions and revisions during the evaluation planning and implementation phases. The evaluations discussed in earlier chapters have also been con- ducted in a wide variety of contexts and countries across the globe, with diverse cultural groups who use different languages and live in different socioeconomic conditions. These contextual factors infl uence what is chosen to be evaluated and how that determination is made.

Evaluation planning can begin in many different ways: a phone call from a person previously unknown to you who says, “I have a program that needs to be evaluated”; an email from someone who is preparing a proposal to develop a new program that needs an evaluation plan; or a request to expand on previous evaluation work with members of a community with whom you have an ongoing relationship. What these beginning points have in common is that you, as the evaluator, are interacting with another person or per- sons. Hence issues of human relations are inevitably part of the process of planning an evaluation. A second important point to note is that evaluands come in all stages of being implemented—from existing only as an idea in a principal investigator’s head, to a fi rmly established program or one that is undergoing changes, to a more dynamic organization that wants to be in a mode of continuous learning.

Identifying Stakeholders

Once the initial contact has been made between a client and an evaluator, both parties need to consider who needs to be involved in the process of planning the evaluation. As defi ned in Chapter 1, stakeholders are people who have a stake in the program: They fund, administer, provide services, receive services, or are denied access to services. It is usually wise to spend some time and effort thinking about which stakeholders need to be included at the very beginning; this can help avoid political disasters at the end of evalu- ations if the proper people were not involved. On a more positive note, the quality of the evaluation will be enhanced with representation of diverse interests, especially by inclu-

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

210 PLANNING EVALUATIONS

sion of traditionally marginalized groups. Appropriate stakeholders are sometimes identi- fi ed by default, including only those who have power in positions related to the evaluation. The selection of stakeholders can also be an evolving process, with some stakeholders identifi ed early in the process and others added as the relevant issues become clarifi ed. In relatively small projects, the identifi cation of stakeholders may be fairly straightforward. However, in larger projects, strategies for selection of representatives from stakeholder groups will probably need to be employed.

Identifi cation of stakeholders is context- specifi c. Two lists of categories of stakehold- ers are displayed in Box 7.1; these lists will give you an idea of how many and what types of diverse groups can be considered in identifying stakeholders. The fi rst list is based on a study of projects specifi cally focused on substance abuse prevention (Center for Substance Abuse Prevention [CSAP], 2008). The second list of stakeholders is based on the UN- Women’s (2014) Guide for the Evaluation of Programmes and Projects with a Gender, Human Rights and Intercultural Perspective, which details how evaluations should incorporate prin- ciples of gender equality, women’s rights, and the empowerment of women in all initiatives they support and fund. Box 7.1 lists the four groups of stakeholders UN-Women and all UN agencies identify and include throughout all evaluation processes.

Box 7.1. Two Samples of Stakeholders for Evaluations, Listed by Category

Substance abuse prevention (based on CSAP, 2008)

Integration of gender in policy for poverty reduction strategies (based on UN-Women, 2014)

Law enforcement

Education

Youth

Criminal justice

Civic organizations

Parents

Faith-based organizations

Elderly persons

Businesses

Human service providers

Health care providers

Military

Colleges and universities

Ethnic groups

Government

Elected offi cials

Child care providers

Various ministry offi cials, such as fi nance, economic planning, and others (health, education, trade, industry, labor, social development, natural resources, and environment)

Elected offi cials

Civil society (e.g., NGOs, community-based organizations, faith- based groups, trade unions, private sector associations), with specifi c attention given to relevant dimensions of diversity within these groups (e.g., rural–urban, disability groups, women’s groups)

World Bank staff involved in poverty reduction planning, especially those responsible for the World Bank Joint Staff Assessments/Joint Staff Advisory Notes, because they assess the quality of poverty reduction plans and make their recommendations for funding or debt reduction to the World Bank and International Monetary Fund

International agencies, such as United Nations agencies and international donor agencies (e.g., CARE, Oxfam, Save the Children, ActionAid)

Representatives from the sectoral groups that represent infrastructure, agriculture, education, health, and employment

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 211

Broad categories that are contextually relevant can be helpful in identifying stake- holders for specifi c evaluation studies. Evaluators can determine which stakeholder groups have relevance by recalling their own experiences in particular contexts, reading literature related to the particular context, conferring with knowledgeable members of the community, and asking for specifi c recommendations to represent diverse viewpoints. Evaluators should be aware of the need to include stakeholders who represent diverse perspectives and positions of power. They should also be aware of the need to provide support for those stakeholders who require it for authentic participation. This support might take the form of transportation, stipends, a safe meeting environment, interpreters, food, or child care. Evaluators working with stakeholders must pay careful attention to their interpersonal skills, because human relations are critical in conducting high- quality evaluations, as discussed in the next section.

· · · · · · · · · · · · E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·

Identifying Stakeholders

1. Machik is an NGO that is building new opportunities for education and training with Tibetans living in a small, isolated village in a deep valley. With support from donors, they have opened the Ruth Walter Chungba Primary School in this rural community. Imagine that Machik has asked you to evaluate the impact the school has made on the community. You need to decide with the school authorities and the donors who the stakeholders are in this community. Who would you ask to participate in this study, and why? (Read about the school and watch a video at this website: www.machik.org/index.php?option=com_content&task= view&id=24&It emid=50.)

2. You have been hired by a school system to evaluate a new reading program for use in elementary schools. How would you begin your identifi cation of appropri- ate stakeholders for this evaluation?

Human Relations

The nature of the relationship between the evaluator and stakeholders is an area of ten- sion in the evaluation community, as exemplifi ed by the different paradigmatic perspec- tives on this topic:

 Methods Branch evaluators tend to favor having a distant relationship, in the belief that this will protect the evaluator from developing biases toward particular stake- holder groups.

 Use Branch evaluators see the necessity of forming a relationship with the stakehold- ers who are the primary intended users, so the evaluator can be responsive to their needs and thus enhance the possible use of the fi ndings.

 Values Branch evaluators believe that the evaluator needs to be involved with the

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

212 PLANNING EVALUATIONS

community suffi ciently to reveal the viewpoints of different stakeholder groups accu- rately.

 The Social Justice Branch evaluators directly address differences in power between themselves and various stakeholder groups, with a conscious awareness of the need to include the full range of stakeholders, especially those who have traditionally been excluded from decision- making positions into the process.

These differences in the nature of evaluator– stakeholder relationships lead to differences in the processes used to defi ne the evaluand and understand its context.

· · · · · · · · · · · · E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·

Human Relations Skills for Evaluators

Two eminent scholars in the evaluation community see the importance of human relations very differently. Read the two following passages and discuss your own thoughts and positioning with regard to this issue. First, Patton (as a contributor to Donaldson, Patton, Fetterman, & Scriven, 2010) writes:

Human beings are in a relationship to each other and that relationship includes both cognitive and emotional dynamics. The interpersonal relationship between the evalua- tor and intended users matters and affects use. That interpersonal relationship is not just intellectual. It is also political, psychological, emotional, and affected by status and self- interest on all sides. What the astute evaluator has to be able to do, which includes the essential competencies to do that, is to be able to engage in relationships. (p. 25)

In contrast, Scriven (also as a contributor to Donaldson et al., 2010) writes that inter- personal skills are not necessarily important for evaluators:

Michael [Patton] fi nds one of these to be a great strength, namely having lots of inter- personal skills. Forget it, guys! The way that evaluation works, and always will, is that it inhabits ninety niches. One of those niches is to be found in Washington in every agency, e.g., in the offi ce of its inspector- general. Here are to be found the desk evalua- tors. Most of them don’t have to have interpersonal skills any more than anyone in any kind of offi ce job; and they don’t need them. All they’re doing is analyzing the reports, and they’re very important people because they’re the fi rst line of advice and back-up to the decision makers. What we need from them is good analytic skills. It’s not that I don’t think that it’s a good thing to have good interpersonal skills; it is that one must not put them in as minimum requirements for every evaluator. (p. 24)

Now answer the following questions:

1. What do you think about these two positions?

2. What merits do their arguments have?

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 213

3. Do you personally agree with one more than the other?

4. What are your reasons for your own positioning on the topic of human relations skills in evaluation?

Interacting with Stakeholders

Kirkhart (2005) has noted that the validity of an evaluation is infl uenced by “interper- sonal justifi cation” (i.e., the quality of the interactions between and among participants and the evaluator). Evaluators bring their own cultural lenses to the planning process, and these affect their interactions with stakeholders in terms of who is involved in the process and how. Lincoln (1995) has reinforced the importance of the quality of human relations in evaluation by suggesting that an evaluator needs to know the community “well enough” to link the evaluation results to positive action within the community. Evaluators must critically examine the meaning of “well enough”; what does this mean? Indigenous researchers provide insights into the nature of relationships that they would interpret as indicating that an evaluator is appropriately situated to work in their communities.

Lessons from the Maori

Cram (2009) and Smith (2012), who work in the Indigenous Maori community in New Zealand (Aotearoa), have provided guidance to the meaning of kaupapa Maori (which means “a Maori way”). Kaupapa Maori can be applied to many aspects of life; it implies the development of a relationship that is respectful of Maori cultural, social, and economic well-being. Cram (2009) provides a list of cultural values that she translates into expecta- tions for evaluators’ interactions in their community. These include the following:

 Aroha ki e tangata (respect for people). Evaluators establish relationships with peo- ple via situating themselves within the history of the community (genealogically, if possible; through personal connections if no genealogical link is present), with the assistance of the community elders. Another aspect of respect for people is to be knowledgeable about appropriate rituals in terms of entering the community (such as who to contact, how to approach people, bringing of gifts, etc.).

 He kanohi kitea (a voice may be heard, but a voice must be seen). Maori people expect that an evaluator will come into their community to allow the community members to see for themselves who this person is. Community meetings, called hui, are often used as a forum for evaluators to meet stakeholders, explain the study, and ask permission to proceed.

 Titro, whakarongo . . . korero (watch, listen . . . talk). An evaluator shows respect for Maori people by listening to what they say before he/she talks. This process of fi rst looking and listening conveys the value that the evaluator places on the contribu- tions of the community members.

 Manaaki kit e tangata (looking after people). In the context of the evaluation, the essential meaning of this concept is that the evaluator establishes a reciprocal rela-

aori

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

214 PLANNING EVALUATIONS

tionship with the stakeholders. The stakeholders are providing access to their com- munity and information in the form of data; the evaluator can offer small gifts or services, capacity- building activities, networking, and access to the evaluation fi ndings.

 Kaua e takahia te mana o te tangata (do not trample on the mana [authority] of the people). Maori people want to know what an evaluator is saying about them before the results are released outside the community. As most communities would, the Maori do not want to be portrayed as having something wrong with them (a defi - cit view). Rather, they want to be portrayed in a balanced way, with both their strengths and their challenges.

 Kia mahaki (be humble). An evaluator should share the results with the Maori com- munity in a way that helps the community take action on its own behalf. The com- munity members can be provided with the tools necessary to fi ght for their own rights and challenge oppressive systems.

· · · · · · · · · · · · E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·

Maori Cultural Values and Evaluation

1. Reciprocity is seen as valuable in evaluations conducted in the Maori community. How would this principle translate to evaluation situations outside the Maori community?

2. What is your opinion with regard to the implications of applying these Maori cultural values in other evaluation contexts?

3. What could evaluators learn about the establishment of relationships with stake- holders from these Maori cultural values?

4. What might some evaluators fi nd objectionable concerning the Maori’s expecta- tions of the evaluators’ interactions in their community? Why would they object?

5. What do you know about yourself that might enhance or inhibit your ability to work in an evaluation context that requires attention to and respect for cultural values and backgrounds?

6. Symonette (2004) suggests that evaluators need to be aware of who they are them- selves, as well as who they are in relation to community:

Even more important for the viability, vitality, productivity and trust- building capacity of a transaction and relationship cultivation is multilateral self- awareness: self in context and self as pivotal instrument. Who do those that one is seeking to communicate with and engage perceive the evaluator as being? . . . Regardless of the truth value of such perceptions, they still rule until authentically engaged in ways that speak into the listen- ing. (p. 100)

How would you answer this question: Who do others think that you are? If you are in an evaluator role, who do others think you are?

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 215

(cont.)

Power and Privilege

Power and privilege are concepts discussed in prior chapters. Here the emphasis is on (1) strategies for evaluators to use to bring themselves and the communities with which they work to consciousness about the dynamics of power and privilege, as well as on (2) mean- ingful ways to engage those who have traditionally had less power in evaluation contexts. Two action researchers, Heron and Reason (2006), provide the following strategies for designing evaluations that include self- refl ection and that monitor evaluators’ engage- ment with communities in culturally respectful ways:

 Research cycling. Evaluators should be prepared to go through the inquiry process several times. This cycling process allows for repeated episodes of action and refl ec- tion that can help refi ne understandings and reduce distortions.

 Authentic collaboration. Evaluators and stakeholders need to devise strategies for interactions that allow for the development of an egalitarian relationship. The interaction dynamic needs to be designed so that stakeholders are motivated to have sustained involvement and allow every voice to be expressed.

 Challenging consensus collusion. Individuals have the right to challenge the assump- tions that underlie the knowledge created or the process by which it was created.

 Managing distress. Group processes typically have moments of stress and tension; a process needs to be in place to handle this distress respectfully.

 Refl ection and action. A cyclical process that includes phases of action and refl ection allows needed changes to occur.

 Chaos and order. Refl ective action is diffi cult when a system is in total chaos. Evalu- ators should encourage divergent thinking and also bring the system back into balance so that the group can move forward toward its goals.

· · · · · · · · · · · · E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·

Power and Privilege

1. How do we understand the dynamics of power when participatory methods are employed by the powerful?

2. Whose voices are raised, and whose are heard?

3. How are these voices mediated as issues of representation become more complex with the use of participatory methods in larger-scale planning and consultation exercises?

4. The culturally responsive approach to evaluation places emphasis on matching the characteristics of the evaluation team with those of the community, particu- larly in terms of race. Frierson et al. (2002) suggest that data will not be valid if they are collected by people who are not attuned to the program’s cultural con- text. What if you are a member of the community? How does that prepare you to

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

216 PLANNING EVALUATIONS

work in that community? What if you are not a member of a community? To what extent is it necessary to share salient characteristics of a community?

5. Recall the discussion of cultural competence in Chapter 1. How does cultural competence come into the discussion of interactions in evaluation contexts?

6. When evaluators enter a community, they may fi nd that they hold power in a way they have not before. For example, an elderly female evaluator may be more respected in this community than in her home culture. List situations where you must be cognizant of the increased or decreased power you hold as a result of personal characteristics that may affect your relationship with the stakeholders (age, gender, education, ethnicity, sexual orientation, etc.).

Developing Partnerships/Relationships

A large community of immigrants and refugees settled outside Lowell, Massachusetts, in a relocation effort for people from Laos who had assisted the United States in the years preceding the Vietnam War. When the United States lost the war, the government fol- lowed through on its promise to move members of the Laotian community who had been their allies to a safe place. The presence of such a large community in what had previously been a very white, working-class, mainstream American community did not go unnoticed by researchers. Researchers motivated by a desire to create knowledge, to work with an exotic community, or even simply to do good inundated the community with their study teams. Silka (2005) and her colleagues at the Center for Family, Work and Community at the University of Massachusetts, Lowell, noted that the immigrants and refugees were not benefi ting from the research. They developed a model for partnership research and evaluation between a consortium of universities and the Laotian community, in order to protect the community from exploitative research that did not directly benefi t the commu- nity. Silka and her colleagues have developed a set of tip sheets to guide researchers and evaluators who conduct studies in the Laotian community; several of these tip sheets are summarized in Box 7.2. They have wider applicability in the development of partnerships with other communities as well.

Box 7.2. Developing Ethical Partnerships: Tip Sheet Summaries

 Initiating Partnerships: Gathering the Players, by Darcie Boyer. This is the initial step in the process of acting on a felt need, identifying others who share a concern in the community and in the research or evaluation world, fi nding appropriate ways to con- tact and communicate with potential partners, and planning to have a community meeting to discuss the potential partnership.

 Ethical Considerations in Participatory Research: The Researcher’s Point of View, by Maryjane Costello. Researchers need to be aware of the diversity of perceptions as to what constitutes ethi- cal practice in various communities.

 Partnership-Based Research: How the Community Balances Power within a Research Partnership, by

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 217

G. Martin Sirait. Partnerships should be arranged

so that both researchers and participants are rec-

ognized as having power in that context.

 Everything You Always Wanted to Know about IRBs, by Sokmeakara Chiev. IRBs, or institutional review

boards, are mandated by U.S. federal legislation for

any organization that receives federal funds to do

research. Communities can institute IRBs of their

own with membership from within their cultural

group.

 Overcoming the Roadblocks to Partnership, by Marie Martinelli. Communities can ensure that they derive benefi ts from proposed research or evaluation by forming community advisory boards, actively participating in the planning process, and considering successful models of partnerships that might transfer to their own situation.

 Knowledge Creation in Research Partnerships, by Pascal Garbani. Researchers need to work together to create knowledge in a manner that respects dif- ferences between and within groups.

Source: Based on Center for Family, Work and Community at the University of Massachusetts, Lowell (2004). The Center for Community Research and Engagement’s home page is www.uml.edu/research/ccre.

Many Indigenous peoples prefer to speak of “relationships” rather than “partner- ships.” For example, Maori, Native Americans, and Africans share an emphasis on con- nectivity and extend it beyond relationships among human beings to include the wider environment, ancestors, and inanimate objects. For them, “partnership” implies more of a contractual relationship that may still refl ect inequities and exploitation. “Relationship” means that there is a deeper connection at multiple levels in terms of where we are from and who our people are. It means that the evaluators understand the culturally appropri- ate ways of a community and see the evaluation as a journey that they take together with community members, with opportunities for mutual learning, participant control, and evaluator accountability (Cram, 2009).

Partnerships or relationships are not easy to develop and may not be smooth through- out their existence. Kirkhart (2005) suggests the following considerations that are related to effective partnerships and relationships. First, relationships in evaluation take time and effort to develop. Evaluators often work in compressed time frames with limited budgets that constrain their ability to be responsive to multicultural dimensions. Second, cultural responsiveness requires knowledge, emotions, and skills. These are complex and not easily taught. Third, evaluators need to be able to interact with the stakeholders in the evalu- ation in ways that are culturally respectful, cognizant of the strength in the community, and facilitate desired change. This means that they need to be fl exible with the design and implementation of the evaluation in order to be responsive to these factors. Finally, evaluators, particularly if they are from outside the community, need to avoid cultural arrogance in several forms: imposing their own cultural beliefs on the stakeholders, pre- imposing a design on the evaluation, or mistakenly thinking that they accurately under- stand the culture in which they are working.

Evaluators can also work with community members on capacity building. The capac- ity building can be reciprocal, in that the evaluators have knowledge and skills to teach from their perspective, and the community members have knowledge, skills, and attitudes to teach from theirs. Teams of evaluators can be formed that allow strengths from all

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

218 PLANNING EVALUATIONS

sides to be represented in the evaluation planning. Caldwell et al. (2005) describe effective evaluator teams formed with academic and tribal representatives. They do point out that one challenge with this approach arises from concerns about confi dentiality and anonym- ity, especially in small communities where identities can be recognized readily.

· · · · · · · · · · · · E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·

Developing Partnerships

Think about the evaluation you intend to plan.

1. At what point will you involve the community?

2. How will you prepare yourself for meeting the community (by reading about the culture, etc.)?

3. How will you approach that community?

4. What benefi ts do you see for the community?

5. How will you demonstrate your respect for its culture and traditions?

The Evaluand and Its Context

The theme of AEA’s annual meeting in 2009 was “Context and Evaluation.” Debra J. Rog, the 2009 president of AEA, defi ned context in these terms:

Context typically refers to the setting (time and place) and broader environment in which the focus of the evaluation (evaluand) is located. Context also can refer to the historical con- text of the problem or phenomenon that the program or policy targets as well as the policy and decision- making context enveloping the evaluation. Context has multiple layers and is dynamic, changing over time. (Rog, 2009, p. 1)

The contrast in terms of how evaluators from different branches view context was cap- tured in the opening plenary session of the 2009 AEA meeting. Bickman (2009), a theo- rist from the Methods Branch, said that context was always something that he called “extraneous variables”—in other words, variables that were not of central concern but had to be controlled, so that the validity of the intervention could be determined apart from contextual factors. His perspective contrasted sharply with that of Bledsoe (2009), who is situated in the Social Justice Branch. She indicated that understanding the context was critical to understanding the experiences of the less powerful in the evaluations that she conducted, in order to challenge assumptions by the more powerful. With those two anchor points, we now explore several types of contextual variables and the implications of these variables for the identifi cation of the evaluand and the methods used in the evalu- ation.

Contextual variables include those associated with the local setting (time and place),

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 219

as well as with the broader context—the history of the problem and its proposed solutions, as well as politics and legislation that have relevance for the evaluand. The range of stake- holders and their cultural differences are also contextual variables that need to be con- sidered. These contextual variables infl uence who is involved (stakeholders), how they are involved, the evaluation questions, the type of evaluation undertaken, use of evaluation fi ndings, and decisions about analysis and dissemination of results. The following ques- tions can help stimulate your thinking about contextual variables and their implications:

 What dimensions of context infl uence the type of evaluation questions that can be addressed?

 How does the nature of the political context infl uence utilization? How does it interact with the type of evaluation conducted?

 What dimensions of context infl uence the choice of methods?

 How does culture within context affect evaluation practice?

 How do our evaluation theories guide us in thinking about context?

 How can we learn about context in multisite studies?

 What are the implications of a context- sensitive evaluation for analysis and dis- semination?

 How can we incorporate context into our evaluation inquiries?

Here is an example from the Hawaiian housing study (Stuffl ebeam et al., 2002; see Chapter 4, Box 4.3) of the identifi cation of contextual variables. The local setting for the housing project was on Oahu’s Waianae Coast, one of the most depressed and crime- ridden areas in the state. The project stretched over 7 years. The funding agency placed high value on self-help and sustainability; this value system infl u- enced the design of the program as well as the evaluation. Contextual variables of particu- lar importance centered on the characteristics of the intended benefi ciaries: specifi cally, the extent of their needs and their abilities to follow through on the expectations for helping to build and pay for their houses. These contextual variables infl uenced who was fi nally accepted as the target audience and how local people were used in the role of data collectors. As noted in Box 4.3, the original intent of the program was to serve the poorest families. However, these families could not get the mortgages, so the focus of the project was shifted to the working poor.

· · · · · · · · · · · · E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·

Questions about Context

Refl ect on the excerpt of Rog’s (2009) explanation of context and the discussion of contextual variables in this section. Now return to the sample studies summarized in boxes in Chapters 3–6. Use the questions listed earlier in this section to analyze rel- evant contextual variables in at least one sample study. Think about how the authors either considered or did not consider these contextual variables.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

220 PLANNING EVALUATIONS

Sources That Inform the Identifi cation of the Evaluand and Context

Developing a focused identifi cation of the context and the evaluand can be approached through a number of different strategies:

 Funding agencies establish priorities and provide information in requests for proposals (RFPs) about the context and the program that needs to be evaluated. Another version of a funding agency request is a request for a program to be devel- oped with the requirement for an evaluation plan in the proposal.

 Traditional scholarly literature reviews can provide valuable information about the context and the evaluand in terms of what is already known about the setting and the program. This type of resource is generally found through databases of articles available in university and sometimes community libraries, or online for a fee.

 Theoretical frameworks for evaluation approaches can provide guidance regard- ing the variables that are important (e.g., an Indigenous evaluation will emphasize specifi cs of the targeted culture), as well as a basis for decisions about appropriate components of a program. Theoretical frameworks can inform the evaluator and stakeholders about power differences on the basis of race/ethnicity, gender, sexual identities, disabilities/deafness, religion, class/socioeconomic status, and other characteristics associated with discrimination and oppression.

 Web-based resources are now available (sometimes overwhelmingly!). Here, an evaluator can read about past evaluations, recommended evaluation strategies for this type of evaluand, and relevant contextual factors. Web-based resources can also include databases such as those posted by the U.S. Census Bureau (2017), the Central Intelligence Agency (CIA) and their World Factbook (CIA Factbook, 2017), the U.S. Department of Education’s (2017) evaluation reports, and USAID’s Devel- opment Experience Clearinghouse (2017) evaluations.

 “Grey literature” (i.e., that which is not published) can be a valuable resource, espe- cially to gain the perspectives of those who have not been in the privileged schol- arly or technological circles that would be represented in the fi rst several strate- gies. This literature can include program- produced documents such as brochures, project reports, self- studies, past evaluations, conference papers, policy statements, newsletters, newspapers, fact sheets, and more.

 Group and individual strategies can be used, such as interviews, surveys, focus groups, concept mapping, and outcome mapping, as well as Indigenous methods based on traditional community meeting ceremonies and rituals.

 Advisory boards are commonly used to guide evaluators throughout the process of planning and implementing an evaluation.

 New technological tools such as satellite imagery and mapping can be used to provide valuable contextual information about the locations of roads, buildings, services, and natural terrain.

We discuss all of these strategies in more detail below.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 221

(cont.)

Funding Agencies

Funding agencies typically include government agencies and foundations. The U.S. gov- ernment has a website that lists opportunities to apply for more than $400 billion in fed- eral monies from over 1,000 different programs (www.grants.gov). In addition, many agen- cies offer their own funding opportunities on their websites (e.g., the U.S. Department of Education). Obtaining funds from federal agencies usually brings a fairly prescriptive set of requirements for how the funds can be used. On the other hand, foundations also offer many potential funding opportunities through a web portal (http://foundationcenter.org/ fi nd- funding); larger foundations offer such opportunities at their own websites. Founda- tions tend to have priority interest areas, but they are generally more fl exible than govern- ment granting agencies. Box 7.3 provides contrasting statements from a federal agency’s and a foundation’s RFPs.

Box 7.3. Government and Foundation RFPs

The U.S. Department of Justice (2009) offers funding for a tribal youth program that includes the following pro- gram requirements:

[The Offi ce of Juvenile Justice and Delinquency Prevention] seeks applicants to establish or expand a mentoring program that offers a mixture of core services and engages youth with activities that enable them to practice healthy behaviors within a positive pro- social peer group. The target population should be youth at risk of gang activity, delinquency, and youth violence.

The goals of this mentoring program are to prevent gang activity, delinquency, and violence by doing the following:

(1) Offering at-risk youth core services that fulfi ll their adolescent developmental needs within the context of a positive pro- social peer group, including:

 A multi-modal mixture of services that may include, but is not limited to, life skills and psycho- educational training, mental health counseling, job placement, community service projects, and structured afterschool recreational, educational, and artistic/culturally enhancing activities.

 Emphasizing long-term relationships with mentors and key staff, who are nurturing and supportive adults.

(2) Developing structured mentoring relationships that include the following:

 A relationship that lasts 2 or more years with signifi cant contact between the mentor and mentee where the mentee views the mentor as a friend, not an authority fi gure.

 Signifi cant training for the mentor.

 Oversight of the mentoring relationship.

 Data collection to track the relationship and positive outcomes arising from the mentoring relationship.

 Structured activities for the mentors and mentees to participate in together.

The Ford Foundation (2010) also supports grantees to develop and implement projects for youth mentoring, but it does not have explicit requirements about the nature of the program. Rather, it has issued this broad statement:

We make grants to develop new ideas and strengthen organizations that reduce poverty and injustice and pro- mote democratic values, international cooperation and human achievement. To achieve these goals, we take varied approaches to our work, including supporting emerging leaders; working with social justice movements and net-

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

222 PLANNING EVALUATIONS

Box 7.3 (cont.)

works; sponsoring research and dialogue; creating new organizations; and supporting innovations that improve lives. These methods of problem- solving refl ect our values and the diverse ways in which we support grantees.

The foundation also describes a model of philanthropy that it has pursued for more than 70 years: to be a long- term and fl exible partner for innovative leaders of thought and action. Lasting change in diffi cult areas, such as the reduction of poverty, protection of human rights, and establishment of democratic governance after a dictatorship, requires decades of effort. It involves sustained work with successive generations of innovators, thinkers, and activists as they pursue transformational and ambitious goals.

Cheek (cited in Mertens, 2009, p. 112) offers the following cautionary questions to consider before accepting money from a funding agency:

 Who owns the data and what can you do with the data?

 What if the funder wants to suppress results of the study? Or wants to exclude parts of the results?

 What exactly is the deliverable (e.g., product expected by the funder)?

 In what time frame?

 Reporting requirements?

 What if there is a disagreement about the way the research or evaluation should proceed?

Scholarly Literature

Many funding agencies require a scholarly review of literature on the evaluation topic in order to provide evidence of knowledge in the fi eld, of the need for the proposed project, and directions to inform the proposed scope of work. Searching databases is very easy for evaluators in the developed world, especially those who work in universities. A list of commonly used databases is provided in Box 7.4. These are generally searchable for free at universities and for a modest fee for people in other settings. Most of these databases can be searched by topic, author, or title. Many databases now have full text documents electronically available to users, eliminating the need to actually visit the library to obtain the documents.

Box 7.4. Scholarly Databases

Psychology

The American Psychological Association (APA) pro- duces the following databases:

 PsycARTICLES. This database contains full text articles from 42 journals published by APA and related organizations. The dates of coverage

vary; the earliest articles are from 1988, but APA is developing PsycARCHIVES, which has over 100 years of content coverage.

 PsycINFO. This database indexes and abstracts over 1,300 journals, books, and book chapters in psychology and related disciplines (1887–pres- ent).

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 223

 PsycBOOKS. Textbooks published by APA and selected classic books from other publishers are found in this database.

Social Science

 Social Science Journals (ProQuest). Social sci- ence journal articles published from 1994 to the present.

 Sociological Abstracts. This is an online resource for researchers, professionals, and students in sociology and related disciplines. Sociologi- cal Abstracts includes citations and abstracts from over 2,000 journals, plus relevant disser- tation listings, abstracts of conference papers and selected books, citations of book reviews and other media, and citations and abstracts from Social Planning/Policy and Development Abstracts.

 Social Work Abstracts. Index to articles from social work and other related journals on topics such as homelessness, AIDS, child and family

welfare, aging, substance abuse, legislation, community organization, and more.

Education

 Education Database (ProQuest). Indexes more than 750 titles on education, including primary-, secondary-, and university-level topics. Almost 500 titles include full text.

 Educational Resources Information Center (ERIC). A bibliographic database covering the U.S. literature on education; a key source for researchers, teachers, policy makers, librarians, journalists, students, parents, and the general public. Accessible to the public at www.eric. ed.gov.

Dissertations and Theses

 ProQuest Dissertations and Theses. An index of dissertations and theses published in the United States and internationally.

Lawless and Pellegrino (2007) describe an evaluation they were planning to deter- mine how to prepare teachers to use technology in their classrooms to enhance learning. They began with a very extensive literature review, which focused on “what is known and unknown about professional development to support the integration of technology into teaching and learning. To answer such questions, we have assembled bodies of literature that are relevant to the design of research studies, the evaluation of the quality of the evi- dence obtained therein, and the possible utility of conclusions” (p. 577). To this end, they examined a multipart literature: what constitutes professional development, how technol- ogy is integrated into the classroom, what infl uences teachers to adopt technology, the multiple roles that technology can play in this context, the quality of previous research on this topic, and the long-term impacts technology has had on teachers and administrators. They used this literature review to “lay out the kinds of questions that should be asked in evaluating how states, districts, and schools have invested their technology integration funds and the nature of the research designs and sources of evidence that might be used to better answer questions about what is effective and why” (p. 578).

In an evaluation of the sustainability of health projects, Scheirer (2005) provides this description of her literature search strategy:

The search was conducted using the search string “sustainability OR routinization OR insti- tutionalization AND health OR healthcare,” in all major relevant bibliographic databases, for the years 1990 to 2003, including PubMed, ProQuest, the Librarians Index to the Internet,

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

224 PLANNING EVALUATIONS

and NLM Gateway. The abstracts of potentially relevant citations were examined to determine if the original research included data collected about any aspect of sustainability after the initial funding had ended. Full texts of all relevant articles were then obtained. A few studies were already known to me from prior related work. In addition, reference lists of obtained articles were examined for any additional studies, such as those using different terminology. The systematic review did not include articles or how-to-do-it commentaries about sustain- ability that did not report empirical data, although these articles were consulted for their con- ceptual frameworks and approaches. These procedures yielded 19 studies that met the criteria for inclusion: reporting data collected about the status and/or infl uences on health program sustainability (including case studies). The review included all available studies that met these criteria, not a sample of them. (p. 327)

The use of scholarly literature is a critical part of enhancing our understanding of the context in which the evaluation is taking place. However, it is limited by the fact that various gatekeepers decide what will be published and what will be archived in a database. Therefore, evaluators should be cognizant of this limitation and engage in other types of search strategies to identify important contextual variables.

Theoretical Frameworks

The theorists whose work is described in Chapters 3–6 provide evaluators with a multitude of theoretical frameworks from which to choose in their planning work. These theories can range from theories of literacy development to theories of community involvement. Theories provide a framework for thinking, highlight relevant concepts, and suggest dynamic relationships between those concepts. Here are some examples of evaluations that used theoretical concepts:

 Bowman’s (2005) evaluation of a tribal education model in a technical college in Wisconsin was based on an Indigenous theory from the Native American com- munity. The geographic coverage area of the technical college included members of three tribes. The evaluators sought out each tribe’s individual customs, culture, language, and epistemological views based on their tribal traditions.

 Donaldson and Gooler (2002) conducted a theory-based evaluation of a job search training program in California. The underlying theory of the program was based on identifying the skills and psychological factors that were necessary for the par- ticipants to fi nd employment and improve their mental health. The theory held that the participants needed to increase their job search confi dence, their job search skills, and their problem- solving strategies in order to achieve the intended outcomes.

 Campbell et al.’s (2014) study of the effectiveness of an intervention to sup- port victims of sexual assault (see Chapter 6, Box 6.9) used a feminist theo- retical framework, which focused on power differentials in the planning, implementation, and use of the evaluation.

 Brady and O’Regan (2009) used Rhodes’s model of mentoring as a theoreti- cal framework for their youth mentoring evaluand. This model is presented graphically in Chapter 3, Box 3.3.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 225

Web-Based Resources

The proliferation of web-based resources sometimes makes me wonder what we would do if we didn’t have the World Wide Web anymore. This is probably unimaginable to many people younger than I am, and I admit that life would be a lot harder for me if it hap- pened. The major search engines of today may not be the major search engines of tomor- row. The two major search engines that I currently use (www.google.com and www.bing. com) provide access to printed documents, pictures, graphics, images, news, videos, dis- cussion groups, maps, and more. Evaluators can locate a great deal of information about contexts of evaluations and experiences with similar evaluands through web searching. Here are two examples:

 Fredericks et al. (2008): “The evaluation relied on information being col- lected from a number of data sources, including case records, which con- tained demographics and disability diagnoses data; Medicaid billing and expendi- ture data” (p. 225).

 Sharma and Deepak (2001) gathered contextual data for their evalua- tion of CBR in Vietnam (see Chapter 4, Box 4.12) from several websites, including the World Bank, the Central Intelligence Agency (CIA), and UNICEF. They were able to report on the gross national product of Vietnam, the density of its population, its population growth rate, and other demographics such as health indicators, age, life expectancy, infant mortality, literacy rates, access to clean water, and government budgets.

“Grey Literature”

Evaluators should always seek program documents that have been produced before the start of the evaluation process. The quantity and quality of these documents will vary widely, depending on the history of the evaluand. Even if a new program is planned, it is probably going to occur in a context that has some kind of paper trail. When I conducted an evaluation of a residential school for the deaf, I asked to see their self-study report and their accreditation report. In addition, I asked to see the curriculum guides and the stu- dent conduct rules. All of these documents gave me an overview of the evaluation context. The APA (www.apa.org/psycextra) has listed the following documents as examples of “grey literature”: research reports, policy statements, annual reports, curricula materials, stan- dards, videos, conference papers and abstracts, fact sheets, consumer brochures, newslet- ters, pamphlets, directories, popular magazines, white papers, and grant information. Examples of using “grey literature” in evaluation practice include the following:

 Mertens et al. (2007; see Chapter 6, Box 6.8) read over the RFP for the teacher training program that they evaluated, as well as the university’s proposal and annual reports for the 6 years prior to the evaluation.

 Bowman (2005) located and reviewed the initial needs assessment that was con- ducted in Wisconsin and was used as the basis for the development of the tribal education model for on- and off- campus activities. She was also able to determine that there had been no electronic, print, or annual data since the time of that report until she undertook her evaluation study in 2004.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

226 PLANNING EVALUATIONS

 Brady and O’Regan (2009; see Chapter 3, Box 3.3) cited the Atlantic Philanthro- pies annual report for 2007 as a source of historical information that set the context for their evaluation of the youth mentoring program in Ireland. The Atlantic Philanthropies foundation has funded programs to improve peo- ple’s lives through education and knowledge creation since the 1990s. The founda- tion reported that early initiatives in this area were not as effective as they had hoped because of lack of coordination, depending on volunteers, and relying on multiple unpredictable funding sources. Within the Foroige agency in Ireland, the foundation funded a pilot project of a BBBS model of youth mentoring.

Group and Individual Strategies

Evaluators can use group and individual strategies such as concept mapping, brainstorm- ing, interviews, surveys, and focus groups, as well as Indigenous methods based on tradi- tional community meeting ceremonies and rituals. Steps for conducting group and indi- vidual interviews are described in Chapter 10 on data collection. Here we provide examples of the use of these strategies and Indigenous methods for the purpose of determining the evaluand and its context.

Bowman (2005) included the use of focus groups and individual interviews in the Native American community in order to determine what their needs were for tribal- related education. She integrated the medicine wheel into the interviews (similar to the Cross et al. [2000] study summarized in Chapter 6, Box 6.6). She structured the questions based on the four quadrants of the medicine wheel. In addition, she pro- vided time for informal interaction following the focus group process to allow people to socialize and share experiences that might not have surfaced during the focus group. The data from the focus groups and individual interviews were used to develop recommenda- tions for changes in the tribal education model, the evaluand of interest in this study.

Africans have traditional tribal gatherings that can be used as a basis for dialogue about context and needs (Chilisa, 2011). The group gatherings in Botswana are called kgotla; these involve the village council in the main village, with the chief or his assistant in charge of the process. Smaller kgotla can be held in outlying areas with the head tribes- man as the facilitator, or even in extended families with the elders facilitating the process. These gatherings can be used to identify problems and potential solutions. One downside to this process is that it has traditionally excluded women and children. Therefore, evalu- ators will need to work with the communities to develop appropriate strategies for all stakeholders’ views to be represented.

Concept Mapping

Trochim (1989) developed the technique of “concept mapping,” which has been applied in many different contexts. The steps in the process involve having participants brain- storm either possible outcomes or specifi c factors that infl uence those outcomes. The next step is to edit the statements to reduce repetition. Participants are then asked to rate the outcomes on two dimensions— importance (compared to other factors) and feasibil- ity over the next few years—on 5-point scales where 5 indicates “extremely important” or “extremely feasible.” Sophisticated statistical procedures (multidimensional scaling and

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 227

hierarchical cluster analysis, discussed in Chapter 12) are then applied to the data to pro- duce confi gurations revealing which of the statements are rated most similarly. Different types of maps can be used to demonstrate how the statements can be organized and used to understand the underlying theory of the project.

Trochim, Milstein, Wood, Jackson, and Pressler (2004) used concept mapping with the Hawaii Department of Health to determine factors of importance that affect individu- als’ behaviors related to avoidance of tobacco, improvement of nutrition, and increased physical activity. Project participants brainstormed factors that they believed infl uenced individuals’ behaviors, and then rated those factors according to their importance and feasibility. The concept mapping revealed that factors could be categorized in terms of policies and laws, environmental infrastructure, children and schools, coalitions and col- laborations, community infrastructure, information and communication, and access. These results were used by the state’s governor in the offi cial state plan, approved by the legislature, and used to create sustainable change in Hawaii.

Outcome Mapping

Buskens and Earl (2008; see Chapter 6, Box 6.10) offer a strategy similar to con- cept mapping called “outcome mapping.” These two strategies are similar in many respects; however, Buskens and Earl offer insights into the application of outcome mapping within the context of transformative participatory evaluations in international development. Outcome mapping deliberately involves subgroups of stakeholders in the process of determining how interventions fi t into the overall development process. It begins with four questions (Buskens & Earl, 2008, p. 174):

1. What is the program’s vision?

2. Who are its boundary partners?

3. What changes in behavior are being sought?

4. How can the program best contribute to these changes?

“Boundary partners” are defi ned as “the individuals, groups, or organizations with whom the program works directly and with whom the program anticipates opportunities for infl uence” (p. 190). Boundary partners are similar to stakeholders; however, Buskens and Earl make the distinction that boundary partners are the subgroups interacting most closely with each other. Hence, instead of having big stakeholder meetings with every- one represented, they tend to have team meetings of relevant boundary partners. For example, the core management team for the IFRP had the following boundary partners (Buskens & Earl, 2008, p. 183):

 Action researchers

 Training development team

 IFRP trainers

 IFRP desk researchers

 Funders

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

228 PLANNING EVALUATIONS

 Motivational Interviewing Southern African Network (MISA)

 Department of Family Medicine at University of Stellenbosch

 Health researchers in southern Africa

The action researchers had their closest associations with the nurse counselors and the project management team members, who constituted their boundary partners. The bound- ary partners for the mothers who participated in the project were the nurse counselors with whom they worked. These teams deliberated on the program’s vision and desired changes in behavior. Buskens and Earl then discussed how the program could provide the conditions necessary for that change to occur. Outcome mapping typically hopes to observe outcomes as not only a change in behavior but also changes in relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution (Wilson-Grau & Britt, 2013). The outcome- mapping process is dynamic and ongoing, allowing the boundary partners to examine their progress, to make adjustments to the intervention as deemed necessary, to plan for the next step and wider adaptation, or to scale up their project.

Advisory Boards

Evaluators often work with advisory boards as a way to get input from representatives of various stakeholder groups. It would not be possible to work with all stakeholders in a national-level study (or a state-level or community-level study, in many instances). Hence the use of an advisory board can allow for important dimensions of the community to be represented. Mertens (2000) worked with an advisory board in a national evaluation of court access for deaf and hard-of- hearing people. The advisory board included representa- tives of the deaf and hard-of- hearing communities who were diverse in various respects: their choice of communication mode and language (sign language, reading lips, use of voice); backgrounds with the court (attorneys, judges, judicial educators, police offi cers, and interpreters); and hearing status (hearing, hard of hearing, and deaf). This group was able to provide guidance in regard to the diversity of experiences that deaf and hard-of- hearing people encounter in the courts. The group also emphasized the importance of understanding these diverse experiences in order to develop an intervention that could improve court access.

Technological Tools: Satellite Imagery and Mapping

Satellite imagery and mapping are valuable tools that can be used to display current conditions, as well as to compare past and current conditions. An organization called Information Technology for Humanitarian Assistance, Cooperation and Action (http:// ithacaweb.org/international- cooperation) provided information to help aid agencies plan how to respond when the island country of Haiti was struck by a massive earthquake on January 12, 2010. This organization used geomapping technology to post before-and- after pictures on its website of the areas hit by the earthquake. The before- earthquake satellite photos showed roads, airports, various types of buildings (public and private), and water and electricity centers. The photos taken after the earthquake showed how extensive the damage was to all these facilities. Electricity was not available; telephone

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 229

cables were damaged; the airport had no fuel or lights, and the road from there into the city was destroyed; the water supply collapsed, and wells were contaminated; the prisons broke open, and the prisoners who survived the quake escaped. The geomapping tool thus provided information that was invaluable in helping the aid agencies identify and respond to the conditions on the ground, especially since communication systems were not functioning.

Note that many of these strategies for identifi cation of context and evaluand are revisited in our Chapter 8 discussion of the approach to evaluation known as “needs and assets assessment.”

Depicting the Evaluand

In most evaluation planning, the evaluand, as the entity that is being evaluated, needs to be specifi ed early in the evaluation planning process. The exception to this specifi ca- tion might occur in developmental evaluations in which there is no static evaluand, or in transformative cyclical evaluations in which the evaluand might be developed based on fi ndings from early stages of the evaluation. As mentioned at the beginning of this chap- ter, evaluands can range in defi nition from a gleam in a proposing investigator’s eye to a well- established program. It is sometimes easier to describe an evaluand that has a long history and ample extant information, although this is not always the case. Sometimes a program that has been around for a while has developed layers of complexity that were not present in the original plans, requiring evaluators to do a bit of investigative work. Programs that are under development may also exist differently in the minds of different stakeholders. One of the greatest services an evaluator can provide in such circumstances is to facilitate discussions among the various stakeholder groups to identify what the vari- ous components of the evaluand are, how they work together, and what resources are needed and available to lead to the desired outcomes. Portrayals of evaluands should be considered as working models that will change over time; however, in order to plan an evaluation, a preliminary portrayal of the evaluand is needed.

Evaluands can be depicted in many ways: descriptively or graphically, as static or dynamic entities. Descriptive portrayals of evaluands are typically given as narratives; the object of the evaluation is described, along with the major players and goals. Graphic portrayals of evaluands have typically taken the form of logic models or logical frame- works (the latter is sometimes shortened to log frame, the terminology used in the inter- national development community for logic models). Evaluators from all branches can use all of these approaches to depicting evaluands; however, they may use them a bit differently. A Methods Branch evaluator might view the logic model as needing to be followed without changes in order to assure the fi delity of the treatment intervention. A Values Branch evaluator would probably be more comfortable with a fl exible view of the logic model, allowing it to evolve as the study progresses. Use Branch evaluators would want the logic model to be viewed as useful to their primary intended user and would therefore be amenable to changes as needed. A Social Justice Branch evaluator would see the logic model as a best guess at the beginning of the project and would want to leave room for changes based on fi ndings from communities throughout the process of the evaluation.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

230 PLANNING EVALUATIONS

Logic Models and Log Frames

Logic models are most closely tied to theory-based evaluation approaches (although they are used in many evaluation approaches), because the essence of theory-based evaluation is to reveal the underlying theory of how the program intends to achieve its intended out- comes. For example, if I want youth to refrain from using illegal drugs, what is my theory as to how to accomplish that outcome? The logic model is supposed to make the program’s theory of change explicit. A theory of change describes how the activities, resources, and contextual factors work together to achieve the intended outcomes.

The W. K. Kellogg Foundation (WKKF, 2004b) has published a logic model develop- ment guide that starts with a very simple depiction of a logic model. This includes two main components: what the program people plan to do (resources/inputs and activities) and what their intended results are (output, outcomes, impact). This elementary depiction of a logic model is shown in Figure 7.1.

“Resources” or “inputs” are those human, fi nancial, and community resources that are needed for the evaluand, such as funding, partnering organizations, staff, volunteers, time, facilities, equipment, and supplies. They can also include wider contextual factors, such as attitudes, policies, laws, regulations, and geography. “Activities” include the pro- cesses, events, technology, and actions that are part of the program implementation. These can include such components as education and training services, counseling, or health screening; products such as curriculum materials, training materials, or brochures; and infrastructure such as new networks, organizations, or relationships. “Outputs” are prod- ucts of the activities and include the quantity and quality of the services delivered by the program, such as the number of workshops taught or the number of participants served. “Outcomes” are the changes in individual participants in terms of behaviors, knowledge, skills, or attitudes. These can be short term or long term. “Impact” is the desired change on a broader level for organizations or communities, such as reduction of poverty or increase in health.

The most basic format for a logic model is the outcomes-based logic model, which starts with stakeholders’ identifying those outcomes and impacts that are important to them. Any of the group processes described earlier in this chapter can be used for this purpose. For example, Fredericks et al. (2008; see Chapter 3, Box 3.5) described a logic model for a project that was supposed to improve services and quality of life for people with developmental disabilities. The stakeholders included a state-level steering

Figure 7.1. Basic logic model template. Source: Based on WKKF (2004, pp. 1 and 17).

Your planned work

Your intended results

Resources/ inputs

Activities Outputs Impact Outcomes

What changes do you

expect in 7–10 years?

What activities do you need

to conduct?

What evidence of service delivery is

there?

What do you need to

accomplish your activities?

What changes do you expect in 1–3 and then

4–6 years?

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 231

committee and a fi nance team from the state agency in charge of the program. The proj- ect had specifi ed goals: “to increase the individualization of service planning and delivery, increase administrative effi ciencies, increase person- centered planning, increase con- sumer choice, increase community integration, and improve the quality of life for consum- ers—in terms of home, relationships, personal life, work and school, and community” (Center for Policy Research, cited in Fredericks et al., 2008, p. 254). The evaluators and the steering committee worked together to develop the logic model displayed in Box 7.5.

Box 7.5. Logic Model from the Fredericks et al. (2008) Quality-of-Life Study

Inputs (What is going into the system?)

Process (What is it that we are doing?)

Outputs and short-term outcomes (How will we know when we have done this?)

Long-term outcomes and impacts (Why are we doing this?)

 Training for staff to ensure more individualized services

 Resources to train and retain qualifi ed staff

 Implementing sites will redesign service delivery efforts based on an individualized service environment

 Increases in person- centered planning

 Increases in community integration

 Increases in the individualization of service planning and delivery

 Increases in administrative effi ciencies

 Links to community partners that will allow consumers to be more involved in the community, both socially and in a work setting

 Increased choices for consumers

 Implementing sites will provide services according to the performance contract

 Implementing sites will provide services to individuals currently not being served

 Implementing sites will serve individuals with a full range of disabilities

 Implementing sites will use a new budgeting procedure

 Increases in consumer choice

 Increases in the number of people being served

 Financial predictability, as measured by stability in the budgets

 Increases in the quality of life for consumers— in terms of home, relationships, personal life, work and school, and community

Source: Fredericks et al. (2008, p. 255). Copyright © 2008 the American Evaluation Association. Reprinted by permission.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

232 PLANNING EVALUATIONS

The WKKF (2004b) logic development guide offers another, more intricate template for a theory-based logic model. Like the simpler logic model just presented, this theory- based logic model explains what the project wants to accomplish and how it will accom- plish those intended results, but it does so in greater detail and complexity. The theory- based approach begins by clarifying the assumptions that underlie the decisions to plan and implement the evaluand. A template for this type of logic model appears in Figure 7.2. The development of the theory-based logic model follows these steps:

1. Identify the problem or issue. Why is this evaluand needed? What are the condi- tions in the community that give rise to the need for this program (e.g., high levels of poverty, increased rate of infection from HIV/AIDS, low literacy levels)?

2. List the community’s needs and assets. This means listing both the strengths and challenges in the community. For example, strengths might include networks of health care workers, expressed desire to work for change, or access to funds. Chal- lenges might include poor infrastructure in terms of transportation or school buildings or clean drinking water. Part of the contextual analysis should pay atten- tion to issues of power and infl uences of discrimination and oppression in the evaluation context.

3. Specify the desired results in terms of outputs, outcomes, and impact. As explained above for the outcomes-based logic model, outputs might be services delivered, workshops provided, or number of participants trained. Outcomes are short-term results in the form of changes in individuals’ behaviors, skills, effi ciency, literacy levels, or disease prevention or treatment. The impacts are the longer-term goals of the project (e.g., reduction of poverty, violence, economic hardship, or hunger).

Figure 7.2. Theory-based logic model template. Source: Based on WKKF (2004, p. 28).

Strategies

Problem or issue

Influential factors

Community needs/assets

Assumptions

Desired results

(outputs, outcomes, and impact)

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 233

4. Identify infl uential factors—both those that are facilitative and those that are bar- riers to change. These can include legislation or policies that either mandate or inhibit the changes that are needed, a history of political stability or civil unrest, economic upturns or downturns, natural disasters, and political or community leadership.

5. Determine strategies (activities) that are needed to achieve the desired results. These might include development of recruitment or training materials, provision of services to enhance skills or health, or enhancement of infrastructure or tech- nology.

6. State the assumptions that underlie the project. Why do the stakeholders believe that this course of action in this context will garner the results they desire? What are the principles, beliefs, or ideas that are guiding this project?

An example of a theory-based logic model is displayed in Figure 7.3. This fi gure is adapted from the work of Kathleen Donnelly- Wijting (2007) for an evaluation of an HIV/ AIDS prevention program for deaf youth in South Africa.

Another example, in Box 7.6, is from Hamilton County, Ohio, which participated in the U.S. Department of Housing and Urban Development’s (HUD) Lesbian, Gay, Bisex-

Figure 7.3. Theory-based logic model for HIV/AIDS prevention for youth in South Africa. Source: Adapted from Donnelly- Wijting (2007). Used by permission of Kathleen Donnelly- Wijting.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

234 PLANNING EVALUATIONS

ual, Transgender, and Questioning (LGBTQ) Youth Homelessness Prevention Initiative. LGBTQ youth were dramatically overrepresented in the population of youth experiencing homelessness, because there were few systems and services designed to meet their needs. The goals of this initiative were to learn more about (1) preventing homelessness for LGBTQ youth and (2) intervening early to prevent chronic homelessness among LGBTQ youth. The initiative involved a deep and diverse list of stakeholders who had a vested interest in the issue, and together they created a theory, on which they based their logic model, of how to resolve LGBTQ youth homelessness.

Box 7.6. Hamilton County Safe and Supported Community Plan to Prevent Homelessness for Lesbian, Gay, Bisexual, Transgender,

and Questioning Youth

Narrative Description of the Evaluand and Theory of Change

The Hamilton County Safe and Supported Community Plan has eight key goals:

1. Facilitate greater community awareness of issues contributing to LGBTQ youth homeless- ness and the Initiative’s efforts to address these issues.

2. Facilitate greater local collaboration among stakeholders, including youth, community members, youth- serving agencies, and staff of youth- chosen spaces.

3. Improve data quality on sexual orientation and gender identity.

4. Use risk and protective factors for screening and assessment of youth at risk of or experienc- ing episodic homelessness.

5. Improve the quality of interventions to reduce risks and build protective factors that can pre- vent LGBTQ youth homelessness.

6. Support positive outcomes for LGBTQ youth in the areas of well-being, permanent connections, stable housing, and education/employment.

7. Obtain new funding and in-kind resources to support plan implementation.

8. Evaluate the initiative including its progress and outcomes.

Safe and Supported Theory of Change: How and Why an Approach Will Produce Change

To prevent LGBTQ youth homelessness:

 Start with a needs assessment, understanding of local community context, and a collaborative planning process with stakeholders and youth representing the community.

 To identify and implement strategies that leverage local strengths and address gaps for preventing LGBTQ youth homelessness and address challenges contributing to LGBTQ youth homelessness.

 Through increased resources for youth, families, schools, communities and peer groups.

 Through cultural competency training and awareness building for families, schools, com- munities, and peer groups.

 Through changes in policies, procedures, and systems.

So that we build protective factors and reduce risk factors associated with LGBTQ youth homelessness, such as:

1. Improve social climate, including inclusivity of policies, effectiveness of resources, and sup- port/acceptance of LGBTQ identity.

2. Nurture youth who are motivated by self-

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 235

(cont.)

acceptance and belonging to a community to seek social and emotional well-being, perma- nent connections, stable housing, and educa- tion/employment.

3. Nurture a community that provides a safety net of social and emotional well-being, permanent

connections, stable housing, and education/ employment opportunities so youth do not experience homelessness.

4. Increase the ability of families to accept and support difference to create a safe space for youth and prevent episodes of homelessness.

Abbreviated Logic Model

Contextual Factors Contextual Factors

Community context

Availability of and access to culturally competent services, programs, shelters, and housing

Availability of data

Economic development and fi nancial resources

Geography

Leadership

Collaboration in the community across youth- serving systems (e.g., education, juvenile justice, law enforcement, mental health, faith-based) and “turf” concerns

Culture

Advocacy efforts and politics

Community awareness of prevalence and causes of LGBTQ youth homelessness

Social attitudes toward LGBTQ

Client context

Socioeconomic demographics (age, race, etc.)

Awareness of and willingness to access supports

Previous access to supports

Protective factors (e.g., employment, positive friends, school connection, supportive adults, survival skills)

Risk factors (e.g., emotional distress, family rejection, lack of stable housing, substance use, mental health challenges, physical factors)

Coming out status

Federal context

HUD, DOE, HHS, DOJ support for the initiative

DOE requiring diversity training for all school staff

Inputs, Activities, and Outputs

Inputs Priority Activities Outputs

Initiative planning team (~30 members), including youth participants

Lighthouse staff (2)

Strategies to end homelessness staff (1)

Technical assistance (TA) team (3) and other federal TA

Group site

Needs assessment

SWOT analysis

Local collaboration

Steering committee meetings (monthly)

Community meetings (4)

More clearly defi ning CQI process (formal change management process)

Needs assessment

Needs assessment fi ndings

Local plan development

Analysis of local data— report

Theory of change

Logic model

Strategic plan

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

236 PLANNING EVALUATIONS

Box 7.6 (cont.)

Inputs Priority Activities Outputs

Coordination of existing funding

Exploring new funding

Local plan development

Six-month strategic planning process involving the systems and providers serving LGBTQ and homeless youth

Financial plan

Local plan implementation

Outputs based on fi nal local plan

Leadership team meetings (biweekly)

Identify funding sources

Local toolkit for corporate response

Development and advocacy of funding strategies

Local plan implementation

Two years of implementation

Plan strategies and activities

Community advisory group

Local plan evaluation

Outcomes and Impact

Short-term outcomes (months 1–6)

Intermediate outcomes (months 7–18)

Long-term outcomes (months 19+)

Identifi cation of community need(s) using data

Participation of LGBTQ homeless youth in planning

Increased community engagement

Increased participant and community awareness of LGBTQ homelessness

Identifi cation of evidence-based or promising practices

Identifi cation and promotion of existing resources

Identifi cation of new funding sources

Reduced number of LGBTQ youth who become homeless

Strengthened relationships among youth and key partners and within each group

Expanded screening and assessment opportunities

Increase cultural competency at initiative partner agencies

Increased participation in LGBTQ competency training for foster parents and JFS workers

Increased number of LGBTQ youth in stable housing, permanent connections, social and emotional well-being, and education/ employment

Increased community acceptance and adult support of LGBTQ youth

Improved response to risk and protective factors of LGBTQ youth at risk of or experiencing homelessness

Implemented interventions and countywide programs to address specifi c needs of youth

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 237

Short-term outcomes (months 1–6)

Intermediate outcomes (months 7–18)

Long-term outcomes (months 19+)

Increased number of foster and adoptive families that support LGBTQ foster youth and increased matches between youth and these families

Improved LGBTQ client services and satisfaction at Sheakley Center

Decreased number of LGBTQ youth who become homeless

Improved access to community supports and resources for LGBTQ youth

More positive school environment for LGBTQ youth

Improved social and emotional well-being among LGBTQ youth at risk of homelessness

Secure funding for initiative recommendations

Expanded dialogue to share and explore perceptions of LGBTQ youth and related issues

Improved understanding of the prevalence of LGBTQ foster youth in Hamilton County

Improved data depth and quality (completeness, accuracy, timeliness)

Source: Hicks and Alspaugh (2014). Copyright © 2014 Meredith Hicks and Meradith Alspaugh. Reprinted by permission.

In addition to the WKKF (2004b) development guide for logic models, a number of other guides are available online:

 The Harvard Family Research Project has a guide for developing logic models. The logic model development process is illustrated with an example of a districtwide family engagement program (https://eric.ed.gov/?id=ED507500).

 The Aspen Institute has developed a tool that includes step-by-step instructions on the development of a logic model within the world of philanthropy. Continuous Prog- ress, a branch of the Aspen Institute’s Global Interdependence Initiative, just launched its Advocacy Progress Planner (www.aspeninstitute.org/programs/aspen-planning-and-eval- uation-program/tools). Funded by the California Endowment and the William and Flora Hewlett Foundation, this tool illustrates the range of possible outcomes and target audi- ences that might be relevant to a certain advocacy or policy change strategy. The model helps a user focus on identifying the proper goals of any advocacy effort, which depends on where the issue stands in the policy process.

 CAPT presents a planning framework for prevention programs (www.samhsa.gov/ capt/applying- strategic- prevention- framework). Many of the steps fi t into the logic model system. Step 1 is to assess the community’s needs and readiness for an intervention. Step 2 is to mobilize the community and build capacity as necessary. Step 3 is called “planning” and includes a description of the program, activities, and strategies. The website gives many examples of best practices from the National Institute on Drug Abuse, CSAP, the

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

238 PLANNING EVALUATIONS

National Center for the Advancement of Prevention, the Offi ce of Juvenile Justice and Delinquency Prevention, the Department of Education, and the Centers for Disease Con- trol and Prevention (CDC). Step 4 is to implement the program, and Step 5 is to evaluate the program’s results and sustainability.

In the fi eld of international development, logical frameworks (log frames) are used instead of logic models. Baker (2000) describes log frames as statements of objectives that lead to the identifi cation of outputs and impact indicators.

The use of a logical (log) framework approach provides a good and commonly used tool for identifying the goals of the project and the information needs around which the evaluation can be constructed. The log frame, increasingly used at the World Bank, is based on a simple four-by-four matrix that matches information on project objectives with how performance will be tracked using milestones and work schedules, what impact project outputs will have on a benefi ciary institution or system and how that will be measured, and how inputs are used to deliver outputs. . . . In other words, it is assumed that the project’s intended impact is a func- tion of the project’s outputs as well as a series of other factors. The outputs, in turn, are a func- tion of the project’s inputs and factors outside the project. Quantifi able measures should then be identifi ed for each link in the project cycle. This approach does not preclude the evaluator from also looking at the unintended impacts of a project but serves to keep the objectives of the evaluation clear and focused. Qualitative techniques are also useful in eliciting participa- tion in clarifying the objectives of the evaluation and resulting impact indicators. (p. 19)

Davies (2005) also describes a logical framework as a 4 × 4 planning matrix:

The four columns are the Narrative—a description of expected changes, Objectively Verifi - able Indicators—of those changes, Means of Verifi cation—of those indicators, and Assump- tions about external infl uences on the expected changes, both positive and negative. The four rows are the Activities, which lead via Assumptions on that row to the Output, which leads via Assumptions on that row to the Purpose, which leads via Assumptions on that row to the Goal. (p. 147)

· · · · · · · · · · · · E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·

Using a Logic Model

Logic Model: Stopping Teens from Texting While Driving

Situation: A high school in Montgomery County is mourning the death of one senior who died in a car accident as he was texting while driving. The problem seems to be complex: Many teens text while they drive; their parents text while driving; teens see other drivers texting while driving; the local police department does not seem to be ticketing or consistently ticketing drivers, despite the law prohibiting driving and texting; and there are limited consequences for the few teens who have been caught texting.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 239

The Montgomery County Teen Unit (MCTU) is planning a campaign to begin a program to teach the teens and the community at large about the dangers of texting while driving. The following table lists the inputs and processes as well as the out- puts/short-term outcomes and impacts/long-term outcomes. What would be some other outputs and short-term outcomes, and some other long-term outcomes and impacts?

Inputs Processes (activities)

• Montgomery County grants • Private funding (telephone

companies)

• Parents • Montgomery High School • Equipment • Volunteers (parents, police,

community members, teens)

• Community partners • Existing resources • MCTU staff • Materials • Time

MCTU will:

• Develop teaching units with driving schools • Create literature with teens • Create public service announcements at high

school’s TV lab

• Engage youth and build relationships • Write grants for funding • Collaborate with county judges for consistent

punishments and education

• Conduct training for cellphone providers • Work with police on vigilant and consistent

enforcement

• Discuss initiative at county hall meetings • Deliver prevention education programs

Outputs and short-term outcomes Long-term outcomes and impacts

Increased knowledge about the danger of texting while driving

Decrease in the number of teens who text while driving after fi rst probation

Name others: Name others:

In the international development context, evaluators focus on the United Nations Sustainable Development Goals (SDGs; these are listed in Chapter 1, Box 1.5). These give evaluators direction in terms of their goals and targets, as well as the indicators they can use to determine whether those goals and targets are being achieved. The World Bank and the United Nations have developed electronic databases that provide helpful informa- tion in planning an evaluation for an international development project.

The United Nations developed the Sustainable Development Knowledge Platform, which includes a global database and a metadata repository that contains information about progress toward the achievement of the SDGs by country or geographic area accord-

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

240 PLANNING EVALUATIONS

ing to each SDG indicator. The World Bank’s World Development Indicators (WDIs) is another database that planners can use to target disparities associated with the most vul- nerable groups, thus enhancing the possibility of designing interventions that are appro- priate within each country’s context.

Here is a list of databases that international development evaluators may fi nd useful if they are working on evaluations related to the SDGs:

1. The SDG Indicators Global Database (https://unstats.un.org/sdgs/indicators/ database) allows planners access to UN system data used to prepare for the secretary- general’s annual report on “Progress towards the Sustainable Develop- ment Goals” by SDG indicator and country or geographic area.

2. The World Bank’s WDI database (http://databank.worldbank.org/data/reports. aspx?source= world- development- indicators) contains current national, regional, and global estimates of development indicators collected from offi cially recog- nized international data sources, disaggregated by sex, age, economics, and urban or rural location. The WDI has been updated to include more indicators that refl ect the SDGs.

3. The World Bank also offers 150 maps and data visualizations of the progress of countries achieving the 17 SDG goals in their online Atlas of Sustainable Develop- ment Goals 2018 (http://datatopics.worldbank.org/sdgatlas). The atlas is meant to “help policy makers, managers, and the public alike better understand them (the SDGs). The Atlas helps quantify progress, highlight some of the key issues, and identify the gaps that still remain.”

Evaluators can use these databases to provide context for their evaluation planning, as well as to inform stakeholders about the extent of needs within various populations.

Descriptive Depictions of the Evaluand

Evaluators always have a descriptive depiction of the evaluand; it can stand alone or sup- port the graphic depiction of the evaluand in a logic model. All the examples of evaluations presented in this and earlier chapters have either a descriptive depiction of the evaluand or a descriptive and graphic depiction. One framework that is useful for conceptualizing a description of the evaluand is the CIPP model developed by Stuffl ebeam (see Chapter 4). Box 7.7 contains examples of the types of variables that might be considered for each aspect of the model, as well as applications of these to the evaluand description of a self- help program for women adjusting to breast cancer and its treatment (Sidani & Sechrest, 1999). It provided information about the course of treatment, belief in self, and improving problem- solving and cognitive reframing skills. The course had three components: (1) The cognitive component provided the knowledge needed to understand the condition, treat- ment, and self-care strategies; (2) the behavioral component addressed women’s skills nec- essary for active participation in their own care, problem solving, and stress management; and (3) the psychological component helped women deal with their feelings. The course used three teaching modes (interactive, didactic, and hands-on experience).

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 241

(cont.)

Box 7.7. Evaluand Descriptions Based on the CIPP Model

Component Variables Example from Sidani and Sechrest (1999)

Context Presenting problem; characteristics of the setting (physical and psychosocial features of the environment; social, political, and economic context of the program).

Setting: accessibility, material resources needed to deliver the services; the physical layout and attractiveness of the setting; organizational culture; composition of and working relationships among the staff; norms and policies.

Women with breast cancer receiving therapy.

Physical side effects; need for management to minimize effect on daily functioning.

Setting: Classroom in a quiet setting; written materials; seating arrangements to facilitate discussion; audiovisual materials; space and equipment for demonstrations and hands-on learning.

Input Critical inputs needed to produce the desired results, including client characteristics (e.g., demographics, personality traits, personal beliefs, employment status, level of anxiety, stage of the disease).

Resources available to clients (internal and external support factors); access to treatment.

Characteristics of the staff: personal and professional attributes, competency, gender.

Clients: Age, gender, educational level, traits such as sense of control, cultural values, and beliefs.

Staff: Communication abilities, demeanor, education background, level of competence or expertise in provided services, preferences for types of treatment, beliefs and attitudes toward target population. Staff members (women) delivering the courses: knowledge about breast cancer and self- help strategies; sensitivity to clients; good communication and teaching skills.

Teaching protocol: objectives, content, learning activities, logistical instructions, training for instructors.

Process Mediating processes, targeted activities, quality of implementation; quantity of process delivered (dosage/strength); frequency, duration; which clients received which components of the project at which dosage; sequence of change expected.

The self-help program had three components: cognitive, behavioral, and psychological. The course was given over six sessions (90 minutes each, once a week). The theoretical process involved this chain of events: attending course, increasing knowledge, engaging in self- care, decreasing uncertainty, improving affect, improving quality of life.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

242 PLANNING EVALUATIONS

Box 7.7 (cont.)

Component Variables Example from Sidani and Sechrest (1999)

Product The expected outcomes; reasons why the program was implemented; criteria to judge the effectiveness of the program; nature, timing, and pattern of change expected. (Nature of outcomes included particular changes in the clients’ lives or condition; timing refers to when the change was expected to occur— immediately, short term, or long term.)

The self-help program expected positive changes in the quality of life about 6 months after the training; it should continue into the future. Improved quality of life was contingent upon the women’s improvement in self-care and affect and the reduction of uncertainties.

Mixing Things Up

As most people know, life rarely follows a linear pathway. Hence the use of linear mod- els to depict evaluands is limited, because they do not portray deviations from what was planned or iterative changes that occur during the life of a program. A logic model is linear and suggests that action fl ows in one direction. However, the intended outcomes can focus on changes in participants, as well as changes in staff members as they progress through the project as well. These could lead to additional changes in the program that are not depicted in the logic model. Davies (2004) asserts that linear models are inadequate to depict the complexity of evaluands throughout the life of a project. He suggests that evaluators consider using more complex modeling strategies based on network analysis.

This chapter includes an example of an evaluand that was depicted in both narrative and graphic form using the WFFK logic development model by a county in Ohio to pre- vent homelessness for LGBTQ youth (Hicks & Alspaugh, 2014) (Box 7.6). Included in the plan is the list of diverse stakeholders who participated, contextual considerations, their theory of change, a complete logic model, and detailed short- and long-term outcomes.

Planning Your Evaluation: Stakeholders, Context, and Evaluand

Choose an evaluand for which you can develop an evaluation plan. This may be a pro- gram that you experienced at some time in your past, something related to your current position, or even a new idea that you would like to develop. Using one of the logic models presented in this chapter, develop a logic model for your evaluand, at least as you pres- ently understand it. Your understanding is expected to change throughout the planning process; therefore, be prepared to be fl exible with this part of the evaluation. Identify potential stakeholders for this evaluand; to the extent feasible, involve the stakeholders in the process of developing the evaluand. After you develop the logic model, write a narra- tive that explains the context of the evaluand and also provides additional details of what

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

Working with Stakeholders 243

is depicted in the logic model. Share this narrative with a peer; obtain feedback as to the clarity and completeness of your depiction of the context and evaluand. Make revisions as necessary. If possible, obtain feedback from the stakeholders about your logic model and narrative.

Moving On to the Next Chapter This chapter rests on the assumption that evaluators and stakeholders know what the evaluand should be or is. However, that is not necessarily the case. In Chapter 8, we look at strategies evaluators can use to provide information to stakeholders who are in the process of designing a new intervention or making substantial changes in an existing evaluand. This approach to evaluation is called “needs and assets assess- ment.” We also consider other evaluation purposes and questions that might be used to guide the evaluation; we focus on how answers to those questions might be used to make changes in the organization.

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .

244

Preparing to Read Chapter Eight

As you are halfway through the book, you now have a good understanding of the landscape of the evaluation fi eld, its history, its currently used paradigms, and the different theories and approaches in evaluation. In Chapter 7, you learned how to identify the stakeholders and establish the context of the evaluand. Do you think by now you can list why evaluations are done?

1. Imagine that your school wants to establish a no-texting policy during classes or meetings. Try to list as many purposes for an evaluation of this type of initiative as you can.

2. Consider the following purposes for an evaluation of the no-texting policy:

 Is this a good policy?

 How well was it implemented?

 What were the results of implementing the policy?

3. What kind of data would you collect in order to address these purposes for the evaluation?

Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice, second edition : A comprehensive guide. Guilford Publications. Created from templeuniv-ebooks on 2022-07-27 00:55:54.

C o p yr

ig h t ©

2 0 1 8 . G

u ilf

o rd

P u b lic

a tio

n s.

A ll

ri g h ts

r e se

rv e d .