Developing a Program Evaluation
Yaya0625
Reading2week6.pdf
The Step-by-Step Guide to EVALUATION H o w t o B e c o m e S a v v y E v a l u a t i o n C o n s u m e r s
The W.K. Kellogg Foundation, founded in 1930 by breakfast cereal pioneer Will Keith Kellogg, is among the largest philanthropic foundations in the United States.
Based in Battle Creek, Michv, WKKF engages with communities in priority places across the country and internationally to create conditions that propel vulnerable children
to realize their full potential in school, work and life.
© 2017 W.K. Kellogg Foundation.
This publication may be shared without charge for educational, charitable, nonprofit or other non-commer- cial purposes. Such use must include acknowledgement that it is “with the permission of the W.K. Kellogg
Foundation.” All other use is prohibited without the prior written permission of the W.K. Kellogg Foundation. For permission requests, please contact [email protected] or 269-969-2079.
The Step-by-Step Guide to Evaluation | page i
Foreword At the W.K. Kellogg Foundation, we believe that evaluation is an effective management tool to both inform
strategy development and track the progress and impact of strategy implementation. We have long been
committed to supporting our grantees’ ability to derive and share lessons learned from their work. To that
end, the foundation published the first Evaluation Handbook (The Kellogg Foundation, 1998) almost two
decades ago to guide evaluation for our grantees. Since that time, as the discipline of evaluation grew and
expanded, the demand for evaluation has risen. More and more nonprofit leaders and practitioners strive
to design evidence-based programs, and more and more funders require their grantees to provide evidence
to demonstrate the success of their funded work. The democratization of evaluation makes it necessary that
evaluation is both rigorous and practical. How to achieve the balance motivated us to update the handbook.
Over the years, the foundation has learned a lot from our grantees about the challenges of evaluation. This
handbook is our continuous effort to demystify evaluation and facilitate its use, for the foundation’s grantees
and for all organizations committed to learning and strengthening their work. It is designed for people who
have little to no exposure to formal evaluation training and provides a starting point for them as they
consider evaluating their work. It is intended to help them become more informed consumers of evaluation.
Evaluations can be simple or extensive depending on the scope and complexity of the work being evaluated.
The scope of the evaluation could potentially include a single program, a multi-site initiative, or a
multifaceted strategy aimed at systems and community change. Regardless of the complexity of the effort,
the basics for evaluating it remain the same and this handbook was written to impart information about
these basics. This handbook complements two other products produced by the Kellogg Foundation: the Logic
Model Development Guide (The Kellogg Foundation, 2004) and the Systems-Oriented Evaluation Guidebook
(The Kellogg Foundation, 2005).
We would like to thank Community Science for writing and HGF, Inc., for designing this handbook, WKKF staff
for their leadership in developing this resource, as well as the following individuals who served as advisors:
Jara Dean Coffey, Traci Endo, Johanna Morariu and Jianping Shen.
Huilan Krenn, Ph.D.
Director of Learning & Impact
W.K. Kellogg Foundation
September 2017
Page ii | The Step-by-Step Guide to Evaluation
Contents
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Chapter 1: Principles to Guide Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 W.K. Kellogg Foundation’s Principles about Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Community Engagement, Racial Equity and Evaluation . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Importance of Culture in Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2 American Evaluation Association’s Guiding Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 2: Definition of Evaluation and the Evolution of Evaluation Practice . . . . . . . . . . . . . . . . . 13
2.1 What is Evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.1 Definition of Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.2 Definition of Evaluative Thinking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Evolution of Evaluation Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Chapter 3: Evaluation Types, Methodologies and Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1 Evaluation Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Evaluation Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 Evaluation Methodologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Chapter 4: Overview of the Evaluation Process that Reflects Evaluative Thinking . . . . . . . . . . . . . . 37
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.1 Overview of the Evaluation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2 Creating a Learning Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.3 Stages of the Evaluation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.4 Navigating Evaluation Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4.1 Capacity, Program, or Population Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.4.2 Focus or Unit of Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.4.3 Internal Versus External Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.4.4 Power Dynamics Among Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
The Step-by-Step Guide to Evaluation | page iii
Chapter 5: Preparing for the Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.1 Why You Might Consider an Evaluation of Your Strategy, Initiative or Program . . . . . . . . 66
5.1.1 Considerations if Your Evaluation
is Part of a National or Statewide Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.1.2 Considerations if the Findings from Your Evaluation
Have Significant Implications for a Theory, Practice or Policy . . . . . . . . . . . . . . . 68
5.1.3 Getting Ready for Evaluation Whether or Not It is a Funding Requirement . . . . . 69
5.2 Who Benefits From the Evaluation and What Are the Potential Risks? . . . . . . . . . . . . . . . 70
5.2.1 Consideration of Different Types of Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.2 Use of Evaluation by Different Types of Stakeholders . . . . . . . . . . . . . . . . . . . . . . 73
5.3 Consider Your Organization’s Capacity to Participate in the Evaluation . . . . . . . . . . . . . . 75
5.3.1 A Staff Person Dedicated to Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.3.2 Budget for Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.3.3 A Culture of Data and Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.3.4 Share and Protect Data Across Program Elements or Program Locations . . . . . . 79
Chapter 6: Determine Stakeholders and Engage Them in the Evaluation . . . . . . . . . . . . . . . . . . . . 83
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.1 Evaluators as Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.1.1 Keep Your Evaluator Informed on a Frequent, Regular Basis . . . . . . . . . . . . . . . . 84
6.1.2 Two Types of Evaluators: Internal and External . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.1.3 The Evaluator’s Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.1.4 The Evaluator’s Cultural Competency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
6.2 Other Key Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6.2.1 How to Identify Key Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.2.2 How to Engage Key Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
Chapter 7: Developing a Logic Model, Evaluation Questions,
Measurement Framework and Evaluation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.1 Overview of Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.2 Use the Logic Model throughout Your Effort’s Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.2.1 Use of Logic Models to Design Your Strategy, Initiative or Program . . . . . . . . . . 109
7.2.2 Use of Logic Models to Inform Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Page iv | The Step-by-Step Guide to Evaluation
7.2.3 Use of Logic Models to Develop Evaluation Questions . . . . . . . . . . . . . . . . . . . . 110
7.2.4 Use of Logic Models to Develop an Evaluation Plan . . . . . . . . . . . . . . . . . . . . . . 110
7.2.5 Use of Logic Models to Create an Outline for Your Evaluation Report . . . . . . . . . 111
7.2.6 Use of Logic Models To Improve Your Strategy, Initiative or Program . . . . . . . . . . 111
7.3 What You Can Expect From Logic Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.3.1 It is a Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.3.2 The Process Can Surface Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113
7.3.3 The Process Can Bring About Innovation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4 How to Go About the Logic Modeling Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4.1 Whom to Involve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4.2 How to Involve Them . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
7.5 Components of the Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
7.5.1 Alignment Between Strategies or Activities and Outcomes . . . . . . . . . . . . . . . . 118
7.6 From Logic Model to Evaluation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.6.1 Two Different Ways to Formulate Evaluation Questions
About Your Strategy, Initiative or Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.6.2 Typical Scenarios When Formulating Evaluation Questions . . . . . . . . . . . . . . . . 123
7.6.3 Other Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.7 From Logic Model to Evaluation Questions to Measurement Framework . . . . . . . . . . . . 124
7.7.1 Key Components of the Measurement Framework. . . . . . . . . . . . . . . . . . . . . . . . 125
7.7.2 How to Use a Measurement Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.7.3 Considerations for Developing the Measurement Framework . . . . . . . . . . . . . . 127
7.7.4 Taking a SMART Approach to Measurement Framework Development . . . . . . . 127
7.8 Your Evaluation Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7.9 Budgeting for Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
7.9.1 Types of Expenses Typically Associated With Evaluation . . . . . . . . . . . . . . . . . . 134
7.9.2 Time Estimates for Specific Evaluation Activities . . . . . . . . . . . . . . . . . . . . . . . . 136
Chapter 8: Data Collection and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
8.1 Determine Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
8.1.1 What You Need to Ask Before Collecting Any Data . . . . . . . . . . . . . . . . . . . . . . . 143
8.1.2 Use of Quantitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.1.3 Use of Qualitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
The Step-by-Step Guide to Evaluation | page v
8.1.4 Use of Mixed Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.2 Critical Considerations in Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
8.2.1 Who the Data Collector Is . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
8.2.2 How to Engage Populations Who Have Been Traditionally
Excluded or Treated as Invisible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
8.2.3 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
8.2.4 Confidentiality and Anonymity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
8.2.5 Obtaining Consent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
8.2.6 Institutional Review Board (IRB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8.3 Analyzing and Interpreting Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
8.3.1 Quantitative Data Analysis and Interpretation of Results . . . . . . . . . . . . . . . . . . 167
8.3.2 Qualitative Data Analysis and Interpretation of Results . . . . . . . . . . . . . . . . . . . 175
Chapter 9: Summarize, Communicate and Reflect on Evaluation Findings . . . . . . . . . . . . . . . . . . 183
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9.1 Communicate and Report Your Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
9.1.1 What You Need to Ask Before Putting the Findings Together . . . . . . . . . . . . . . . 184
9.1.2 Different Communicating and Reporting Formats and Options . . . . . . . . . . . . . 187
9.2 Considerations in Developing and Implementing a Communications Plan . . . . . . . . . . 188
9.3 Keeping It Simple . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
9.4 Ways to Display Your Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.4.1 Displays of Patterns and Trends Over Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.4.2 Displays of Distribution and Spread of Responses . . . . . . . . . . . . . . . . . . . . . . . 194
9.4.3 Displays of Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
9.4.4 Displays of Frequency of Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
9.4.5 Displays of Themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
9.5 Reflecting on Your Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
9.5.1 Use of Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
9.5.2 Considerations in Preparation for Reflecting on,
Discussing and Using Evaluation Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Guide to Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Page vi | The Step-by-Step Guide to Evaluation
What is evaluation? How can it help your organization? And how can you work more effectively with your evaluator? This handbook is designed to demystify evaluation and help you get the most out of evaluation for your organization.
The Step-by-Step Guide to Evaluation | page vii
Overview
What is evaluation? How can it help your organization? And how
can you work more effectively with your evaluator? This
handbook is designed to demystify evaluation and help you get
the most out of evaluation for your organization.
The evaluation profession is multi-faceted and can be
characterized partly by its theoretical debates, ethical
considerations and proprietary interests. This handbook will
not attempt to provide in-depth evaluation information that has
little relevance to the day-to-day evaluation needs of nonprofit
directors and staff, nor will it transform anyone into an
evaluator. On the contrary, the goal of this handbook is to
educate busy nonprofit directors and staff, such as yourself,
about the essential elements of evaluation, so you can work
more effectively with trained evaluators; hold evaluators
accountable to the highest standards of quality, integrity and
competency; and maximize the usefulness of evaluation to your
organization.
The scope of your evaluation may include a single program, a
multi-site initiative or a multifaceted strategy aimed at systems
and community change. Regardless of the complexity of the
effort, the basics for evaluating it remain the same and this
handbook was written to impart information about these basics.
Before discussing each chapter in detail, here are a few things to
know about how this handbook is organized:
When the term “you” is used in this guide, it refers to you -
LOOK FOR THESE
This symbol
indicates a new
term is introduced.
The handbook
provides many definitions, all of
which are combined into one list
(“Glossary”) at the end of the
guide. This can be printed as a
reference.
This symbol is
used to summarize
tips.
This symbol refers
to an important
point.
Page viii | The Step-by-Step Guide to Evaluation
the reader or user of this handbook.
“Highlights” will be included in most chapters to provide a
concise summary of key points.
Brief examples are included within the chapters to make it
easier to understand the information.
Short exercises are included at the end of chapters to help
you retain its key points.
Many publications were consulted in creating this handbook.
You will find the references to these publications at the end
of the handbook.
An annotated resource guide for evaluation was created to
supplement this handbook; you will find it at the end of the
handbook. The resources are organized by evaluation topic
as well as issue such as education, poverty, health and
workforce development. Additionally, relevant and practical
resources cited in each chapter are included here.
This handbook contains nine chapters.
Chapter 1 synthesizes the W.K. Kellogg Foundation’s philosophy
concerning evaluation and the American Evaluation
Association’s guiding principles.
Chapters 2 and 3 explain essential information about the
evaluation field. As mentioned before, the evaluation field is
vast, so only some aspects of evaluation are summarized here.
Chapter 2 lays out the historical development and significant
contributions leading to the current status of the evaluation
field. Chapter 3 describes the various evaluation types, method-
ologies and approaches.
The Step-by-Step Guide to Evaluation | page ix
Chapter 4 is a brief overview of the overall evaluation process and its stages, which align with the
subsequent chapters.
Chapters 5 through 9 provide more detail about each stage in the evaluation process and explain
what you should know to work most effectively with an evaluator or how to use evaluation in your
work. Chapter 5 discusses some key considerations when planning your evaluation. Chapter 6
explains how to determine your evaluation stakeholders and engage them in the evaluation
process. Chapter 7 focuses on developing and using a logic model and on generating evaluation
questions, a measurement framework and an evaluation plan. Chapter 8 provides steps for
determining what type of data to collect and guidance on analyzing and interpreting the data.
Finally, in Chapter 9, you will learn considerations for summarizing and communicating your
evaluation findings.
To the extent possible, each chapter is written as a stand-alone section. Depending on what you
want to learn or what you already know, you can select the chapters you want to read. Thus, you
will find some information (e.g., definitions, concepts, resources) repeated in several chapters.
Page x | The Step-by-Step Guide to Evaluation
1
The Step-by-Step Guide to Evaluation | page 1
Principles to Guide Evaluation
introduction This chapter is intended to
help you understand the W.K.
Kellogg Foundation’s (WKKF)
principles and the American
Evaluation Association’s
principles concerning
evaluation. As a current or
future grantee of the
foundation, you could find the
foundation’s principles
informative. Accordingly, the
guiding principles—upheld
by the American Evaluation
Association—will help you, as
consumers of evaluation,
to know what to expect of
professional of evaluators as
you work alongside them.
The foundation emphasizes
the necessity of incorporating
culture, community engagement
and racial equity into efforts to
effect community and systems
change; as such, you can see
these topics throughout the
handbook.
1.1 w.k. kellogg foundation’s principles about evaluation
The Kellogg Foundation believes that all people have the
inherent capacity to effect change in their lives, in their
organizations and in their communities. What often limits the
ability to effect change is the availability of tools and resources
to actualize community desire for change. The foundation is
therefore committed to supporting the change process by
providing tools and resources that assist individuals,
organizations and communities in the generation and practical
application of knowledge that will result in improvements in the
quality of life for children and inform the narratives about certain
groups of people. Evaluation is one such tool.
Evaluation supports the ability to monitor and measure the
quality, pace and direction of change that individuals,
communities and organizations undertake. It does this by
systematically generating knowledge that can support learning,
How this chapter is organized…
1.1 W.K. Kellogg Foundation’s Principles about Evaluation
1.1.1 Community Engagement, Racial Equity and Evaluation
1.1.2 Importance of Culture in Evaluation
1.2 American Evaluation Association’s Guiding Principles
Exercises
Page 2 | The Step-by-Step Guide to Evaluation
quality improvement and good judgment in decision-making. Evaluation also can align purpose,
action and impact to ensure that longer-term change at the societal level unfolds progressively.
The foundation believes the following principles should guide evaluation:
Planning for evaluation should begin the moment new strategies, initiatives and programs are
conceptualized, evaluation findings should be used both to strengthen strategy development
and programming along the way, and to measure the extent of change.
Evaluation should support an organization’s efforts to become stronger and more effective and
should enhance its ability to obtain and effectively use new resources.
Evaluation should be adapted to the contexts of the community being evaluated and to the
important outcomes identified by the community (e.g., policy, impact on equity). Mixed
methods and different perspectives (e.g., a racial equity lens) can help capture the reality and
outcomes experienced by community members.
Evaluation should be designed to address real issues and to provide staff and stakeholders
with reliable information to address problems and to build on strengths and opportunities.
Mixed methods intentionally use two or more kinds of data gathering and analysis
tools — typically a combination of qualitative (e.g., focus groups and
interviews) and quantitative (e.g., multiple choice surveys and assessments) —
in the same evaluation.
A stakeholder is any person or group who has an interest in the strategy, initiative or
program being evaluated or in the results of the evaluation, including the evaluator.
Evaluation should invite multiple perspectives and involve a representation of people who care
about and benefit from the program.
Evaluation should be flexible and adaptable; strategies, initiatives and programs don’t exist
in a vacuum and events such as staff turnover, elections, legislation and economic recession
can affect their implementation and outcomes. Therefore, evaluators and implementers must
be flexible and work together to adapt to such events and respond to the needs of community
members.
The Step-by-Step Guide to Evaluation | page 3
Evaluation should build the skills, knowledge and perspectives of individuals to self-reflect,
dialogue and act based on data and knowledge. This strengthens the capacity of all
participants to establish a learning environment and work together to solve problems.
1.1.1
Community Engagement, Racial Equity and Evaluation
Community engagement and racial equity are central to the Kellogg Foundation’s work. In
evaluation, they are interconnected, especially in community change initiatives because people
of color usually make up Kellogg’s stakeholders and are usually the ones left out of discussions
about the evaluation design and implementation. Consider, for instance, an initiative designed to
improve the identification, referral and treatment of children exposed to violence in a Native
American community. Due to historical trauma experienced by Native American communities,
the evaluator must engage tribal leaders in discussions about the best ways to measure the
identification, referral and treatment, along with realistic expectations for change over the
initiative’s grant period. The evaluator should not assume the success measures for this
community are the same as for other communities because of the history, governance structure,
location, socioeconomic conditions and traditional norms of the tribe. This means paying explicit
attention to factors such as:
How tribal leaders and community members talk about the violence they experience.
What tribal and nontribal mental health resources exist.
What nontribal mental health resources are available and how culturally competent are the
resources.
Who needs to give the evaluator permission to collect data.
Who owns the data that is collected.
How to ensure the findings are not misused.
Page 4 | The Step-by-Step Guide to Evaluation
In evaluation, community engagement can look like the following:
Advice on the most appropriate markers of change or indicators for their community as well as
the amount of change that is desired and possible.
Help with determining the best data collection methods and data sources for their community.
Assistance with data collection, provided that those assisting receive proper training
and support.
Discussion about findings and what they mean for the community.
Transformation of the findings into useable knowledge and products that can be leveraged for
advocacy and other types of action.
Community engagement in evaluation is commonly referred to as stakeholder engagement. Some evaluators emphasize they practice community-based participatory research or a participatory approach. Regardless of what it is called, the points mentioned above should still be considered.
Indicators are markers of progress toward the change you hope to make with your
strategy, initiative or program.
Funders, evaluators and nonprofit directors must pay attention to power dynamics when engaging
community in evaluation. Specifically, they should:
Determine who constitutes members of the “community” - for example, residents only;
residents and directors and staff of public, nonprofit and private organizations that live and
work in a place; all the individuals and the elected officials who represent their interests at
local and state levels; or networks of people connected through a shared identity or goal.
Ensure that people who traditionally have been excluded
from decisions that affect their lives - typically people of
color; low-income families; youth; lesbian, gay, bisexual,
transgender and queer (LGBTQ) people; and people with
different disabilities - are engaged in a way that values and
is not dismissive of their perspectives.
Avoid only listening to leaders of large, well-endowed and
established institutions.
Create various avenues for engaging people in the
community, from social media sites to town hall meetings
and small group discussions convened by individuals
The Step-by-Step Guide to Evaluation | page 5
trusted in the community, especially by the groups of people mentioned above, and conducted
at safe and convenient locations.
The funders and professionals designing and implementing the evaluation will need to consider
the above points and determine the most feasible, meaningful way to engage the community given
the resources available and timeframe for the evaluation.
Racial equity in evaluation can look like the following:
Framing evaluation questions in a way that doesn’t make assumptions and perpetuate
stereotypes about a particular group of people. For example, if all youth of color are lumped
together and labeled “at-risk youth,” the evaluation could end up looking only at negative
behaviors and how the initiative improved the behaviors and neglect to identify positive
behaviors and how the initiative helped to reinforce these behaviors. Don’t use words in the
questions that make a judgment - good or bad - about a particular group of people.
Ensuring the evaluation process doesn’t maintain the status quo by excluding people of color
as decision-makers, problem-solvers, advisors, information sources or end-users of
the results.
Recognizing and incorporating the knowledge of people of color into the evaluation design
and implementation.
Designing data collection protocols and instruments in the languages spoken by the desired
respondents and at the appropriate reading level.
Accounting for situations and events in the community that could affect racial and ethnic
disparities in health, education, socio-economic status and other well-being outcomes in the
evaluation design if the situations and events are targets of change (e.g., leadership changes,
closing of a factory that was the biggest employer in the town, sudden influx of new
immigrants, budget cuts, etc.). A good evaluation should consider all these possibilities as
“alternative explanations” for the outcomes in order to determine the extent to which the
intervention contributed to the changes observed.
Bringing attention to the systems, policies and social norms that affect racial and ethnic dis-
parities in health, education, socio-economic status and other well-being
outcomes in the evaluation design and implementation. (See the Kellogg Foundation’s 2016
guide on systems-oriented and culturally responsive evaluation.)
Page 6 | The Step-by-Step Guide to Evaluation
Culture is a set of socially transmitted and learned behavior patterns, beliefs,
institutions and all other products of human work and thought that characterize the
functioning of a particular population, profession, organization or community.
Culture is continually evolving.
1.1.2
Importance of Culture in Evaluation
The word culture is often used without fully understanding its meaning. Depending on the
environment or context, it can be used to imply race, ethnicity, sexual orientation, gender identity,
disability and income group. But, “culture” is more than that.
Individuals who share the same culture form relationships that become the basis for a sense of
community. Individuals typically participate in multiple communities simultaneously. These
communities may face racial and other inequities because their cultures have historically been
Resources on designing and managing equity-focused evaluation include:
Borgman-Arboleda, C. & Clark, H. (2010). Considering evaluation: Thoughts for social change and movement-building groups, by C. Borgman-Arboleda and H. Clark, published by ActKnowledge in 2010. Available at http://www.actknowledge.org.
How To Design And Manage Equity-Focused Evaluation by M. Bamberger and M. Segone, published UNICEF in 2011. Available at http://betterevaluation.org/resource/guide/ design_manage_equity_focused_evaluation.
The Step-by-Step Guide to Evaluation | page 7
suppressed, ignored, dismissed or undervalued. Consequently, people in these communities have
been excluded from decisions that affect their lives and from resources and opportunities that
would enable them to have the best life possible. Culturally competent evaluators are committed
to, and skilled in engaging people in these communities in the planning and implementation of an
evaluation. They also promote racial, gender and other forms of equity through the evaluation.
While training in cultural competency for evaluators has been receiving more attention, culture and
context still are not adequately addressed in evaluation, which can affect the accuracy of findings.
Culture and context can be considered in evaluation design and implementation in ways such as:
Engaging respected leaders in an African American community to facilitate entry into the
community for data collection and other evaluation activities.
Respecting how Muslim women might respond to a female or male evaluator, as their
responses are regulated by a set of behavior patterns and beliefs about gender roles in their
culture.
Bearing in mind the way African Americans and Central American immigrants and refugees
respond to the question, “How many family members live with you?” is shaped by their cultural
values about family, which typically extend beyond the nuclear family.
Understanding that appropriate terms to describe a geographic area vary in urban versus rural
settings (e.g., the term “neighborhood” would not be appropriate in a rural community).
Recognizing the possibility that a white evaluator may not have the skills to respond to racist
comments made by a white respondent who assumed the evaluator and respondent think alike
about other racial groups because they have the same skin color.
Recognizing that in a focus group or in interviews using inappropriate terms such as “black
boys” to refer to African American men can be offensive and will diminish the study’s
effectiveness.
Also, your work and its evaluation don’t exist in a vacuum; organizations and communities have
cultures that are affected by dynamics within, around and between your organization and
community and the formal systems (e.g., education system, health system) and informal systems
(e.g., faith beliefs, extended family networks) within which they are situated. Context refers to the
combination of factors or circumstances surrounding the strategy, initiative or program and the
evaluation that could influence implementation, results and use of the findings.
Page 8 | The Step-by-Step Guide to Evaluation
Make sure that the appropriate terms familiar to the participants and community
members are used in materials for the evaluation, from introductory letters to survey
questionnaires (e.g., the term “household” may be less familiar to some cultural groups than
simply “family”).
Design data collection methods that are easy and comfortable for the participants (e.g.,
an online survey might not be convenient for people with limited or no access to
computers).
Ensure that questionnaires are in a language understood by potential respondents (e.g.,
in the language potential respondents are proficient in, and at a reading level that they
can understand).
Engage an interpreter for focus groups and interviews when necessary.
Interpret the findings to ensure that any community conditions (e.g., leadership chang-
es in the local government, closing a factory that was the biggest employer in the city,
passing of legislation, sudden influx of new immigrants or refugees) that could have
affected the results (e.g., the loss of a major employer in the city that might have
contributed to poor participation in a program for mothers who had to work more hours
to help supplement their household income) are discussed in the evaluation report.
checklist You Can Work With an Evaluator to:
Useful resources on the role of culture in evaluation include:
The Importance of Culture in Evaluation, by Kien Lee, published by The Colorado Trust in 2007. Available at http://www.communityscience.com/pdfs/CrossCulturalGuide.r3.pdf.
A Guide to Culturally Response Evaluations, by H.T. Frierson, S. Hood and G.B. Hughes, included in the National Science Foundation’s 2002 User-Friendly Handbook for Project Evaluation. Available at http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5.pdf.
The Step-by-Step Guide to Evaluation | page 9
1.2 american evaluation association’s guiding principles
Section 1.1 describes the Kellogg Foundation’s values concerning evaluation. Equally important is
how the American Evaluation Association - the national association for evaluation in the United
States - sets guidelines for evaluators (see www.eval.org for the Association’s guiding principles
for evaluators). The association’s mission is “to improve evaluation practices and methods, in-
crease evaluation use, promote evaluation as a profession and support the contribution of eval-
uation to the generation of theory and knowledge about effective human action.” To this end, the
American Evaluation Association created the following five principles to guide the professional
practice of evaluators and to inform evaluation clients and the general public about the principles
they can expect professional evaluators to uphold. Detailed explanations about the principles can
be downloaded from the association’s website mentioned above; they are abbreviated here.
Systematic Inquiry: Evaluators are expected to conduct systematic, data-based inquiries where
they discuss with you the strengths and limitations of the proposed evaluation design, adhere to
the highest technical standards in employing their methods and communicate their methods and
approaches accurately and in sufficient detail. The inquiries can be based on both qualitative and
quantitative data.
Competence: Evaluators should possess the qualifications, including cultural competency, needed
to undertake the tasks proposed in the evaluation; practice within the limits of their professional
training and competence; and participate in continuing professional development.
Cultural competency is to the commitment and ability (e.g., knowledge, skills) to
respect and engage with diverse segments of communities and to include the
contextual and cultural dimensions relevant to these diverse segments in the
evaluation design and process.
Integrity and Honesty: Evaluators are expected to behave with honesty and integrity during the
entire evaluation process. They are responsible for initiating discussion to clarify the costs, tasks
to be undertaken, limitations of methodology, scope of results likely to be obtained and uses of
data resulting from the evaluation. They should disclose any roles or relationships they have that
Page 10 | The Step-by-Step Guide to Evaluation
could pose a conflict of interest (or appearance of a conflict) with their role as an evaluator. Finally,
they are expected to inform you if certain procedures or activities likely will produce misleading
evaluative information or conclusions.
Respect for People: Evaluators should respect the security, dignity and self-worth of respondents,
program participants, clients and other evaluation stakeholders. This means that they understand
and account for cultural and contextual factors that could influence the results of a study (e.g.,
languages spoken, geographic location, timing, political and social climate, economic conditions
and other relevant activities in progress at the same time). They also are responsible for
communicating clearly the risks, harms and burdens that could affect participants in the
evaluation. Evaluators should help foster social equity by ensuring the people who share
information benefit from their contributions.
Responsibilities for General and Public Welfare: Evaluators are expected to consider relevant
perspectives and interests of the full range of stakeholders and allow all relevant stakeholders
access to evaluative information in ways that respect people and honor promises of confidentiality.
Additionally, evaluators should not ignore any clear threats to the public good that could surface
during the evaluation process.
Why are these principles important? These principles help you, as consumers of evaluation, to
know what is professionally expected of evaluators as you consider and select an evaluator to work
with you.
The Step-by-Step Guide to Evaluation | page 11
exercises
1. This group exercise is best done after the staff members in your organization have read Chapter
1 of this handbook. Think about a strategy, initiative or program in which you are involved that
you’d like to be evaluated. Ask yourselves these questions:
A. How can the community be engaged and what factors have to be considered to maximize
members’ engagement?
B. How can you engage the community in conversations about the history of racial
discrimination and oppression, community asset and experiences with larger systems
(e.g., education, law enforcement, health, housing) that can help you understand the
community context from the community members’ perspective?
2. You learned that your evaluator submitted a manuscript for publication to a peer-reviewed
journal based on the evaluation of your initiative. You asked to see the manuscript and after
reading it felt that it misrepresented the initiative’s impact. Which of the guiding principles can
be used to raise and frame your concerns to the evaluator?
Page 12 | The Step-by-Step Guide to Evaluation
3. The evaluation of your collaborative’s advocacy effort for early childhood education is delayed
because several collaborative members did not provide the information needed to the
evaluator. You get a call from your project officer at the funding agency who said he heard from
the evaluator about problems with the collaborative and he wanted to know what was going
on. The project officer suggested he attend the next collaborative meeting to let everyone know
they must cooperate with the evaluator. As the collaborative’s coordinator, how would you han-
dle this situation? (Hint: What information do you need to further understand what is going on?
What power dynamics could be at play here?)
The Step-by-Step Guide to Evaluation | page 13
2 Definition of Evaluation and the Evolution of Evaluation Practice
Page 14 | The Step-by-Step Guide to Evaluation
2.1 what is evaluation?
Two important terms are “evaluation” and “evaluative thinking.”
2.1.1
Definition of Evaluation
The purpose of evaluation is to facilitate learning and improve
your strategy, initiative or program. This learning happens
through a process of collecting and summarizing evidence that
leads to conclusions about the value, merit, significance or
quality of an effort. Funders, program implementers and
policymakers often evaluate the strategies, initiatives or pro-
grams they support to determine whether they are worth con-
tinuing support. Although this is not the only use of evaluation,
many people tend to look at evaluation as something that
results in a judgment about the merit of their performance and
work. Often they dislike evaluation because the judgment could
seem unfair for various reasons. For example, they might believe
that inappropriate metrics were used in forming the judgment,
or the evaluator failed to understand what they are trying to do,
or they could resent findings that show their work did not have
the anticipated impact.
You have probably encountered these situations before. You
might have been hesitant to tell your funders anything that
didn’t work; you could lack confidence that your evaluator will
help you frame the findings within the proper context; worst of
all, you might not see evaluation as worthwhile for any purpose
other than to fulfill your funding requirement. Consequently, the
power of evaluation to facilitate learning and improve your strat-
egy, initiative or program is diminished.
How this chapter is organized…
2.1 What is Evaluation?
2.1.1 Definition of Evaluation
2.1.2 Definition of Evaluative
Thinking
2.2 Evolution of Evaluation
Practice
Highlights
Exercises
The Step-by-Step Guide to Evaluation | page 15
Some funders will continue to use evaluation to judge their grantees’ performance and worth. This use of evaluation will not go away, but it should not be the sole purpose of evaluation. The aim of this handbook is to hone your ability to work with evaluators or with your staff (if you don’t hire an external evaluator) to measure the “right thing”, and to effectively communicate and use the results to become better at what you do. This includes communicating to funders - and the people you work with or serve - about why something did or did not work.
2.1.2
Definition of Evaluative Thinking
Underlying evaluation is a way of thinking about what results are expected, how results can be
achieved and what data or evidence are needed to inform future actions so that results can be im-
proved. This is called evaluative thinking. At its core are dialogue, reflection, learning and improv-
ing. If you consider evaluation not just as an inquiry that leads to a judgment (did the
program or organization perform as expected?) but also as evaluative thinking, you will become
more comfortable with evaluation and might even embrace it as part of your organization’s culture
and daily operations. Evaluative thinking is a “muscle” that needs to be exercised regularly to
become better and stronger.
Page 16 | The Step-by-Step Guide to Evaluation
Whether you perform an evaluation yourself or work with an external evaluator, in order to benefit
further from evaluative thinking and evaluation you might need a stronger understanding of the
process and a system for planning and implementing it. Also, evaluative thinking should be
integrated into the day-to-day operations of your organization to improve your use of data and
data-informed decision-making and ultimately your efforts, services and products.
Evaluative thinking is about understanding — using a systematic process of
collecting and analyzing data instead of a set of disorganized, random opinions —
and telling the story about your strategy, initiative or program. It is based on the
belief that a systematic process is valuable and necessary. This involves identifying
assumptions about what you think works and doesn’t work and why; posing
thoughtful questions about what you expect to see differently during and after you
implement your effort; pursuing deeper understanding through reflection and
dialogue; communicating what was learned without underestimation or
exaggeration; and making informed decisions in preparation for action. Evaluative
thinking also has the potential to shift the narratives about certain issues and groups
of people by challenging widespread assumptions associated with them, providing
data to support alternative explanations, and shifting mindsets through education
and learning.
The Step-by-Step Guide to Evaluation | page 17
Create a learning agenda for your staff, board members or partners and designate a staff
person, board member or committee to be responsible for the learning agenda.
Collect, analyze and share data about the performance of the strategy, initiative or
program on a regular basis. This can range from basic data, such as attendance of
program participants, to more complex data, such as reading and math proficiency scores
for third graders in the school in which you work.
Allocate time at staff, team or collaborative meetings for structured discussion about
what everyone learned and how they can apply their new knowledge. Good questions to
ask to structure this discussion include:
What do you think the data tell you about what is happening?
Did the data surprise you? Why or why not?
What should you do differently, if anything?
What support do you need to overcome the challenges you face?
What additional information do you need to know or learn about what you are
doing or not doing?
checklist Some ways to integrate evaluative thinking into your organization and day-to-day operations:
A useful resource on evaluative thinking is Evaluative Thinking published by the Bruner Foundation. Available at http://www.evaluativethinking.org/docs/EvaluativeThinking. bulletin.2.pdf
Page 18 | The Step-by-Step Guide to Evaluation
2.2 evolution of evaluation practice
You might ask yourself why you need to know about the history of evaluation. One reason is to
understand the motivation behind evaluation and its evolution over time. This information will
increase your insight into how the funding, political and social contexts within which evaluations
are conducted changed over time and will help you know what to expect of evaluators today and as
the field grows.
Evaluation in the United States gained prominence in the 1960s in response to: efforts to
strengthen the U.S. defense system, new laws to serve racial and ethnic minorities and persons
with disabilities equitably and to requirements of the Great Society programs. In the 1970s, eval-
uation was used to assess if educational and social organizations used resources appropriately
and achieved their objectives. In the 1980s, evaluation helped promote excellence in response to
global competition. In the 1990s, evaluation was employed to ensure quality, competitiveness and
equity in service delivery, and the United Way of America started encouraging the nonprofit organi-
zations it funded to use a program outcome evaluation approach.
Evaluation in the 21st century has become increasingly diverse in its range of applications, from
defense policy to social media technology. Also, the world has become more complex with global
migration, widening gaps between rich and poor and threats to democracy. In response, the array
of evaluation methodologies also has had to increase. Innovative approaches, such as
developmental evaluation and culturally responsive evaluation, have emerged.
Developmental evaluation is an approach used to support innovation within an
organization and in its strategies, initiatives and programs. Efforts that are innovative
often are in a state of continuous development and adaptation, and they frequently
unfold in a changing and unpredictable environment.
A useful resource about developmental evaluation is A Developmental Evaluation Primer by J.A.A. Gamble, published by The J.W. McConnell Family Foundation in 2008. Available at http://tamarackcci.ca/files/developmental_evaluation_primer.pdf.
The Step-by-Step Guide to Evaluation | page 19
Culturally responsive evaluation recognizes that cultural values, beliefs and context
lie at the heart of any evaluation effort.
Resources about culturally responsive evaluation include:
Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and Practice, by S. Hood, R. Hopson and H. Frierson, published by Information Age Publishing in 2015.
Handbook of Critical and Indigenous Methodologies, by N.K. Denzin, Y.S. Lincoln and L.T. Smith, published by SAGE Publications in 2008.
Practical Strategies for Culturally Competent Evaluation, by the Centers for Disease Control and Prevention, published in 2014. Available at http://www.cdc.gov/dhdsp/docs/cultural_ competence_guide.pdf.
The Center for Culturally Responsive Evaluation and Assessment (http://education.illinois. edu/crea).
In summary, evaluation today extends beyond the limits of its
original intent and approaches.
Experimental designs with control and treatment groups are
no longer the only or best ways to evaluate programs aimed at
improving the quality of life for children and their families. These
designs are well suited for stable programs in a fairly controlled
environment and where the goal is to determine if the program
led to the anticipated outcomes — to prove the causal link
between the program and outcomes. For example, an
experimental design may be appropriate for determining if an
academic enrichment program implemented by the Boys and
Girls Club led to improved grades. All other things being equal,
the children in the program (“treatment”) can be compared to
the children in another Boys and Girls Club that did not have the
program (“control”).
Page 20 | The Step-by-Step Guide to Evaluation
Experimental designs assess the causal effects of a program by comparing two groups
of people: one group receives the intervention (“treatment group”) and one does not
(“control group”).
Experimental designs, however, are not appropriate for complex community and systems change
initiatives. Cross-case study methodology is more appropriate for such initiatives. This
methodology, which builds on case study methodology, involves several cases and looks for
patterns across them to understand where the commonalities and differences lie and the possible
reasons for them. For example, this design is ideal for evaluating an initiative intended to
strengthen community collaboration to improve early childhood education in five communities
within a state. Each community makes up a “case,” and the five cases can be compared to one
another to look for common and unique patterns rooted in each community’s unique history,
culture and context. This design focuses on determining the initiative’s contribution to changes
in early childhood education. In a complex initiative such as this, determining causality is not the
goal since there are many factors that could contribute to the changes. You may have heard the
phrase “contribution versus attribution,” which refers to this sort of situation.
Another change in the evaluation field is the increasing attention paid to building the knowledge
and skills of nonprofit leaders and staff so they can effectively use evaluation to fulfill funders’
requirements; to inform their program design, implementation and improvement; and to
strengthen the organization. Evaluation capacity-building is an intentional process to increase
individual motivation, knowledge and skills and to enhance a group’s or organization’s ability to
conduct or use evaluation. Evaluation capacity-building efforts have led to positive results;
however, it takes more than knowledge and skills. A shift in organizational culture is needed for
the feedback loop between evaluation and program to become ingrained in the organization.
The Step-by-Step Guide to Evaluation | page 21
Finally, a relatively new development in the evaluation field is the emphasis placed on the role of
culture and context, particularly the traditions, histories and norms of people who have
traditionally been excluded or oppressed. This emphasis is important for three reasons:
Evaluation generates knowledge, and knowledge is power. Virtually every phase of the evalua-
tion process has political implications which will affect focus, decisions made, external percep-
tions of the program or organization and determination of those whose interests are advanced
and whose are ignored.
The traditions, histories and norms of a group of people affect the change desired and
achieved. Sometimes, expectations about outcomes are not realistic because more time and
effort are required to lay the groundwork for the change desired.
The attention paid to the role of culture in evaluation also raises questions about how personal
biases and institutional practices may devalue the perspectives of people historically excluded
from decisions that affect their lives. These potential biases and practices can be
acknowledged and factored into the evaluation design, implementation and reporting.
A useful resource about the history of developing culturally responsive evaluators is the New Directions for Evaluation issue on Building a New Generation of Culturally Responsive Evaluators Through AEA’s Graduate Education Diversity Internship Program (Number 143, Fall 2014).
Page 22 | The Step-by-Step Guide to Evaluation
HIGHTLIGHTS
Underlying evaluation is evaluative
thinking - using a systematic process of
collecting and analyzing data rather than
a set of disorganized, random opinions
in order to tell the story about a program,
strategy, policy or organization. This
process involves identifying assumptions
about what you think works, what doesn’t
and why; posing thoughtful questions
about the difference(s) you expect to see
during and after you implement your
strategy, initiative or program; pursuing
deeper understanding through reflection
and dialogue; communicating what was
learned without underestimation or
exaggeration; and making informed
decisions in preparation for action.
Recent developments in the evalua-
tion field include culturally responsive
evaluation and evaluation capacity-
building. These developments were
necessary in response to contemporary
issues such as global migration, widening
gaps between rich and poor and threats to
democracy. As consumers of evaluation,
you should be aware that their recent
prominence in the profession means that
not all evaluators are trained to be
culturally responsive or to build
organizations’ evaluation capacity.
exercises 1. Which of the following is a way to integrate evaluative thinking into your day-to-day operations:
A. Allocating time for regular and structured discussion during staff or team meetings
B. Collecting, analyzing and sharing data about the program once a year with program staff
C. Getting the evaluator you contracted to present to the staff once a year about the
evaluation findings
D. Disconnecting the evaluation of programs in your organization from your organization’s
overall learning agenda
E. None of the above
The Step-by-Step Guide to Evaluation | page 23
2. Consider the current, day-to-day operations in your organization. What opportunities are there
to integrate evaluative thinking? What are the first steps you can take?
3. You work for the city government. The mayor asked you to evaluate the effectiveness of the new
city plan to invest in mixed-income neighborhoods and more transportation options for resi-
dents. To ensure the inquiry process reflects evaluative thinking, which of the following should
you NOT do:
A. Work with elected and informal leaders who represent the affected areas to organize town
hall meetings and develop guiding questions to facilitate dialogues about the experiences
of residents
B. Convene resident leaders to discuss what a successful mixed-income neighborhood and
effective transportation options look like in order to determine success measures
C. Work with a professional to design and administer a community survey without getting
input on the survey from key stakeholders
D. Contact the community leaders you know, ask them what they think of the plan and ask that
they identify other leaders to whom you can speak
E. In addition to gathering the data, develop a plan to summarize and communicate the
findings to city council members and facilitate a discussion for improvement
4. Think about a time when you evaluated your strategy, initiative or program. Considering the
definition of evaluative thinking and the previous tips on how to integrate evaluative thinking
into your daily operations, what could you have done differently to more effectively use the
evaluation for learning and improvement?
Answers: 1A; 3C
Page 24 | The Step-by-Step Guide to Evaluation
3
The Step-by-Step Guide to Evaluation | page 25
introduction Whether your organization receives funding from a private foundation or public agency, as a leader
within the organization, you want to introduce and integrate evaluative thinking into your daily work
to cultivate a learning organization. This means evaluating your strategy, initiative and program to learn
how to strengthen and improve them. It also means assessing and improving how your organization gen-
erates and applies knowledge. Working with local stakeholders, including staff and board members, and
sometimes community leaders, you typically have to make the following decisions:
The type of evaluation to conduct based on the developmental stage of your strategy, initiative or
program as well as the budget and timeframe.
Considerations of racial equity and culture in your evaluation.
Engagement of community in the evaluation process.
The best evaluator to hire.
How you will learn about the actions you could take based on the evaluation findings.
You can approach evaluation in different ways, depending on your
evaluation goals and the stage of your strategy, initiative or program.
Even if an external evaluator is hired to conduct the evaluation,
you are expected to take an active role in making these decisions.
You and your staff know the program, initiative or strategy better
than anyone else and will be the primary users of the evaluation’s
findings. And there is no “one size fits all” evaluation methodology.
The best evaluations are designed to provide you with the
information you need and to match the scope and complexity of
your strategy, initiative or program’s major activities.
Evaluation Types, Methodologies and Approaches
How this chapter is organized…
3.1 Evaluation Type
3.2 Evaluation Approaches
3.3 Evaluation Methodologies
Highlights
Exercises
Page 26 | The Step-by-Step Guide to Evaluation
3.1 evaluation type
What is the purpose of your evaluation? To prove that you achieve the outcomes as intended or to
monitor if you are doing what you said you would do in your grant application? How often have you
been confused about whether or not you should be asking for a process evaluation or an outcome
evaluation? What about performance monitoring - how is this different from process or outcome
evaluation?
The type of evaluation you want depends on the stage that your strategy, initiative or program is
in and the purpose of the evaluation. Evaluation type is not the same thing as evaluation
methodology. Once you determine the type you should use, and depending on what you want to
accomplish through the evaluation process (e.g., build staff capacity, empower community
leaders, emphasize community assets), you can decide on the approach you want to take, After
that you can select from different methodologies, such as case study or quasi-experimental design.
Essentially, there are three major types of evaluation and each serves a specific function and
answers certain questions:
Performance monitoring
Process or formative evaluation
Outcome or summative evaluation
Sometimes, it could be necessary to conduct these three types of evaluation simultaneously.
The Step-by-Step Guide to Evaluation | page 27
exhibit 3-1: types of evaluation
Purpose Kinds of Questions Answered Timing
Pe rf
or m
an ce
M on
it or
in g
Ensures accountability for program activities.
Demonstrates that resources for the strategy, initiative or program, are used as intend- ed and managed well.
Monitors and reports on progress toward pre-estab- lished goals.
Provides early warning to funder and management of problems.
Have activities for the strategy, initiative or program been conducted as planned?
Have products and services been generated and provided by the effort as planned?
Has the effort accomplished what it set out to do?
Performance monitoring can be conducted throughout the strategy, initiative or program period, from beginning to end.
Pr oc
es s
or F
or m
at iv
e Ev
al ua
ti on
Seeks to understand if a strategy, initiative or program is being implemented as planned and according to schedule.
Assesses if the effort is producing the intended outputs.
Identifies strengths and weaknesses of the effort.
Critical for informing adjustments to the effort.
Has the strategy, initiative or program been implemented as planned and if not, why?
What has worked or not worked and why?
What needs to be improved and how?
Process or formative evaluation should be conducted at the start-up period and while the strategy, initiative or program elements are still being adapted.
O ut
co m
e or
S um
m at
iv e
Ev
al ua
ti on
Investigates whether the strategy, initiative or program achieved the desired outcomes and what made it effective or ineffective.
Assesses if the effort is sustainable and replicable.
What changes did the strategy, initiative or program cause or contribute to?
How did the effort cause or contribute to the changes?
How is the effort going to be sustained and replicated?
Outcome or summative evaluation should be conducted when immediate and intermediate outcomes are expected to emerge, usually after the effort has been going on for awhile, or when it is considered “mature” or “stable” (i.e., no longer being adapted and adjusted).
Page 28 | The Step-by-Step Guide to Evaluation
3.2 evaluation approaches
Once you have determined if you want to conduct performance
monitoring, a process/formative evaluation, an outcome/
summative evaluation, or some combination of the three types
of evaluation, you need to consider how you want to
approach the evaluation. In other words, what do you want to
accomplish through the evaluation process? This decision will
depend on your values, especially about who your stakeholders
are and how you engage them in the evaluation process.
An evaluation approach is the way one goes about designing,
implementing and using the evaluation. Why is it important
for you to know about the various approaches? Evaluators are
trained in certain approaches and have their own values about
how to approach an evaluation. They tend to be partial to certain
approaches whose underlying philosophy they share. You don’t
need to know everything about the various approaches. But
when an evaluator says, “I practice empowerment evaluation,”
or “I practice culturally responsive evaluation,” you do need to
understand the implications for you, your organization, your
program and the people who are supposed to benefit from
the program.
Usually, the data collection methods used for each type of evaluation and the various approaches to evaluation are the same, such as interviews, focus groups and surveys. The dif- ference lies in how the different methods are combined (e.g., which is the main method) to answer the evaluation questions.
Exhibit 3-2 describes some common evaluation approaches that you likely will encounter. While they have similarities, each approach has important distinguishing factors. You don’t have to decide which approach is the best one to use, but you must know what to expect for your organization; your strategy, initiative or program; and your evaluation. This material is reflected in the right-hand column.
If the evaluator doesn’t specify any such approach, don’t worry! It usually means that the professional is not partial to one approach and should be willing to work with you to design an evaluation that works for you, your organization and your community.
The Step-by-Step Guide to Evaluation | page 29
exhibit 3-2: unique attributes of some common evaluation approaches and what you can expect from them
Brief Description Distinguishing Attributes What to Expect
A pp
re ci
at iv
e In
qu ir
y
Focuses on a vision for what the outcomes should be and look like and then a plan toward achieving that vision. Evaluation users identify where they have had good practices in their initiative, strategy or program and how to increase these practices. This does not mean that needs or deficiencies are not addressed in this methodology; they just don’t become the major objects of inquiry. The emphasis is on a positive holistic vision versus addressing discrete problems.
The inquiry process itself is an intervention for change. The process involves answering the following types of questions:
What was your vision for what you wanted to achieve?
As you reflect on your experience with the program, what was a high point?
What did you feel was most successful?
What are the most outstanding stories or moments that made you proud?
Questions that focus on positive topics.
Positive-oriented vocabulary and rejection of a deficiency model.
Emphasis on what worked, how things can be better and ways to practice and sustain the solutions.
Cu lt
ur al
ly R
es po
ns iv
e Ev
al ua
ti on
Recognizes that cultural values, beliefs and context lie at the heart of any evaluation effort.
Explicitly ensures that the voices of people who have been historically excluded are integrated into the design, planning and implementation of the evaluation. Incorporates concepts of oppression into the design.
Allocation of time and resources to intentionally understand the program, the people affected by the program and the history of the place where the people and program are located.
Close attention to the experiences, assumptions and biases of the evaluator to ensure genuine connection to the context in which the evaluation is occurring.
Page 30 | The Step-by-Step Guide to Evaluation
Brief Description Distinguishing Attributes What to Expect
D ev
el op
m en
ta l E
va lu
at io
n
Supports the process of innovation within an organization and in its activities. Initiatives that are innovative often are in a state of continuous development and adaptation and they frequently unfold in a changing and unpredictable environment.
Emphasizes learning and use of learning to continuously refine the innovation and to be responsive to the environment and context within which the innovation is operating.
Frequent inclusion of evaluation on program agenda to continuously benefit from the findings and lessons generated by the evaluator.
Active engagement of the evaluator alongside the program team, technical assistance providers and any other supports.
Em po
w er
m en
t E va
lu at
io n
Provides organizations with the tools and knowledge that allow them to improve their programs through self-evaluation and reflection. The evaluator often serves as a coach or additional facilitator, depending on the organization’s capacity.
Program staff and participants are involved, through facilitated dialogues and discussions, in articulating the program, setting priorities for the evaluation and determining the measures and evidence needed to monitor the program’s progress and success.
Involves time on the part of staff, organizational leadership and program participants.
Builds the skills of those involved to ask evaluation questions, collect and analyze data, interpret the findings and learn from the experience and findings.
Sy st
em s-
or ie
nt ed
Ev
al ua
ti on
Views a program, initiative, strategy or policy as part of social and natural systems and subsystems that affect and are affected by the effort’s capacity to achieve its goals.
Emphasizes boundaries, relationships and perspectives within and across the systems and subsystems, and the program, initiative, strategy, policy or other entities.
Attention to the multiple systems of which the program, initiative, strategy, policy or other entity is a part.
The Step-by-Step Guide to Evaluation | page 31
Useful resources about these evaluation approaches include:
A Developmental Evaluation Primer, by J.A.A. Gamble, published by The J.W. McConnell Family Foundation in 2008. Available at http://tamarackcci.ca/files/developmental_ evaluation_primer.pdf.
An Introduction to Empowerment Evaluation, by B. Sherriff and S. Porter. Available at http://www.mrc.ac.za/crime/evaluation.pdf.
The Center for Appreciative Inquiry (http://www.centerforappreciativeinquiry.net/).
The Center for Culturally Responsive Evaluation and Assessment (http://education.illinois.edu/crea).
Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and Practice, by S. Hood, R. Hopson and H. Frierson, published by Information Age Publishing in 2015.
3.3 evaluation methodologies
Methods refer to qualitative or quantitative techniques for collecting and analyzing data.
Procedures refer to steps that must be followed to ensure an effective evaluation (e.g., selection of
people to include in the evaluation or from whom to collect data). Exhibit 3-3 describes common
evaluation methodologies and what each means for determining outcomes.
A methodology is a set or system of methods and procedures used to answer
evaluation questions.
Page 32 | The Step-by-Step Guide to Evaluation
exhibit 3-3: types of evaluation methodologies
Brief Description What It Means for Determining the Outcomes
Ca se
S tu
dy
Focuses on understanding a unit (program, site or situation) in its context, using a combination of qualitative and quantitative data. It is ideal for studying a program in-depth and how it unfolds and performs in a particular context. The findings cannot be generalized (or applied) to other situations.
The interaction between the program and context is explicitly studied. Multiple data sources are required to inform the analysis and findings. The outcomes are described within the context in which they occurred.
Cr os
s- ca
se S
tu dy
Takes several cases and looks for patterns across them to understand where the commonalities and differences lie and the possible reasons for the differences. Suitable for evaluating a program or initiative taking place in multiple sites, while accounting for the unique conditions in each site and the common conditions across them.
One or more outcomes can be explored across a number of cases to determine how they unfold in the same way or in different ways across different contexts.
Ex pe
ri m
en ta
l D es
ig n Assesses the causal effects of a program by comparing
two groups of participants - one that receives the intervention (“treatment group”) and one that does not (“control group”). Also known as randomized control trials (RCTs). Program participants are assigned at random to the treatment and control groups. The intervention must be delivered consistently to everyone in the treatment group.
Demonstrates the outcomes were achieved because of the program.
Q ua
si -e
xp er
im en
ta l D
es ig
n Is similar to experimental design except there is no random assignment of participants into treatment and control groups. Before-and-after program comparison (sometimes referred to simply as pre- and post-testing) is one of the most common forms of quasi-experimental design. Careful understanding of factors that could influence the results is critical to accurately interpret the results.
Useful for showing a certain level of evidence about the degree to which the program caused the outcomes, but avoids the ethical concerns involved in withholding or delaying treatment or substitut- ing a less effective treatment for one group of participants (as in the case of experimental design).
The Step-by-Step Guide to Evaluation | page 33
Brief Description What It Means for Determining the Outcomes
O ut
co m
e M
ap pi
ng
Assists in understanding how a strategy, initiative or program affects the individuals, organizations and groups (also referred to as “boundary partners”) it influences or with whom it interacts. Outcomes are defined as changes in the behavior, relationships, activities or actions of the people, groups and organizations of these boundary partners. Useful for answering four questions about the program: What is the vision the program is contributing to? Who are the program’s boundary partners? What are the changes are being sought and occurring? How is the program contributing to the change process?
Boundary partners are engaged in the process of determining the outcomes and in self-reflection and monitoring. The outcomes must be framed in terms of observable changes in behavior, relationships, activities and actions.
Let’s consider this example. You have been implementing a district-wide academic enrichment
program for two years and you now have funding to evaluate it. Here are some suggestions for how
you can prepare for the evaluation:
First, you probably want to conduct a process/formative and outcome/summative evaluation to
help you understand the extent to which the program was implemented as planned and the extent
to which the intended outcomes were achieved.
Second, you might prefer an external evaluator who practices culturally responsive evaluation
because the enrichment program takes place in schools located in communities that vary in
history, demographics and geographic settings. Or you might not have any preference for the
approach as long as the evaluator works with you to answer the evaluation questions and takes
into consideration the culture and context of the participating schools.
Third, you want to combine two methodologies: 1) a quasi-experimental design to compare the
grades of youth across the four schools in the district before and after participation in the
enrichment program and 2) a case study of the youth in two of the schools to understand more
deeply what contributed to the change (or lack of change).
Combining methodologies would strengthen your ability to practice evaluative thinking because
each methodology generates different information to help tell your story. Combining
Page 34 | The Step-by-Step Guide to Evaluation
methodologies, however, requires more resources and an evaluator with expertise in both
qualitative and quantitative methods. Alternatively, if you have resources, you could hire two
separate evaluators, but make sure you bridge their efforts so that at the end, you have one
coherent story and not two separate summaries to share. You also could ask the first evaluator you
hire to subcontract to a second evaluator with the required expertise.
Useful resources about evaluation methodologies include:
Case Study Research: Design and Methods, by R.K. Yin, published by SAGE, Inc. in 2009.
Cross-case Methodology: Bringing Rigour to Community and Systems Change Research and Evaluation, by K.S. Lee and D.M. Chavis, published in the Journal of Community and Applied Social Psychology, 2011, Vol. 22, Issue 5. Available at http://onlinelibrary.wiley.com/ doi/10.1002/casp.1131/full.
Outcome Mapping Learning Community at http://www.outcomemapping.ca/.
Outcome Mapping, by S. Earl, F. Carden and T. Smutylo, published by the International Development Research Centre in 2001. Available at http://www.outcomemapping.ca/ download/OM_English_final.pdf.
Study Designs for Program Evaluation, published by Aguirre Division, JBS International in 2006. Available at http://www.pacenterofexcellence.pitt.edu/documents/study_designs_for_ evaluation.pdf.
HIGHLIGHTS
At the start, you should determine what
type(s) of evaluation you need -
performance monitoring, formative or
process evaluation, or summative or
outcome evaluation. The evaluation type
depends on the stage of your program.
Sometimes, evaluators have a preference
for a certain evaluation approach. One
approach is not necessarily better than
the other. It is important to understand
the unique strengths of each approach
and which one best suits your evaluation
goals.
Once you have determined the type(s) of
evaluation you need, you can combine
different evaluation methodologies,
depending on your resources, to answer
your evaluation questions.
The Step-by-Step Guide to Evaluation | page 35
exercises
1. evaluation seeks to know if the program caused or contributed to
behavioral changes among the youth it serves.
A. Outcome
B. Performance monitoring
C. Process
D. Proactive
E. None of the above
2. evaluation seeks to know if the program met the milestones in its
work plan.
A. Outcome
B. Performance monitoring
C. Process
D. Proactive
E. None of the above
3. Which of the following is true about process or formative evaluation?
A. It is less time-consuming
B. It is easy to implement
C. It can be conducted throughout the program period, from beginning to end
D. It should be conducted only when the program is getting started
E. It should be conducted at the end of the program
4. Let’s consider the example below.
The advocacy initiative to fight for early childhood education grew out of an interest shared by
a group of nonprofit organizations. A collaborative was established and your organization
decided to be both the fiscal agent for the grant from the community foundation and the
Answers: 1A; 2B; 3C
Page 36 | The Step-by-Step Guide to Evaluation
coordinator for the collaborative and its activities. According to the work plan approved by the
community foundation, for the first year, you and the collaborating partners would develop an
infrastructure for decision-making, communications and data sharing. Once the infrastructure
has been established, the collaborative would develop an advocacy agenda and action plan.
The remaining two years of the grant would focus on implementing the agenda and action plan,
including affecting policies and practices relevant to early childhood education.
The community foundation was interested in the following evaluation questions:
What facilitated or challenged the establishment of the collaborative and its preparation
for action?
How was the community engaged in developing the advocacy agenda?
What policy and other outcomes were achieved or not achieved by the end of three years?
Why?
Given the information above, answer the following:
A. What type of evaluation should you use - performance monitoring, formative, summative or
some combination of these? Why?
B. What evaluation approach do you think would be suitable for this evaluation? Why?
The Step-by-Step Guide to Evaluation | page 37
4 Overview of the Evaluation Process that Reflects Evaluative Thinking
Page 38 | The Step-by-Step Guide to Evaluation
introduction Evaluation can be somewhat intimidating for those unfamiliar with
its usefulness or general process, which has several stages
These stages reflect evaluative thinking:
Identifying assumptions about what you think works and
doesn’t work and why.
Posing thoughtful questions about what change you expect to
see during and after you implement your strategy, initiative or
program.
Pursuing deeper understanding through reflection and dialogue
Communicating what was learned without underestimation or
exaggeration (i.e., without playing down or overstating what
you did or achieved).
Making informed decisions about future action.
Regardless of whether you are evaluating a strategy, initiative,
program, or even your organization - you can apply the same
sequence of evaluation steps. Each step, however, will be more
complicated as the entity to be evaluated becomes more complex;
evaluation of a single project is simpler than evaluating a multi-site
initiative, a strategy, or an effort to change a policy, system or
community. If planned properly, evaluation easily can be understood
and received positively, especially if you engage your stakeholders
throughout the evaluation process.
How this chapter is organized…
4.1 Overview of the
Evaluation Process
4.2 Creating a Learning
Organization
4.3 Stages of the Evaluation
Process
4.4 Navigating Evaluation
Choices
4.4.1 Capacity, Program or
Population Outcomes
4.4.2 Focus or Unit of Analysis
4.4.3 Internal Versus External
Evaluation
4.4.4 Power Dynamics Among
Stakeholders
Highlights
Exercises
The Step-by-Step Guide to Evaluation | page 39
4.1 overview of the evaluation process
An evaluation that reflects evaluative thinking is the systematic process of telling your story by:
Identifying assumptions about why you think your strategy, initiative or program will work.
Determining what change you expect to see during and after you implement what you set out
to do.
Collecting and analyzing data to understand what happened.
Communicating, interpreting and reflecting on the results.
Making informed decisions to improve what you plan to do next.
Evaluation should be viewed as a collaborative process that involves any or all of these
stakeholders, depending on their roles: funders (public, private); staff; board members; program
participants or constituents of your organization; community leaders; collaborating organizations
or partners; and policymakers. If done properly, it helps tell the story of your strategy, initiative or
program through a continuous cycle of asking, planning, acting, reflecting and improving.
Evaluation should emphasize utility (i.e., findings should be practical and useful for end users)
and inform decision-making and capacity building strategies for involved parties. Therefore your
evaluation should consider:
Implementation and impact as an ongoing process and what your actions should build on
instead of replace.
Existing information systems to promote improvements that you can sustain over time.
Previous lessons learned from research along with the current values and realities of all the
stakeholders involved.
A logic model to illustrate how your strategy, initiative or program is supposed to create change
with a design that is driven by questions you want to answer, existing capacity,
feasibility and appropriateness of approach and methodology.
Page 40 | The Step-by-Step Guide to Evaluation
A logic model is a graphic representation of the theory of change that illustrates the
linkages among resources, activities, outputs, audiences and short-, intermediate-
and long-term outcomes.
A PREPARE for conducting an evaluation
B DETERMINE stakeholders and how and when to engage them
C IDENTIFY assumptions and determine what will be di�erent (theory of change and logic model)
D DEVELOP evaluation plan (logic model, measurement, framework, etc.)E COLLECT and analyze data
F COMMUNICATE results and understand what happened (interpret �ndings and facilitate learning)
G MAKE informed decisions (improve actions and next steps)
Stakeholders
Exhibit 4-1: evaluation process that reflects evaluative thinking
The Step-by-Step Guide to Evaluation | page 41
Useful resources for planning an evaluation include:
Introduction to Program Evaluation for Public Health Programs: A Self-study Guide, by the Centers for Disease Control and Prevention, published in 2011. Available at http://www.cdc.gov/ eval/guide/cdcevalmanual.pdf.
Planning an Evaluation: Using the Rainbow Framework, by Better Evaluation, published in 2013. Available at http://betterevaluation.org/sites/default/files/Rainbow%20Framework%20 -%20compact%20version.pdf.
Project/Programme Monitoring and Evaluation Guide, by the International Federation of the Red Cross. Available at http://www.ifrc.org/global/publications/monitoring/ifrc-me-guide- 8-2011.pdf.
Evaluating Community Programs and Initiatives, by The Community Toolbox. Available at http://ctb.ku.edu/en/tablecontents/index.aspx.
4.2 creating a learning organization
Evaluative thinking and evaluation contribute to an organization’s capacity to become a “learning
organization.” A learning organization is one in which the leadership and staff continually develop
their ability to achieve their desired results through new and betters ways of thinking and
problem-solving.
For successful implementation and use of evaluation, your organization should develop or
strengthen the following:
Knowledge about the purpose, benefits and risks of evaluation
Commitment to use data to support decision-making regularly and consistently and to support
a culture of evaluative thinking
Page 42 | The Step-by-Step Guide to Evaluation
Internal systems and processes to support learning
Skills in managing, interpreting and using information
Resources to hire an internal evaluator (a staff person with the knowledge and skills to do
evaluation) or an external evaluator (an evaluator who is not an employee) and support staff
and other stakeholder involvement
Relationships with individuals and organizations with expertise in evaluation.
A useful resource about learning organizations is Peter Senge and the Learning Organization, by M.K. Smith, published in 2001. Available at http://infed.org/mobi/peter-senge-and-the- learning-organization/
4.3 stages of the evaluation process
The process starts with preparing for the evaluation (A). This involves asking questions, including:
Who is funding the evaluation and why?
Who benefits from the evaluation?
What are the potential risks?
What else is happening at the same time that could affect what you are doing and the
evaluation?
What type of evaluation you need - performance monitoring, formative/process summative/
outcome or all three?
What evaluation approach you prefer, if any, and which evaluation methodology is most
appropriate for your needs?
Do you need an internal or an external evaluator?
The Step-by-Step Guide to Evaluation | page 43
During this stage, the capacity of the organization or collaborative (if it involves partners) should
also be assessed so the evaluation can be most appropriate and useful. Chapter 5 describes in
more depth what you need to consider when preparing for an evaluation.
Once you have prepared for the evaluation, you want to determine who are the stakeholders of the
evaluation and how and when to engage them (B). For the evaluation process to be a collaborative,
useful learning process, all stakeholders should be identified and engaged accordingly to provide
multiple perspectives about the main issues that could affect the evaluation, and about what they
want to know from the evaluation. Otherwise, the evaluation is likely to be designed based on the
needs and interests of only a select few stakeholders - usually the ones with the most
power - and could miss other important questions and issues of stakeholders who are not included
in the design and planning process.
Critical questions specifically related to the evaluation design that stakeholders can help answer
include, but are not limited to:
What results do you expect from the strategy, initiative or program?
What does success look like? How do you know when you have achieved it?
What factors might help or hinder that success?
Who or what are the best data sources?
Will the evaluation be beneficial or harmful to the community and why?
What are some of the traditions, social norms, or dynamics that should be considered when
the evaluator or staff are collecting data?
Is there potential for the evaluation or evaluator to be perceived negatively, why and what can
be done to reduce this likelihood?
Page 44 | The Step-by-Step Guide to Evaluation
Additional questions that should be asked of you and
stakeholders at the outset also could include:
What do you, other staff, partners and key stakeholders need
to know more about?
What decisions do you feel you need to make, for which you
need more information?
What will you do with the answers to your questions? (Play
out various scenarios, depending on the various answers
you might find.)
Who will make the decisions and when?
What issues related to these decisions are likely to surface?
How are decisions made in this organization or
collaborative?
What other factors may affect the decision-making process?
How will you and others know if you used the evaluation
results and process as planned?
Staff, partners and stakeholders are more likely to use
evaluation if they understand and feel ownership over the
evaluation process. Therefore, the more people who have
information about the evaluation and actively participate in the
process, the easier it will be to use the results for program
improvement and decision-making. However, because of the
many individual and organizational obstacles to using
information and testing assumptions about an effort’s
effectiveness (e.g., fear of being judged, concern about the time
and effort involved, resistance to change, dysfunctional
communication and information-sharing systems, unempowered
staff, turf issues among partners), you will need to engage in
discussion and reflection about the specific obstacles to using
information within your organization or collaborative.
The Step-by-Step Guide to Evaluation | page 45
Identify stakeholder role(s) in evaluation planning, implementation, interpretation of
results and decision-making about next steps.
Review list of stakeholders to ensure all appropriate stakeholders are included.
Understand and respect stakeholders’ values.
Create a plan for stakeholder involvement.
Identify areas for stakeholder input.
Bring stakeholders together as needed.
Target key stakeholders for regular participation.
Ask stakeholders to suggest evaluation questions.
checklist Tips for engaging stakeholders:
The next major stage in the process is to identify the assumptions underlying your idea for why the
strategy, initiative or program will bring about the change you want to see and what will be differ-
ent as a result (C). Developing a theory of change and logic model is very useful during this stage
of the evaluation process.
A theory of change explains the links between activities and outcomes and how
and why the desired change is expected to come about, based on past research or
experiences. A logic model is a graphic representation of the theory of change that
illustrates the linkages among resources, activities, outputs, audiences and short-,
intermediate- and long-term outcomes.
Page 46 | The Step-by-Step Guide to Evaluation
Useful resources for developing a theory of change and logic model include:
Logic Model Development Guide, published by the Kellogg Foundation in 2004. Available at http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation- logic-model-development-guide.aspx.
Point K Tools: Logic Model Builder, by Innovation Network. Available at http://www.innonet.org/?section_id=64&content_id=185.
Theory of Change: Guided Example - Project Superwomen, published by ActKnowledge and the Aspen Institute Roundtable on Community Change in 2003. Available at http://www.theoryofchange.org/pdf/Superwomen_Example.pdf.
Theory of Change: A Practical Tool for Action, Results and Learning, by J. Reisman and A. Gienapp, published by Organizational Research Services in 2004. Available at http://www.aecf.org/upload/publicationfiles/cc2977k440.pdf.
Many evaluation experts agree that using a logic model is an effective way to help facilitate
success because the model is developed by all stakeholders, including staff, participants and the
evaluator. (For more detail on logic models, see the W.K. Kellogg Foundation’s Logic Model
Development Guide.) Consequently, a model provides participants with the opportunity to share
and discuss their assumptions, develop a common understanding of the change process and
become clearer about terms and definitions. The logic model also:
helps organize and systematize planning, management and evaluation functions;
guides the development of a measurement framework, which identifies the relevant indicators,
measures, data sources, data collection methods and frequency of data collection for each
component of the logic model;
makes explicit the contextual conditions that need to be changed or that could influence the
change process and outcomes; and
guides the reporting of the evaluation findings and reflection process.
It doesn’t matter if you are evaluating a strategy, initiative or program, or even your organization.
If you are trying to impact outcomes at the individual, organization, system or community level, a
logic model is helpful. Developing a logic model with stakeholders helps you, your staff, your board
members and other participants continually learn about what you are trying to accomplish and how
to become better at it.
The Step-by-Step Guide to Evaluation | page 47
Once the theory of change and logic model have been developed, evaluation questions can be
determined and a measurement framework can then be created to identify the data sources;
frequency of data collection; and the qualitative and quantitative measures of change (D). All of
this information, along with data collection methods, analysis strategies and plans for reporting
and communicating the findings, are compiled to form the evaluation plan.
After the evaluation plan is completed, data collection can start, followed by analysis (E).
Some analyses can be linear and straightforward. For example, if the goal of the evaluation is to
find out the impact of your organization’s after-school program, then the sequence of activity is
more or less linear, as noted below.
Collect data from the youth and parents before the program starts.
Collect data halfway through the program or at the end of the program (or both).
Analyze the data using statistics (e.g., percentages) and coded qualitative data.
Qualitative data is information that can be
collected or captured in text form.
Other types of analyses can be more complicated. For
example, if the goal of the evaluation is to find out the
impact of your policy advocacy or systems change
strategy, then the sequence of activity would be:
Map the system before your strategy.
Determine what parts of the system you want
to change.
Identify the factors that could facilitate or prevent
that change and whether the factors are within or
outside your control.
Page 48 | The Step-by-Step Guide to Evaluation
Collect data from multiple sources at key junctures to help you understand if, and when, the
change occurred.
Analyze the data using qualitative and/or quantitative methods.
Map the system again to see if and where the change happened.
If you are collecting survey data, review and “clean” data before beginning your analysis.
If you don’t have experience with survey data, you should consult with your evaluator
about cleaning the data and what to do if the data are incomplete or missing (e.g., some
people did not answer all the questions).
If you are collecting qualitative data, review the notes and if necessary, follow up with
the individual or review the situation to fill in any information gaps. If you are working
with an evaluator to collect qualitative data, ask him or her to explain the procedures for
ensuring high quality data.
Leave enough time and money for analysis. Qualitative analysis requires more resources
because it takes more time to read, code and summarize text than it does to enter the data
and calculate the percentages or other statistics involved in quantitative analysis.
Be clear about the use of the word “significant” to describe your findings. If it is not a
statistically significant finding but is still important, you can use the term “practically
significant” or “programmatically significant.”
checklist Tips for analysis:
The next few stages after data collection and analysis relate to communication and interpretation
of findings (F) and making informed decisions about improvements and next steps (G). Chapter 9
discusses the reporting, interpretation and reflection of evaluation findings in depth. This should
involve your stakeholders who could have insights into findings (e.g., why things turned out the
way they did), facilitating everyone’s learning and determining implications of findings and
The Step-by-Step Guide to Evaluation | page 49
lessons learned. The reflection and dialogue, especially about whether the change process
occurred the way everyone expected, can then inform decisions about improvements and next
steps. At this stage, it is helpful to go back to the theory of change and logic model and adjust your
effort based on the new insights.
4.4 navigating evaluation choices
Tensions often arise in evaluation because individual stakeholders have their own priorities for
the evaluation. The funder, evaluator, implementers and other key stakeholders should
continuously communicate to clarify and manage expectations. As events unfold around the
strategy, initiative or program causing it to adapt to the stakeholders’ needs, these discussions are
critical to prevent misunderstandings. The following subsections cover the typical tensions that
emerge and the typical choices the funder, evaluator and other key stakeholders have to make
during evaluation.
Let’s use this example to illustrate the typical tensions: You received a three-year grant to build
your organization’s cultural competency to better respond to the health needs of recent
immigrants and refugees from Southeast Asia (namely Vietnam, Cambodia, Laos and Myanmar)
and help reduce the health disparities they encounter. This grant is part of a national initiative
aimed at building the capacity of health organizations to respond to the growing diversity of
communities across the country. The funder expects to see improved capacity of the funded
organizations to work with racially, ethnically and culturally diverse communities as well as
reductions in health disparities.
Page 50 | The Step-by-Step Guide to Evaluation
4.4.1
Capacity, Program or Population Outcomes
Capacity outcomes. You should consider the following before even starting the evaluation:
Clarify early on the answer to these questions: “Capacity for what or to what end?” and
“Whose capacity?” Your funder, any of the consultants they hire and you must agree about
what capacity outcome is realistic to expect within the grant period and to what end. Using the
example above:
First clarify “Whose capacity is being built?” In the example above, it is clearly your
organization’s capacity.
The “capacity” is your organization’s ability to respond to the health needs of diverse popu-
lations and the “end” is a reduction in health disparities. In three years, it might be feasi-
ble to observe changes in your organization’s board and staff composition that were made
to be more culturally diverse, and in how your organization engages the community (the
capacity outcomes). However, it might not be possible to observe changes in the health
disparities (the ultimate change or difference you want to make) experienced by Southeast
Asians as a population in that time period.
Let’s consider another example - one that reflects a common issue in the field of community and
systems change: Funders in this field often make grants to build community capacity to mobilize
residents to address a problem they face, such as inadequate early childhood education. In this
case, funders, grantees and key community stakeholders must agree on what capacity outcome
is realistic to expect within three years, what difference they expect to see (if any) with early child-
hood education in their community and, equally important in this situation, what constitutes
“community.” Does “community” refer to nonprofit organizations, residents, public agencies,
business owners, etc.? Answering this question early is important because the coalition’s success
could be wrongly judged by the composition of its membership. In one to two years, it may be
feasible to create a community coalition that focuses on early childhood education issues and on
the development and distribution of a report card (posted online and disseminated in print
format) on early childhood education outcomes - a form of community capacity. In three years,
the coalition could have developed the ability to engage and mobilize the larger community into
action. However, it might not be possible within that time period to observe child outcomes (e.g.,
readiness to learn, emotional well-being) that can be traced back to the coalition.
The Step-by-Step Guide to Evaluation | page 51
A theory of change and logic model can help show the link between improved capacity and the
changes you anticipate making in people’s lives, in a policy, system or community, and the
sequence of events between the improvement and the changes.
Program (also strategy or initiative) outcomes. These outcomes are the benefits that participants
experience as a result of the strategy, initiative or program; the outcomes are limited to
participants or the people “touched” by the strategy, initiative or program. In the example
above regarding a three-year grant to build your organization’s cultural competency to be able to
respond better to the health needs of recent immigrants and refugees from Southeast Asia:
The outcomes are limited to immigrants and refugees who received services from your
organization.
The outcomes can tell if your organization was effective in engaging them as recipients of your
services and improving health outcomes such as reduction in their blood sugar and blood
pressure levels, fewer asthma attacks and regular and on-time mammograms and pap smears
among the women.
Let’s say you decided to expand your effort to include - with additional resources from another
funder - a community-wide campaign to promote pedestrian safety after learning about the
number of accidents among recent immigrants and refugees. The campaign could result in few-
er pedestrian-related injuries or deaths. The results of the campaign are limited to the people
who were “exposed” to the campaign.
Population outcomes. Population outcomes are:
Changes in the condition or well-being of children, families or communities (e.g., increase in
employment and graduation rates or decreased infant mortality or teen pregnancy rates).
Long-term results of the combined efforts of a number of different strategies, initiatives,
programs and organizations. It takes a comprehensive, multi-year effort to try to change
population outcomes.
If you have a multifaceted strategy aimed at changing systems and that is scalable, sustainable
and impactful at the population level, then the evaluation of your strategy should consider
outcomes such as increased high school graduation or decreased infant mortality rates. If you are
implementing a single program designed to improve the academic achievement of a group of high
school students, it is not realistic to expect outcomes for anyone else beyond the students with
whom you are working. In the example above about a three-year grant to build your organization’s
Page 52 | The Step-by-Step Guide to Evaluation
cultural competency, it would not be realistic to expect changes in health disparities at the popu-
lation level because it would require more than the efforts of your organization to bring about such
change. Instead, it would require a large-scale effort to change systems and community conditions.
4.4.2
Focus or Unit of Analysis
The focus or unit of analysis is the target of change and the
entity that is being analyzed in the evaluation. This entity could be individuals, an organization,
a program model, system, policy or community; the evaluation can focus on one or more of these
units at the same time. In the example previously
mentioned regarding building the cultural competency of health
organizations responding to the needs of Southeast Asian
immigrants and refugees, the units of analysis are both the
organization and the immigrants and refugees served.
Other examples include the following scenarios:
If your program is designed to improve the reading
proficiency of children, the units of analysis are the program and the children.
If you are trying to change a school district policy to
purchase local fresh produce and get rid of vending
machines that sell unhealthy snacks, the units of analysis are the policy, system (from how the
produce is delivered to how the food is prepared and served to children in the schools), the
organization (the schools) and the individual (the children).
Depending on the scope and timeframe of the effort and expectations of all the key stakeholders,
you can decide to focus on the system, policy, organization, individual or all four.
The Step-by-Step Guide to Evaluation | page 53
When the focus or unit of analysis is the individual:
You are documenting the changes that individuals experience.
Your evaluation will usually want to assess two things: (1) the degree to which the strategy,
initiative or program is being implemented accordingly and (2) if the individual participants
experienced the desired outcomes. In the earlier example about your organization and the
grant to improve its cultural competency to better serve Southeast Asian immigrants and
refugees, this would include
Individual
Program
Organization
Community
Policy
Larger Formal Systems
CO M
PL EX
IT Y
O F
M ET
H O
D O
LO G
IE S
Exhibit 4-2: different units of analysis
Page 54 | The Step-by-Step Guide to Evaluation
measuring whether individuals served improved their blood sugar and blood pressure
levels or have more regular mammograms. Other types of individual-level outcomes could be:
The young men participating in the program have better grades as a result of the mentoring
activities.
The parents are more involved in their children’s school as a result of the parent leadership
institute.
The heads of households report they buy more fresh fruits and vegetables per week due to
the healthy eating initiative.
The low-income families report using the services offered by the banks in their community.
When the focus or unit of analysis is the organization:
In the example used about increasing cultural competency, you might examine changes in your
organization’s priorities, policies and practices.
To evaluate change at the organization level, you want to hire evaluators skilled in evaluation
and knowledgeable about organizational development. These evaluators should be able to
collect and analyze data on key outcome indicators such as how successful your organization
has been in engaging and sustaining the involvement of different groups of people and the
degree to which staff of diverse racial, ethnic and cultural backgrounds feel valued
and respected.
When the focus or unit of analysis is an initiative or program:
Some grants involve the development and implementation of a new initiative or program
designed to meet the needs of a particular target population. In the cultural competency
The Step-by-Step Guide to Evaluation | page 55
example above, this could include testing a new curriculum for training staff to become more
culturally competent in working with the Southeast Asian community.
Other grants may adapt existing initiatives or programs for new locations or client groups. For
example, you may work with a community to create and test a cultural and linguistic adaptation
of a financial literacy program tailored to the community’s needs.
The above initiatives or programs share common traits: They are creating and pilot testing new
initiatives or programs without a proven track record of effectiveness. As a result, they need
ongoing monitoring and improvement as the initiative or program is refined.
To evaluate these new or adapted initiatives or programs, you could conduct a case study to
understand:
How the effort was implemented.
What elements made it effective.
What worked and didn’t work and why.
What knowledge, skills and other capacities are required of the staff to implement the
effort.
Most important, how to refine the effort.
Much of the data collected could be qualitative. When the initiative’s or program’s elements
have become final and stable, you can conduct an outcome or summative evaluation and
combine the use of a quasi-experimental design with the case study methodology to assess its
effectiveness.
A quasi-experimental design assesses the causal effects of a program by comparing
two groups of participants (a “treatment” group and a “comparison” group) or by
comparing data collected from one group of participants before and after they par-
ticipated in the program. There is no random assignment of participants into the two
groups, unlike studies using an experimental design.
In a different situation, you might select and implement an initiative or program that already
has been proven effective through rigorous summative/outcome evaluations (also called an
evidence-based program). You could be using a financial literacy curriculum that already has
Page 56 | The Step-by-Step Guide to Evaluation
been tested and shown to be effective. The curriculum is ready for an outcome or summative
evaluation and you can use before-and-after comparisons (i.e., a type of quasi-experimental
design) to determine its effectiveness. You also can combine this methodology with a case
study to collect additional information to answer questions about the quality of the curriculum.
When the focus or unit of analysis is a policy:
These could be situations that involve education campaigns or other mobilization efforts to
advocate for policy change at the local, county or state levels. In the pedestrian safety
campaign example used previously in this chapter, you might discover there is no policy about
use of mobile phones while driving and decide to get involved in an effort to advocate for a
policy to stop their use. This type of initiative calls for evaluators with the policy evaluation
expertise and experience to conduct case studies of the local transportation system, political
engagement, community mobilization process and policy outcomes.
The evaluation will have to rely on multiple data sources and methods (e.g., interviews with
legislators, focus group with advocates, survey of community residents) to assess how the
campaign and external circumstances shaped policy decisions.
Sometimes, when evaluating policy change, you might not observe a new or revised policy,
but instead could find that a policy contrary to what you were advocating for was blocked. This
could still constitute a positive outcome.
Be clear from the outset about your theory of change, logic model and outcome measures.
Useful resources about evaluating policy change include:
Ten Theories to Inform Advocacy and Policy Change Efforts, written by S. Stachowiak and published by Center for Evaluation Innovation and ORS Impact in 2013. Available at http:// www.evaluationinnovation.org/sites/default/files/Pathways%20for%20Change.pdf.
The Evaluation Exchange, Advocacy and Policy Change, Volume XIII, No. 1, published by The Harvard Family Research Project in 2007. Available at http://www.hfrp.org/var/hfrp/ storage/original/application/6bdf92c3d7e970e7270588109e23b678.pdf.
The Step-by-Step Guide to Evaluation | page 57
When the focus or unit of analysis is system:
Some grants might be designed to improve a local system. For example, there could be a
health equity initiative that calls for increased access to health care for underserved groups
such as the immigrants and refugees served; low-income families; racial and ethnic
minorities; and people with different sexual orientation. This means changing aspects of the
health system, including how people are reached and informed about their eligibility for health
insurance; how they enroll in insurance plans; their understanding and use of preventive care
services; and the cultural and linguistic competency of health care providers.
Another example could be a food system initiative that requires schools to purchase fresh
produce from local farmers and change the lunch menus for children in subsidized school
lunch programs. This requires changing school district policies, ensuring that local farmers are
trained and equipped to handle and deliver food to schools, and educating children and their
parents about healthy eating.
You can conduct both formative/process and summative/outcome evaluations and employ an
outcome mapping methodology, as many partners must be engaged at different points in the
system to help make the change.
You want to create a picture or map of the parts of the system you are trying to change early in
the effort. Otherwise, you risk evaluating the wrong thing.
You also want to clarify what the change is. For example, the relationship among the parts that
you are attempting to change; new or improved functions of those parts; or new or improved
policies, procedures or practices that support those parts of the system. This clarity will guide
your evaluation questions and data collection and analysis. If you have the time and resources,
you also can assess the experiences of the people who interacted with the system before and
after your effort to change the system or parts of the system.
Page 58 | The Step-by-Step Guide to Evaluation
Useful resources for evaluating systems change include:
A Practical Guide to Evaluating Systems Change in a Human Services System Context, written by Latham and published by Center for Evaluation Innovation and Learning for Action in 2014. Available at http://www.evaluationinnovation.org/publications/practical-guide- evaluating-systems-change-human-services-system-context.
Evaluating Systems Change: A Planning Guide, written by M. Hargreaves and published by Mathematica Policy Research in 2010. Available at https://www.mathematica-mpr.com/
our-publications-and-findings/publications/evaluating-system-change-a-planning-guide.
When the focus or unit of analysis is a community:
In comprehensive community change initiatives, the unit of analysis is the community. (This
would not be the case in the example about your organization’s cultural competency.)
You want to take the time to define “community” at the beginning of the initiative so the
evaluation does not look for change in the wrong places.
Similar to when the unit of analysis is a system, you want to clarify the nature of the change
- for example, change in relationships among the public, private and nonprofit organizations
that serve the community; or in the physical structure or social norms of the community; or in
the ways different racial and ethnic groups interact, collaborate and relate to each other.
Again, you should have a theory of change and logic model for what you are trying to do, and
the expected outcomes.
Communitywide surveys are commonly used in evaluations where the unit of analysis is the
community, and they can be expensive to administer. Therefore, you must use an appropriate
sampling strategy to select a group of people from the larger community (the sample) that you
believe will reveal information about the larger group (the population).
Useful resources for evaluating community and systems change include:
Evaluating Community Change, published by Grantmakers for Effective Organizations in 2014. Available at http://docs.geofunders.org/?filename=geo2014_indicators_framework.pdf.
Approaches to Measuring: Community Change Indicators, written by L. Weaver, P. Born and D. Whaley, published by Tamarack - An Institute for Community Engagement in 2010. Available at http://tamarackcci.ca/resource-library/evaluation/approaches-measuring- community-change.
The Step-by-Step Guide to Evaluation | page 59
4.4.3
Internal Versus External Evaluation
Internal evaluation is conducted by a staff person within the
organization that is conducting the program or entity being
evaluated, whereas an external evaluation is conducted by
an evaluator who is not an employee of that organization.
Whether an organization should conduct an internal or
external evaluation usually depends on the available
resources, qualifications of the internal or external
evaluator, scope of the evaluation and the funder’s
requirements. However, other factors are equally important in
making this decision:
External evaluators can bring a broader perspective while
internal evaluators tend to have intimate knowledge about
the context that the strategy, initiative or program is
operating within.
External evaluators can be perceived as threatening
while internal evaluators can be perceived as being less
objective.
Sometimes, you have no choice because the funder requires
an external evaluation. Regardless of whether an internal or
an external evaluator is selected, clear lines of accountability
must be established from the outset.
Useful resources on hiring evaluators include:
• Hiring and Working with an Evaluator, by the Juvenile Justice Center published in 2016. Available at http://www.jrsa.org/pubs/juv-justice/evaluator.pdf.
• Selecting an Evaluator, by Community Science for the W.K. Kellogg Foundation. Available from [email protected].
Page 60 | The Step-by-Step Guide to Evaluation
4.4.4
Power Dynamics among Stakeholders
As mentioned before, evaluation generates knowledge, and knowledge is power. For example,
when a group of people in a community learn about how well or poorly certain teaching techniques
in an early childhood education program are working, they have the “power” to use this knowledge
to advocate for change. This characteristic of evaluation engenders power dynamics based on:
Who is a part of the evaluation.
How the data were collected and from whom.
Who owns the data.
Who gets to see and use the findings.
Who benefits or fails to benefit from the findings.
Examples of where power affects the evaluation process and evaluation findings include these
actual scenarios:
An evaluator makes decisions about the measures of success without consulting the program
staff or community leaders, and suggests to the staff and leaders they don’t have the
educational qualifications to make the decision.
In a breach of ethics, a program director tells service recipients they will get additional services
if they participate in the evaluation.
Community leaders assume the evaluation report is too “sophisticated” for community mem-
bers and therefore don’t think it’s worth sharing.
Leaders hold discussions to address the findings at times and locations that are inconvenient
for the families who are affected by the findings.
The Step-by-Step Guide to Evaluation | page 61
Identify and engage key stakeholders early on.
Place equal value on experiential and theoretical knowledge without diminishing the
effect of either.
Take the time to think carefully as you develop and frame the evaluation questions to
avoid making wrong or biased assumptions about the people who are supposed to
benefit from the program, initiative, strategy or organization.
Integrate into the evaluation plan a communications strategy about how the findings will
be used and who will convey those findings.
checklist Tips to manage the power dynamics:
HIGHLIGHTS
A good evaluation is useful and
responsive to the organization’s needs;
works within the organization’s capacity
to manage and use information; engages
an organization’s designated staff in deci-
sions about the evaluations; and is scientif-
ically rigorous and honest.
Evaluation can help tell the story of your
program, initiative, strategy or organiza-
tion through a continuous cycle of asking,
planning, acting, reflecting and improving.
Becoming familiar with the cycle of eval-
uation and its stages will help you plan
your evaluation.
Take time to consider the tensions that
often come up in an evaluation - level of
change, focus or unit of analysis, use of an
internal or external evaluator and power
dynamics among stakeholders.
You, your stakeholders and evaluator
should address the question of usefulness
from the outset.
Page 62 | The Step-by-Step Guide to Evaluation
exercises
1. An organization with a culture of data and learning is more likely to benefit from evaluation.
A. True B. False
2. Consider the culture of data and learning in your organization. Does it exist? If so, what are
some of the characteristics of this culture in your organization? If not, what can you do to
promote such a culture in your organization?
3. You can’t develop a logic model until after you see how the program works.
A. True B. False
4. What program or initiative are you involved in that could benefit from a logic model? Why?
5. Let’s consider this example adapted from the American Evaluation Association
(see www.eval.org): A state department of education is funding an initiative to improve
academic outcomes through the development and implementation of school improvement
plans by school districts. The department contracted a technical assistance provider to assist
grantees. You received a two-year grant to implement the initiative in your school district. The
state department of education provided $45,000 for a two-year evaluation to address the
following evaluation questions:
How were schools responding to and using the technical assistance services?
Were schools changing practices with respect to planning, implementation and
The Step-by-Step Guide to Evaluation | page 63
monitoring?
Did the use of the technical assistance provider’s services have an effect on academic
outcomes?
The evaluation team proposed a methodology to answer the primary evaluation questions. The
methodology included five data collection methods:
A telephone survey of all team members from the five school planning teams in your
school district.
A review of each local school’s pre- and post-initiative school improvement plan.
An analysis of school-level academic outcomes before and after the grant.
Focus groups with principals from all five schools.
Focus groups with a sample of parents from all five schools.
The principals, the parents, the five schools and the school district can be viewed as:
A. Units of analysis
B. Separate entities
C. Stakeholders with conflicting interests
D. All of the above
E. None of the above
6. Consider the choices you have to make about evaluation as described in Section 4.4. Have you
encountered situations where you have had to make decisions about these choices? What did
you decide and why? Knowing what you know now, might you have selected a different choice
and why?
Answers: 1A; 3B; 5D
Page 64 | The Step-by-Step Guide to Evaluation
5
The Step-by-Step Guide to Evaluation | page 65
Preparing for the Evaluation
How this chapter is organized…
5.1 Why Might You Consider
an Evaluation of Your
Strategy, Initiative or
Program?
5.1.1 Considerations if Your
Evaluation Is Part of a
National or Statewide
Evaluation
5.1.2 Considerations if the
Findings from Your
Evaluation Have
Significant Implications
on a Theory, Practice
or Policy
5.1.3 Getting Ready for
Evaluation Whether or
Not It Is A Funding
Requirement
5.2 Who Benefits from the
Evaluation and What are
the Potential Risks?
5.2.1 Consideration of Different
Types of Stakeholders
5.2.2 Use of Evaluation by
Different Types of
Stakeholders
continued on following page
introduction Whether you have conducted
evaluations before or not, you must have asked
yourself, “Who should be involved in the evaluation?” “Who do I
know with the skills and expertise to conduct an evaluation?” “Can
we do it ourselves without hiring someone?” “How much does an
evaluation cost?” These are important questions you should consider
before proceeding with
evaluating your strategy, initiative or program. This is the first step
in evaluative thinking and preparing for conducting an evaluation.
A PREPARE for conducting an evaluation
B DETERMINE stakeholders and how and when to engage them
C IDENTIFY assumptions and determine what will be di�erent (theory of change and logic model)
D DEVELOP evaluation plan (logic model, measurement, framework, etc.)E COLLECT and analyze data
F COMMUNICATE results and understand what happened (interpret �ndings and facilitate learning)
G MAKE informed decisions (improve actions and next steps)
Stakeholders
Page 66 | The Step-by-Step Guide to Evaluation
5.1 why you might consider an evaluation of your strategy, initiative or program
Depending on who is funding the evaluation, the requirements
could vary greatly:
Sometimes, if evaluation is a grant requirement, you might
find it difficult to look beyond doing just what is needed to
fulfill the requirement, especially if you have limited
capacity. You collect the data because your funder expects
it and you report the data at the end. You might or might
not take the time to consider how to use the data more
effectively to improve your program.
If it is a not a requirement and you truly see the value of
evaluation, you might need to take time to convince others,
such as decision-makers in the organization or staff, about
the value.
Your program may be a model program that a funder is
interested in replicating. If so, the context of the evaluation
could be more politically charged than usual, so you need to
take extra measures to ensure the findings are
communicated effectively and properly and not
inappropriately taken out of context.
Therefore, here are some questions to ask at this stage:
Is this part of a grant requirement? What data are you
required to report? What type of evaluation are you required
to conduct?
Is this part of a bigger evaluation, such as a national or
statewide evaluation? Are you required to collect data in a
certain way and at a certain time to support the national or
statewide evaluation?
continued from previous page
5.3 Considering Your
Organization’s
Capacity to Participate
in the Evaluation
5.3.1 A Staff Person Dedicated
to Evaluation
5.3.2 Budget for Evaluation
5.3.3 A Culture of Data and
Learning
5.3.4 Sharing and Protecting
Data across Program
Elements or Program
Locations
Highlights
Exercises
The Step-by-Step Guide to Evaluation | page 67
If it is part of a national or statewide evaluation, how can you use the required collected data
for your own purposes as well, or supplement the data for your own information needs?
Does your program have significant implications for a theory, practice or policy? If yes, who will
be scrutinizing the results?
If it is not a grant requirement, what needs to be done in your organization to ensure the
decision-makers and staff are on board? What do you need to prepare the organization
for evaluation?
If your organization decided to do the evaluation solely for its own purposes, you will have
more control over the entire evaluation process. If, however, the evaluation is part of a grant
requirement, you must work with the funder’s expectations about reporting, or with the evaluator
assigned by the funder. This does not mean that you give up control of the evaluation process. On
the contrary, you must be just as vigilant about ensuring the responsiveness of the evaluation to
your organization’s information needs.
5.1.1
Considerations if Your Evaluation is Part of a National or Statewide Evaluation
Here are some questions you should ask the funder or the evaluator assigned by the funder, if they
have not already provided this information:
Is there a logic model for the overall initiative, and how does your program’s logic model fit
into it?
What are the evaluation questions for the national or statewide evaluation?
What data are you required to collect? When do you have to report the data?
How does your program evaluation or data inform the larger evaluation?
How will the data you contribute be used?
What are nonnegotiable and negotiable in terms of evaluation? Examples include:
Are there instruments you must use provided by the national or statewide evaluator? Have
they been tested and validated for populations such as the ones you are serving?
Can you tailor these instruments to suit your community context and culture?
When do you have to provide the data to the national or statewide evaluator? How flexible
is the timeline, so it aligns with your program and needs?
Page 68 | The Step-by-Step Guide to Evaluation
When will you be able to get feedback or information about what the national or statewide
evaluation found and learned?
Asking and answering the above questions early is important so you are not surprised by a request
from the funder or national or statewide evaluator. Also, the funder or evaluator may not know
what it takes to carry out the work on the ground or the local considerations that could affect the
national or statewide evaluation. They might not even be aware of how their expectations could
affect your program’s implementation or your organization’s capacity.
On the other hand, you might not have thought about collecting certain types of data, and
understanding what the funder or evaluator wants could trigger an idea that might be very useful to
your strategy, initiative, program or organization. It is not unusual for a funder and
national or statewide evaluator to consider adding questions or adjusting their design after
speaking with local program directors and staff.
5.1.2 Considerations if the Findings from Your Evaluation Have Significant Implications for a Theory, Practice or Policy
Sometimes, your strategy, initiative or program may be primed for replication or expansion
because someone thinks it works or is partial to what you are trying to do. The evaluation findings
could have implications on a theory, practice or policy. If this is the situation, the evaluation could
be considered a high-stakes one with lots of potential users. Knowing this early can help you plan
how the findings will be communicated, by whom and when. Such planning is crucial to ensure the
findings are shared properly - not taken out of context, exaggerated or dismissed. Here are some
questions to guide your approach:
How interactive should the communication be for each of the intended audiences? For exam-
ple, written and print materials are least interactive, while discussions and working
sessions are most interactive. Verbal and video presentations fall somewhere in the middle.
A useful resource about communicating evaluation findings is Evaluating Strategies for Communicating and Reporting, by R. Torres, H. Preskill and M. Piontek, M., published by SAGE in 2005.
What are the risks in sharing the findings that could affect the strategy, initiative, program,
organization or the community you serve?
The Step-by-Step Guide to Evaluation | page 69
Can the findings be taken out of context and harm the effort, organization or community?
What can you do to mitigate the risks and consequences?
What roles do you, your staff, your board members and your internal or external evaluator have
in summarizing and communicating the findings and insights?
Who would be the most effective messenger of the information?
Do you need a facilitator skilled in adult learning techniques to assist with discussions,
working sessions and even verbal presentations?
Should you provide training for the messengers as spokespersons for the media and other
types of inquiries?
5.1.3
Getting Ready for Evaluation Whether or Not It is a Funding Requirement
You may realize the value of evaluation and choose to pursue it even if not required by your funder.
If this is the case, you need to make sure your board members, other decision-makers and staff
have equally bought into its value. Attending to staff commitment is especially important because
implementing an evaluation likely will add to their responsibilities and, perhaps, stretch their
already limited resources. Some steps to prepare your organization for evaluation include:
Look at the functions within the organization and discuss with the staff responsible for each
function what information they wish they had to perform their roles more effectively.
Introduce the idea of evaluation and how it could benefit the organization and its efforts.
Ask staff about their concerns regarding evaluation and find out what negative or positive
experiences they have had with evaluation.
Explore ways in which data collection can be integrated into the staff’s daily functions without
overburdening them and how they might use the data to support their work.
Form an evaluation and learning committee with staff from different levels of the organization
or with different functions, and charge the committee with finding out more about evaluation.
Address the concerns that were raised in both driving and guiding the evaluation
planning process.
Page 70 | The Step-by-Step Guide to Evaluation
5.2 who benefits from the evaluation and what are the potential risks?
As multiple stakeholders are involved in evaluation, you must
carefully consider how the information and findings you share
with the stakeholders, including your evaluator, can be used (or
misused). Check thoroughly whether anything in the data you
collect for the evaluation can be taken out of context or
misconstrued in a way that is culturally incompetent or
harmful to your community. If you have a designated evaluator,
you should work with him or her on this issue.
To begin to answer the question, “Who benefits from the
evaluation and what are the potential risks?” consider who is
interested in learning about the evaluation of your strategy,
initiative or program. Additionally who cares about the answers
to questions such as:
Are you running your effort as you had planned? Who stands
to benefit or lose from the way you run your effort?
Are the participants benefiting from your effort as desired?
Who stands to benefit or lose if they do or do not?
Is your effort making an impact? Who stands to benefit or
lose from the way your effort does or does not make an
impact?
Might the results affect policies? Who stands to benefit or
lose if it does or does not?
5.2.1
Consideration of Different Types of Stakeholders
Who are the individuals who could affect or be affected by the
strategy, initiative or program and its evaluation?
The Step-by-Step Guide to Evaluation | page 71
They can include:
Funders (public, private, nonprofit)
Program staff
Board members
Program participants or constituents of your organization
Community leaders
Collaborating organizations or partners
Other organizations serving the same target population
Policymakers
exhibit 5-1: examples of reasons why different stakeholders might be interested in the
evaluation results of the after-school mentoring program for youth living in low-income
neighborhoods in an urban area
Stakeholders Why are they interested in learning about how your program is doing?
Funders They want to know if their investments were put to good use. If the mentoring program is not reaching expectations, such as increasing academic achievement, they would want to know why and make decisions about continuing to fund the program or building the program’s capacity (or not) based on the findings.
Program staff They want to know if they are doing their work properly and if they are bringing changes to the youth’s lives as planned. If not, they would want to use the information to improve their program and revisit its theory and logic model.
Board members They want to learn about the progress of the program and the impact it is making (or not) in the lives of the youth served. If it is having an impact, they would want to use this information for their fundraising efforts. If not, they would want to address the issues the program is facing.
Program participants or constituents
The youth want to tell you if and how the program is helping them meet their academic or career goals. They want to know if the findings actually match what they experienced. If not, they would want to help the program capture more accurately the outcomes they experienced.
Page 72 | The Step-by-Step Guide to Evaluation
Stakeholders Why are they interested in learning about how your program is doing?
Community leaders They want to know if the program is making a difference in their communities or if it is doing more harm than good. For example, is it shedding a positive light on the community or bringing bad publicity to the community by emphasizing that there are many at risk-youth present? If the program is working, they could use the information to advocate for sustaining or expanding the program. If not, they could advocate for changes such as bringing in a new executive team to manage the program.
Collaborating organizations or partners
They want to know if their support is bringing changes to the lives of the youth as planned. If not, they would want to use the information to improve their support for the program.
Other organizations serving the same target populations
They want to know whether the program achieved the intended goals and compare the results to their program to determine if your program supports or does not support their agenda.
Policymakers They want to know whether the program supports or opposes their policy positions about after-school mentoring programs. They would want to know if there are supporting stories they can use for their advocacy or alternative explanations to make their arguments stronger.
Useful resources for engaging stakeholders in evaluation include:
A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, by H. Preskill and N. Jones, published by the Robert Wood Johnson Foundation in 2009. Available at http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging- stakeholders-in-developing-evalua.html.
Do Nothing About Me Without Me, by J. C. Bourns, published by Grantmakers for Effective Organizations and Interaction Institute for Social Change in 2010. Available at http://docs. geofunders.org/?filename=do_nothing_about_me_without_me.pdf.
Stakeholder Engagement Tool, by S. Salentine and A. Johnston, published in MEASURE Evaluation and USAID in 2011. Available at http://www.cpc.unc.edu/measure/resources/ publications/ms-11-46-e.
The Step-by-Step Guide to Evaluation | page 73
5.2.2
Use of Evaluation by Different Types of Stakeholders
Different stakeholders probably would use evaluation for different purposes, producing certain
benefits but also potential risks that could be avoided if you had a strategy to deal with them first.
For example, if you are evaluating one of your after-school mentoring programs for youth living in
low-income neighborhoods, consider how the results would affect your stakeholders.
Exhibit 5-2 is a table to help you think through the benefits and risks for each group and how you
can avoid the risks, to the best of your ability, as every situation is different. Exhibit 5-2 does not
include every potential scenario but gives general examples. Please take the time to complete such
a table carefully in your evaluations and customize with specific action steps for your context. You
and your evaluator should complete the table together and work with each group of stakeholders
to plan and take steps to prevent misunderstandings about the evaluation and misuse of the
findings.
exhibit 5-2: benefits and potential risks of the evaluation
Stakeholders group
The evaluation results can help this group by …
The evaluation results can put this group at risk by …
To prevent potential harm, you should …
Funders Demonstrating to the foundation staff that the grants are effective.
Highlighting that the foundation may be funding grants that do not work, and the program officers who favor the grant programs may have to end them.
Engage the program officers early on to understand what they consider success and discuss their information needs.
Program staff Showing that their work is making a difference.
Showing that their work is not making a difference. Consequently, some of the grants may not be renewed, which could lead to budget and staff cuts.
Work with the staff early on to ensure the evaluation questions align with grant requirements.
Board members Identifying areas where the nonprofit organization is making a difference.
Identifying areas of weaknesses in the nonprofit organization’s leadership, administrative and financial systems and day-to-day operations.
Engage board members in the design and implementation of the evaluation.
Page 74 | The Step-by-Step Guide to Evaluation
Stakeholders group
The evaluation results can help this group by …
The evaluation results can put this group at risk by …
To prevent potential harm, you should …
Program participants or constituents
Strengthening, continuing or expanding programs that they can continue to participate in and benefit from.
Eliminating opportunities afforded by the programs.
Engage them early on to help determine what success looks like as a result of their participation in the programs and ensure their vision is aligned with the funder’s expectations.
Community leaders
Bringing resources to the community.
Eliminating or reducing existing resources.
Obtain their perspectives about larger community forces related to the issues the programs are intended to address.
Collaborating organizations and partners
Demonstrating that they can and should work together to achieve change.
Surfacing conflicts that make it less desirable for them to work together in the future.
Work with all the partners to ensure their information needs, roles and respon- sibilities are considered in the evaluation design and implementation.
Other organizations that serve the same target populations
Demonstrating your program supports their agenda and the target populations they serve.
Diminishing the need or impact of their programs and competing with them for grants.
Conduct an environmental scan of organizations that serve the same target pop- ulations and understand how their agendas relate to yours.
Policymakers Elevating what works and providing data that support their policy agenda.
Bringing attention to problems in the districts or communities for which they are responsible and challenging their leadership and policies.
Speak to policymakers to understand their policy agenda and how the evaluation findings might or might not support their information needs and agendas.
The Step-by-Step Guide to Evaluation | page 75
5.3 consider your organization’s capacity to participate in the evaluation
To make the most of an evaluation, invest in building your organization’s capacity not only to
become an informed consumer of evaluation but also to develop and maintain a learning system
and culture. You have to set this tone at your organization; the evaluator cannot do this for you.
However, you can work with the evaluator to discuss when findings may be available and how to
best present the findings.
The following checklist in Exhibit 5-3 can help determine your organization’s evaluation capacity.
For each question, research the organization’s history of doing these tasks. Whether the staff had a
positive or negative experience the last time they attempted these activities, likely there are
lessons to build on. By finding out what might have been done before, you can avoid reinventing
the wheel or repeating the same mistakes.
Page 76 | The Step-by-Step Guide to Evaluation
exhibit 5-3: checklist to help your organization assess its evaluation capacity
Questions Yes No
Does your organization have leaders who are willing to take risks; share their learning with funders, board members, and staff; and adapt based on data and knowledge?
Ask leaders to acknowledge the importance of evaluation. Engage leaders in sharing and reflecting on the evaluation findings.
Prepare a presentation to leaders about how data and information could improve the organization’s efforts to become more impactful.
Engage leaders in a discussion about their concerns and fears about sharing the evaluation findings and actions that can be taken to alleviate their concerns.
Does your organization have a staff person who has some percentage of time dedicated to providing information to the evaluator or funder as well as summarizing and sharing the information in-house?
Engage this person in the evaluation you are considering.
Find out what helps or challenges his or her tasks and consider how to build on what is working and develop strategies to deal with the challenges.
Assess the existing staff’s skills to determine if someone from within the organization has the capacity to fulfill this function.
Develop a job description.
Does your organization allocate funds to support evaluation?
Find out if the funds are a set-aside and can be used for evaluation in general.
If not, and it is dependent on the grant, do a budget analysis to determine how much has been spent historically to do the evaluation.
Determine if the funds can be used more efficiently to expand the evaluation work, and if not, speak to your board about what you can do to expand the budget.
Speak to your board about allocating funds for evaluation.
Speak to your funder to find out if they can supplement the grant or allow you to reallocate a portion of the grant for evaluation.
The Step-by-Step Guide to Evaluation | page 77
Questions Yes No
Does your organization have a culture of using data and learning?
Establish a learning committee and task it with developing and implementing a learning plan by:
Speaking to staff and finding out what is facilitating the culture and what is challenging it.
Building on what is working and developing strategies to deal with the challenges.
Establish a learning committee. Task it with developing and implementing a learning plan by conducting an assessment and asking staff when and how they use data in their work and the challenges for making best use of the data.
Does your organization share data across program or program locations and have a system or process for protecting the data?
Using the learning committee, find out what is working and not working, including what tools are being used to share data and, most important, to what degree the data are protected to prevent loss and violation of confidentiality.
Using the learning committee, find out what type of software program or other tool is needed for data sharing and protection.
5.3.1
A Staff Person Dedicated to Evaluation
If your organization has an internal evaluator, that is great. However, many organizations cannot
afford an internal evaluator, so they hire an evaluator or work with an evaluator appointed by
the funder. Often, program staff are overstretched and overburdened by data requirements. This
problem becomes greater when staff does not have the capacity to handle data issues. In this case,
having a dedicated staff member who has the capacity to work with data would benefit the
organization. This staff person should be knowledgeable about various types of data, data
collection methods, data requirements, data analyses and data reporting. This person serves as a
data expert who works with the evaluator or the funder to provide the needed information
(e.g., raw data, summaries of data). This person can attend webinars or workshops conducted
by organizations such as the American Evaluation Association (http://www.eval.org) and
The Evaluator’s Institute (http://tei.gwu.edu).
Page 78 | The Step-by-Step Guide to Evaluation
5.3.2
Budget for Evaluation
An organization serious about evaluation will have a line item
in their budget for evaluation. This may include salaries for an
internal evaluator or data staff. This line item also could include
a budget to offset the costs for conducting an evaluation.
Expenses might include hiring an evaluator, cost of data system
or software, training of staff or volunteers, travel costs for
evaluation (e.g., travel expenses for evaluation participants),
translator or interpreter costs, etc. Section 7.9 in Chapter 7
provides more detailed explanation about how to budget for
certain types of expenses and activities in evaluation.
5.3.3
A Culture of Data and Learning
An organization with a culture of data and learning is more likely
to benefit from evaluation. It becomes a learning organization,
i.e., an organization in which leadership and staff continually
develop their abilities to achieve the results desired. In such
an organization, staff has a strong commitment to using data
for program improvement. An organization that emphasizes
learning will be more likely to use evaluation findings to improve
their programs or services. Learning organizations tend to have
systems already in place for data collection. If yours is not such
an organization, you should work with staff to build this culture,
which starts with organization leadership who models the be-
havior by asking questions, using data to inform decisions and
making improvements based on data. Leadership also
must sanction staff time to collect data where necessary, to
attend training about data use and to conduct reflections and
learning meetings.
The Step-by-Step Guide to Evaluation | page 79
5.3.4
Share and Protect Data Across Program Elements or Program Locations
Your organization can become a better consumer of evaluation by investing in a system that will
allow data sharing across program elements or program locations. This also will prevent your staff
from working in isolation and will benefit program participants by giving you a more
comprehensive understanding of their needs or progress.
When your organization participates in evaluation, you will be collecting and sharing a lot of data.
You should consider ways to protect the participants’ data - a step that is both ethical and
reassuring, as it also will help ease your participants’ worry about the safety of their data. This
might involve having password-protected computers or locked cabinets for data storage. You also
could consider having a data-sharing plan with evaluators or funders.
HIGHLIGHTS
Depending on who is funding the
evaluation, the requirements might vary
greatly. Be vigilant about understanding
these requirements and be engaged in
the process to prepare for the evaluation -
even when there is an external evaluator
assigned to your organization by
the funder.
Carefully consider how the information
and findings you share with the
stakeholders, including your evaluator,
can be used (or misused). Check
thoroughly whether anything in what you
collect for the evaluation can be taken out
of context or misconstrued in a way that
is culturally incompetent or harmful to
your community.
Invest in building your organization’s
capacity both to become an informed
consumer of evaluation and to develop
and maintain a learning system and
culture.
Whether you hire an internal or external
evaluation depends on the resources you
have, your staff members’ knowledge and
skills related to evaluation, your funder’s
requirements and the importance of an
independent perspective.
Page 80 | The Step-by-Step Guide to Evaluation
exercises
1. When you hire an external evaluator, you can give up control of the evaluation to him/her and
participate in the evaluation process only when asked by the evaluator.
A. True B. False
2. Consider a time when you hired an external evaluator or used an internal evaluator. What were
the pros and cons? If you could do it all over again, would you make a different decision after
reading Chapter 5? Why or why not?
3. It is perfectly fine to share data with anyone in your organization regardless of whether or not
they work on the program.
A. True B. False
4. What policies and procedures do you have in place to protect the data your organization col-
lects and uses? If none, what policies and procedures might you consider?
Answers: 1B; 3B
The Step-by-Step Guide to Evaluation | page 81
5. Your organization received a grant to increase low-income families’ access to healthy and fresh
food. The grant was awarded as part of a national initiative to decrease obesity through the use
of environmental change strategies. The award letter stated that you would be required to
participate in a national evaluation and nothing else. You have a conference call with the
funder in a week. What questions should you ask the funder about your organization’s
participation in the national evaluation?
Page 82 | The Step-by-Step Guide to Evaluation
6
The Step-by-Step Guide to Evaluation | page 83
Determine Stakeholders and Engage Them in the Evaluation
How this chapter is organized…
6.1 Evaluators as
Stakeholders
6.1.1 Keeping Your Evaluator
Informed on a Frequent,
Regular Basis
6.1.2 Two Types of Evaluators:
Internal and External
6.1.3 The Evaluator’s Role
6.1.4 The Evaluator’s Cultural
Competency
6.2 Other Key Stakeholders
6.2.1 How To Identify Key
Stakeholders
6.2.2 How To Engage Key
Stakeholders
Highlights
Exercises
A PREPARE for conducting an evaluation
B DETERMINE stakeholders and how and when to engage them
C IDENTIFY assumptions and determine what will be di�erent (theory of change and logic model)
D DEVELOP evaluation plan (logic model, measurement, framework, etc.)E COLLECT and analyze data
F COMMUNICATE results and understand what happened (interpret �ndings and facilitate learning)
G MAKE informed decisions (improve actions and next steps)
Stakeholders
introduction An evaluation is not simply
a matter of looking at your
organization, strategy,
initiative or program
and saying, “Here is
what we want to
know.” The evaluation
should include
understanding and
considering the priorities
and concerns of the various
people who have some vested
interest in what you do. Typical
stakeholders include your board of directors, staff, current or
potential funders, technical assistance providers, policymakers and
program participants. In most cases, evaluators also are considered
stakeholders because they are directly involved with the effort.
A stakeholder is any person or group who has an
interest in the strategy, initiative or program
being evaluated or in the results of the evaluation,
including the evaluator.
Involving stakeholders is important because they can help (or
hinder) an evaluation before it is conducted, while it is being
conducted and after the results are produced and ready for use.
Because nonprofit organizations often tackle complex issues,
stakeholders take on particular importance in ensuring the right
evaluation questions are identified and that evaluation results will
be used to make a difference. Stakeholders are much more likely to
support the evaluation and act on the results and recommendations
if they are involved in the evaluation process from the beginning.
Page 84 | The Step-by-Step Guide to Evaluation
if they are involved in the evaluation process from the beginning.
6.1 evaluators as stakeholders
As mentioned earlier, you will want to engage various
stakeholders in your evaluation. This includes the evaluator, who is not often thought of as a stake-
holder. However,
evaluators are not neutral third parties. Evaluators have a vested interest in what they are doing
and care about doing it well. With help from the organization’s executive director and staff, they
also can bring together all the other stakeholders needed for the evaluation.
6.1.1
Keep Your Evaluator Informed on a Frequent, Regular Basis
Evaluators play a very important role in your efforts, from helping you design the research to inter-
preting and communicating the results. You have a responsibility to keep the evaluator informed
on a frequent and regular basis about what is happening in your strategy, initiative or program. You
are immersed in your effort’s implementation and pay close attention to the need to adjust when
you see something is not working. Communicating these adjustments to your evaluator will make
his or her work more efficient and effective.
Let’s say you are implementing a financial literacy program. One aspect involves partnering with
a financial institution to host and conduct a 12-week-long financial literacy academy. However,
after you received the grant and were ready to start, the financial institution that committed to the
partnership pulled out of the agreement. You approached several banks and credit unions in the
community, but none had the staff capacity to do it. In the end, you decided to implement the cur-
riculum yourself and shortened it to 10 weeks to fit the program schedule.
In the middle of this challenging situation, while spending more time than you anticipated seeking
a partner and then revising the curriculum, you might have forgotten to inform the evaluator of the
change. Or you might not have realized the implications for the evaluation and therefore didn’t
inform the evaluator.
The Step-by-Step Guide to Evaluation | page 85
Here’s why it is important to have spoken to the evaluator: A survey questionnaire - already tested
and validated by the people who designed the academy - was used to evaluate the academy. Once
the model for the academy had been changed, the questionnaire was no longer valid. It needed
to be revised to fit your new approach and the results cannot be compared to those from another
study that used the academy model. The evaluation design would have to be adjusted.
An evaluator sometimes lives elsewhere and you won’t see him or her frequently. In such cases,
maintain regular contact by telephone and email.
6.1.2
Two Types of Evaluators: Internal and External
You might have heard the terms “internal” or “external” evaluator. You should understand the
difference and when to use an internal or external evaluator or, in some cases, both.
Internal Evaluator
It is perfectly fine to assign the responsibility for evaluation to a current staff member or to hire an
evaluator to join your staff. This internal evaluator can serve as both an evaluator and a staff
member with other responsibilities. This option has a number of advantages:
An internal evaluator can be less expensive because you’re not spending resources to hire
someone.
You are using existing resources and building the evaluation capacity within your own
organization.
Most importantly, as the internal evaluator works within the program, he or she might be more
familiar with the program, its staff and community members; could have access to
organizational resources; and might have more opportunities for informal feedback with
program stakeholders.
However, keep in mind the following disadvantages to hiring an internal evaluator:
The evaluator might be constrained by demands of other existing job responsibilities, leaving
insufficient time to focus on the evaluation.
The internal evaluator could lack the outside perspective and technical skills of an external
evaluator.
Page 86 | The Step-by-Step Guide to Evaluation
The internal evaluator might find it harder to remain objective, especially if evaluating a pro-
gram that he or she is working on.
The internal evaluator could also want to avoid negative conclusions, which reflect badly on
that person’s own work, other people in the program or the organization as a whole.
Nonetheless, if the strategy, initiative or program is not too complex and has budget constraints,
assigning or hiring a staff person with the right skills to be an internal evaluator is advisable.
The Step-by-Step Guide to Evaluation | page 87
checklist Things to consider for an internal evaluator:
University degrees in evaluation are not common. Many people now working as
evaluators have previously held managerial and administrative roles or conducted applied
social research. Therefore, consider hiring those who do not label themselves as
professional evaluators, but who have conducted evaluation tasks for similar programs.
If you cannot afford to hire someone for evaluation alone, then consider what other skills
the person needs. Assess the candidates’ prior experience working with the sectors and
populations you serve.
Also assess candidates’ ability to build bridges with program staff. Look for experience
with facilitating learning with groups. However, even when staff members have the above
qualities, you should still consider some of these questions:
Do they have the technical expertise?
Can they be objective? Do their roles in the organization create some bias,
predisposition or even prejudice?
Could they be influenced by others in the organization who desire a certain
evaluation outcome?
Are the internal politics such that a valid and fair evaluation is difficult, if
not impossible?
Will their findings have credibility, both within the organization and to
external parties?
The following principles are important for an effective internal evaluation:
Involve staff as much as possible.
Link the evaluation to the organization’s learning agenda. (If you don’t have a learning agenda,
you might want to develop one.)
Page 88 | The Step-by-Step Guide to Evaluation
Dedicate a staff person or form a group of staff members to be responsible for the day-to-day
management of the evaluation.
Strive for consensus on the evaluation plan and make sure your staff are involved in the plan.
Provide necessary resources for the evaluation and the staff person or persons responsible for
the evaluation.
Develop a process to regularly share updates about the evaluation’s progress.
Allocate time for staff to reflect on the evaluation’s findings and to discuss the findings’
implications on their work.
Develop a systematic plan by which improvements to your organization’s strategies, programs,
initiatives and policies can be made.
External Evaluator
In many cases when you have larger grants or a complex strategy, initiative or program, you might
want or need to hire external evaluators because:
These professionals often have assets including technical expertise and experience not
available in your organization.
They also could have networks that bring additional useful expertise and knowledge.
External evaluators might have broader evaluation expertise than internal evaluators,
particularly if they specialize in program evaluation or have conducted extensive research on
the subject matter related to your effort.
External evaluators also could bring a different perspective to the evaluation because they are
not directly affiliated with your organization or effort.
At the same time, here are some drawbacks to hiring external evaluators:
External evaluators could be detached from the daily operations of the strategy, initiative or
program and thus have limited knowledge of the effort’s needs and goals and limited access to
the effort’s activities.
External evaluators might not understand the community as well as you do.
External evaluators can be more costly than internal evaluators.
External evaluators have less opportunity to develop internal evaluation capacity.
The Step-by-Step Guide to Evaluation | page 89
You and your external evaluator will need time to build trust and good communications. It is
common for external evaluators to create some initial anxiety for you and your staff; everyone will
need to work hard at developing a working relationship.
checklist Considerations for selecting an external evaluator:
Assess the candidates’ prior experiences working with the sectors and populations you
serve and their understanding of the socioeconomic, demographic, historical, cultural and
political factors that shape the conditions in which the strategy, initiative or program is
operating. This means ensuring culturally competent evaluators - those who understand
how various groups perceive an intervention, communicate their views and experiences
and are affected by the factors mentioned above. Culturally competent evaluators are
particularly effective because they:
Keep an open mind.
Avoid making assumptions about a particular group of people.
Understand how their own cultural background, biases and worldviews could affect
their interactions with program participants and other stakeholders.
Gather appropriate data to draw conclusions by using methods that are respectful of
other cultures.
Look for candidates whose experience suggests the capability to devise and manage a
variety of evaluation designs and tasks, and who can clearly articulate which design is
most appropriate for your effort. Ask questions that require the candidates to describe
their experiences working with similar populations, and how they would apply their
understanding of the nuances of working with these groups to your evaluation.
Page 90 | The Step-by-Step Guide to Evaluation
exhibit 6-1: advantages and disadvantages of internal and external evaluators
Advantages Disadvantages
Internal Evaluator Costs less.
Knows the effort being evaluated.
Can easily help coordinate key stakeholders.
Helps build capacity of organization to conduct future evaluations.
Might not have enough time to work on the evaluation due to other responsibilities in the organization.
Could lack expertise and technical skills of an external evaluator.
Might lack objectivity.
Could raise doubts among potential funders about the objectivity and validity of the evaluation.
Might inhibit candor of clients who do not express themselves honestly to someone on the staff whom they will see again.
External Evaluator Brings technical expertise that might not be available within your organization.
Could have networks with additional expertise.
Could be more efficient due to experience with evaluation.
Could be more objective.
Could have greater credibility with potential funders.
Might not have enough understanding of the effort or context.
Can be more expensive.
Might have less opportunity to devel- op internal evaluation capacity.
Could cause anxiety among staff members who feel they are under a microscope from an outsider.
Might cause staff to be less engaged, seeing the evaluation as the external evaluator’s responsibility.
Useful resources on hiring evaluators include:
Hiring and Working with an Evaluator, by the Juvenile Justice Center, published in 2016. Available at http://www.jrsa.org/pubs/juv-justice/evaluator.pdf.
How Do I Work with An Internal Evaluator or External Evaluator Contractor, by E.F. Chappelle, published in 2011. Available at http://www.cdc.gov/dhdsp/pubs/docs/ cb_january_11_2011.pdf.
Selecting an Evaluator, by Community Science for the W.K. Kellogg Foundation. Available from [email protected].
The Step-by-Step Guide to Evaluation | page 91
6.1.3
The Evaluator’s Role
With your staff and stakeholders, think through all of the potential evaluator roles and
relationships and determine which configuration best fits your particular situation, the purpose of
the evaluation and the questions you are attempting to address.
Remember the primary goals of evaluation are (a) that stakeholders are engaged, active
participants in the process; and (b) that the evaluation process and findings will be meaningful
and useful to those ultimately responsible for improving and assessing the strategy, initiative or
program. In the end, there is no one way to conduct an evaluation. Given that premise, the critical
skills of the evaluator you choose should include:
The ability to listen, negotiate and bring together multiple perspectives, particularly those that
have been historically excluded.
The ability to analyze the specific situation.
The ability to assist in developing a design with the program team that leads to the most useful
and important information and final products.
Another important evaluator role is
the relationship between the evaluator
and primary stakeholders, including the
program team. Questions to consider
include:
How can this relationship be highly
interactive?
How much control should the
evaluator have over the evaluation
process relative to that of the
stakeholders or program team?
How actively involved should key staff
and stakeholders be in the evaluation
process?
Page 92 | The Step-by-Step Guide to Evaluation
Depending on the primary purpose of the evaluation, you might
also look for an evaluator with different strengths. For instance:
If the evaluation purpose is to determine the worth or merit
of a program, you could look for an evaluator with
methodological expertise and experience.
If the evaluation is focused on facilitating improvements to a
new program, you could look for someone with strong
facilitation skills and knowledge of organizational
development.
6.1.4
The Evaluator’s Cultural Competency
Most importantly, the evaluator should be culturally competent.
This means you should hire someone who has skills that enable
him or her to:
Have an open mind about people who may seem different.
Avoid making assumptions about other people’s worldviews
and behaviors.
Understand how historical and contextual factors (e.g.,
structural racism, poverty, sense of hopelessness) could
affect the evaluation process and results.
Appreciate and account for the strengths and assets of
people who, despite the inequities they experience, strive to
overcome their day-to-day challenges and aspire to have a
higher quality of life.
Gather the right data to draw appropriate conclusions that
consider the context within which the program is operating.
Overall, among the skills and expertise wanted
from a good evaluator are the ability to work with you, program staff and other stakeholders, and the ability to practice cultural competency. The selected evaluator must be able to listen, bring together several perspectives, integrate stakeholder input in the evalua- tion design and prioritize anal- yses that will be most applicable to your organization’s information needs.
However, remember that hiring an external evaluator should not mean the organization suspends its engagement in the evaluation. The evaluator brings requisite methodological skills, competencies and experiences, but your staff and evaluation committee (if you have one) will remain critical players in the evaluation process by determining the measures of success, “making meaning” of the data analysis, extracting and discerning lessons learned and making decisions based on the evaluation findings. Indeed, all of the organization’s stakeholders need to be active and engaged in the process.
The Step-by-Step Guide to Evaluation | page 93
In addition to these skills, culturally competent evaluators typically:
Go out of their way to demystify evaluations by explaining the evaluation design and process in
terms that people with no training in evaluation can understand.
Ensure the results are explained in plain and simple language so people with no social science
background can understand.
Encourage people to ask questions and share their views about what the results might mean
to them.
Useful resources for the role of culture in evaluation and culturally competent evaluations include:
The Importance of Culture in Evaluation, by Kien Lee, published by The Colorado Trust in 2007. Available at http://www.communityscience.com/pdfs/CrossCulturalGuide.r3.pdf.
A Guide to Conducting Culturally Responsive Evaluations by H.T. Frierson, S. Hood and G.B. Hughes, included in the National Science Foundation’s 2002 User-Friendly Handbook for Project Evaluation. Available at https://www.nsf.gov/pubs/2002/nsf02057/ nsf02057_5.pdf.
6.2 other key stakeholders
As mentioned before, in addition to the evaluator, all evaluations have multiple stakeholders. They
include:
funders (public, private);
program staff;
board members;
program participants or constituents of your organization;
community leaders;
collaborating organizations or partners; and
policymakers.
Page 94 | The Step-by-Step Guide to Evaluation
For example, if your strategy, initiative or program focuses on improving the competencies of
high school teachers in your community, the range of stakeholders would include the program
implementers, participating teachers, members of the school board, school administrators,
parents, participating students, funders and the evaluator.
exhibit 6-2: types of stakeholders
Types of Stakeholders Definition Examples
Implementers Those directly involved in operating and implementing the program.
Program directors and managers
Trainers who work with the teachers
Partners Those who actively support and are invested in the program.
Organizations that are advocating for a higher quality education system
Parents
Representatives of the school system
Policymakers Those in a position to decide something about your program (e.g., whether or not it should be expanded to other school districts).
School board
School administrators (e.g., principal)
State education department director
Program participants
Those being served or affected by your programs.
Students in the schools with the program
Teachers
Funders Those who fund the program. Public agencies
Private foundations
Remember to ensure you have gathered multiple perspectives about the main issues by involving
as many stakeholders as possible in initial evaluation discussions. Otherwise, the evaluation is
likely to be designed based on the needs and interests of only a few stakeholders - usually the
ones with the most power - and may miss other important questions and issues of stakeholders
not included at the table. There are many advantages to involving stakeholders:
The Step-by-Step Guide to Evaluation | page 95
Stakeholders can help improve the quality of decision-
making, since those with a vested interest contribute from
the initial stages.
Stakeholders can build trust that leads to acceptance of both
positive and negative feedback.
Involving stakeholders ensures the information
gathered is more reliable and comes from different
perspectives.
Involving stakeholders creates opportunities for you, your
stakeholders and decision-makers to learn from each other
by exchanging information and experiences.
6.2.1
How to Identify Key Stakeholders
Roles of various stakeholders
Determine what your needs are. At this point, begin to assess
whom you should involve. Since stakeholders offer different
kinds of value, your reasons for inviting certain individuals or
groups to participate should include the following:
They have content knowledge of the program being
evaluated (e.g., staff and any expert consultants you hired).
They represent diverse perspectives and experiences, so
they can raise questions and ideas that reflect all sides of
the issue (e.g., community leaders, political representatives,
target population, partner organizations, etc.).
They are affected by the program (e.g., program
participants).
They are in positions of influence and can raise questions
relevant to politicians, elected officials and other change
agents (e.g., respected community leaders, advocates).
When involving stakeholders from the beginning of
the evaluation process, you are more likely to:
Reduce stakeholders’ distrust and fear of evaluation.
Increase stakeholders’ awareness of, and commitment to, the evaluation process.
Increase the chances stakeholders will support your evaluation efforts, advocate for your effort and adhere to subsequent recommendations.
Increase the chances the evaluation findings will be used.
Increase the credibility of your evaluation findings.
If you do not engage stakeholders, your evaluation runs the risk of missing important elements of the strategy, initiative or program. In that case, evaluation findings might be ignored, criticized or resisted because your evaluation did not consider stakeholder concerns or priorities.
Page 96 | The Step-by-Step Guide to Evaluation
They are proponents of evaluation and offer support throughout the evaluation’s design and
implementation (e.g., your funders and board of directors).
They are responsible for decisions about the evaluation and program (e.g., organizational
leadership, program director).
6.2.2
How to Engage Key Stakeholders
Several factors should be considered in determining the best strategy for engaging your
stakeholders. These considerations will help you decide whether to work with stakeholders in
person or virtually, and which engagement methods - individual meetings, group meetings or
surveys - best fit your circumstances.
Imagine that your funders have asked you, as part of the grant requirement, to evaluate the
curriculum they funded to improve the competencies of high school teachers. You have
considered who could benefit from the evaluation (e.g., students, the designer of the curriculum)
and whose interests could be at risk due to the evaluation (e.g., principals, teachers). You need
stakeholders to support your evaluation, so at this point here are some considerations when
engaging stakeholders:
a. Budget: Do you have the budget to cover the costs of gathering input from stakeholders? While
you might want to engage every stakeholder in your community who is vested in improving the
quality of education, you should consider what financial resources you have to engage these
stakeholders. For example, it will cost more money to travel to engage some stakeholders.
Consequently, you might consider limiting the number of stakeholders or conducting virtual
meetings to gather their input.
b. Time: Some stakeholders, such as legislators and elected officials, require a fair amount of
time to engage as key informants. Determine whether you have the time to wait (in some cases
several months) to book an appointment to see them. In this case, you could consider sending
them a short list of questions and asking them to respond by email, or requesting a 15-minute
telephone call to discuss their perspectives.
The Step-by-Step Guide to Evaluation | page 97
c. Geographic locations: Bringing people physically together
can be valuable. Visual cues and building relationships are
critical throughout an initiative and its evaluation. However,
depending on the budget, time constraints and locations
of stakeholders, getting them t ogether in the same room
might not be efficient or feasible. Using a virtual engagement
strategy such as a webinar could be your best option.
d. Engagement methods: Another consideration when
engaging stakeholders is the methods you would use to
bring them on board. You should determine whether you
need to engage the stakeholders face-to-face, or via email,
webinar or a survey. Your budget and time should factor into
your decisions about these methods. For example, if you
have limited resources, you might want to engage
stakeholders who are easier to access (e.g., via email) and
require minimal resources to engage. You might have to
allocate funds to reimburse people for their transportation
and childcare expenses if you involve certain types of
stakeholders (e.g., youth, parents) face-to-face.
No magic formula exists to determine how
many stakeholders should be included. However, keep in mind that larger groups (more than nine) could take longer to reach decisions than smaller groups and might make the process more complicated. Also, consider carefully what will enable and motivate the stakeholder to participate. Community members might need transportation or childcare. Elected leaders could need specific and clear explanation about how their involvement relates to concerns of their constituents. Program staff might need an explanation about how their involvement can make their jobs easier.
Page 98 | The Step-by-Step Guide to Evaluation
Exhibit 6-3: template for engaging stakeholders
Who is the stakeholder?
What is the best way to engage the stakeholder?
What expenses and other special considerations are there?
What is the stakeholder’s role?
How frequently should you communicate with them?
Now that you have identified other key stakeholders for the evaluation, how do you invite them to
participate? Here is one strategy:
Invite the identified stakeholders to a meeting, or series of meetings, depending on their avail-
able time. At this meeting ensure that you:
Brief stakeholders on the strategy, initiative or program and the evaluation. One major
objective of this meeting should be to obtain a clear understanding of stakeholder
interests, perceptions and concerns related to the effort and evaluation.
Have all stakeholders clearly identify and agree to their roles and responsibilities related to
the evaluation before it begins.
Assure stakeholders that you will strive to maintain open communications and address
their concerns. Clarify to the group when they will hear from you, including when you will
seek their input and involvement during the evaluation process.
The Step-by-Step Guide to Evaluation | page 99
Emphasize to the stakeholder group that while their input and recommendations are very
valuable, it may not be feasible to implement all of their recommendations. Emphasize that
these decisions are driven by the availability of program resources (i.e., program staff, time
and budget priorities).
After this initial meeting, and when the selected stakeholders are on board and the theory of
change and logic model have been developed, you can begin identifying the most important
questions you want the evaluation to answer.
Involving the stakeholders in developing the questions is important. Effective evaluation
questions can guide how to examine your data to determine if your program is accomplishing what
it should. Having stakeholders assist in developing the questions will not only strengthen their
buy-in and support, but their perspectives could allow you to look at the strategy, initiative or
program and evaluation from a new angle. Also, some stakeholders may “hear” the questions
differently, especially if the questions’ phrasings suggest inappropriate assumptions about the
program participants or the community.
Consider this question, for example: “How has the program affected the academic performance of
high-risk youth?” This question labels the program youth as “high-risk,” which usually raises
concerns among advocates and leaders who work with the youth because of the general and
negative connotation associated with the term. For that reason, the question could be better
phrased as, “How has the program improved the academic performance of participating youth?”
Engaging different stakeholders in developing the evaluation questions provides an opportunity
for this sort of discussion, which also fosters learning by the organization staff, the evaluator and
the stakeholders.
Page 100 | The Step-by-Step Guide to Evaluation
HIGHLIGHTS
Evaluators are stakeholders, too.
Consider: What are their interests? How
might this affect how the evaluation is
designed - questions posed from a
particular focus and what interpretations
are made?
There are advantages and
disadvantages to hiring an external
evaluator or an internal evaluator. You
should consider the type of evaluation you
plan to conduct, the funds available for
the evaluation, the staff’s current capacity
to participate in the evaluation and the
intended use of the findings.
Stakeholders will have different,
sometimes even contradictory, interests
and views. They also hold different levels
of power. Program directors have more
power than staff. Legislators have more
power than primary grade students.
Funders have a particular kind of power.
Ask yourself: Which stakeholders are not
being heard in this process? Why not?
Where can we build consensus and how
can we prioritize the issues?
Examine the values embedded in the
questions being asked. Whose values are
they? How do other stakeholders,
particularly program participants, think
and feel about this set of values? Are there
different or better questions the evaluation
team members and other stakeholders
could build consensus around?
Useful resources for engaging stakeholders in evaluation include:
A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, by H. Preskill and N. Jones, published by the Robert Wood Johnson Foundation in 2009. Available at http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging- stakeholders-in-developing-evalua.html.
Do Nothing About Me Without Me, by J. C. Bourns, published by Grantmakers for Effective Organizations and Interaction Institute for Social Change in 2010. Available at http://docs. geofunders.org/?filename=do_nothing_about_me_without_me.pdf.
Stakeholder Engagement Tool, by S. Salentine and A. Johnston, published in MEASURE Evaluation and USAID in 2011. Available at http://www.cpc.unc.edu/measure/resources/ publications/ms-11-46-e.
The Step-by-Step Guide to Evaluation | page 101
exercises
1. When engaging stakeholders
A. Ask stakeholders about the best way to involve and communicate with them
B. Respect only those stakeholders’ values that are close to yours
C. Be clear about their role in the evaluation
D. Both A and B
E. Both A and C
2. If your organization hires an experienced evaluator with an excellent track record, there is no
need to involve program stakeholders.
A. True B. False
3. Stakeholders would more likely fully support your evaluation, if you involve them during the
stage.
A. Planning
B. Data collection
C. Analysis
D. Interpretation
E. All of the above
4. Consider a program, initiative, strategy or policy that you would like to evaluate. Who are the
stakeholders and what are the best ways to engage them?
5. You are working with an external evaluator to evaluate a curriculum to improve the quality of
teaching in high schools. A recent report by an independent research association has shown
decreasing high school graduation rates during the past five years for your community. The
report has raised a lot of concerns among parents, principals and local elected officials. Your
Answers: 1E; 2B; 3E
Page 102 | The Step-by-Step Guide to Evaluation
organization adopted the evidence-based curriculum from a national organization. If success-
ful, the curriculum may be expanded to other school districts. To evaluate the curriculum’s
implementation and effectiveness, your evaluator proposed the following activities: interviews
with principals; focus groups with teachers and parents; observations during instruction; and
content analysis of lesson plans before and after the curriculum’s implementation. Apply the
template intruduced previously [Exhibit 6-3] to determine how to engage the stakeholders in
this question.
Who is the stakeholder?
What is the best way to engage the stakeholder?
What expenses and other special considerations are there?
What is the stakeholder’s role?
How frequently should you communicate with them?
The Step-by-Step Guide to Evaluation | page 103
7 Developing a Logic Model, Evaluation Questions, Measurement Framework and Evaluation Plan
Page 104 | The Step-by-Step Guide to Evaluation
A PREPARE for conducting an evaluation
B DETERMINE stakeholders and how and when to engage them
C IDENTIFY assumptions and determine what will be di�erent (theory of change and logic model)
D DEVELOP evaluation plan (logic model, measurement, framework, etc.)E COLLECT and analyze data
F COMMUNICATE results and understand what happened (interpret �ndings and facilitate learning)
G MAKE informed decisions (improve actions and next steps)
Stakeholders
introduction The Kellogg Foundation Logic Model Development Guide (available at www.wkkf.org/resource-
directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide) is an excellent
companion for this chapter Rather than repeat the information in that guide, this chapter focuses on the
use and process of developing a logic model and how to generate evaluation questions and a
measurement framework from it. You should combine all of these components to form an evaluation
plan.
Evaluation - especially logic models - is a learning
and management tool that can be used throughout the
life of a strategy, initiative or program, whatever
your stake in it. Using the logic model results in
effective design of the effort and offers greater
learning opportunities, better documentation of
outcomes and shared knowledge about what
works and why.
More important, the logic model helps to
ensure that evaluative thinking is integrated
into your evaluation design and implementation.
Evaluative thinking is a systematic process of collecting and analyzing data in order
to tell the story about your strategy, initiative, program, policy or organization. It
is based on the belief that a systematic process is valuable and necessary. Such a
process involves identifying assumptions about what you think works and doesn’t
work and why; posing thoughtful questions about what change you expect to see
during and after you implement your effort; pursuing deeper understanding through
reflection and dialogue; communicating what was learned without underestimation
or exaggeration; and making informed decisions in preparation for action.
The Step-by-Step Guide to Evaluation | page 105
How this chapter is organized…
7.1 Overview of Logic Model
7.2 Use The Logic Model Throughout Your
Effort’s Life Cycle
7.2.1 Use of Logic Models to Design Your
Strategy, Initiative or Program
7.2.2 Use of Logic Models to Inform
Implementation
7.2.3 Use of Logic Models to Develop Evaluation
Questions
7.2.4 Use of Logic Models to Develop an
Evaluation Plan
7.2.5 Use of Logic Models to Create an Outline for
Your Evaluation Report
7.2.6 Use of Logic Models to Improve Your
Strategy, Initiative or Program
7.3 What You Can Expect from Logic Modeling
7.3.1 It is a Process
7.3.2 The Process Can Surface Differences
7.3.3 The Process Can Bring About Innovation
7.4 How to Go About the Logic Modeling
Process
7.4.1 Whom to Involve
7.4.2 How to Involve Them
7.5 Components of the Logic Model
7.5.1 Alignment between Strategies or Activities
and Outcomes
7.6 From Logic Model to Evaluation Questions
7.6.1 Two Different Ways to Formulate Evaluation
Questions About Your Strategy, Initiative
or Program
7.6.2 Typical Scenarios When Formulating
Evaluation Questions
7.6.3 Other Considerations
7.7 From Logic Model to Evaluation Questions
to Measurement Framework
7.7.1 Key Components of the Measurement
Framework
7.7.2 How to Use a Measurement Framework
7.7.3 Considerations for Developing the
Measurement Framework
7.7.4 Taking a SMART Approach to Measurement
Framework Development
7.8 Your Evaluation Plan
7.9 Budgeting for Evaluation
7.9.1 Types of Expenses Typically
Associated with Evaluation
7.9.2 Time Estimates for Specific Evaluation
Activities
Highlights
Exercises
Page 106 | The Step-by-Step Guide to Evaluation
7.1 overview of logic model
You design and implement your strategy, initiative or program the way you do for a reason. For
example based on your experience and findings from past research studies, you believe that
adequate information about how to purchase a home and the services offered by financial
institutions are not accessible to working class African-American families. Your effort in Rainbow
County seeks to increase the number and percent of working class African-American
homeowners in the county. To this end, your effort includes services to educate families about
budgeting, savings and improving their credit scores; acquiring financing; and understanding what
it means to be a homeowner. You also plan to work with the lending institutions in the county to
improve their lending policies, practices and services. Your theory of change is that by working
with the families and lending institutions, more residents will be able to purchase homes;
homeownership supports asset building and family economic security. When you draw the
pathway of change for the effort, you are in fact creating a logic model.
A theory of change is a narrative that explains the links between program strategies
or activities and outcomes, and how and why the desired change is expected to
come about.
In the above example, the rationale for your effort would be explained in more detail, including
citations of supporting studies and assumptions you are making. This information becomes the
basis of your theory of change.
A logic model is a graphic representation of the theory of change that illustrates the
linkages among program resources, activities, outputs, audiences and short-,
intermediate- and long-term outcomes related to a specific problem or situation.
The Step-by-Step Guide to Evaluation | page 107
Useful resources for developing a theory of change and logic model include:
Logic Model Development Guide published by the Kellogg Foundation in 2004. Available at http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation- logic-model-development-guide.aspx.
Point K Tools: Logic Model Builder, by Innovation Network. Available at http://www.innonet.org/?section_id=64&content_id=185.
Theory of Change: Guided Example - Project Superwomen published by ActKnowledge and the Aspen Institute Roundtable on Community Change in 2003. Available at http://www.theoryofchange.org/pdf/Superwomen_Example.pdf.
Theory of Change: A Practical Tool for Action, Results and Learning, by J. Reisman and A. Gienapp, published by Organizational Research Services in 2004. Available at http://www.aecf.org/upload/publicationfiles/cc2977k440.pdf.
Your logic model for the above theory could look like. The previous the logic model that shows
how (A) leads to (B) and (B) to (C) and (C) to (D) and so on through (G). The effort may be affected
by the context, history, culture and other factors surrounding it (G). A logic model usually includes
components such as resources/inputs, activities, outputs and outcomes, and the outcomes can be
divided into immediate, intermediate and long-term outcomes (explained further in section 7.2).
The logic model operationalizes your theory of change. During the evaluation, you collect data to
test the theory. In most cases, practitioners, evaluators and even funders tend to use “logic mod-
el” and “theory of change” interchangeably. However, they are complementary and should be used
together, not separately or as a substitute for each other.
Page 108 | The Step-by-Step Guide to Evaluation
G Context, History, Culture, and Other Factors
Exhibit 7-1: sample program logic model
A
Funding
Staff
Consultation from a housing
intermediary
Resources/ Inputs
Activities Outputs Immediate Outcomes
Intermediate Outcomes
Long-term Outcomes
B
Design and conduct 10 courses for
working class African
American families
Reach out to and work with
lending institutions
Review existing lending policies
C
Number of participants
who completed the
courses
Participants’ needs were
met
Number of partnerships with lending institutions
D
Increased knowledge
among families about process and resources for purchasing
a home
Improved financial
literacy among families
Actions by lending
institutions to reassess their policies and
practices
E1 Increased
savings
E2 Improved
credit scores
E3 Increased loan approval rate
E4 Better terms for mortgage loans
E5 Changes in
lending policies and practices
F
Increase in percentage of homeowner- ship among
working class African
Americans in the county
Theories of change link outcomes and activities to explain HOW and WHY the desired change is expected to come about, while logic models graphically illustrate program com- ponents such as inputs, activities and outcomes; creating one will help you and your stake-
holders clearly identify inputs, activities and outcomes. Also, the arrows show the connections between program components should you decide to include them in the logic model. Some people could interpret the arrows as causal links (i.e., component D causes component E) but your evaluation might not be able to prove this because 1) the design you selected (e.g., you and your evaluator decided a case study design was the best one to answer the evaluation questions) or 2) your initiative is complex and many factors contribute to the desired outcomes. Additionally, arrows that point in only one direction could suggest a linear flow in how events and outcomes unfold and that is probably not how they occur. Once you have completed the evaluation and have better insight into what happened, you could revisit the logic model and illustrate the arrows or flow and then test the logic model again.
The Step-by-Step Guide to Evaluation | page 109
7.2 use the logic model throughout your effort’s life cycle
Using a logic model is an effective way to ensure a strategy, initiative or program’s success. A logic
model is a “living” document that should not be shelved and forgotten once developed but used
and reviewed regularly throughout the life of the effort. The logic model, in particular, is useful for
informing the design of a strategy, initiative or program; its implementation; the plan for
evaluating it; communication of the evaluation findings; and its improvement.
7.2.1
Use of Logic Models to Design Your Strategy, Initiative or Program
When you are designing your strategy, initiative or program or writing a funding proposal for it,
developing a theory of change and logic model as part of the process can be helpful. Some funders
request a logic model as part of the application. At this stage, you might need to do some research
about the theories and best practices that support the outcomes you want to achieve: What
strategies and activities have been shown to be most effective in achieving those outcomes?
Once you have that information, you can begin to sketch out a logic model to show the path and
progression of change. The proposal writing should become easier if you can describe the logic
model and the theory underlying it.
Sometimes after you are awarded the grant, you could have an opportunity to review the logic mod-
el with other grant stakeholders (e.g., evaluator, technical assistance provider, foundation or
government program officer). This is a great opportunity to take advantage of the expertise of these
stakeholders. They could serve as a sounding board. They might have questions about your logic
model and even counterarguments to your logic. If you are working with an external evaluator at
this point, involve that person in this logic modeling process, so you and the evaluator have a
shared understanding of your strategy, initiative or program.
7.2.2
Use of Logic Models to Inform Implementation
A logic model is not an evaluation tool, but a planning tool to ensure you are implementing the
strategy, initiative or program as illustrated in the logic model. The components of the logic
Page 110 | The Step-by-Step Guide to Evaluation
model - inputs; strategies/activities; outputs;
immediate, intermediate and long-term outcomes;
and contextual conditions - act like ingredients in
a recipe for change. You also can use the model to
determine if your effort is being implemented as
planned. If not, you can use the data gathered to
make decisions for midcourse adjustments. This is
frequently referred to as a process of “formative
evaluation”.
7.2.3
Use of Logic Models to Develop Evaluation
Questions
Evaluation questions can be generated from a logic
model. The logic model acts like a hypothesis: It
allows you to see which stage of development you are
in with regard to your strategy, initiative or program,
and therefore what types of questions to ask. It also
helps you craft specific questions. For instance, without a logic model, you could simply ask the
question, “Did the program achieve the intended outcomes?” With a logic model, can see that only
certain outcomes should be expected after one year of implementation. Thus, your questions can
be more precise and easier to answer.
7.2.4
Use of Logic Models to Develop an Evaluation Plan
Out of the logic model, you can create a measurement framework (see Section 7.7) and then
explain how you will collect and analyze data about the measures in the framework. This forms the
basis of your evaluation plan. Also, it’s important to include in your plan ways to use and
communicate the findings. Otherwise, the findings can be used inappropriately without your
consent or knowledge.
The Step-by-Step Guide to Evaluation | page 111
7.2.5
Use of Logic Models to Create an Outline for Your Evaluation Report
Other applications of the logic model could be used to organize your reporting, and also used like
a hypothesis. In your report, you can use the model to show how you tested the hypothesis and
how your evaluation findings support, or do not support, the hypothesis. The typical sections of
your report could include:
Underlying assumptions, components and context for the strategy, initiative or program
Resources used to support the effort
Activities
Results (outputs, immediate and intermediate outcomes)
Facilitating factors and challenges
Recommendations for improvement
7.2.6
Use of Logic Models to Improve Your Strategy, Initiative or Program
The evaluation findings tell you the degree to which your theory of change, illustrated in the logic
model, is true. Remember the earlier recommendation that the logic model should not be shelved
and forgotten? At this point, bring it out again and see if what you expected to happen when you
created the logic model actually happened. Were the activities implemented as planned? Did
your strategy, initiative or program benefit people the way you thought it would? Did the activities
contribute to the results you desired? If not, assess where the effort did not unfold as expected and
what improvements are needed. After you make the improvements to the effort, you should revisit
the measurement framework and evaluation plan, and make the necessary and corresponding
modifications.
Page 112 | The Step-by-Step Guide to Evaluation
The logic modeling process also can happen during a strategic planning process for your organization. while the text focuses more on a strategy, initiative or program logic model, an organization can have a logic model, too. This logic model would show how the multiple
strategies and programs of your organization contribute to the organization’s mission and goals. If your organization is going through a strategic planning process and you are working with an evaluator, make sure the evaluator and strategic planner work together, becuase they could have different skills. The strategic planner might be skilled in the process and mapping out the strategies and programs while the evaluator could be better at identifying the metrics for monitoring your organization’s goals and objectives.
Involving an evaluator during your organization’s logic modeling or strategic planning process can benefit your organization. An evaluator working with this process should have good facilitation skills, such as:
Ability to guide people through a process of learning or planning together Ability to create a safe space for various stakeholders to share different perspectives Ability to shift roles between facilitator and expert Knowledge of logic modeling and strategic planning process Expertise in topic areas of interest (e.g., health, education, economic development)
Logic modeling can occur at various times in the organization and program cycles. Logic modeling
should happen, at a minimum at the beginning of the program cycle. However, logic modeling
should not end there. You should revisit the logic model as often as needed. Remember, your logic
model is a “living” document. Revisit, adapt or change when appropriate.
7.3 what you can expect from logic modeling
7.3.1
It is a Process
Often times the two words “logic model” could make people cringe! This could be due to their
previous experience, often involving painstakingly long and frustrating conversations with an
“outsider” trying to describe things that seem obvious. Sometimes the process is difficult because
the evaluator asks question after question trying to understand what the plan is. The evaluator
The Step-by-Step Guide to Evaluation | page 113
might seem to be criticizing every part of the program, but he or she really is just trying to gain a
clearer picture of the theory of change that may only exist in someone’s mind. Or frustrations could
arise because the evaluator dismisses the staff’s input or jumps to wrong conclusions about the
strategy. In some situations, program leaders might believe everyone understands the program the
same way, only to find out during the first logic modeling meeting they are wrong. Do these
scenarios seem familiar?
Developing a logic model does not happen in just one meeting. Several meetings and iterations of
the logic model probably will be needed to get the logic model just right for your strategy, initiative
or program. Contrary to popular belief, logic modeling is a process that involves time, energy and
thought; it is not a task to complete in a two-hour meeting.
Logic models are not evaluation tools; they are learning and management tools that should be
used throughout the life of a strategy, initiative or program. A logic modeling process should facili-
tate effective planning, implementation, evaluation and improvement of your effort.
7.3.2
The Process Can Surface Differences
The logic modeling process will surface differences in viewpoints about everything from definitions
to assumptions about the change process. This dynamic is normal. In fact, it’s helpful when dis-
agreements arise while creating a logic model for your strategy, initiative or program.
Precision is required regarding definitions and concepts.
For example, if the activity consists of instructional courses about homeownership, you will be
asked during the logic modeling process to identify the instructors, how frequently each course
occurs (number of sessions per week and number of hours per session), how many participants per
session and how improvement in knowledge about homeownership is measured. Achieving this
sort of precision requires time to discuss and decide. You could be surprised how many different
ways of thinking might exist among your staff and other stakeholders. Hence, good facilitation is
necessary when developing a logic model.
Page 114 | The Step-by-Step Guide to Evaluation
7.3.3
The Process Can Bring about Innovation
This process is intended to explore the pathway to change from various angles, and even surface
innovative ideas on how the strategy, initiative or program can be better designed or otherwise
improved. For instance, with the previous homeownership example, someone might suggest that
homeownership alone cannot lead to family economic security; economic and workforce
development must also be addressed. That person could suggest adding a component to include
job training.
7.4 how to go about the logic modeling process
7.4.1
Whom to Involve
Ideally, logic modeling should be done with key stakeholders of your strategy, initiative or program,
such as key staff, the organization’s leadership, program participants, community leaders and, of
course, the evaluator. Sometimes your board members and funders might have to be involved in
the process.
The Step-by-Step Guide to Evaluation | page 115
7.4.2
How to Involve Them
You should work closely with the evaluator to design the process because you know which
stakeholders to involve and the evaluator understands how the process should unfold. Together,
you can design an effective process and agenda. There are different ways to engage stakeholders
in the process, all of which require dialogue and interaction. Here are three examples:
If you decide to bring everyone together for several hours and it’s a fairly large group, you
should design a process that allows for small group discussions. These small groups can be
organized by stakeholder type (e.g., all the program participants in one group) or as a mix of
different stakeholders (e.g., two program participants, two staff members, two board
members).
Keep in Mind: Differences in power among stakeholders can cause some people to be less
vocal and others to be inadvertently “louder.” You can do four things to manage this:
Make sure the stakeholders with less power (e.g., program participants) outnumber the
ones with more power (e.g., staff or board members).
Orient and prepare the stakeholder group with less power before the meeting so they know
what to expect and feel confident about their contribution to the process.
Use a discussion technique that gives everyone a chance to talk.
Use exercises that allow participants to write down their thoughts and turn in the written
notes; this could help increase participation from those who are shy about speaking in
front of a group.
If you have participants who speak a language other than English, use an interpreter and
simultaneous interpretation equipment. Also, conduct half the meeting in one language
and the other half in the other spoken language and require people not proficient in the
language to wear wireless headsets. This way, everyone experiences what it is like to not
understand another person’s language, which could help level the playing field.
If you decide not to bring everyone together, you can design a process consisting of several
separate discussions and then combine the feedback to formulate the logic model. Then, share
the draft logic model with everyone, get comments and revise the logic model accordingly.
Page 116 | The Step-by-Step Guide to Evaluation
Keep in Mind: If you do it this way, explain to everyone why some comments might not have
been integrated into the revised version.
Sometimes people engage better when they can react to something. If you think this approach
will work for your strategy, initiative or program, you can create a draft logic model and then
convene the stakeholders to discuss it. Work with the evaluator to develop a set of questions to
structure and guide the discussion.
Keep in Mind: If you do it this way, be sure to stress the version they are reacting to is a draft
and be clear about the parameters for revisions. You don’t want people to think they can
change whatever they want and risk disengaging them when their input is not integrated into
the revised version.
7.5 components of the logic model
Basic components of the logic model, as described in the
Kellogg Foundation’s Logic Model Development Guide, include:
Resources/Inputs: The human, financial, organizational and
community resources which a program, initiative, strategy or
policy has for the work.
Strategies or Activities: The processes, tools, events,
technology and actions that are an intentional part of the
implementation. These interventions are used to bring about
the intended changes or results.
Strategies or activities are the processes,
techniques, tools, events, technology and actions
of the planned program, used to bring about the
intended changes or outcomes.
Outputs: The direct products of activities; might include
types, levels and targets of services to be delivered by the
strategy, initiative or program.
The Step-by-Step Guide to Evaluation | page 117
Short-term (sometimes called immediate and intermediate outcomes) and long-term
outcomes: The specific changes in participants’ behavior, knowledge, skills, status and level of
functioning, or in policies, procedures and practices. Short-term outcomes should be
attainable within one to three years, while longer-term outcomes should be achievable within a
four- to six-year timeframe. These timeframes reflect the range of typical funding cycles.
Immediate Outcomes or Short-Term Outcomes: Immediate changes or benefits
expected - usually within one to two years - as a result of successful implementation
of the strategy.
Intermediate Outcomes, which also can be considered Short-Term Outcomes:
Changes or benefits, usually within one to two years of the immediate outcomes.
Long-Term Outcomes: Lasting changes with organizational, community or
systems-level benefits (e.g., improved social conditions, reduced rate of a particular
health outcome).
Context: The relevant demographic, economic, community, historical, cultural, political or other
social factors that influence the activities and outcomes of a strategy, initiative or program.
Contextual factors can include the racial and ethnic makeup of a community, geographic
region, economic opportunity, the history of discrimination, language barriers, political
climate, access to health care, quality education or employment and impoverished conditions.
The context informs the types of activities planned. In place-based work, these contextual vari-
ables might serve as outcome variables as well (i.e., the efforts are intended to change these
variables as they exist in the targeted place).
The components could vary based on the theory (or the change process) of the strategy, initiative
or program. Each term is explained more in detail in the W.K. Kellogg Foundation Logic Model
Development Guide (http://www.smartgivers.org/uploads/logicmodelguidepdf.pdf ).
Page 118 | The Step-by-Step Guide to Evaluation
7.5.1
Alignment between Strategies or Activities and Outcomes
One of the most useful things about the logic model is that it provides a visual tool to ensure that
your strategies or activities and immediate, intermediate and long-term outcomes are aligned.
In other words, you can trace the connection between all these components. Sometimes, one
strategy or activity can lead to several immediate outcomes and one intermediate outcome. You
can have different combinations; what is important is the “logic” of their connections.
Let’s use a pervious example. You are implementing an initiative in Rainbow County to increase
the number and percent of working class African-American homeowners in the county. Your effort
includes services to educate families about budgeting, saving and improving their credit scores;
acquiring financing; and understanding what it means to be a homeowner. You also plan to work
with the county lending institutions to improve their lending policies, practices and services.
The Step-by-Step Guide to Evaluation | page 119
exhibit 7-2: alignment between activities and outcomes
Where there is alignment ...
Activity Immediate Outcome Intermediate Outcome Long-Term Outcome
Design and conduct 10 courses for working class African American families
Increased knowledge among families about process and resources for purchasing a home
Increased savings
Improved credit scores
Increased loan approval rate
Increase in percentage of homeownership among working class African- American families in the county
Where there is misalignment that requires adjustment ...
Activity Immediate Outcome Intermediate Outcome Long-Term Outcome
Design and conduct 10 courses for working class African American families
Increased knowledge among families about process and resources for purchasing a home
Residents purchase homes
Decrease in poverty
This might not be realistic to achieve;
rather, it could be more feasible to observe residents engaged in steps
toward homeownership.
Homeownership alone will not lead to decrease in poverty.
7.6 from logic model to evaluation questions Drafting evaluation questions most likely will require numerous meetings with the evaluator and
other key stakeholders, as appropriate. Evaluation questions will depend on the phase the
strategy, initiative or program is in, the particular local circumstances and the ultimate purpose of
the evaluation. For example, your effort could benefit from a process or formative evaluation, but
may not be ready for an outcome or summative evaluation at this time.
A logic model can help generate evaluation questions. You can see what phase the effort is in and
therefore, which questions to ask and when it is best to answer them. Also, the model helps clarify
which variables are critical to achieving the desired outcomes. The array of questions you might
Page 120 | The Step-by-Step Guide to Evaluation
want to answer about your effort could be vast. A logic model helps narrow the array in a systemat-
ic way by highlighting the connections between strategies or activities and outcomes, and between
strategies or activities and the assumptions underlying the program. You will be better able to
address questions such as:
How is the strategy, initiative or program supposed to work?
Where do the assumptions in the model hold and where do they break down?
Where are the gaps or unrealistic assumptions in the model?
Which pieces of the model seem to be yielding the strongest outcomes or relationships to one
another?
Which pieces of the model are not functioning in practice?
Are there key assumptions that have not been embedded in the effort that should be?
By organizing evaluation questions based on the logic model, you are better able to determine
which questions to target in an evaluation.
There is another consideration in developing evaluation questions. The questions also depend
on the type of evaluation you want to conduct - a performance evaluation, a process or formative
evaluation or an outcome or summative evaluation.
A performance evaluation is concerned with:
Ensuring accountability
Demonstrating that resources are used as intended and managed well
Monitoring and reporting on progress toward established goals
Providing early warning of problems to funder and management
A process or formative evaluation is concerned with:
Understanding if a strategy, initiative or program is being implemented as planned and
according to schedule
Assessing if the effort is producing the intended outputs
Identifying strengths and weaknesses of the effort
Informing mid-course adjustments
The Step-by-Step Guide to Evaluation | page 121
An outcome or summative evaluation is concerned with:
Investigating whether the strategy, initiative or program achieved the desired outcomes and
what made it effective or ineffective
Making mid-course adjustments to the effort
Assessing if the effort is sustainable and replicable
A useful resource for engaging stakeholders in developing evaluation questions is:
A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, by H. Preskill and N. Jones, published by the Robert Wood Johnson Foundation in 2009. Available at http://www.rwjf.org/en/library/research/2009/12/a-practical-guide-for-engaging- stakeholders-in-developing-evalua.html.
7.6.1
Two Different Ways to Formulate Evaluation Questions About Your Strategy, Initiative
or Program
Here is a method to help formulate evaluation questions. Look at the logic model and start with the
following five elements:
Who - Who was your strategy initiative, or program intended to benefit?
What - What was the effort intended to do? What was the context within which the effort took
place and how could it have affected its implementation and outcomes?
When - When did activities take place? When did the desired changes start to occur?
Why - Why is the effort important to your organization or community? Why might it be important
to people in other organizations or communities?
How - How is the effort intended to affect the desired changes or bring about the desired out-
comes?
Here are examples of evaluation questions to ask for the previous sample scenario:
Who - Who, among the African American families reached, successfully completed the courses?
Who, among the lending institutions in your county, were open to partnering with your
organization?
Page 122 | The Step-by-Step Guide to Evaluation
What - What components of the program were successfully implemented? What components
were most useful and least useful to families? What are the lessons learned from the program
that could help other nonprofit organizations with a similar goal?
When - When (at what stage in the partnership) did a lending institution begin to reassess its
policies, practices and services? When did families begin to take steps toward homeowner-
ship?
Why - Why did the lending institutions decide to participate in your program? Why did some
families complete or not complete all the courses?
How - How did the economic crisis affect the families’ ability to improve their financial situation
enough to purchase a home? Did the initiative contribute to the desired intermediate
outcomes (i.e., families took steps toward purchasing a house) and if “yes,” how?
Another method for formulating evaluation questions is to consider different aspects of your
strategy, initiative or program and generate questions about each of these aspects.
exhibit 7-3: examples of evaluation questions about different aspects of your effort
Aspect of Your Effort Example of Evaluation Question
Theory of Change and Logic Model
Was the theory of change and logic model correct?
What aspects of the theory and logic model did not happen in practice and why or why not?
Implementation Was the effort implemented as intended?
Why or why not?
Results and Outcomes To what extent did the effort lead to the anticipated results?
What was the change and to what extent did the effort contribute to the change?
What difference did the effort make to the organization, participants and community?
The Step-by-Step Guide to Evaluation | page 123
Aspect of Your Effort Example of Evaluation Question
Context What demographic, economic, community, historical, cultural, political or other social factors could have influenced the effort’s implementation and outcomes?
What organizational factors (e.g., staff capacity, leadership, resources) might have affected the effort’s implementation and outcomes?
Learnings What worked and what did not?
What were unintended consequences or benefits?
Investment Was the effort cost effective?
Was there another alternative that could have represented a better investment?
What Next Can the effort be scaled up?
Can the effort be replicated elsewhere?
Is the change self-sustaining or does it require continued intervention?
7.6.2
Typical Scenarios When Formulating Evaluation Questions
What happens when different stakeholders have different questions and your list of questions
becomes too long?
At this point you want to remind your stakeholders about the objective of the evaluation. You can
point to and use the logic model to focus their questions. You also can estimate the cost of the
data collection and analysis needed to answer the questions, and explain that your evaluation
funds are limited.
How many evaluation questions should there be?
Again, remind stakeholders that questions should focus on and align with the information you
want to obtain about the program. Therefore, try to avoid unrelated questions and questions you
cannot address due to time and budget constraints.
Typically, three to five evaluation questions are recommended.
Page 124 | The Step-by-Step Guide to Evaluation
7.6.3
Other Considerations When Formulating Evaluation Questions
Commitment letter — Check your funding or grant agreement, which could contain a list of
evaluation questions the funder would like the program or initiative to address. This practice is
typical of most foundations and public agencies when making grants. However, the evaluation
does not have to be limited to answering just these questions. You can add your own according to
your own learning needs.
National evaluation — Sometimes, you are part of a larger multi-site initiative where an evaluator
is hired by the funder and is responsible for evaluating the entire initiative. That evaluator also will
have a set of evaluation questions. You could ask the same questions about your program site
(versus the entire initiative) or you might add questions specific to your program. It’s best not to
have completely different questions. That way, you can be efficient and collect data for both the
national and your own evaluation.
7.7 from logic model to evaluation questions to measurement framework
Logic modeling is the first and most important step in planning for your evaluation. With your logic
model and evaluation questions, you then can plan your evaluation. Using your logic model and
knowing the questions you want answered, you can develop a measurement framework, as an
evaluation planning tool.
The Step-by-Step Guide to Evaluation | page 125
Developing a measurement framework will allow you to determine how to assess progress toward
achieving outcomes and answer the evaluation questions.
With a measurement framework for your effort in hand, you get a clear picture of how to conduct
your evaluation. The measurement framework provides another opportunity for stakeholders to
further define outcomes. With it, you can consider what the outcome means in more concrete
terms.
7.7.1
Key Components of the Measurement Framework
Seven key components make up the measurement framework:
Outputs are direct products of activities and may include types, levels and targets of services
to be delivered by the strategy, initiative or program.
Outcomes are the immediate, intermediate and long-term changes or benefits you need to
document. These outcomes should be the same ones identified in the logic model.
Indicators are markers of progress toward the change you hope to make with your strategy,
initiative or program.
Measures of change are values - quantitative and qualitative - that can be used to assess
whether the progress was made.
Data collection methods are the strategies for collecting data. This could include quantitative
methods, such as conducting surveys or analyzing existing data, or qualitative methods, such
as conducting interviews or a document analysis.
Data sources are the locations from which (e.g., national database, program survey), or people
from whom, (e.g., program participants), you will obtain data.
Data collection frequency is how often you plan to collect data.
Page 126 | The Step-by-Step Guide to Evaluation
Indicators and measures of change are sometimes used interchangeably by various people, including funders and evaluators. Don’t be confused. What is most important to know is when there is progress toward the desired outcomes and how you will measure that progress.
The measure can be expressed in numerical form (e.g., percent change) or in words (e.g., first-time events, such as the first time participants felt empowered to save for a house because they now understood the process and had the skills to go through the steps).
7.7.2
How to Use a Measurement Framework
Once you have identified your outputs and immediate, intermediate and long-term outcomes,
you can list each output and outcome on the measurement framework in the first column. After
you have listed each one, you can make a clear plan for assessing progress toward that particular
output or outcome. This involves moving across the rows of the measurement framework from left
to right to identify indicators, measures of change, data collection methods, data sources and data
collection frequency for each outcome. Please note that as you complete the measurement
framework, some components could contain overlapping responses. For example, the data source
for two outcomes may be the same.
The Step-by-Step Guide to Evaluation | page 127
7.7.3
Considerations for Developing the Measurement Framework
Take contextual factors into account: Identify realistic indicators and measures of change. As
previously discussed, it is essential to consider positive and negative contextual conditions
that could affect the strategy, initiative or program’s success.
Involve key stakeholders: Involve individuals who are key stakeholders in the effort’s
implementation and evaluation. This will provide another opportunity to incorporate feedback
from individuals with diverse perspectives.
Understand your capacity: Be sure you have the needed capacity (human power, skills, time,
etc.) to collect and analyze data to measure the outcomes identified. If necessary,
consider how you can acquire additional resources, personnel or training to do so.
Create a living document: Like your logic model, your measurement framework is a living
document. It is a tool for planning, but should be regularly modified it based on changes in
your goals, activities, organization’s capacity or information gained from the data you are
collecting. When you adjust the logic model and evaluation questions, you also should adjust
the measurement framework.
7.7.4
Taking a SMART Approach to Measurement Framework Development
Let’s return to the previous example of the effort in Rainbow County to increase the number and
percent of working class African American homeowners in the county. Based on your experience
and findings from past research studies, you believe that instruction about what it takes to
purchase a home and about the services offered by financial institutions are not accessible to
working class African American families. To remedy those circumstances, your effort includes
services to educate families about budgeting, savings and improving their credit scores; acquiring
financing; and understanding what it means to be a homeowner. You also plan to work with the
county institutions to improve their lending policies, practices and services. Your theory of change
is that by working with the families and lending institutions, more residents will be able to
purchase homes. Hence, homeownership supports asset building and family economic security.
Page 128 | The Step-by-Step Guide to Evaluation
A SMART approach means:
Specific outputs and outcomes clearly state the issue of focus, target group and
timeframe. For example, in the above scenario, the outputs are the number of
courses per week or per month, the length of the courses, the content taught
and the average number of attendees. The outcomes specifically focus on the
individuals who are enrolled in the program.
Measurable outputs and outcomes are ones where you can clearly assess the
change that has occurred among the people affected by the effort. For instance,
in the Rainbow County example, participants’ understanding of steps in the
homeownership preparation process can be assessed using pre- and post-tests.
Achievable outputs and outcomes take into account the scale and scope of
outcomes that can be achieved based on time and resources available. For
instance, you can expect the outcomes are achievable because the program has
the resources to provide X number of courses over Y number of months for
families to learn the steps in the homeownership process.
Relevant outputs and outcomes (immediate, intermediate and long-term) work
toward your desired change in an incremental manner. For the effort, this
involves small steps to increasing the number of homeowners in Rainbow
County.
Time-specific outputs and outcomes mean the expected timeframe for changes
should be clear and realistic. In the Rainbow County example, the
program is implemented over the course of two years, taking into consideration
the amount of time potential homeowners might need to learn about and
engage in the homeownership process, and the amount of time to engage
lending institutions. Additionally, the measurement framework provides a tool
for laying out how impacts unfold and evolve. You also could find it helpful to
conduct research to determine how long to implement programs before you can
to observe and measure changes.
The Step-by-Step Guide to Evaluation | page 129
exhibit 7-4: sample measurement framework tool
Output/ Outcome Indicator
Measures of Change
Data Collection Methods Data Sources
Frequency of Data Collection
Specific outputs or changes (should be same information in your logic model)
Markers toward progress
Value for assessing progress
How data will be collected
Where data will be obtained from
How often data will be collected
Output
Number of participants who completed courses
Completion of courses
Number of participants who completed X percentage of courses
Average number of participants per course
Program staff records and attendance logs
Program staff records and attendance logs
Monthly (or according to the course schedule)
Immediate Outcome
Improved financial literacy among families
Use of banking services
A family budget
First time families opened a bank account and created a family budget
Interviews Course participants
Twice throughout the program
Intermediate Outcome
Increased savings
More money in the bank
Percentage of change in funds in the bank
Adherence to family budget
Interviews
Log to track monthly savings
Course graduates
Every six months after family graduates from program
Page 130 | The Step-by-Step Guide to Evaluation
Output/ Outcome Indicator
Measures of Change
Data Collection Methods Data Sources
Frequency of Data Collection
Long-Term Outcome
Increase the number of working class African- American home owners in Rainbow County
Percent homeowner- ship among African- American families
Homeowner- ship rates, disaggregated by race
Collect home ownership data from the county
County records Every two years
A useful resource on developing measurement frameworks is Building a Common Outcome Framework to Measure Nonprofit Performance, published by The Urban Institute in 2006. Available at http://www.urban.org/uploadedpdf/411404_nonprofit_performance.pdf.
7.8 your evaluation plan
Using the logic model, the evaluation questions, and the measurement framework tool as the basic
components, you can develop an evaluation plan that pulls all of these and more together. A good
evaluation plan should have the following elements:
Background Information about the strategy, initiative or program: Purpose of the evaluation,
origins of the effort, goals, theory of change and logic model.
Evaluation Questions: Specific questions that are measurable. These might need to be
prioritized in order to focus resources and keep evaluation manageable.
Evaluation Design: Data collection methods; types of data that will be collected; sampling
procedures; analysis approach; steps taken to ensure accuracy, validity and reliability; and
limitations. The measurement framework should be incorporated into this section.
Timeline: Completion dates and time ranges for key steps and deliverables.
Plan for communicating findings and using results to inform work: Details regarding what
products will be developed and what will be included in each product.
The Step-by-Step Guide to Evaluation | page 131
Budgetary Information (see Section 7.9 for more detailed information about how to develop an
evaluation budget): Could include expenses for staff time, consultants’ time, travel,
communications, supplies and other costs (e.g., incentives for participants, translation and
interpretation time).
Evaluator/evaluation team: Specify who is responsible for conducting the evaluation process
and what this role entails. This might be an external evaluator, internal evaluator or internal
evaluator with an external consultant.
Parts of your evaluation plan can be copied and used to write your evaluation report.
exhibit 7-5: side-by-side comparison of an evaluation plan and an evaluation report
Evaluation Plan Evaluation Report
Cover Page: Includes clear title; name and location of the strategy, initiative or program; period to be covered by evaluation.
Cover Page: Includes clear title; name and location of the strategy, initiative or program; period covered by evaluation or date evaluation was completed.
Background Information of the Effort: Purpose of the evaluation, origins and goals of effort, activities and services, including theory of change and logic model.
Executive Summary: Brief, stand-alone description of program, outline of evaluation purpose and goals, methods, summary of findings and recommendations.
Evaluation Questions: Specific questions that are measurable, might need to be prioritized to focus resources and keep evaluation manageable.
Introduction and Background: Purpose of the evaluation, origins and goals of effort, target population, activities and services, review of related research, evaluation questions and overview and description of report.
Evaluation Design: Data collection methods; types of data to be collected; sampling procedures; analysis approach; steps taken to ensure accuracy, validity and reliability; and limitations.
Evaluation Design: Data collection methods, types of data collected, sampling procedures; analysis approach; steps taken to ensure accuracy, validity, and reliability; and limitations (include the theory of change and logic model).
Page 132 | The Step-by-Step Guide to Evaluation
Evaluation Plan Evaluation Report
Timeline:Completion dates or time ranges for key steps and deliverables.
Evaluation Results: Evaluation findings, evaluation questions addressed, visual representations of results (e.g., charts, graphs, etc.).
Plan for Communicating Findings and Using Results to Inform Work: Details regarding what products will be developed and what will be included in each product. Potential users and intended audience include program staff and administration, program participants, community leaders, funders, public officials and partner organizations.
Summary, Conclusion and Recommendations: Summary of results, implications of findings, factors that could have shaped results, clear and actionable recommendations.
Budgetary Information: Might include expenses for staff time, consultants’ time, travel, communications, supplies and other administrative costs. (If the plan is to be shared with a wider group of people, information about staff and consultants’ time may be excluded.)
Evaluator and Evaluation Team: Specifications regarding who is responsible for conducting the evaluation process and what this role entails. This may be an external evaluator, internal evaluator or internal evaluator with an external consultant.
Various, surprising things can affect how long it takes to implement and complete an evaluation.
When developing the timeline of the evaluation, you should consider the following:
Factors that could affect respondent availability (school year cycles, summer months, etc.)
Factors that might affect data collection (data collector trainings, translation time of instru-
ments, Institutional Review Board (IRB) approval, data availability, etc.)
Upcoming opportunities to share findings (e.g., board meetings, annual meetings or funding
cycles, etc.)
Institutional Review Boards (IRBs) are entities set up to protect the rights and welfare
of people who participate in research. Evaluations of programs involving Native
Americans/Alaska Natives also require permission from their tribal governments.
The Step-by-Step Guide to Evaluation | page 133
Data collection that puts participants at any type of risk requires IRB review and approval.
Universities usually have their own IRBs and faculty or students must submit an application to the
IRB for approval before starting research. American Indian/Alaska Native communities usually have
their own IRBs and procedures to protect their members from being harmed by research. There also
are independent review boards if you (or your evaluator) are not affiliated with a university or
working with a tribal nation. Plan for this step and the amount of time it could take to get
approval, which can affect the schedule for data collection and analysis. This topic is addressed in
more detail in Chapter 8.
7.9 budgeting for evaluation
How much will the evaluation of your strategy, initiative or program cost? How much should you
budget for the evaluation when you submit your grant application? The cost of an evaluation
can vary widely, depending on the type of evaluation you want to conduct (e.g., performance
monitoring, process or formative evaluation, outcome or summative evaluation), the approach you
want to take (e.g., empowerment evaluation, developmental evaluation) and the most appropriate
evaluation methodology for your strategy, initiative or program (e.g., case study, quasi-
experimental design). Chapter 3 provides a more detailed explanation of what the different types,
approaches and methodologies mean for your organization.
Page 134 | The Step-by-Step Guide to Evaluation
A general guideline is to allocate 5 to 10 percent of your total program cost for performance
monitoring and/or a process or formative evaluation. That is, if your strategy, initiative or program
requires $100,000 to operate, you can estimate evaluation to cost between $5,000 and $10,000.
Consequently, your overall estimated budget for program and evaluation would range from
$105,000 to $110,000. If you plan to conduct an outcome or summative evaluation, consider
allocating 15 to 20 percent of your total program cost to evaluation.
Useful resources on budgeting for evaluation include:
The Evaluation Part of a Proposal Budget. Available at http://www.opportunitystudies.org/ repository/File/evaluation/evaluation_budgeting.pdf.
Evaluation on a Shoestring Budget, by Wilder Research, published in 2009. Available at https://www.wilder.org/Wilder-Research/Publications/Studies/Program%20Evaluation %20and%20Research%20Tips/Evaluation%20on%20a%20Shoestring%20Budget%20-%20 Tips%20From%20the%20Wilder%20Research%20Program%20Evaluation%20Series,%20 Fact%20Sheet.pdf.
7.9.1
Types of Expenses Typically Associated with Evaluation
If you hire an independent evaluator, you want to ensure these expenses, where applicable, are
covered in the budget the evaluator submits. This list can help guide your allocation of resources
and review of budgets from evaluators.
Consider labor costs for external evaluator, internal evaluator and other staff and consultants.
External evaluator (hourly or daily rate multiplied by the number of hours or days required
for the work)
Internal evaluator or staff person responsible for the evaluation (hourly rate multiplied by
the number of hours required for the work)
Staff person or other consultants who will assist with tasks such as data collection, facilita-
tion of organizational learning or transformation of findings into products to share with the
public (hourly rate multiplied by the number of hours required for the work)
The Step-by-Step Guide to Evaluation | page 135
Compensate people from whom you collect data.
Appreciation for their time in the form of cash or gift
cards (make sure it is for supplies they need and from a
store they can easily access)
Travel expenses for bus or cab fare
Childcare expenses, either included in the cash incentive
or value of the gift card or as payment for someone to
look after the participants’ children during meetings and
focus groups
Translation or interpreting fees if you need to hire translators
or interpreters to reach populations whose primary language
is not English. You might need interpreters to be present
during interviews, focus groups or other types of data
collection activities.
Most translators charge by the word or page or a flat fee.
The translation fees will vary depending on the
complexity of the subject matter, number of words to
be translated into another language, amount of time to
complete the
assignment and language combination.
Most interpreters charge by the hour or day. The fees will
depend on the geographical location, the nature of the
assignment and interpreter’s qualifications.
Supplies will be needed
Duplication of paper-and-pencil survey questionnaires,
consent forms and any other materials needed for
data collection
Refreshments and food at data collection venues
Postage if the questionnaires are distributed by mail and
returned in self-addressed stamped envelopes
It’s important to appropriately compensate, based
on a living wage, for the respondents’ time. There might be some implications for people who receive public benefits, so be sure to look into this and, if necessary, find alternative and appropriate ways to compensate them for their time and effort.
A useful resource on compensation is Compensation for Research
Subjects, published by the Committee for Protection of Human Subjects at the University of California, Berkeley. Available at http://cphs.berkeley.edu/ compensation.pdf. Your IRB should also have guidelines that you can follow.
A useful resource on translation fees can be found at http://www. affordablelanguageservices. com/translation-costs. Some state government websites also contain information about interpreting fees.
Page 136 | The Step-by-Step Guide to Evaluation
Institutional Review Board (IRB) fees could be an expense.
If the evaluator or somebody in your organization is affiliated with a university, the fees may
be waived
If not, you can use an independent, private IRB that charges a fee for its services and
typical charges range from about $500 to more than $1,000
7.9.2
Time Estimates for Specific Evaluation Activities
Proposed budgets, including evaluation budgets, are based on a set of assumptions. If you are
developing an evaluation budget, keep track of your assumptions. If you are working with an
external evaluator, it might be helpful to ask the evaluator to clarify his or her assumptions. In
addition to time cost and the expense of designing instruments, preparing for data collection and/
or analyzing the data, typical data collection activities include:
Administration of paper-and-pencil surveys.
Depending on the length of the survey questionnaire and the amount of time required to
help the respondent complete the questionnaire, it could take between 15 and 30 minutes,
on average, to administer each questionnaire. It imight take longer if the questionnaire is
complex.
For example, if you administer the survey at a community event where you expect to get 30
people to complete the questionnaire, the total time required for the survey can range from
approximately 7.5 hours (15 minutes x 30 people) to 15 hours (30 minutes x 30 people). If
you engage two administrators, the total number of hours for the survey won’t change, but
the time spent by each administrator will be shorter.
Administration of online surveys.
The cost of administering an online survey can be divided into three parts:
i. Compiling the list of email addresses to send the questionnaire, which might take
30 minutes or more, depending on the total number of recipients.
ii. Identifying the nonrespondents and sending them reminders, which could take
30 minutes or more, depending on the number of nonrespondents.
The Step-by-Step Guide to Evaluation | page 137
iii. Identifying the people who remained unresponsive even after sending them email
reminders and contacting them by telephone to encourage them to respond; this could
take five to 10 minutes per call, on average.
For example, if you administer an online survey to 30 people, the activity will require
approximately 2.5 hours (30 minutes to compile their email addresses, 15 minutes to send
one reminder and 10 minutes to contact 10 people, assuming that one-third of the sample
will remain unresponsive).
Implementation of focus groups.
Scheduling a focus group could take two to four hours, depending on the number of
schedules to coordinate, and it might take one hour to secure a location for the focus
group.
A focus group usually takes one to two hours, on average, not including set up and
take down.
There should be a facilitator and a note-taker at the focus group, which typically translates
into two to four hours for staff time per focus group.
Some evaluators also might want to record the focus group discussion and have a formal
transcription of the discussion, which could increase the cost of the focus group.
Conducting interviews.
Scheduling an in-person or telephone interview usually takes 15 minutes on average per
subject.
An interview typically takes between 30 minutes and one hour.
There should be an interviewer and a note-taker, which typically averages one to two hours
of staff time per interview.
Some evaluators also might want to record the interview and have a formal transcription of
the interview, which could increase the cost.
Page 138 | The Step-by-Step Guide to Evaluation
HIGHLIGHTS
A logic model is an important living
document for your strategy, initiative or
program. However, there should be equal, if
not more, emphasis on the process of logic
modeling.
Engaging stakeholders in the logic modeling
process strengthens your strategy, initiative
or program.
The logic modeling process will help surface
the various assumptions stakeholders might
hold, promote shared understanding,
resolve disagreements and generate new
and innovative ideas.
Good facilitation is key to a successful logic
modeling process.
A logic model can help generate evaluation
questions. You can see what phase the
program is in and therefore, which
questions to ask and when it is best to
answer them.
To formulate evaluation questions, look at
your logic model and ask:
Who your effort benefits What the effort is intended to do When activities occur Why the effort is important How the effort will bring about the
desired change
A measurement framework can be
generated from the logic model. A
measurement framework enables you to
determine how to assess your progress
toward achieving outcomes. The
measurement framework includes seven
main components: outputs; outcomes;
indicators; measures of change; data
collection methods; data sources; and data
collection frequency.
Using the logic model, evaluation questions
and the measurement framework as basic
components, you can develop an evaluation
plan, which pulls this information - and
more - together.
The cost of an evaluation can vary
widely, depending on the type evaluation
you conduct, the approach you take and the
most appropriate evaluation methodology
for your strategy, initiative or program.
A proposed evaluation budget is based
on a set of assumptions. It is important to
document the assumptions or clarify the
assumptions in your budget or with your
external evaluator.
The Step-by-Step Guide to Evaluation | page 139
exercises
1. Processes, tools, events, technology and actions that are parts of a program are examples of
.
A. Outputs
B. Resources/inputs
C. Program impact
D. Short-term outcomes
E. Activities
2. A logic model can be used to .
A. Design your program
B. Inform implementation
C. Develop evaluation questions
D. Create an outline for your evaluation report
E. All the above
3. Outcomes are .
A. Attendance rates for a parenting program
B. Total number of sessions conducted for a leadership program
C. Changes in youths’ attitudes and behaviors related to eating healthy
D. All of the above
E. None of the above
4. Long-term outcomes are typically achieved in a year or less.
A. True B. False
Answers: 1E; 2E; 3C; 4B
Page 140 | The Step-by-Step Guide to Evaluation
5. Consider your program or initiative. Are the strategies or activities aligned with the outcomes
you are trying to achieve? Is there a logical connection between them? If not, what needs to be
adjusted?
6. Consider a measurement framework that you may have developed for a program or initiative.
After reading Chapter 7, is there anything you would change in the measurement framework?
7. Consider an evaluation that you conducted in the past. Were there any expenses you wished
you had budgeted for? If you were to develop a budget for a similar evaluation today, what
labor and other expenses would you include in the budget?
The Step-by-Step Guide to Evaluation | page 141
8 Data Collection and Analysis
Page 142 | The Step-by-Step Guide to Evaluation
A PREPARE for conducting an evaluation
B DETERMINE stakeholders and how and when to engage them
C IDENTIFY assumptions and determine what will be di�erent (theory of change and logic model)
D DEVELOP evaluation plan (logic model, measurement, framework, etc.)E COLLECT and analyze data
F COMMUNICATE results and understand what happened (interpret �ndings and facilitate learning)
G MAKE informed decisions (improve actions and next steps)
Stakeholders
introduction You might have wondered why some evaluations
use surveys
while others use focus groups. Is one better than
the other? Not always. The data collection method
depends on what you want to know, how much
money and time you have and what the people
involved are comfortable with. This is your part
in the decision-making process. You should be
aware, nevertheless, of the tension among
funders, policymakers and evaluators regarding
quantitative and qualitative methods and data.
Some people prefer numbers over narratives and
might discount one form of data over the other.
You should not let their preference sway you from
what is best for your effort and evaluation;
nevertheless, you should be prepared to justify
your choice of method or methods. Systematic data collection and analysis is
essential to evaluative thinking and brings rigor to the story you can tell about your program, strategy,
policy or organization.
Evaluative thinking is a systematic process of collecting and analyzing data to tell the
story about your strategy, initiative, program, policy or organization. It is based on the
belief that a systematic process is valuable and necessary, which involves identifying
assumptions about what you think works and doesn’t work and why; posing thoughtful
questions about what you expect to see change during and after implementing your
effort; pursuing deeper understanding through reflection and dialogue; communicat-
ing what was learned without underestimation or exaggeration; and making informed
decisions in preparation for action.
Important considerations to include when designing your data collection or reviewing your evaluator’s
design:
Who will collect the data - your program staff or the evaluator? If the former, do they have the neces-
sary skills? If the latter, are funds available to hire the person?
How will you reach certain populations who are uncomfortable with, and distrustful of, people who
want information from them?
The Step-by-Step Guide to Evaluation | page 143
What knowledge, skills and other resources (e.g., staff capacity,
money and technology) are needed and available for data collec-
tion?
How much time will it take and what will be the final result?
What does it mean to maintain confidentiality and anonymity for
your participants?
What is an Institutional Review Board (IRB), do you need such a
review and how do you get it?
8.1 determine data collection methods
8.1.1
What You Need to Ask Before Collecting Any Data
There are many ways to collect data, some more costly and
time-consuming than others. Therefore, before you start
collecting data, it is imperative to consider the following
questions with stakeholders such as your organization’s
leadership, key staff, board members or an advisory committee.
How this chapter is organized…
8.1 Determine Data Collection Methods
8.1.1 What You Need to Ask Before Collecting Any Data
8.1.2 Use of Quantitative Methods
8.1.3 Use of Qualitative Methods
8.1.4 Use of Mixed Methods
8.2 Critical Considerations in Data Collection
8.2.1 Who the Data Collector Is
8.2.2 Howto Engage Populations Who Have Been Traditionally Excluded or Treated as Invisible
8.2.3 Resources
8.2.4 Confidentiality and Anonymity
8.2.5 Obtaining Consent
8.2.6 Institutional Review
Board (IRB)
8.3. Analyzing and Interpreting Data
8.3.1 Quantitative Data Analysis and Interpretation of Results
8.3.2 Qualitative Data Analysis and Interpretation of Results
Highlights
Exercises
Page 144 | The Step-by-Step Guide to Evaluation
exhibit 8-1: essential questions to ask before collecting data
Evaluation Purpose
What are the evaluation questions? Which methods will help answer them and provide the most reliable and valid data?
Who is the primary audience for the findings? What types of data will make the most sense and be most useful to them?
Data Sources
Who is providing the data? Who are the potential respondents? How many respondents are needed?
Timing
How long is the strategy, initiative or program?
How much time was budgeted for data collection and analysis?
When is the right time to collect the data?
When will the results become available for reporting to the funder, board members, community leaders and other stakeholders?
Resources
What is the evaluation budget? How much of it is for data collection and analysis versus reporting and dissemination of findings?
Who should collect the data - staff or someone else? Does staff have the time and skills? Should an external evaluator be hired?
Source: The table was adapted the Evaluation Toolkit, published by the Pell Institute and Pathways to College Network. Available at http://toolkit.pellinstitute.org/evaluation-guide/collect-data/
A resource on data collection methods is the User-Friendly Handbook for Project Evaluation, published by the National Science Foundation in 2002. Available at http://www.nsf.gov/ pubs/2002/nsf02057/nsf02057_5.pdf.
The Step-by-Step Guide to Evaluation | page 145
8.1.2
Use of Quantitative Methods
Determining if you should use quantitative methods.
Depending on the answers to the questions raised in Exhibit 8-1, quantitative methods such as
surveys or assessments could be right for you. Or qualitative methods such as interviews or focus
groups might suit your needs better.
Types of questions that quantitative methods can help answer
At the most basic level, quantitative methods are concerned with what, who and when. Therefore,
you should consider quantitative methods if your evaluation questions include inquiries about
who participated and benefited from your program; what they learned or got from your program;
what changes were brought about by your strategy, initiative or program; and when the changes
occurred. Also, if you need to generalize (apply to other settings) the findings or make predictions
about the results of your organization’s work, quantitative methods are better.
Examples of quantitative data include:
Number of people who attended program activities over the course of the year - participation
rate.
Whether or not program participants developed new knowledge and skills - percent change in
knowledge and skills before and after participation.
Number of students in your school district who graduated in any given year, including those
who enrolled in ninth grade for the first time, plus number of transfer students who joined the
class, minus the students who left - high school graduation rate.
Number of deaths of infants younger than 1 year old per 1,000 live births in your county - infant
mortality rate.
Primary audience for the findings
Quantitative methods generate data that appeal to people who prefer information that quantifies
impact and provides the “bottom line.”
Potential respondents and sample size
Quantitative methods can be helpful because it’s relatively less expensive to administer a survey
than to conduct interviews or focus groups with a lot of people (think about the time it takes to
send a survey questionnaire to 50 people, compared with talking to 50 people for 30 to 45 minutes
Page 146 | The Step-by-Step Guide to Evaluation
each). Simultaneously, you should consider whether potential respondents will be uncomfortable
with surveys and questions that ask them to rate something on a scale of 1 to 10. This is usually
true for people with low literacy, very limited English proficiency and distrust of surveys because
they don’t know where the data go and how the data are used. If the potential respondents are
uncomfortable, you might not get a good response rate or high quality data. In this case, conduct-
ing interviews or focus groups with fewer respondents could be preferable because it would yield
better quality data.
Amount of time for data collection and analysis
Quantitative methods are useful if the amount of time to collect and analyze data is very limited.
A survey with close-ended questions (i.e., you provide response options that respondents can
select) that ask people to rate something takes less time to administer than scheduling and
conducting interviews and focus groups. Running calculations of frequencies, averages or
percentages also will take less time than reviewing, coding and analyzing qualitative data (i.e.,
notes from discussions and interviews). Quantitative methods are equally useful if the program
extends over two or more years or if the change is expected to occur over the course of several
years (e.g., changes in high school graduation rate), because you can compare baseline data with
subsequent data.
Closed-ended questions provide discrete, multiple-choice answers that respondents
can select.
Budget and other resources
In general, under comparable conditions (i.e., amount of time for data collection, sample size),
quantitative methods can be less expensive than qualitative or mixed methods (which use a
combination of quantitative and qualitative methods) for several reasons. Secondary data such
as number of emergency room visits or crimes committed are usually available at minimal or no
charge. Surveys can be administered online or by mail, which costs less than the typical resources
needed to travel to an interview or focus group. An analyst needs less time to calculate
frequencies and percentages than to read and code text from interview and focus group
transcripts. Also, a different set of skills is required to do these calculations than to code text and
generate themes.
The Step-by-Step Guide to Evaluation | page 147
exhibit 8-2: advantages and disadvantages of quantitative methods
Advantages Disadvantages
You can collect data from a large sample of people. Sensitive information on topics such domestic violence, drug use, racism, immigration status and other personal matters are difficult to obtain.
You can analyze the data relatively quickly and easily, especially if you are using software packages such as Excel, STATA, SPSS, etc. .
They generally do not explain the reasons for responses.
It does not require a lot of money if a survey is used and administered online or by mail.
They may not be comfortable or familiar for certain groups of people who distrust mail or telephone surveys, have low literacy, or come from oral societies where written words are not part of their traditions.
The results can be generalized if the sample is representative of the study population. (There are specific statistical methods to calculate the rep- resentativeness of a sample; ask someone with expertise in sampling.)
Different types of quantitative data collection methods.
There are two major ways for collecting quantitative data: (a) surveys and (b) tests and
assessments.
Surveys
Surveys are one of the most popular ways to collect quantitative data. In a survey, a questionnaire
is distributed to a group of people to complete. While such questionnaires could include
open-ended questions, closed-ended questions are typically used to collect quantitative data.
Statistical analysis can be applied easily to responses to closed-ended questions.
Open-ended questions that ask respondents to resply through written text.
Page 148 | The Step-by-Step Guide to Evaluation
Make the questions short and clear, ideally no more than 20 words. Be sure to give the
respondents all the information they will need to answer the questions (e.g., select the best
answer, select all the options that apply).
Start with questions that are not personal and fairly easy to answer.
Avoid questions that have more than one central idea or theme (sometimes referred to as
“double-barreled” questions). Example of a double-barreled question:
Did the voter education campaign increase your knowledge about how to register to
vote and to learn about the candidates running for governor?
This question contains two ideas; the voter education campaign could have affected the
person’s knowledge about each one differently.
Do not use jargon or vocabulary that your respondents might not know.
Avoid inexact words that are open to interpretation (e.g., “generally,” “usually,”
“average,” “typically,” “often” and “rarely”).
Make sure the answer to one question relates smoothly to the next. For example, if
necessary add, “If yes, did you … ?” or “If no, did you … ?”
Give a time period if asking respondents to think about something they did before (e.g.,
during the last six months, how often did you … ?).
Think carefully about terms or concepts that could be interpreted differently in other
cultures. If needed, consult someone who shares the same culture as your potential
respondents (e.g., family in some cultures refers to both nuclear and extended family
members; “house” and “home” mean the same thing in some cultures and languages while
you, in fact, might be interested in the structure of a building - “house” - versus the feel-
ing of belonging - “home”).
checklist Simple rules to follow when developing a survey:
The Step-by-Step Guide to Evaluation | page 149
exhibit 8-3: when to use various survey methods
Survey Method When to Use
Mail surveys You have a complete and accurate mailing list.
The people you want to survey have some interest in the survey topic and are likely to respond.
The people you want to survey prefer this method (e.g., less tech-savvy individuals).
You want to give respondents time to consider their answers or other information when completing the survey.
Telephone surveys You need results relatively quickly.
The people you want to survey have telephones.
The people you want to survey have difficulty using written or online surveys (e.g., respondents with low literacy rates, poor eyesight, or no access to computers).
The survey is relatively short to discuss by telephone.
Internet or web-based surveys
You need results relatively quickly.
The people you want to survey are regular Internet users.
Your survey is short and simple.
Your survey is more complex with skip patterns (e.g., responses to a question determine which questions to be answered later, etc.). This is easier to do when the questionnaire is in electronic format than in hardcopy format.
Handout surveys (the questionnaire is handed out for people to complete in paper form or on a tablet [e.g., iPad])
You want to capitalize on who is available (e.g., people at a conference, community festival, etc.).
The people you want to survey might not be available or accessible to you again.
Page 150 | The Step-by-Step Guide to Evaluation
Survey Method When to Use
Face-to-face surveys (you go over the questionnaire in person)
Your survey questions are too complex and may need in-person explanation.
There is concern that people would not respond willingly unless someone they trust is present to reassure them about the content of the questions.
The people you want to survey are unreachable by phone, email or Internet.
Budget is not an issue and you have trained interviewers who can administer the survey consistently and properly.
Source: E. Taylor-Powell & C. Herman (2000). Collecting Evaluation Data: Surveys. Available: http://learningstore.uwex.edu/assets/pdfs/g3658-10.pdf
Tests and assessments
Tests and assessments can be useful tools in evaluation that can generate information about
changes in health status or behavior that your organization, strategy or program helped to make.
However, developing the right tests and assessments requires expertise and specialized training
to properly design, administer and analyze. Some tests and assessments may already exist, and
the data are available for use, (e.g., SAT scores, blood sugar levels and community walkability
indexes).
exhibit 8-4: different types of tests and assessments
Physiological health status tests
Can be used to reveal the health needs of a certain population or indicate the extent of a particular health problem in a target population or community. Examples of these tests are broad-based screenings — such as cholesterol, blood sugar or blood-pressure readings — and birthweights of babies.
Physiological tests also can be used to measure your strategy, initiative or program’s outcomes. For instance, if you are operating a prenatal care program, you can compare the birthweights of infants born to mothers in your effort with mothers who are not in a prenatal care program. Statistical tests for significance can be applied to this kind of data to further confirm the effects of your effort.
The Step-by-Step Guide to Evaluation | page 151
Knowledge or achievement tests
Can be used to measure changes in participants’ knowledge. Through testing before and after the program, you can assess what the participants need to learn, and then measure what they have actually learned. Be aware, however, that a person’s knowledge does not mean the person is using that knowl- edge in everyday life.
Another type of knowledge or achievement testing is done through observation, as when a staff member observes a mother interacting with her child during a home visit. The observation includes a set of behaviors the staff member must be trained to look for and record.
Other types of assessments and inventories
Can be used to measure need and assess outcomes. You may develop your own instruments to determine, for example, if your clients are satisfied with the services you offer. Standardized questionnaires developed by health researchers on such topics as patient satisfaction, general health (including items on physical, emotional and social function), mental health, depression and disability status also can be used. Such standardized tests also exist for studying children’s experiences with schools, qualifications of teachers and other education-related matters.
There are advantages to using a questionnaire that has already been developed and field-tested. For example, you don’t have to spend time and money creating your own. However, keep in mind that standardized assessments might not adequately reflect the important and unique aspects of your initiative program or the situation or culture of the people you serve. If you are using an existing assessment, find out where it was tested before and if it is culturally appropriate for the group of people you hope to survey.
Useful resources on analyzing quantitative data include:
Analyzing Quantitative Data, by E. Taylor-Powell, published by the University of Wisconsin Cooperative Extension. Available at http://learningstore.uwex.edu/assets/pdfs/G3658-6. pdf.
Analyzing and Interpreting Data, published by Wilder Research. Available at http://www. evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09.pdf.
Page 152 | The Step-by-Step Guide to Evaluation
8.1.3
Use of Qualitative Methods
Determining if you should use qualitative methods. Depending on the answers to the questions raised in subsection 8.1.1, qualitative methods could
be right for you.
Types of questions that qualitative methods can help answer
At the most basic level, qualitative methods are concerned with why and how and are useful for
in-depth study of a particular issue rather than a broad study. Therefore, if your evaluation
questions include inquiries about how the participants in your strategy, initiative or program
applied what they learned in their daily life, or why the community leaders had difficulty mobilizing
the residents to take action on an issue, you should consider qualitative methods. Qualitative data
also provide contextual information about your organization, effort and community, which could
mean more to the program director who must make recommendations for improvement or to the
policymaker who must revise an existing policy.
Examples of qualitative data include:
Factors that motivated participants to attend (or not) activities during the year
Ways in which program participants applied their new knowledge and skills
Reasons for the decrease in high school dropout rate over the past two years
Reasons for the increase in number of deaths in your county over the past five years among
infants younger than one year old
Primary audience for the findings
Qualitative methods generate data that appeal to audiences who prefer information in the form
of case studies, stories and rich descriptions, and who are curious to know what lies behind data
trends and statistics.
Potential respondents and sample size
Qualitative methods are helpful if you are working with a smaller number of people, mainly
because conducting interviews and focus groups with a lot of people can be expensive and time
The Step-by-Step Guide to Evaluation | page 153
consuming. Qualitative methods also help if the people
you want to collect information from feel more comfortable
expressing their opinions verbally than in written form.
Finally, qualitative methods are useful when the information
to be collected contains a lot of nuances due to cultural and
language differences.
Program length and amount of time for data collection
Qualitative methods can be useful regardless of program
length (e.g., a summer youth program, a six-month training
institute, or a year-long prevention program). The evaluation
plan should allow enough time to analyze the data because
coding text and identifying the themes that emerge from the
coding is more time-consuming than calculating
frequencies, averages or percentages.
Budget and other resources
In general, under comparable conditions (i.e., length of
program, amount of time for data collection, sample size),
qualitative methods can be more expensive than
quantitative or mixed methods for several reasons. There
may be travel costs for the data collector and for the
respondents, childcare expenses for parents who
participate in a focus group, or interpreter fees for
people who don’t speak English. An analyst also needs
more time to read and code text from interview and focus
group transcripts and observation notes. Finally, a different
set of skills is required to code text and generate themes
than to calculate frequencies and percentages.
Page 154 | The Step-by-Step Guide to Evaluation
exhibit 8-5: advantages and disadvantages of qualitative methods
Advantages Disadvantages
Provides understanding and description of in-depth experiences by individuals in your strategy, initiative or program.
Not useful if you want to generalize findings to the whole study population or community (i.e., findings may be relevant only to the people your organization or program serves).
Provides you or the evaluator with an opportunity to explain definitions or questions that are unclear to participants.
Participants may not feel comfortable verbalizing and discussing sensitive topics.
You or the evaluator can easily guide and redirect questions in real time.
Collecting and analyzing data can be expensive and time consuming.
Findings may be easier to interpret for some of your stakeholders who are uncomfortable with numbers and other forms of quantitative data.
A useful approach when no readily available, field-tested survey questionnaires or assessment tools exist for the topic you want to explore.
Different types of qualitative data collection methods
There are three major ways for collecting qualitative data: interviews, focus groups and
observations. Each method has pros and cons, depending on whom you want to collect data from
and other factors.
Interviews
There are two types of interviews: structured and semi-structured. In structured interviews, the
questions are written out exactly the way they should be asked, and the interviewer should ask
every respondent the questions in the same order.
In a semi-structured interview, topics are listed and examples of probes are provided, and the
interview becomes more of a discussion. The interviewer can tailor the questions - to a certain
degree - to the respondent’s role in the program and cultural and language background.
The Step-by-Step Guide to Evaluation | page 155
Interviews also can be conducted by telephone or in person. The former can save time and money
(e.g., no travel to the interview site). The latter is helpful if respondents’ body language is
important to observe and if you expect respondents to be more engaged by encountering the inter-
viewer face-to-face.
Be very familiar with the interview questions. Read them several times before the
interview to ensure comfort with the questions and their flow.
Be prepared to rephrase questions or ask them out of order if you are conducting a
semi-structured interview. Respondents may need questions reworded to understand
their meaning. They also could answer a question listed later in the interview guide in
their initial response; if this happens, you don’t want to ask that particular question when
you come to it in the guide and act as if you didn’t hear the respondent’s answer in the
first place. You want to either rephrase the question to see if you get the same answer,
skip it, or reflect on what the respondent said earlier and ask him or her if you
understood it correctly.
Be relaxed. Respondents can sense your comfort or anxiety. The more relaxed you are, the
more relaxed respondents will be. This helps the interview flow smoothly.
Avoid rushing respondents. Allow enough time for respondents to answer questions
before assuming they do not have something to say.
Maintain eye contact. Don’t be so involved with your notes that you appear disengaged.
Avoid making any faces or sounds that could be perceived by respondents as
disapproval, approval, or any sort of judgment about what they said.
If you want to record an interview, ask for permission first. If you see indications that the
presence of a tape recorder makes the respondent uncomfortable, consider turning off
the recorder.
checklist Tips for conducting interviews:
Page 156 | The Step-by-Step Guide to Evaluation
Focus groups
Focus groups are structured discussions to understand people’s perspectives, experiences or
knowledge about a particular topic. A moderator suggests topics and facilitates the discussion.
The goal is to discover the how and why of something, to get contextual responses rather than
“yes” or “no” answers. You can use a focus group to answer questions that you might ask in an
interview, such as:
What motivated you to participate in the program?
Why do you think more and more infants younger than 1 year old are dying in your county over
the past five years (after you show them the infant mortality rate)?
How are you using the knowledge and skills you learned from the program?
Generally, you can ask the same questions in focus groups that you might ask an individual during
an interview. However, focus groups are different from interviews in that:
Participants can bounce or build off each other’s comments and provide richer information.
The setting provides an opportunity to observe interactions and dynamics among people that
mirror real life.
People can get uncomfortable when they have to answer certain questions in front of other
people (e.g., personal questions about their families or anything about finances or
legal matters).
The unit of analysis is the group and not individuals in the group. Consequently, you can’t
summarize the findings by stating things like, “Four people indicated they now know how to
read food labels while the remaining six people said they continue to struggle with
understanding the labels.” Instead, the finding should read, “Participants were mixed in their
ability to understand food labels after the program.”
Focus groups usually consist of six to 12 people. These groups are not directed at, or focused on,
getting consensus.
The Step-by-Step Guide to Evaluation | page 157
Ask no more than five questions and make sure they are not phrased in a way that solic-
its a “yes” or “no” response, but instead generates discussion.
Be very familiar with the questions. Read them several times before the focus group to
ensure comfort with the questions and their flow.
Have a process for managing the group dynamics. For example, if someone in the group
doesn’t say anything, plan how to engage that person. Alternatively, if someone
dominates the discussion, plan how to give others a chance to talk.
If you want to record the focus group, make sure everyone in the group gives permission.
Position the recorder so everyone’s voices will be captured clearly. If it appears the
presence of a tape recorder makes even one respondent uncomfortable, consider taking
handwritten notes or turning off the recorder.
checklist Tips for conducting focus groups:
Focus groups require excellent facilitation skills. Sometimes, focus groups can surface tensions, conflicts or other emotions, and the person conducting the focus group must be prepared to deal with the situation. To do this, you or your staff must carefully consider
what could go wrong during the focus group, work with the evaluator or facilitator to prepare for uncomfortable moments (e.g., two people getting into an argument, someone bursts into tears) and agree on how to handle them.
Page 158 | The Step-by-Step Guide to Evaluation
Observations
Observations are structured means of recording the actions and interactions of participants in an
evaluation. They provide an opportunity to collect data on a range of behaviors, capture
interactions and openly explore the topic of interest in the evaluation. Observations can be
quantified or described qualitatively. Observations as a data collection method can be useful if you
want to understand the context within which an activity for the strategy, initiative or program takes
place. For instance, observations are common in evaluations of classroom instruction techniques
and child development activities.
Use a structured protocol that makes it easier for the observer to gather and record
observations. The protocol could include criteria for the setting, quality of interactions,
content of activities, behaviors of participants, and anything else that is supposed to be
part of the program and, most important, observable.
Avoid asking the observer to record things that cannot be observed, such as what people
are thinking or why they did something.
Be clear about the role of the observer and how the observer should behave in the setting.
checklist Tips on using observations:
Review of artifacts (e.g., documents, recordings, videos)
This method is important because artifacts can be a source of data for the evaluation. Artifacts
include mission statements, organizational charts, annual reports, activity schedules, audio
recordings, diaries, videos, grant proposals, participant utilization records, promotional
literature, etc.
Such materials can enable you to learn about the history, philosophy, goals, outcomes and
challenges of a particular program; these materials provide clues about important shifts in
program development or maturation. Document reviews also can help you formulate questions for
a survey or an interview. Bear in mind that written documents do not necessarily provide
The Step-by-Step Guide to Evaluation | page 159
comprehensive or correct answers to specific problems because they could contain errors,
omissions or exaggerations. They simply provide one form of information and should be used
carefully and in connection with other types of data.
Useful resources on qualitative data analysis include:
Analyzing and Interpreting Data, published by Wilder Research. Available at http://www. evaluatod.org/assets/resources/evaluation-guides/analyzing-interpretingdata-8-09.pdf.
Qualitative Data Analysis, by M. Miles, M. Huberman and J. Saldaña, published by SAGE Publications in 2014.
8.1.4
Use of Mixed Methods
Over the past three decades, a trend in evaluation has been to shift toward mixing quantitative and
qualitative methods into a single evaluation called mixed method evaluation. This approach makes
sense because each method has its own strengths and weaknesses, as discussed in
previous subsections. Combining them can lead to a stronger, more complete evaluation than a
conventional evaluation that uses only one method.
A mixed method evaluation systematically integrates two or more evaluation methods,
usually drawing on both quantitative and qualitative data, such as using surveys and
focus groups in one evaluation study.
Using examples provided previously about quantitative and qualitative methods, you could use
mixed methods in the following ways:
Track and analyze the attendance of people in your program over the past six months. You
notice participants from a particular neighborhood don’t attend regularly and see a dip in
attendance in January and February. You can conduct focus groups to find out why.
Administer a survey questionnaire and ask program participants about a set of behaviors. You
can then conduct interviews with a subset of participants to find out how your program helped,
or did not help, them change those behaviors.
Page 160 | The Step-by-Step Guide to Evaluation
Monitor the high school graduation rate in your community. You see a decrease over the
past five years and conduct a series of focus groups with various people such as principals,
teachers, parents and students, to find out what factors they believe are contributing to the
decrease.
Observe an increase in the number of deaths in your county of infants younger than 1 year per
1,000 live births. You can conduct interviews with parents whose infants have died (remember,
don’t conduct a focus group because of the sensitivity of the subject) and focus groups with
service providers to find out what they believe is contributing to the increase.
exhibit 8-6: advantages and disadvantages of mixed methods
Advantages Disadvantages
Provides a more complete understanding of the program, its implementation and outcomes than quantitative or qualitative methods alone.
Can be more expensive and time consuming than quantitative or qualitative methods alone.
Can be more efficient when the findings from one method inform the content for another method. For instance, the answers gathered from a focus group can be used to develop response options for a survey questionnaire.
It can be difficult to find an internal or external evaluator with expertise in both qualitative and quantitative methods.
Enables triangulation where a finding can be con- firmed by multiple data sources.
The Step-by-Step Guide to Evaluation | page 161
8.2 critical considerations in data collection
Several considerations in data collection don’t always get
enough attention early on in the evaluation process. These
considerations are especially important when working
in communities that have traditionally been excluded or
devalued in evaluation and research studies.
8.2.1
Who the Data Collector Is
The decision about who will collect the data must be
carefully made. You want respondents to feel comfortable
with the person asking for data and to respond honestly and
confidently. You don’t want respondents to be intimidated
or fearful. A data collector can be someone external to your organization or someone within your
organization (e.g., staff person, board member or volunteer). You should make your decision based
on several characteristics about the data collector. The data collector should:
Not be invested in getting positive or negative answers from respondents and thus be
compelled to sway the respondent.
Not be someone a participant feels compelled to please (e.g., someone who has power to
provide or withhold service).
Have the skills to interact respectfully with respondents (e.g., not “talk down” to people) and
have the humility to recognize that he or she does not know everything there is to know about
another group of people, especially if they come from a different cultural background
(e.g., don’t assume that everyone greets each other with a handshake or a hug).
Be able to follow instructions and protocols consistently.
Be discreet and not share any information provided by one respondent with another
respondent or with people other than the evaluation team.
Page 162 | The Step-by-Step Guide to Evaluation
8.2.2
How to Engage Populations Who Have Been Traditionally Excluded or Treated as Invisible
If your evaluation involves such populations (e.g., undocumented immigrants, low-income
families, people of color, people with different sexual orientation, people with disabilities, victims
of violence and incarcerated populations), keep in mind several critical considerations throughout
the evaluation process, including:
Their past experiences with researchers and evaluators (e.g., is this a community that has been
studied before by all sorts of research or evaluation groups).
Logistical and other challenges faced if they have to travel to another location for an
interview or focus group (e.g., they have to take two buses to get there, they don’t have money
for bus fare, they need someone to watch their children while attending the focus group).
Cultural differences that could cause discomfort (e.g., women from some cultures are not
allowed to be left alone to talk to male data collectors; undocumented immigrants might be
afraid to speak to anyone they don’t know; some words, phrases and gestures could be
disrespectful in another culture).
Language differences that could cause discomfort and misunderstandings (e.g., is an
interpreter needed, will the interpreter keep the information he or she hears confidential, is
the translation accurate and appropriate for that particular cultural group).
Literacy level, which could cause discomfort and embarrassment if someone cannot read or
write well.
Framing of findings within context to not perpetuate stereotypes or suggest the victims are to
be blamed for their situations.
The Step-by-Step Guide to Evaluation | page 163
8.2.3
Resources
Certain expenses must be carefully accounted for and not forgotten when you create the evaluation
budget, including:
Professional translation or interpreting fees
Most translators charge by the word or page while interpreters charge by the hour. They
also could charge a fixed fee depending on the assignment. Get two to three quotes for
comparison.
Expert review of the instrument’s reading level
Teachers of English as a Second Language (ESL), Adult Basic Education (ABE) and General
Education Development (GED®) test (or other high school equivalency preparation) are
good resources in your community for this purpose. Yhey typically charge by the hour or you
can negotiate a fixed fee for services.
Transportation
Bus and subway fares vary across cities. Budget a little bit more in case a few people
missed the bus and have to take a taxi.
Childcare assistance
Consider how many childcare providers you could need depending on the number of
children, or include the cost in the stipends or incentives (cash or gift cards)
for participants.
Refreshments and food
Try to negotiate with a local restaurant to cater your event for free or at a discounted price.
Remember to provide healthy and fresh options.
Stipends or incentives (e.g., gift cards, cash)
This can vary depending on how much time and what level of involvement you are asking of
participants. If you decide to use American Express or Visa gift cards, remember to budget
for the fee associated with each card.
Postage if it’s a mail survey
Don’t forget to weigh the introduction letter and the survey questionnaire together to make
sure the postage is sufficient.
Page 164 | The Step-by-Step Guide to Evaluation
8.2.4
Confidentiality and Anonymity Confidentiality and anonymity are essential considerations for you and the evaluator. Your
respondents’ privacy should be protected vigilantly. For example, names of participants should
never be revealed in an evaluation report. The terms “anonymity” and “confidentiality” have
different meanings and should not be used interchangeably.
Anonymity requires you and your evaluator to not know who the participants
are. For instance, you don’t ask respondents to put their names in a survey or identify
themselves in a focus group.
Confidentiality means you and your evaluator know who the participants are,
but you don’t link any answers to the respondents. Any information you have
that contains the person’s name or personal information must be kept in a locked
drawer or stored in a password-protected electronic file.
8.2.5
Obtaining Consent
One of the most important ethical rules governing evaluation is that potential respondents must
give their permission to participate in the evaluation before you survey, interview or observe them.
When you are collecting data from anyone under 18 years, you also must ask their parents for
written consent. You can ask for permission in writing or verbally. However, the request should
include information about:
What you are asking them to do
The benefits and risks of their participation
Who will see the data and how the data will be used
How much time it will take for them to answer the questions
How their participation is voluntary and they can decline or stop in the middle of the process if
it makes them uncomfortable
The Step-by-Step Guide to Evaluation | page 165
8.2.6
Institutional Review Board (IRB)
An Institutional Review Board (IRB) is a committee established
to review and approve research involving human subjects. The
IRB’s purpose is to ensure that all human subject research is
conducted in accordance with federal, institutional and ethical
guidelines.
To determine whether your evaluation needs IRB approval,
you must consult the guidelines on protecting human research
subjects from the following sources: your funding agency; your
agency, organization or institution; and your evaluator (if using
an external evaluator).
Universities have their own IRBs and you have to be affiliated
with the university to use the IRB there. Native American/
Alaskan Native tribes tend to have their own IRBs, and you must
get permission from the tribal government to collect data from
their members. Otherwise, you can pay a private IRB to review
and certify your evaluation design, plan and instruments.
If your evaluation requires IRB approval, make
sure you submit your IRB application well in advance before you begin collecting data (approvals can take anywhere between several days to several months). Also, IRB requires a processing fee, so make sure you allocate resources for IRB approval in your budget.
Page 166 | The Step-by-Step Guide to Evaluation
8.3 analyzing and interpreting data
Now that you have collected the data you need, it’s time to analyze, interpret and make sense of the
data, and then act on what you learned. This process can be complicated and, at times, technical.
For example, take a look at the following excerpt from a study by the Pew Research Center.
The wave of incoming Asians pushed the total number of Asian Americans to a
record 18.2 million, or 5.8 percent, of the total U.S. population, according to census
data. By comparison, non-Hispanic whites (197.5 million) account for 63.3 percent
of the U.S. population, while Hispanics (52 million) and non-Hispanic blacks (38.3
million) account for 16.7 percent and 12.3 percent, respectively.
The influx of Asians reflects a slowdown in illegal immigration while American
employers increase their demand for high-skilled workers.
If you used data collected by the U.S. Census as part of your evaluation, what does the above
passage suggest?
The trends, based on census data, are facts. However, what the Associated Press said - which
attempts to make sense of the data - leaves the impression that all Hispanics may be “illegal” and
all Asians may be “high-skilled workers.” It could create the perception that one group is “good”
and the other is “bad.” Yet, nothing in the data presented suggests this conclusion.
An interpretation like this can affect - in a negative way - how people think about Hispanics and
Asians, as well as how policymakers deal with immigration. Therefore, making sure that you
analyze, interpret and present your data correctly is a critical responsibility.
The Step-by-Step Guide to Evaluation | page 167
8.3.1
Quantitative Data Analysis and Interpretation of Results
Descriptive statistical analysis.
Quantitative analysis is often associated with statistical analysis, and program staff without a
background in statistics could be intimidated by the thought of doing this type of analysis. Yet, this
is the most basic form of quantitative analysis. When you calculate the number and
percentage of responses to a particular question or the average rating for questions about the
usefulness of the training, you are starting to do descriptive statistical analysis. It is used to
examine the responses to a question by calculating and looking at the following things:
Distribution of responses or frequency distribution (e.g., how many people checked response
option 1, response option 2, response option 3, etc.).
Average value, or the mean (i.e., looking at the average rating across the participants’ ratings).
The most common response, or the mode.
The number in the exact middle of the data set, or the median.
The mean, median and mode combined are also referred to as the central tendency and provide
information about the “typical response or score.” Most people don’t use the term central
tendency. You usually hear the terms “frequency of responses,” “mean response” and “most
common response.”
Descriptive statistical analysis also provides another piece of information technically referred to as
Obviously you cannot calculate mean values for
variables such as race and ethnicity, gender, sexual ori- entation or religion because a numerical value for them has no meaning.
variability. This statistic refers to the following:
Spread of your results, including the range (difference
between the highest and lowest scores).
Variance (shows how widely individuals in a group vary in
their responses).
Standard deviation (how close or far a particular response is
from the average response).
Page 168 | The Step-by-Step Guide to Evaluation
Unless you have statistical skills or know how to use a statistical analysis software package such
as SPSS or SAS, you should rely on someone with training in quantitative methods to do this analy-
sis. Nevertheless, it is helpful for you to understand what these terms mean to accurately interpret
your evaluation findings.
You could have heard about the 65, 95 and 99 percent rule. It means that in a normal or bell-
shaped distribution, if you go up and down (or left and right) one standard unit (or one standard
deviation), you will include roughly 65 percent of the cases or responses in the distribution. If you
go up and down two units, you will include roughly 95 percent of the cases or responses, and if you
go up and down three units, you will include roughly 99 percent of the cases or responses.
A normal or bell-shaped distribution is a symmetrical curve that is mathematically
defined based on the mean and standard deviation, as shown below.
−2 −1 +1−3 +30 +2
34.1% 34.1%
13.6% 2.2%
13.6% 0.1%0.1% 2.2%
68% of data
99.7% of data
Standard Deviations
Mean
One Standard Deviation
95% of data
The Step-by-Step Guide to Evaluation | page 169
Thus, when working with someone to interpret quantitative data
using descriptive statistical analysis, you might want to ponder
the following:
When looking only at the average, or mean value (e.g., value
of 3 on a 1 to 5 scale), are most of the responses close to the
average value or are the responses falling on the extreme
ends of the continuum (values of 1 and 5)?
If the standard deviation is a high number, which means
most of the participants responded very differently from
each other, what could have contributed to the wide
variation in their responses?
If the standard deviation is a very low number, it means that
most of the participants responded very similarly to each
other. Should and can the participants be divided into
subgroups and further analysis be conducted to see if there
is a difference between subgroups?
Example #1: You are implementing a leadership institute to
improve parents’ ability to advocate for their children’s
education. As part of the institute, you invite speakers to
present information about particular policies that affect
children’s ability to learn and succeed academically. After each
information session, you survey parents about their new
knowledge. Here is a question from the survey.
Question:
Please indicate the amount of new knowledge you developed
about suspension policies in your child’s school after attending
the information session.
1 2 3 4
None A little Some A lot
You might want to assign a small number to
negative responses so the bigger the number, the more positive the results. You should avoid changing this configuration across questions so respondents don’t get confused.
Page 170 | The Step-by-Step Guide to Evaluation
Here is how you can display the responses to the above question:
NEW KNOWLEDGE ABOUT SUSPENSION POLICIES
N = 30
Amount of New Knowledge Developed No. (%)
None 16 (53.3)
A little 7 (23.3)
Some 3 (10.0)
A lot 2 (6.7)
Missing 2 (6.7)
Total 30 (100.0)
The title should be clear. Always show the total number of respondents so the reader knows the number the percentage is based on.
R round any figures to at least one decimal point because it is more precise.
Always show the number and the percentage; one without the other doesn’t convey the proportion sufficiently (e.g., 50 percent is half of the proportion, which seems large, but depending on the sample size, 50 percent could be only 16 people, which could be a relatively small number if you are attempting to serve 100 people.
Mean (Average) = 1.68, which means that your participants developed a little new knowledge about the suspension policies in their child’s school.
In this example, calculating the median does not provide a meaningful piece of information.
No new knowledge developed was the most common response (or mode).
Example #2: You are implementing a city-wide strategy to promote immigrant integration, which
includes encouraging new immigrants to use public resources in the community and, during the
process, to interact with long-time residents. Here is a question you can ask in a community survey
of immigrants.
Question:
Please indicate how many times in the last three months you visited the public library.
The data you collected from 20 people indicated the following number of visits:
1, 1, 4, 5, 7, 10, 11, 11, 13, 14, 15, 16, 16, 17, 18, 19, 20, 20, 20, 20
The Step-by-Step Guide to Evaluation | page 171
You can create a frequency distribution with this basic display:
NUMBER OF VISITS TO THE PUBLIC LIBRARY LAST THREE MONTHS
N = 20
Number of Visits/Month No. of Respondents (%)
1 2 (10.0)
4 1 (5.0)
etc. etc.
The mean (average) number of visits in the last three months = 12.9 visits, or almost 13 visits; another way to describe this statistic could be “on average, about one time per week.”
If you don’t convert the values into a standardized number, you can’t tell how spread out the
responses are just by looking at them. It would be even harder if you had, say, 100 responses. So,
here is where statisticians (or the evaluator you work with) apply mathematical formulas to
calculate the variance (45.31) and the standard deviation (6.73).
What do the numbers mean?
In this example, the mean is 12.9 and the standard deviation is 6.73. This means that about 65
percent of the responses fall between 6.17 (12.9 minus 6.73) and 19.63 (12.9 plus 6.73). In plain
English, this implies that 65 percent of people surveyed visited the public library between
approximately six and 20 times over the last three months.
If the standard deviation is a small value relative to the average value, it means most respondents
tend to have scores close to the average. This means the variability is low or the respondents
tended to behave similarly. If the standard deviation is a large value relative to the average, it
means there is a lot of variability and most of the respondents tend to have scores that are farther
away from the average (thus the distribution is wider). In the example above, it means the number
of times most of the respondents visited the public library in the last three months varied widely
from the average number.
Page 172 | The Step-by-Step Guide to Evaluation
One can handle missing data in different ways. If you have little missing data - less than 10 percent of the total responses - then you can just report on the number and percentage of missing responses. If there is a higher proportion of missing data, then you might want to
perform imputation, which is the process of replacing missing data with substituted values. These values can be based on different conditions that make sense for the data set. For instance, you can substitute the missing responses from women who make less than $15,000 annual income with the average response of all the women who make less than $15,000. If you have a lot of missing responses in your data set, and you cannot generate decent results without performing imputation, you should consult with a statistician, a trained quantitative analyst, or your evaluator to find out how best to handle the missing data.
Imputation is the process of replacing missing data with substituted values.
Inferential statistics
Evaluators often use inferential statistics to determine if the results and conclusion extend beyond
the data you collected. Inferential statistics can be used to try to infer from the sample data what
the trend might look like in the larger population. It also can be used to make judgments about
whether the difference between two groups of people or two data sets happened by chance (i.e., is
the result statistically significant or not). It takes someone trained in quantitative analytical
techniques or statistics to perform inferential analysis; nevertheless, you should know the basics
to be informed consumers of the conclusion.
Statistical significance means that the difference between two or more groups of
people (or the same group of people at two different times) is not due to chance. More
precisely, the hypothesis or theory that there would be no difference between the two
groups of people is not true.
Things you to know about inferential statistics:
There are five common types of inferential analysis that you are most likely to come across:
Chi squares - comparing two categorical variables, like gender and voting preference
among community members.
The Step-by-Step Guide to Evaluation | page 173
Correlations - indicates that a relationship or pattern exists; it does not mean that one
variable “causes” the other (e.g., you could see a strong positive correlation between
healthy eating and weight loss among your program participants; however, the correlation
will not tell you if healthy eating is the cause for their weight loss).
T-tests - comparing the average values (mean scores) of two groups of people (e.g.,
between men and women) or of two data sets (e.g., scores on a pretest and post-test on
healthy eating behaviors).
Analysis of variance - comparing the average of three or more groups (e.g., between people
of three different age groups) or of three data sets (e.g., scores on healthy eating behaviors
for the same group of people in year one, year two and year three).
Regression - determining whether one variable is a predictor of another (e.g., if the length
of participation [number of weeks] in a healthy eating program is actually a predictor of
weight loss).
It requires a certain sample size or number of respondents to be able to conduct inferential
analysis. The smaller the sample size, the less reliable the results. If your evaluator wants to
conduct inferential analysis, ask if the sample size is large enough for this sort of analysis.
Find out how missing data were handled in the analysis - were the cases with missing data
excluded from the analysis or was imputation conducted to be able to include them? This
affects the sample size used in the analysis.
Sometimes, your evaluator or quantitative data expert may get excited about running
inferential analysis, but it might not be efficient or helpful to do this type of analysis depending
on what you want to know about your organization, strategy or program’s impact. Don’t assume
it is the best way to analyze quantitative data. Ask this question of your evaluator or the staff
person assigned to the evaluation and make sure you get a clear answer: What will the finding
tell you about your strategy, initiative or program’s impact? What will you learn from the find-
ing? Request the answer in plain English and non-statistical terms.
Interpretation
Quantitative findings must be interpreted with the organization, strategy, initiative, program and
other contextual factors in mind — factors that you and your staff know better than most. Therefore,
you absolutely must not leave the interpretation to the evaluator you hire, but work together with
the evaluator to review and interpret the findings.
Page 174 | The Step-by-Step Guide to Evaluation
These questions can guide your interpretation:
What is the magnitude of the effort’s impact, or how effective
was the organization, strategy, initiative or program? This is
typically expressed in terms of effect size.
Effect size is a standard measure that shows the
difference in the mean or average values between
two groups of people or data sets.
If you had a lot of missing data or insufficient responses,
why and what can be done differently to increase the
response rate in the future?
Are the results what you expected when you planned the
strategy or program? If not, what do you think affected the
results? Do you have qualitative data that can provide
insights into the results?
Are the results programmatically significant or not,
regardless of statistical significance, and what does it mean?
For instance, the difference in responses from two groups of
people might not be statistically significant, but could still
be large enough to warrant more tailored approaches or
interventions for each group. It is especially important to
tailor your approaches or interventions to the history,
cultural background and experience of different groups of
people.
What implications do the results have on the organization,
strategy or program? What actions do you need to take,
if any?
Statistical significance is the probability that the
observed difference between two groups of people or data sets is due to chance, but this information is not enough to fully understand the impact of your organization, strategy, initiative or program. This is where knowing the effect size can be useful. The general rule is that a value of 0.8 suggests a relatively large effect (the highest value is 1.0), 0.5 means a moderate effect, and 0.2 means a small effect.
Your evaluation should support your organization, strategy, initiative
or program improvement. Complex statistical analyses of well-designed experimental investigations that are costly but do not lead to improvements are unnecessary It is better to keep it simple. Sometimes, looking at the frequency of responses, average value, most common response and the maximum and minimum responses is enough to give deep insights into your work and help you make decisions about next steps for your organization and your effort.
The Step-by-Step Guide to Evaluation | page 175
8.3.2
Qualitative Data Analysis and Interpretation of Results
Qualitative data usually take the form of text. There are four major steps in qualitative data
analysis are described below.
Review the data
Before performing any qualitative data analysis, you must read and understand the data you have
collected. Look at the question or questions in the interview, focus group or observation the data
are supposed to answer. Check for unclear or incomplete sentences, and get clarity before you
code the data.
Organize the data
You can organize your data in various ways to make it easier to code. For example, you can organize
the data by question or by type of respondent or both.
Code the data
When coding data, you should identify and label the string of text that answers the question or
provides insight about a certain topic. You should label the text consistently across respondents.
There are two basic methods of coding and you could use one or both of these:
Open coding — When you assign codes based on what emerges from the data. Here you must
put aside your presuppositions, expectations of what the findings should look like and
previous knowledge of the subject matter, so themes actually emerge from your data.
Closed coding — When you already have codes prepared beforehand based on the questions
you want to answer.
Sometimes, even after you have coded the data, you might need to revisit the codes to refine them,
especially if you performed closed coding and found something in the data that you did not expect.
Also, sometimes codes flourish in a way that leaves you with too much data. If so, the code needs
to be broken down into subcodes to better organize the data. The rule for coding is to make the
codes fit the data, rather than trying to make your data fit your codes.
Page 176 | The Step-by-Step Guide to Evaluation
Identify and generate themes
After the data have been coded, study the text that were coded and develop a
general statement or theme that reflects what was said or observed. This is usually the hardest
part because you have to read the coded text several times and generate a theme that captures
the essence of the statements or observations collectively. If you want to explain the specifics, you
can provide examples of statements and observations to support the theme. Also, to capture how
strong a theme is, you can report on the number and percentage of responses you coded that
supported the theme. You also might need to note any differences in themes if you group
respondents by race and ethnicity, gender, or any other demographic characteristics.
Question for program staff:
What do you think a high-quality youth mentoring program should have?
Responses:
1. People who look like the youth and can relate to them.
2. Adults who are committed to putting in time to help the youth outside of the program.
3. Structured activities that teach youth life skills.
4. People who are from the neighborhoods the youth come from.
5. People who can relate to the youths’ experiences.
The relevant text can be coded to answer more than one question.
Don’t create long strings of codes, and always keep a list of codes, their definitions and
when and why you revised the code and definition.
Make sure you code enough text to provide the full context of the data (not just a few
words with no meaning when separated from the rest of the text).
Consider connections between codes as well and look for patterns in relationships.
checklist Tips on coding data:
The Step-by-Step Guide to Evaluation | page 177
6. Role models for the youth, people in whom the youth can see something is familiar to them,
maybe they grew up in the same neighborhood or remind them of someone they admire
growing up.
7. At least two hours per week that mentors must spend with the youth, if not more if they care
about their mentee.
8. Adults the youth can call any time, even outside of program activities, when they have
questions.
9. Some of our adult mentors actually take their mentees to baseball games and other kinds of
events, which has really helped build their relationships.
10. Our youth tend to enjoy spending time with people who they can talk to and who take the time
to talk to the youth, mainly because they have experiences they can relate to.
Examples of the types of closed codes and definitions you can develop before analysis:
CARE_ADULTS = Caring adults
STRUC_ACT = Structured activities such as college preparation, understanding of different career
pathways, how to deal with peer pressure, etc. Structured activities mean there is an instructor or
facilitator and a clear start and end time for the activity.
At first glance, it looks like responses 2, 7, 8, 9 and 10 can clearly be coded CARE_ADULTS. But
when you look at the responses more carefully, you would probably realize that “Caring Adults”
may be insufficient to capture the essence of some of the statements; they’re not just about caring
adults, but caring adults the youth can relate to or with whom they’ve shared experiences. So, you
can create a new code, ADULTS_RELATETO, and then code responses 1, 4, 5, 6 and 10 with this new
code. Note that response #10 is coded both as CARE_ADULTS and ADULTS_RELATETO.
Here’s how you can summarize this particular finding:
A total of 10 program staff members were interviewed for their perspectives about the attributes of
a high-quality mentoring program. Adults who really care about their mentees was an important
characteristic mentioned by half of the staff. Equally important was the fact the adults were people
the youth could relate to because they share similar backgrounds or interests.
Page 178 | The Step-by-Step Guide to Evaluation
Interpret the findings
The next step involves making sense of what you found. This means comparing your results to your
expected outcomes, original evaluation questions, the goals and objectives of your program and
current state-of-the-art knowledge (e.g., research about mentoring programs). Some questions to
guide your interpretation include:
Did any of the patterns and themes surprise you?
What are the factors that might explain the deviations?
If you collected quantitative data (that is, you used mixed methods), do the qualitative findings
support the quantitative findings? If not, what are the factors that could explain the differences
(e.g., sampling, the way the questions were asked in the survey compared to the interviews,
etc.)?
Do any interesting stories emerge from the responses?
Do the results suggest any recommendations for improving the program?
Do the results lead to additional questions about the program? Do they suggest additional data
could be needed?
Do you need to change the way the data are collected next time?
Be thoughtful when you are making sense of the data. Don’t rush to conclusions or make
assumptions about what your participants meant to say. Involving other people (e.g., program
staff ) or working with your evaluator to discuss what the findings mean will help you make sense of
the data.
The Step-by-Step Guide to Evaluation | page 179
HIGHLIGHTS
At the most basic level, quantitative
methods are concerned about what, who
and when, and qualitative methods are
useful for determining how and why.
Your questions or what you want to learn
should drive the decision about whether
you should use quantitative or qualitative
methods. One type of method is not
necessarily better than the other. You should
weigh what you want to know and how
much money, staff capacity and time you
have for the evaluation. Sometimes, you
might hire an evaluator who is partial to one
method, but don’t let his or her preference
drive your decision.
Quantitative data collection methods
include surveys, tests and assessments.
Qualitative data collection methods include
interviews, focus groups, observations and
review of artifacts.
Mixed methods evaluation, which combines
quantitative and qualitative methods, can
lead to a stronger, more complete evaluation
than only one method.
Critical considerations in data collection
include who will collect the data, how to
engage populations who have been
traditionally excluded or treated as
invisible, allocation of funds for certain
expenses that you might not think about
(e.g., childcare assistance, transportation,
translation and interpretation), how to
protect the confidentiality and anonymity of
your respondents, permission from
potential respondents to participate in the
evaluation (especially anyone under 18
years) and review of research or evaluation
protocol by an Institutional Review
Board (IRB).
In quantitative analysis, three things are
important to examine: frequency
distribution, central tendency and
variability.
In qualitative analysis, open and closed
coding can be used to analyze the data and
generate themes.
Interpretation of findings means comparing
your results to your expected outcomes,
original evaluation questions, goals and
objectives of your program and current
state-of-the-art knowledge (e.g., research
about mentoring programs).
When collecting, analyzing and interpreting both quantitative and qualitative data, remember that:
Page 180 | The Step-by-Step Guide to Evaluation
exercises
1. Which of the following is an advantage of qualitative methods?
A. Requires strong mathematical skills
B. Can be used to find out why your program works or does not work
C. Likely to cost less than quantitative methods
D. Takes less time than in quantitative methods to analyze the same number of responses
E. None of the above
2. An evaluation design that utilizes both qualitative and quantitative methods is called
.
A. Ethnography
B. Mixed methods evaluation
C. Quasi-experimental evaluation
D. Comprehensive evaluation
E. Summative evaluation
3. When evaluating a disability program with participants who have poor eyesight, which survey
method is recommended? .
A. Internet/computer survey
B. Mail survey
C. Telephone survey
D. Questionnaire survey
E. None of the above
The Step-by-Step Guide to Evaluation | page 181
4. Anonymity requires that .
A. You and the evaluator know the participant
B. All program staff know the participant
C. You know the participant but keep it a secret
D. You don’t link any of the answers to the participant
E. You and evaluator don’t know the participant
5. Think about an evaluation that you’d like to conduct for a program or initiative. What are the
best data collection methods to help answer your evaluation questions and why?
6. Consider an evaluation you conducted in the past. Would you do anything differently after
reading this chapter and why?
Answers: 1B, 2B, 3C, 4E
Page 182 | The Step-by-Step Guide to Evaluation
9
The Step-by-Step Guide to Evaluation | page 183
Summarize, Communicate and Reflect on Evaluation Findings
A PREPARE for conducting an evaluation
B DETERMINE stakeholders and how and when to engage them
C IDENTIFY assumptions and determine what will be di�erent (theory of change and logic model)
D DEVELOP evaluation plan (logic model, measurement, framework, etc.)E COLLECT and analyze data
F COMMUNICATE results and understand what happened (interpret �ndings and facilitate learning)
G MAKE informed decisions (improve actions and next steps)
Stakeholders
introduction Evaluation findings can be communicated in
many different ways to tell the story of your
strategy, initiative or program. You might
have a typical format for presenting your
evaluation findings, nevertheless, you
should know why some formats for
displaying and communicating your
findings could be more effective than
others. One end of the spectrum features
traditional reports. The other end uses
creative means such as photography and
drama, which can help communicate
findings to people who have lower levels of
literacy or come from cultures with strong
oral traditions.
There also are various options for displaying
your evaluation findings to sharpen the story
you want to tell. The field of data visualization and visual analytics has grown due to the availability of
large amounts of data along with technology advancements in accessing, handling and displaying data.
You easily can get caught up in all the options, but need to remain focused on what you want to
communicate, why and to whom.
Attending to this stage of the evaluation process is key because effective summary and communication of
evaluation findings helps:
Convey knowledge
Facilitate understanding
Create meaning
Confirm or challenge theories or previous ways of thinking
Inform decision-making and action
Page 184 | The Step-by-Step Guide to Evaluation
How this chapter is organized…
9.1 Communicate and Report Your Evaluation Findings
9.1.1 What You Need to Ask Before Putting the Findings Together
9.1.2 Different Communicating and Reporting Formats and Options
9.2 Considerations in Developing and Implementing a Communications Plan
9.3 Keeping It Simple
9.4 Ways to Display Your Evaluation Findings
9.4.1 Displays of Patterns and Trends over Time
9.4.2 Displays of Distribution and Spread of Responses
9.4.3 Displays of Comparisons
9.4.4 Displays of Frequency of Words
9.4.5 Displays of Themes
9.5 Reflecting on Your Evaluation Findings
9.5.1 Using Evaluation Findings
9.5.2 Considerations in Preparation for Reflecting on, Discussing, and Using Evaluation Findings
Highlights
Exercises
It’s not necessary to wait until the end of the program to share
findings and insights. You can share findings in the middle of your
program as long as you clarify that they are interim, preliminary
insights.
9.1 communicate and report your evaluation findings
9.1.1
What to Ask Before Putting the Findings Together
After data collection and analysis, you need to determine how to
summarize and communicate the findings and tell the story
effectively to your stakeholders, whether they are funders, board
members, community leaders, staff, volunteers or program
participants. Much of the material for this chapter comes from
Evaluation Strategies for Communicating and Reporting, by R.
Torres, H. Preskill and M. Piontek, published 2005 by SAGE Pub-
lications, and is adapted for this handbook.
The Step-by-Step Guide to Evaluation | page 185
You can use many strategies to summarize and communicate your findings. Sometimes, you could
have much more data than you possibly can share effectively. Therefore, you should begin by
asking the following questions before putting the findings together:
Are you required to submit an evaluation report to the funder about the results and impact of
the strategy, initiative or program?
What are the reporting requirements (where and when the report is due, questions to be
answered, sections that must be included, page limitations, inclusion of graphs and
tables, etc.)?
Who else needs to know the results and impact of the effort and why?
Do you want to inform them about their investments, give them tools to make decisions,
encourage support for the effort, or inform them for other reasons?
Depending on why they’re being informed, which aspects of the results and impact could
be particularly interesting to them?
When is the best time to share the findings and impact with the intended audience?
How much does each intended audience know about the effort?
How interactive do you want the communication to be for each intended audience? For
example, written and print materials are least interactive while discussions and working
sessions are most interactive. Verbal and video presentations fall somewhere in the middle.
What are the risks in sharing findings that could lead to bad consequences for your
organization and the community you serve?
Can the findings be taken out of context and harm the organization or community?
What can you do to mitigate the risks and consequences?
What roles do you, your staff, your board members and your internal or external evaluator have
in summarizing and communicating the findings and insights?
Who are the most effective messengers for the information?
Do you need a facilitator skilled in adult learning techniques to assist with discussions,
working sessions and verbal presentations?
Your answers to these questions will help determine the content and format for summarizing and
communicating the results and impact of the program.
Page 186 | The Step-by-Step Guide to Evaluation
From the outset, emphasize the use of evaluation for learning.
Involve key stakeholders in the evaluation’s design and implementation and
communicate throughout the evaluation process so there are no surprises.
Think about what to say and how to say it from the perspective of the stakeholders
hearing about the evaluation findings.
Share any negative findings through a discussion format so you can effectively facilitate
the learning process and reduce the likelihood of misunderstandings.
Don’t start the report or discussion with negative findings. Instead, lead with positive
findings and use words and phrases such as “accomplishments,” “how we can do better”
and “work in progress.”
checklist Tips for dealing with potentially negative findings:
The Step-by-Step Guide to Evaluation | page 187
9.1.2
Different Communicating and Reporting Formats and Options
exhibit 9-1: overview of communicating and reporting formats (adapted from torres, preskill and piontek, 2005, p. 27)
Need or Purpose
Format Ed uc
at e
or b
ui ld
aw
ar en
es s
Im pr
ov e
pr og
ra m
En su
re a
cc ou
nt ab
ili ty
Le ve
ra ge
s up
po rt
G en
er at
e ne
w
kn ow
le dg
e
R ep
lic at
e an
d sc
al e
pr og
ra m
M ak
e re
co m
m en
da ti
on s
ab ou
t f ut
ur e
di re
ct io
ns
A dj
us t e
va lu
at io
n
de si
gn a
nd
im pl
em en
ta ti
on
Short communications (e.g., memos, emails, blogs, tweets)
Reports (interim and final)
Executive summaries
Newsletters, briefs, brochures, bulletins
Posters
Webinars
Blogs
Discussions (group or individual)
Photography
Drama
Page 188 | The Step-by-Step Guide to Evaluation
When communicating to policymakers, don’t use language that is too technical or present
too many caveats about the statistical results. For this particular audience, a distinction
between statistical and practical importance could be too much to provide. Instead, only
present findings of practical and policy importance.
checklist Tip for communications:
9.2 considerations in developing and implementing a communications plan
Now that you have determined the best way to summarize and communicate the evaluation
findings and insights, you might want to develop a communications plan. The plan doesn’t have
to be extensive or long; its main purpose is to help you organize and put in writing the process for
sharing your evaluation findings and insights. You might want to involve your staff and other
stakeholders in developing and implementing the plan. The plan is an important tool to:
Stay focused on how you ultimately want to use the information. Without a plan, it is easy to
lose sight of this focus. Other priorities such as collecting the data and responding to your
funder’s reporting requirements could overshadow everything else toward the end of your
strategy, initiative or program.
Prompt you to think about the best messengers for your evaluation, potential risks in
sharing the information and strategies you can develop to mitigate the risks and
consequences. For example, if you share information about the vulnerabilities of the children
or youth your effort serves or if you use youth to deliver the message in a presentation, do you
risk further stereotyping about them?
Plan ahead about what resources you could need, including funding and technical expertise.
The Step-by-Step Guide to Evaluation | page 189
At a minimum, your communications plan should address the following:
Background, organization and context of the program, initiative, strategy, or policy
Purpose of the evaluation
Audiences and their specific information needs
Evaluation products, by type of audience
Potential risks and strategies for mitigating them
Budget for products
Additional knowledge, skills, or resources needed
Timeline
9.3 keep it simple
A key principle about effectively communicating your findings
is to keep the displays simple. Avoid cluttering the displays
with lines, colors, shades or anything else that could make it
attractive, but draws attention away from the content. To make
the material easy for the reader to understand, include only the
essential, critical information.
Let’s consider the example that has been used previously. An
initiative in Rainbow County seeks to increase the number and
percent of working class African American homeowners in the
county. The program includes services to educate families about
budgeting, savings and improving their credit scores; acquiring
financing; and understanding what it means to be a homeowner.
It also works with lending institutions to make their products
and services more accessible to working class families. The
initiative tracks where the graduates are in the homeownership
process. Exhibit 9-2 is a typical chart that shows this
Page 190 | The Step-by-Step Guide to Evaluation
information and highlights the people who are in the most advanced stage (i.e., their loan
application has been approved). This chart can be created easily in Microsoft Word. Now, look at
the table in Exhibit 9-3. It’s the same table, but without lines and shading. Ask yourself:
Which one is easier on your eyes?
Do you notice the difference between the two tables?
Does one give you more information than the other?
Does one present the data in a more manageable way than the other?
What is the key point of the tables? Is one table clearer about its message? If so, which one?
Exhibit 9-2: a typical chart in the form of a table
Year Course Was Completed Name Current Stage of Homeownership
Spring 2014 John Sykes Improved credit score
Spring 2014 Antwon Bates Increased savings
Spring 2014 Tony Austin Loan approved
Spring 2014 Laura Garcia Loan approved
Fall 2014 Mary Crawford Improved credit score
Fall 2014 Sarah Jones Loan approved
Fall 2014 Bob Love Increased savings
Fall 2014 Laura Mills Applied for loan
Spring 2015 Cindy Jones Applied for loan
Spring 2015 John Simmons Applied for loan
Fall 2015 Tanesha Williams Have family budget
Fall 2015 Michelle Simms Have family budget
Fall 2015 Mark Sifford Increased savings
The Step-by-Step Guide to Evaluation | page 191
Exhibit 9-3: typical chart in simplified table
Year Course Was Completed Name Current Stage of Homeownership
Spring 2014 John Sykes Improved credit score
Antwon Bates Increased savings
Tony Austin Loan approved
Laura Garcia Loan approved
Fall 2014 Mary Crawford Improved credit score
Sarah Jones Loan approved
Bob Love Increased savings
Laura Mills Applied for loan
Spring 2015 Cindy Jones Applied for loan
John Simmons Applied for loan
Fall 2015 Tanesha Williams Have family budget
Michelle Simms Have family budget
Mark Sifford Increased savings
Page 192 | The Step-by-Step Guide to Evaluation
9.4 ways to display your evaluation findings
After you have a communications plan and understand the importance of presenting findings sim-
ply, you and the evaluator can discuss the best way to display the findings based on
quantitative and qualitative data. Data visualization is a growing field and there are lots of
resources about how to convey your data effectively.
It is impossible to cover the topic fully in this handbook, but some excellent information is available from the following resources:
Workshops and other instructional materials, published by A.K. Emery. Available at http://annkemery.com/introducing-the-essentials/.
Presenting Data Effectively, by S. Evergreen, published by SAGE Publications in 2014. Additional resources are available at http://stephanieevergreen.com.
When displaying findings, your intention is to:
Draw the viewer’s to ponder the content rather than the method, graphic design or
something else.
Avoid any misrepresentation of what the data say.
Provide clear labels to help the viewer understand what he or she is seeing.
Avoid small print that causes the viewer trouble in reading and understanding the data.
9.4.1
Displays of Patterns and Trends Over Time
Displays that are effective for showing patterns or trends over time include graphics that show
the time on one dimension (typically, the horizontal dimension or x-axis), and the item for which
change is being observed on another dimension (typically, the vertical dimension or y-axis).
Examples of such displays, line graphs and slope graphs, are illustrated below. Both graphs
convey the same findings. In the first graph, each line in the slope graph represents one of the
three groups of program participants and changes in each group’s use of banking services before
The Step-by-Step Guide to Evaluation | page 193
and after a six-month financial literacy program. At a glance, the viewer can quickly see that groups
A and C increased their use while group B decreased their use. The graph also shows that group
A benefited the most from the program because the slope of the line is steeper than the line for
group C. This should prompt the viewer to ask why there is a difference between the groups, and
explore what circumstances led to group B’s decreased use of banking services. Answers to these
questions could lead to improvements in the program.
exhibit 9-4a: display of patterns and trends over time using a line graph
C
C
B
B
A
A
exhibit 9-4b: display of patterns and trends over time using a slope graph
Frequency of use of banking services before the program
Frequency of use of banking services after the program
Page 194 | The Step-by-Step Guide to Evaluation
9.4.2
Displays of Distribution and Spread of Responses
Displays that effectively show how responses are distributed along two dimensions include scatter
plots and histograms (see Exhibits 9-5a and 9-5b). For example, you operate an initiative to
prepare young men and boys of color for college. In the scatterplot below, the dots would represent
your program participants. The horizontal dimension or x-axis would represent the
number of times the young men attended the ACT tutorials over a six-month period. The vertical
dimension, or y-axis, would represent their ACT scores in a mock test.
The display does not show that attending tutorials
regularly and consistently will increase ACT test scores, which would suggest a causal relationship. Instead it shows a positive relationship between two behaviors - not that one behavior causes the other. Other factors could be contributing to the increased ACT scores.
At a glance, the viewer can see that a line is implied. This
suggests that attending tutorials and ACT scores are related - the
more times the participant attended the tutorials, the higher
his or her ACT scores. The viewer also can see that a handful of
participants who attended almost all the tutorials earned very
high ACT scores. The reverse is just as clearly true for those
participants who attended very few tutorials and scored low
on their mock ACT tests. This display can be used to make the
case that attending tutorials regularly and consistently can help
increase ACT test scores.
The histogram below also shows distribution of data. For in-
stance, the histogram could be used to show the number of
infants born with certain weight over the course of your effort.
The horizontal, x-axis, would represent frequency of cases for
each of the weight values. A viewer can look at the histogram
and see that most of the infants were born with weights that fell
in the middle range. To take the analysis one step further, you
could compare the values in the middle of the range with the
average weight of a newborn in the state to determine if your
program participants tend to have worse, better, or similar birth
outcomes relative to the state average.
The Step-by-Step Guide to Evaluation | page 195
exhibit 9-5a: displays of distribution and spread of responses using a scatterplot
exhibit 9-5b: displays of distribution and spread of responses using a histogram
Least Frequent Most Frequent
Infant Weight (lbs)
Lowest Score
N um
b er
o f C
as es
Highest Score
Respondents who attended all the tutorials and scored the highest on their tests.
9.4.3
Displays of Comparisons
Displays that help compare two or more groups include bar charts, clustered bar charts, side-by-
side bar charts and stacked bar charts. In these displays, the bars usually represent a categorical
variable (e.g., frequency of breastfeeding) while the length of the bar represents a quantitative val-
ue (e.g., frequency, percentage or rating). The clustered, side-by-side and stacked bar charts can
show a third dimension by using different color bars to represent different groups of people (e.g.,
Page 196 | The Step-by-Step Guide to Evaluation
women in your program compared with women in another program).
For example, in the clustered bar shown in below, the viewer can see that most women in the study
breastfed for more than 12 months. It’s also clear that slightly more women in your program (X)
did this compared with women in program Y. This finding may prompt the viewer to ask “why?” A
discussion about what worked well in your program based on other data you might have collected
could follow.
exhibit 9-6: examples of bar, clustered bar, side-by-side bar and stacked bar charts
> 12 months
> 12 months
> 12 months
> 12 months
6- 12 months
6- 12 months
6- 12 months
6- 12 months
<6 months <6 months
<6 months<6 months
Women in program X Women in program Y Women in program Z
9-6a bar
9-6c side-by-side bar
9-6b clustered bar
9-6d stacked bar
The Step-by-Step Guide to Evaluation | page 197
9.4.4
Displays of Frequency of Words
Word clouds allow you to show your qualitative findings in the simplest way. A word cloud
displays how many times a word has been mentioned in the given text or texts. The words
mentioned more often appear larger on the figure and the words mentioned less often appear
smaller on the figure, as shown below. Word clouds can be created using online applications such
as Wordle (www.wordle.net) or TagCrowd (www.tagcrowd.com). Word clouds are most useful for
displaying findings about responses to the question, “What one word would you use to describe
…?”
exhibit 9-7: example of a word cloud (Taken from Henderson, S., & Segal, E. H. (2013). Visualizing qualitative data in evaluation research. New Directions on Evaluation, p. 56)
Page 198 | The Step-by-Step Guide to Evaluation
For example, the viewer can see in the above graphic that the word “data” was mentioned most fre-
quently in your staff members’ responses to the question, “What is the one word you think about
when we talk about evaluation?” After your staff received extensive training in evaluation, you
could ask the question again and see how they respond by comparing the two word clouds.
What the word cloud doesn’t tell you, however, is whether the word “data” was mentioned in a
positive or negative way. Nor does it show how “data” related to other words used during the
interviews. Therefore, word clouds, while attractive and easy to create, should not be used as a
stand-alone display. You should provide an explanation about the word cloud, going beyond what
the viewer sees.
9.4.5
Displays of Themes
If you would like to convey more about your qualitative findings, you can summarize and display
themes from your qualitative data.
Themes are patterns tfound in qualitative data. The general rule is that a theme is
formed when three or more pieces of evidence point to the same idea.
For example, if three interviewees identified challenges to accessing services, that
would be a theme, whether the challenges were due to restricted service hours (no
extended office hours available) or language barriers (no interpreters or translated
materials available).
A matrix is an effective format for displaying themes derived from qualitative data.
Let’s use this example; Program staff was asked, “What are the three most important
qualities of a youth mentoring program?” The staff also was asked to rate their responses in order
of importance.
Below is how you can describe the findings based on the exhibit. [Note: The darker the shade the
more important it is.]
The Step-by-Step Guide to Evaluation | page 199
Three qualities are essential for a youth mentoring program: structured activities, caring adults in
general and caring adults youth can relate to. Among these three qualities, more than half of the
staff (N=6, 60 percent) believed that “caring adults youth can relate to” is the most important
quality. While “structured activities” is an important quality in youth mentoring programs, it was
rated by staff as the least important among the three qualities.
exhibit 9-8: matrix for communicating qualitative findings
Respondents
Qualities of a Youth Mentoring Program
1 2 3 4 5 6 7 8 9 10
Structured activities
Caring adults in general
Caring adults youth can relate to
9.5 reflecting on your evaluation findings
As stated previously, staff and other stakeholders are more likely to use the evaluation findings if
they understand the purpose of the evaluation and contributed to its design, implementation,
interpretation and use of the findings. To summarize and communicate your findings is not
sufficient. It’s important to reflect upon the findings and their implications and plan ways to put
them to use. Remember, evaluation must provide usable information to equip you to make
informed decisions and shape your programs to be as effective as possible.
However, there are obstacles to reflecting upon the findings and planning ways to use them, such
as:
Fear of being judged by board members, staff or people outside your organization.
Concern about the time and effort involved to convene stakeholders to discuss and reflect on
the findings.
Page 200 | The Step-by-Step Guide to Evaluation
Resistance to change that could impact the way things have
been done in the past.
Inadequate communication and knowledge sharing systems
that affect how, when and with whom information is shared.
Staff who are not interested in the findings for various
reasons.
Organizational limitations such as limited budget and staff
capacity to carry out other functions deemed more
important.
Concern about negative findings.
9.5.1
Use of Evaluation Findings
Improving your strategy, initiative or program
A goal of every evaluation should be to improve the strategy,
initiative or program, and evaluation findings should support
decisions and actions about what makes the effort more
effective. You, your staff and other stakeholders might want to
discuss what the findings say about the strengths and
weaknesses of the effort. Together, you should determine what
to do to adjust and improve the effort.
Being accountable
Evaluation is frequently used to hold the implementers account-
able. Sometimes, you have to adjust your strategy, initiative or
program because you learned that something wasn’t working
(e.g., you had trouble engaging parents with the current out-
reach strategy, or the industry for which you were developing a
workforce suddenly collapsed). If this happens, communicate
with your funder and other stakeholders so they understand why
the outcomes for which you are accountable are no longer valid.
The Step-by-Step Guide to Evaluation | page 201
Educating or building awareness
Evaluation can be used to educate or build stakeholders’ or the public’s awareness of your
strategy, initiative or program, and the issues it addresses. In reflecting on and discussing your
findings with the intent of educating and building awareness, you must think carefully about how
you frame the findings to avoid inadvertently perpetuating stereotypes or painting a negative
image of the people or community you serve. Also, you might want to combine quantitative and
qualitative data to show the numbers and provide the story behind the numbers to make the
information come alive.
Leveraging support
You can also seize opportunities to reflect on and discuss your evaluation findings to leverage
support from stakeholders such as community leaders and funders. In these situations, you want
to communicate clearly the evidence that supports the benefits of your strategy, initiative or
program. In these scenarios, put any negative findings in the context of learning and improvement.
Generating new knowledge
Evaluation can be used to discover and share new knowledge about effective practice. Evaluation
frequently tests a theory of change and the results can generate insights about whether, and under
what conditions the theory holds true. The new knowledge can inform decisions that affect the
lives of the people your strategy, initiative or program serves.
Replicating and scaling the strategy, initiative or program
If the evaluation shows that your strategy, initiative or program is effective and provides insight
into what makes it effective, you could want to consider replicating it in other communities or
increasing the number of people exposed to it. You have to be careful, however, about replicating
it in other communities that have different community and cultural contexts. You should ascertain
the degree to which the elements that worked in your program, initiative or strategy are culturally
appropriate for another group of people living under different conditions.
Developing recommendations for next steps
Your evaluation findings can help you develop recommendations for the future. These
recommendations could pertain to a wide range of issues such as type of support needed from
funders, training and other technical assistance needed to help your organization, staff capacity,
programming and other matters.
Page 202 | The Step-by-Step Guide to Evaluation
Making adjustments to the evaluation design and process, if necessary
In some cases, you could learn the evaluation design or implementation process was not
optimal for answering your evaluation questions. For example, you might have discovered that very
few people responded to the online questionnaire. Your evaluation could have been more effective
if you had used interviews or focus groups, even if it meant fewer respondents. Adjusting the
design of the evaluation midway likely would be difficult, but you should weigh the pros and cons
of continuing the evaluation with limitations or making midcourse corrections. This situation could
be a good learning opportunity for you and your organization. Also, if you see problems with the
way the external evaluator has designed the evaluation, such as not having translated materials
available for people with limited English proficiency, then you should bring up the issue even
though the evaluation is well under way. Flagging these types of limitations is important so the
problem can be resolved immediately and without harming the effectiveness of your evaluation.
9.5.2
Considerations in Preparation for Reflecting on, Discussing and Using Evaluation Findings
Communication avenues for different types of stakeholders
An effective process provides multiple avenues to impact staff, program participants and other
stakeholders in positive ways. Some avenues for consideration include:
Set aside time during staff meetings for reflection and discussion about the evaluation find-
ings. Align the timing of the reflections and discussions with the evaluation’s timeline for
reporting findings.
Host “open houses” or coffee or tea chats to share evaluation findings with program
participants or other stakeholders. This helps keep the communication of findings in a
nonthreatening and less formal discussion format.
Organize and conduct roundtables with various stakeholders to discuss and reflect on
the findings.
Host a webinar and invite participants to ask questions.
Blog about the highlights of the evaluation findings or post them on social media sites. Pose
questions and invite comments from readers and followers.
The Step-by-Step Guide to Evaluation | page 203
An effective process also helps:
Improve communication and understanding between different groups (e.g., between line staff
and managers or between staff and volunteers) about the strategy, initiative
or program.
Improve understanding about the population being served, particularly disenfranchised groups
who are often unheard, misrepresented and misjudged.
Leverage additional resources and other support for the effort.
Facilitate development of knowledge, skills and other competencies among staff, program
participants and other stakeholders.
Below are examples of scenarios where the reflection process and discussion facilitated some of
the above changes.
Example 1
An initial evaluation of a program providing educational services to families and children in an
economically disadvantaged urban community helped program staff discover they were
operating the program based on a set of implicit and unspoken assumptions. The fact that these
assumptions were not put in writing or discussed explicitly as part of the program seemed to be
contributing to problems with new staff members’ ability to understand the program, its goals and
underlying principles. Founding staff used the evaluation findings and insights to create a
Page 204 | The Step-by-Step Guide to Evaluation
historical overview of the program’s origins, the path it had taken to get where it was and the
assumptions underlying the program and its mission. The discussion helped the new staff better
understand the program and created shared understanding between new and longer-serving staff.
Example 2
The evaluator provided a preliminary summary to the staff about what she was learning from the
interviews conducted. One preliminary finding revealed that interviewees had varying
understandings about what the program meant by “creating a more inclusive community.”
African-American interviewees, in particular, questioned the likelihood of this outcome when they
didn’t see their leaders involved in the steering committee for the program. Communicating this
finding to the program staff led to a discussion about what inclusiveness means and how the
African-American interviewees were unaware that the NAACP director and the high school
counselor – both African Americans – served on the committee. The evaluator used this opportu-
nity to ask the staff about what an inclusive community looks like to them, what representation
means and how aware they were of other types of African American leaders in the community. The
discussion resulted in a series of learning sessions in the organization about what it takes to build
an inclusive community.
Requirements for an effective process
Excellent facilitation: If you or your evaluator do not have strong facilitation skills, consider
hiring a professional.
Clear meeting objectives and outcomes: If you are facilitating the reflections and use of
evaluation findings through a meeting or discussion, set clear objectives and learning
outcomes for the process.
Rules of engagement: As in any good meeting, have rules about how participants should en-
gage one another (don’t interrupt when someone is talking, be respectful, etc.).
Attention to power differences: If the meeting involves people with different levels of power –
whether based on rank, position, race, ethnicity or any other characteristic – have procedures
in place to prevent the more powerful from dominating the discussion. Also give those with
less power the opportunity to contribute (e.g., small group discussions, written feedback, go-
ing around the room and inviting each person to say something).
The Step-by-Step Guide to Evaluation | page 205
Potential disagreements: You also want to ensure you have
a process for handling potential disagreements, tensions or
conflicts due to the findings. Take time to identify findings,
especially negative findings, which could surface such
challenges, then work with the facilitator (if you have one) to
plan how to deal with them when they emerge.
Next steps: Summarize the decisions and next steps at the
end of the reflections and discussion.
You might want to train a few people to serve as spokespersons for the program and
evaluation findings if the program is a very visible one (e.g., it’s the first program of
its kind in the community, it is testing a theory that has been heavily publicized in the
media) or if findings are somewhat controversial (e.g., they don’t support the theory, they
bring attention to issues such as structural racism and other inequities). In this way, you
can manage the communications and limit it to the few people who have deep
understanding of the program and the evaluation and who are prepared and skilled in
speaking to journalists and others. You also could plan a press release or a press
conference, so you have more control over the framing of the findings and prevent
their misuse.
checklist Tip for communicating evaluation findings:
Page 206 | The Step-by-Step Guide to Evaluation
HIGHLIGHTS
Before you begin putting the findings
together, be clear about: your funder’s
reporting requirements; stakeholders who
should know about the results and what
you want them to do with the information;
which aspects of the results may be
particularly interesting to them; and the best
time to share the information with them.
Pay close attention to negative findings and
have a strategy for dealing with them.
Develop a communications plan to help you
organize and put in writing the process for
sharing your findings and insights.
Keep displays about your findings simple
to enable viewer to focus on the content.
Start with all the information in the display,
take a step back and then strip the display
of any unnecessary and distracting graph-
ics and other information. Put yourself in
the viewer’s shoes and if the finding could
be misunderstood and make the necessary
adjustments.
If you are replicating your program,
initiative or strategy, check thoroughly to
see if you have sufficient evaluation
information to determine what parts might
need to be adapted for another cultural or
community context.
If you choose to reflect on findings in
a meeting, ensure you plan for good
facilitation; set an agenda with clear
meeting objectives and learning outcomes;
set rules of engagement; include procedures
for handling power differences and dis-
agreements, and identify next steps.
You might want to train a few
spokespersons for your program or
evaluation findings, especially if the
program is a high-stakes or visible program
or if the findings could stir controversy.
The Step-by-Step Guide to Evaluation | page 207
exercises
1. If the evaluation does not produce favorable results, which of the following can help ease the
fear of reporting such findings:
A. Seeing this as an opportunity to learn and make improvements
B. Keeping the findings a secret
C. Starting the report with a list of the negative results
D. Making sure the decision-makers know about the negative results right away
E. None of the above
2. Can you identify three things that are missing from the following data display?
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Health Bene�ts/ Services
Types of Policies
How Health Insurance
Works
Ways to Buy
Insurance
Tax Credit/ Other
Programs
Average
Excellent
Fair 1.76
2.62
1.63
2.53
1.56
2.50
1.62 1.68
2.56 2.56
1.81
2.50
Le ve
l o f K
no w
le dg
e
figure 1: average ratings of knowledge, by gender
Answers:1A; 2 - explanation of what the scores mean, legend to explain the gray and black bars (which one refers to males and which ones to females), and total number of respondents.
Page 208 | The Step-by-Step Guide to Evaluation
3. Take a set of qualitative or quantitative findings from one of your evaluation reports. Consider
how the findings could be displayed to better communicate the story you want to tell about the
program or initiative that was evaluated.
The Step-by-Step Guide to Evaluation | page 209
Conclusion
This handbook was designed to demystify evaluation and familiarize you with its basic elements so
you can integrate evaluative thinking into your daily work and partner more effectively with trained
evaluators. While you may choose to hire an independent evaluator, you can still use the basic
knowledge you learned about evaluation to make decisions regarding:
The most appropriate evaluation type, approach and methodology to assess your strategy,
initiative, program or policy
Ways to engage key stakeholders throughout the evaluation process
Amount of funds to allocate for the evaluation
Culturally appropriate data collection methods
Analysis strategies
Interpretation, use and communication of findings
Practicing evaluative thinking can help strengthen the impact of your organization’s work. By
becoming a more informed consumer of evaluation, you can participate in the evaluation of your
strategy, initiative or program in a more meaningful manner. As a result, you can conduct an
effective evaluation that generates knowledge that will not only benefit your organization but also
the families and communities you work with and serve.
Page 210 | The Step-by-Step Guide to Evaluation
The Step-by-Step Guide to Evaluation | page 211
Glossary
Anonymity Anonymity requires you and your evaluator do not know who the participants are. For instance, you don’t ask respondents to put their names on or identify themselves in a survey or focus group.
Closed-ended Questions Closed-ended questions provide discrete, multiple-choice answers that respondents can select.
Confidentiality Confidentiality means you and your evaluator know who the participants are, but you don’t link any of the answers to the respondent. Any information you have that contains the person’s name or personal information must be kept in a locked drawer or stored in a password-protected electronic file.
Cultural Competency Refers to the ability – stance, knowledge, skills and commitment – to respect and engage with diverse segments of communities and to include the contextual and cultural dimensions relevant to these diverse segments in the evaluation design and process.
Culture A set of socially transmitted and learned behavior patterns, beliefs, institutions and all other products of human work and thought that characterize the functioning of a particular population, profession, organization or community. Culture is continually evolving.
Culturally Responsive
Evaluation
Culturally responsive evaluation recognizes that cultural values, beliefs and context lie at the heart of any evaluation effort.
Developmental Evaluation An approach used to support innovation within an organization and in its strategies, initiatives and programs. Efforts that are innovative are often in a state of continuous development and adaptation, and they frequently unfold in a changing and unpredictable environment.
Page 212 | The Step-by-Step Guide to Evaluation
Effect Size A standard measure that shows the difference in the mean or average values between two groups of people or datasets.
Evaluative Thinking Evaluative thinking is a cognitive process, motivated by inquisitiveness and a belief in the value of evidence, which involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and making informed decisions in preparation for action.
Experimental Design Experimental designs assess the causal effects of a program by com- paring two groups of people - one group receives the intervention (“treatment group”) and one does not (“control group”).
Immediate Outcomes or
Short-Term Outcomes
Immediate changes or benefits expected — usually within one to two years — as a result of successful implementation of the strategy.
Imputation The process of replacing missing data with substituted values.
Intermediate Outcomes or
Short-Term Outcomes
Changes or benefits, usually within one to two years of the immediate outcomes.
Indicators Indicators are markers of progress toward the change you hope to make with your strategy, initiative, or program.
Institutional Review Boards Institutional Review Boards (IRBs) are entities set up to protect the rights and welfare of people who participate in research. Evaluations of programs involving Native Americans/Alaska Natives also require permission from their tribal governments.
Logic Model A logic model is a graphic representation of the theory of change that illustrates the linkages among program resources, activities, outputs, audiences, and short-, intermediate-, and long-term outcomes relat- ed to a specific problem or situation.
The Step-by-Step Guide to Evaluation | page 213
Long-Term Outcomes Lasting changes with organizational, community, or systems-level benefits (e.g., organizational practices or policies, new or modified legislation, improved social conditions). Sometimes, these outcomes might be referred to as impact.
Methodology A set or system of methods and procedures that you use to answer your evaluation questions.
Mixed Methods Study Involve the intentional use of two or more different kinds of data gathering and analysis tools — typically a combination of qualitative (e.g., focus groups and interviews) and quantitative (e.g., multiple choice surveys and assessments) — in the same evaluation.
Normal or Bell-shaped
Distribution
A normal or bell-shaped distribution refers to a symmetrical curve that is mathematically defined based on the mean and standard deviation.
Open-ended Questions Open-ended questions are questions that ask respondents to respond through written text.
Qualitative Data Qualitative data is any information that can be collected or captured in text form.
Quasi-experimental Design Assesses the causal effects of a program by comparing two groups of participants (a “treatment” group and a “comparison” group) or by comparing data collected from one group of participants before and after they participated in the program. There is no random assignment of participants into the two groups, unlike studies using an experimental design.
SMART Metrics Specific, Measurable, Attainable, Realistic and Timely metrics
Stakeholder A stakeholder is defined as any person or group who has an interest in the strategy, initiative or program being evaluated or in the results of your evaluation, including your evaluator.
Page 214 | The Step-by-Step Guide to Evaluation
Statistical Significance The difference between two or more groups of people is a result that is not due to chance, or more precisely, that the hypothesis or theory that there would be no difference between the two groups of people is not true.
Strategies or Activities Strategies or activities are the processes, techniques, tools, events, technology and actions of the planned program, used to bring about the intended program changes or outcomes.
Themes Themes are patterns that you find in your qualitative data. The general rule is that a theme is formed when there are three or more pieces of evidence pointing to the same idea. For example, if three interviewees identified challenges to accessing services, that would be a theme, whether the challenges were due to restricted services hours (no extended office hours available) or language barriers (no interpreters or translated materials available).
Theory of Change A theory of change is a narrative that explains the links between activities and outcomes and how and why the desired change is expected to come about.
The Step-by-Step Guide to Evaluation | page 215
Bibliography Bamberger M, & Segone M. (2011). How to Design and Manage Equity-Focused Evaluation. New York:
UNICEF Evaluation Office.
Chelimsky, E. & Shadish, W. (Eds.) (1997). Evaluation for the 21st Century. Thousand Oaks, CA:
SAGE Publications.
Clark, H. & Anderson, A.A. (2004). Theories of Change and Logic Models: Telling Them Apart. Paper presented
at the American Evaluation Association Conference Presentation, Atlanta, GA.
Child Trends. (2007). Five Steps for Selecting an Evaluator: A Guide for Out-Of-School Time Practitioners.
Retrieved August 26, 2015, from http://www.childtrends.org/wp-content/uploads/2013/04/Child_
Trends-2007_10_01_RB_SelectingEvaluator.pdf.
Cross, E.Y., Katz, J.H., Miller, F.A., & Seashore E.W. (1994). The Promise of Diversity. Burr Ridge, IL: Irwin
Professional Publishing.
Dalton, J., Elias, M., & Wandersman, A. (2001). Community Psychology: Linking Individuals and
Communities. Belmont, CA: Wadsworth.
Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping. Ottawa, Canada: International Development
Research Center.
Fetterman, D., Kaftarian, S., & Wandersman, A. (Eds.) (2001). Empowerment Evaluation: Knowledge and Tools
for Self-Assessment and Accountability. Thousand Oaks, CA: SAGE Publications.
Fitzpatrick, J. (2012). An introduction to context and its role in evaluation practice. New Directions for
Evaluation, 135, 7-24.
Frierson, H.T., Hood, S., & Hughes, G.B. (2002). A guide to conducting culturally responsive evaluations.
In The 2002 User-friendly Handbook for Evaluation (pp. 63-73). Arlington, VA: National Science
Foundation.
Page 216 | The Step-by-Step Guide to Evaluation
Gitlin, A. (Ed.). (1994). Power and Method. New York, NY: Routledge.
Hendricks, M., Plantz, M., & Pritchard, K. (2008). Measuring outcomes of United Way-funded programs:
Expectations and reality. New Directions for Evaluation, 119, 13-35.
Hofstede G. (1997). Culture and Organizations. New York, NY: McGraw-Hill.
Hood, S., Hopson, R. & Frierson, H.T. (Eds.) (2015). Continuing the Journey to Reposition Culture and
Cultural Context in Evaluation Theory and Practice. Greenwich, CT: Information Age.
Labin, S., Duffy, J., Meyers, D., Wandersman, A., & Lesesne, C. (2012). A research synthesis of the
evaluation capacity building literature. American Journal of Evaluation, 33, 1-32.
Lee, K. (2007). The Importance of Culture in Evaluation. Denver, CO: The Colorado Trust.
Lee, K. S., & Chavis, D. M. (2012), Cross-case methodology: Bringing rigour to community and systems
change research and evaluation. Journal of Community and Applied Social Psychology, 22 (5): 428-438.
Mathison, S. (2005). Encyclopedia of Evaluation. Thousand Oaks, CA: SAGE Publications.
Newcomer, K., Hatry, H., & Wholey, J. (2015). Handbook of Practical Program Evaluation, 4th Ed., New
Jersey: John Wiley & Sons, Inc.
Newcomer, K. & Wirtz, P. (2004). Using statistics in evaluation. In J. Wholey, H. Hatry, & K. Newcomer
(Eds.), Handbook of Practical Program Evaluation, 2nd Ed. (pp. 439-459). San Francisco, CA: Jossey Bass.
Owens, J.M. (2007). Program Evaluation. New York, NY: The Guilford Press.
Patrizi, P. & Patton, M.Q. (Eds.) (2010). Evaluating strategy. New Directions for Evaluation, No. 128.
Pawson, R. (2003). Nothing as practical as a good theory. Evaluation, 9, 471-490. doi:
10.1177/1356389003094007.
Posovac, E.J. & Carey, R. G. (2003). Program Evaluation: Methods and Case Studies. Upper Saddle River, NJ:
Prentice Hall.
Revels, M. (February 16, 2016). Focus Group Research: Workshop Material. Atlanta, GA: Author.
The Step-by-Step Guide to Evaluation | page 217
Stufflebeam, D. (2001). Evaluation models. New Directions for Evaluation, 89, 7-98.
Taylor-Powell, E. & Herman, C. (2000). Collecting Evaluation Data: Surveys. Retrieved August 1, 2015
http://learningstore.uwex.edu/assets/pdfs/g3658-10.pdf.
The American Heritage College Dictionary, 3rd Ed. (2000) Boston, MA: Houghton Mifflin.
The Urban Institute. (2006). Building A Common Outcome Framework to Measure Nonprofit Performance.
Washington, DC: The Urban Institute.
Torres, R., Preskill, H., & Piontek, M. (2005). Evaluating Strategies for Communicating and Reporting, 2nd Ed.
Thousand Oaks, CA: SAGE Publications.
Trochim, W. (1999). The Research Methods Knowledge Base. Ithaca, NY: Cornell University.
Voget, W.P. (1999). Dictionary of Statistics & Methodology: A Nontechnical Guide for the Social Sciences.
Thousand Oaks, CA: SAGE Publications.
Yin, R.K. (2003). Case Study Research: Design and Methods, 3rd Ed. Thousand Oaks, CA: SAGE Publications.
Page 218 | The Step-by-Step Guide to Evaluation
The Step-by-Step Guide to Evaluation | page 219
Guide to Evaluation Resources
OVERVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
EVALUATION BASICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Introductory Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
Evaluation for Planning, Development
and Continuous Quality Assurance and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Community Based and Social Change Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . 223
Theories of Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Logic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Engaging Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Cultural Competence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
Racial Equity Lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Hiring an Evaluator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Using and Presenting Evaluation Findings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
APPLIED EVALUATION RESOURCES BY TOPIC AREAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Advocacy and Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
Workforce Development and Employment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Financial Literacy and Family Asset-building . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Early Childhood Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
K-12 and Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Poverty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
Youth Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
SECONDARY DATA SOURCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Juvenile Justice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
Early Childhood Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
K-12 and Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
OTHER DATA-RELATED RESOURCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
Page 220 | The Step-by-Step Guide to Evaluation
OVERVIEW This resource guide includes major resources about evaluation in general, and about specific topics of particular interest related to evaluation in the W.K. Kellogg Foundation’s priority areas. Resources marked with the symbol “” are resources recommended by the Kellogg Foundation because they are particularly useful and relevant to both those with limited experience with evaluation as well as those with ample experience.
EVALUATION BASICS Introductory Evaluation Resources
Better Evaluation (2013). Manage an evaluation or evaluation system. Retrieved from http://betterevaluation.org/sites/default/files/Manage%20-%20Compact.pdf This web-based resource provides practical assistance to nonprofit organizations engaged in evaluation. It is intended to help users’ access information on the important aspects of program evaluation. The sources are listed in the order by which conducting an effective evaluation should be done. Topics range from engaging stakeholders to building evaluation capacity. The hyperlinks allow users to continuously go deeper on a particular topic (e.g., from understanding the basic components of a budget for evaluation to sample budget forms).
Better Evaluation (2013). Planning an evaluation: Using the rainbow framework. Retrieved from http:// betterevaluation.org/sites/default/files/Rainbow%20Framework%20-%20compact%20version.pdf This web-based resource offers a framework that can be used to develop an evaluation plan by prompting the reader to consider a series of key questions. It is broken down into seven evaluation cate- gories: managing an evaluation; defining what is to be evaluated; framing the boundaries of evaluation; describing activities, outcomes, impacts and contexts; understanding causes of outcomes; synthesizing data from evaluation; and reporting and supporting use of evaluation findings.
Centers for Disease Control and Prevention. (2011). Introduction to program evaluation for public health programs: A self-study guide. Retrieved from http://www.cdc.gov/eval/guide/cdcevalmanual.pdf
This report, intended for a beginner-level audience, provides a glossary of key evaluation terms. Several evaluation methods are discussed, including formative, summative, midterm and final evaluation. The Centers for Disease Control and Prevention (CDC) deem triangulation as the most effective data collection method. The publication addresses the need to expand social equity and cultural sensitivity, the benefits of evaluation and the use of evaluation to build capacity.
Glenaffric Ltd. (2007). Six steps to effective evaluation: A Handbook for programme and project managers. Joint Information Systems Committee. Retrieved from http://www.jisc.ac.uk/media/ documents/programmes/reppres/evaluationhandbook.pdf
This handbook outlines an approach to evaluation that is applicable to all JISC development activity and
The Step-by-Step Guide to Evaluation | page 221
relevant to both program and project managers. Although the handbook is intended for JISC managers, those with intermediate level experience with evaluation will find it useful. The handbook provides guid- ance in conducting the following six steps: (1) identify stakeholders, (2) describe project and understand program, (3) design evaluation, (4) gather evidence, (5) analyze results and (6) report findings. The six- step approach promotes the use of formative and summative evaluation and describes effective data collection.
JBS International. (2006). Study designs for program evaluations. Retrieved from http://www.pacenterof excellence.pitt.edu/documents/study_designs_for_evaluation.pdf
This document discusses the following evaluation approaches: exploratory, descriptive, experimental and quasi-experimental. The report is geared more toward evaluators as it describes and provides ex- amples of randomized controlled trials and a number of quasi-experimental designs, such as propensity score matching, pretest and posttest comparisons, and simple differences. Also included in this document is a discussion of factors that may affect internal and external validity of an evaluation.
Pawson, R. (2003). Nothing as practical as a good theory. Evaluation, 9, 471-490. doi: 10.1177/1356389003094007
This article, written for evaluation beginners, explains what evaluation is. Methods of evaluation are discussed in great detail and are supplemented with real examples. The benefits of evaluation are also explained.
Shackman, G. (2009). What is program evaluation? Retrieved from http://gsociology.icaap.org/methods/ basicguideshandouts.html
This short guide provides users with a basic overview of program evaluation, discussing logic models, methods of evaluation, data collection methods and ways to determine whether a program caused its intended outcome.
The Urban Institute. (2006). Building a common outcome framework to measure nonprofit performance. Retrieved from http://www.urban.org/uploadedpdf/411404_nonprofit_performance.pdf
This document, written for nonprofit organizations, provides basic indicators to assess different types of nonprofit programs. It also guides nonprofits in developing their own indicators. Ultimately, this framework provides an approach for assessing programs in a way that is specific, observable, understandable, relevant, time bound and valid.
United Nations Educational, Scientific, and Cultural Organization (2007). Internal Oversight Service Evaluation Section. Retrieved from http://unesdoc.unesco.org/images/0015/001557/155748E.pdf
This handbook, designed to be used by a beginner-level audience, describes the basic evaluation approaches and methods. While it does not touch on social equity or cultural sensitivity, it explains the benefits of evaluation and the use of evaluation to build capacity.
Page 222 | The Step-by-Step Guide to Evaluation
U.S. Health and Human Services, Administration for Children and Families. (2010). The Program manager’s guide to evaluation: Second edition. Retrieved from http://www.acf.hhs.gov/sites/default/files/opre/ program_managers_guide_to_eval2010.pdf
This guide, for people with little to some evaluation experience, explains what program evaluation is, why evaluation is important, how to conduct an evaluation, how to report evaluation findings and how to use evaluation results to improve programs that benefit children and families.
Wall, J. (n.d.). Program evaluation model: 9-step process. Sage Solutions. Retrieved from http://www.janetwall.net/attachments/File/9_Step_Evaluation_Model_Paper.pdf
This publication provides readers with a nine-step model for conducting effective program evaluation. Af- ter a brief introduction to program evaluation, the document describes each step. Sample questions and data collection models are included. This document also lists resources on various topics (e.g., assess- ment and evaluation standards, ethical standards, response rates.).
Evaluation for Planning, Development and Continuous Quality Assurance and Monitoring
Corporation for National and Community Service. (n.d.). Evaluation resources. Available at http://www.nationalservice.gov/resources/evaluation/all-evaluation-resources
This web-based resource contains a series of hyperlinks to the federal agency’s evaluation core curriculum course. The courses are designed to assist nonprofit organizations as they move through each stage of the evaluation process, from planning to use of the results. The narrative instructions are accompanied by slides and samples of evaluation products, such as evaluation plans and statements of work for evaluation.
International Federation of the Red Cross. (n.d.). Project/programme monitoring and evaluation guide. Retrieved by http://www.ifrc.org/global/publications/monitoring/ifrc-me-guide-8-2011.pdf
This handbook aims to promote a common understanding of monitoring and evaluation for the International Federation of Red Cross and Red Crescent Societies (IFRC) projects. The handbook also provides general evaluation information useful to individuals with a novice level of evaluation experience. Several methods of evaluation are discussed in the publication and the IFRC encourages a triangulation method (using several types of data collection methods such as surveys, interviews and self-report) for data collection. This publication stresses the importance of cultural sensitivity when collecting data on sensitive topics.
Prevention by Design. (2006). Evaluation terminology tip sheet. Retrieved from http://socrates.berkeley. edu/~pbd/pdfs/evaluation_terminology.pdf
This document, which can be used by anyone, provides the definitions for common evaluation terminolo- gies that are essential to planning and conducting evaluation work.
The Step-by-Step Guide to Evaluation | page 223
United Nations Development Programme. (2009). Handbook on planning, monitoring and evaluating for development results. Retrieved from http://web.undp.org/evaluation/handbook/documents/english/ pme-handbook.pdf
This handbook is intended to support the United Nations Development Programme (UNDP) in becoming more results-oriented and to improve its focus on development changes and real improvements in people’s lives. The handbook is useful for UNDP staff, managers and executive board, as well as independent evaluators and members of the national, regional and global evaluation community. The handbook categorizes evaluation methods by timing—ex-ante, midterm, final or terminal and ex-post evaluation. According to the document, the data collection method depends upon certain criteria (accessibility to surveys, interviews, self-report, etc.). Though social equity and cultural sensitivity are not addressed, the benefits of evaluation and the use of evaluation to build capacity are discussed in great detail.
Community based and Social Change Evaluation Resources
Borgman-Arboleda, C. & Clark, H. (2010). Considering evaluation: Thoughts for social change and movement-building groups. New York, NY: ActKnowledge. Retrieved from http://www.actknowledge.org
This booklet presents an easy-to-use resource for evaluation and assessments of social justice, social change and movement building work. Different types of evaluation, as well as various methods, are discussed in this publication, from how to collect data to recommending the use of mixed methods in evaluation. Social equity is briefly mentioned in this piece while the various benefits of evaluation are clearly noted.
Crystal Foster, C. & Louie, J. (2010). Grassroots action and learning for social change. Center for Evaluation Innovation. Retrieved from http://www.evaluationinnovation.org/sites/default/files/foster%20louie%20 brief.pdf
This guide examines the similarities and differences between advocacy and organizing, then presents a framework for evaluating community organizing. The guide details what evaluation should be— participatory, prospective, learning-based, real-time, respectful of the culture of organizing, attentive to leadership development and evidence based.
Gelmon, S., Foucek, A., & Waterbury, A. (2005). Program evaluation: Principles and practices (second edition). Portland, OR: Northwest Health Foundation. Retrieved from http://www.northwesthealth.org/ search?q=evaluation%20handbook
This handbook provides a framework for community-based organizations to evaluate and understand the effectiveness of their programs. The handbook provides an overview of basic program evaluation prin- ciples and practices for documenting program progress. Cultural competency and building capacity are briefly discussed.
Page 224 | The Step-by-Step Guide to Evaluation
Grantmakers for Effective Organizations. (n.d.). Four essentials for evaluation. Retrieved from http://docs.geofunders.org/?filename=2012_geo_evaluation_essentials.pdf
This document is aimed at helping grantmakers further their evaluation efforts. The target audience is supporters of evaluation who want to embed these practices more deeply in the work of their organizations. According to this piece, there are four steps critical for evaluations: lead, plan, organize, and share. Readers can also use this text to gain an understanding of evaluation methods and data collection. The document acknowledges the importance of social equity and cultural competence when conducting an evaluation. The benefits of conducting evaluations are discussed throughout this guide as well.
The Community Toolbox. (n.d.). Evaluating community programs and initiatives. Retrieved from http://ctb.ku.edu/en/tablecontents/index.aspx
This toolkit guide addresses how to evaluate a specific program and standards for good evaluations. The piece is geared toward a beginner-level audience and describes evaluation approaches and methods. An entire section of the Community Toolbox is devoted to social equity and cultural competency.
Theories of Changes
Anderson, A. (n.d.). The community builder’s approach to theory of change: A practical guide to theory development. The Aspen Institute Roundtable on Community Change. Retrieved from http://www.dochas.ie/Shared/Files/4/TOC_fac_guide.pdf
This document provides all the information needed to facilitate a theory of change process with a community group by reviewing the major definitions of theories of change, providing important background information for facilitators before they enter a planning session, and offering practical guidance for facilitating planning sessions. The resource guide also provides a resource toolbox for the theory of change facilitator.
Clark, H. & Anderson, A. (2004). Theories of change and logic models: Telling them apart. ActKnowledge & Aspen Roundtable on Community Change. Retrieved from http://www.theoryofchange.org/wp-content/ uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA.pdf
This presentation explains the differences between theories of change and logic models and when to use each.
Community Vision. (2011). Developing a theory of change for your structural change grantmaking. Retrieved from http://www.lgbtfunders.org/commonvision/resources/developing%20a%20theory%20of%20 change.pdf
This guide defines theory of change for evaluation beginners and offers steps for developing an effective theory of change. The presentation also discusses the differences between theories of change and logic models.
The Step-by-Step Guide to Evaluation | page 225
Reisman, J. & Gienapp, A. (2004). Theory of change: A practical tool for action, results and learning. Organizational Research Services. Retrieved from http://www.aecf.org/upload/publicationfiles/ cc2977k440.pdf
This handbook created for the Annie E. Casey Foundation gives an overview of what a theory of change is and how to create one. Included in this handbook are examples of theories of change and tables and graphs that can be used as samples to create theories of change.
The Aspen Institute: Aspen Planning and Evaluation Program. (n.d.). Advocacy progress planner. Retrieved from http://planning.continuousprogress.org/
This online tool can be used by anyone interested in advocacy planning and evaluation.
Logic Models
Centers for Disease Control and Prevention, Division for Heart Disease and Stroke Prevention. (n.d.). Evaluation guide: Developing and using a logic model. Retrieved from http://www.cdc.gov/dhdsp/ programs/spha/evaluation_guides/docs/logic_model.pdf
These guides are a series of evaluation technical assistance tools developed by the CDC Division for Heart Disease and Stroke Prevention to assist in the evaluation of heart disease and stroke prevention activities within states. The guides are intended to offer guidance, provide consistent definitions of terms, and aid skill building on logic modelling.
Center for Program Evaluation and Performance Measurement. (n.d.). Planning the evaluation. Retrieved from http://www.cdc.gov/dhdsp/pubs/docs/cb_feb2014.pdf
This online resource provides a brief overview of what logic models are, why they are used, and how to develop them.
Child Welfare Information Gateway. Logic model builder. (n.d.). Retrieved from https://toolkit.childwelfare.gov/toolkit/
This online toolkit provides step-by-step assistance in developing a logic model to use for program evaluation. This tool allows for the logic model to be downloaded to Microsoft Word so that it can be customized and reformatted.
Hawkins, B., Taylor-Powell, E., & MacDonald, G. (n.d.) Logic Models Library Guide. U.S. Centers for Disease Control and Prevention. Retrieved from http://libguides.library.cdc.gov/logic_model
This web-based library includes a selection of no-cost, open-access materials on logic models for various public health activities, domestic and international. The resource includes materials in four categories: (1) logic models for program planning and implementation, (2) logic models for program evaluation, (3) developing logic models and (4) instructional materials (i.e., how to develop a logic model).
Page 226 | The Step-by-Step Guide to Evaluation
Hyndman, B., Hershfield, L., & Thesenvitz, J. (2001). Logic models workbook. The Health Communication Unit at the Centre for Health Promotion at the University of Toronto. Retrieved from http://www.blazesports.org/wp-content/uploads/2011/02/thcu-logic-model-workbook.pdf
This workbook uses a four-step approach to provide an overview of key concepts and methods to assist health promotion practitioners in the development of program logic models.
Innovation Network. (n.d.). Point K Tools: Logic model builder. Retrieved from http://www.innonet.org/?section_id=64&content_id=185
This web-based workbook assists nonprofit directors and other interested individuals in building a logic model for their program. Registration is required.
Knowlton, L. W. & Phillips, C.C. (2012). The logic model guidebook: Better strategies for great results (Second edition). Los Angeles, CA: SAGE Publications.
This guide provides students, practitioners, and beginning researchers with practical support to develop and improve logic models that reflect knowledge, practice and beliefs.
Milstein, B. & Chapel, T. Developing a logic model or theory of change. The Community Tool Box. Retrieved from http://ctb.ku.edu/en/tablecontents/sub_section_main_1877.aspx
This toolkit, created for nonprofit organizations, provides an overview of what a logic model is, when it can be used, how to create one, what makes the process effective, and the benefits and limitations of logic modeling.
Sundra, D., Scherer, J., & Anderson, L. (2003). A guide on logic model development for CDC’s Prevention Research Centers. Centers for Disease Control and Prevention. Retrieved from https://www.bja.gov/evaluation/guide/documents/cdc-logic-model-development.pdf
This guide assists users in acquiring knowledge and skills for developing logic models for their centers. This guide examines what a logic model is and the benefits of using one, the importance of involving stakeholders in logic model development and the major components of a logic model.
Taylor-Powell, E. & Henert, E. (2008). Developing a logic model: Teaching and training guide. Madison, WI: University of Wisconsin - Extension. Retrieved from http://www.uwex.edu/ces/pdande/evaluation/pdf/ lmguidecomplete.pdf
This guide provides an extensive overview of logic modeling. The guide explains what a logic model is, logic model components and language, benefits of logic models, and how to develop a logic model. Some excerpts of this piece are aimed at an advanced-level audience.
University of Wisconsin-Extension. (2003). Enhancing program performance with logic models. Retrieved from http://www.uwex.edu/ces/pdande/evaluation/pdf/lmcourseall.pdf
This document, which is a companion for an online course, provides recommendations for planning and evaluating education and outreach programs, as well as helping program practitioners use and apply
The Step-by-Step Guide to Evaluation | page 227
logic models. Throughout the PDF, icons are used to indicate additional resources (e.g., link to play audio file, practice activity). With over 200 pages of material, this document provides an in-depth look into logic models.
The World Bank. (n.d.). The LogFrame handbook: A logical framework approach to project cycle management. Retrieved from http://documents.worldbank.org/curated/en/783001468134383368/pdf/ 31240b0LFhandbook.pdf
This handbook summarizes the essential elements of logic model development. Although it provides guidelines for World Bank usage, the handbook is useful for a beginner-level audience.
Westmoreland, H., Lopez, E., & Rosenberg, H. (2009). How to develop a logic model for district-wide family engagement strategies. Harvard Family Research Project. Retrieved from http://www.hfrp.org/ publications-resources/browse-our-publications/how-to-develop-a-logic-model-for-districtwide- family-engagement-strategies
This short step-by-step guide is intended to help users to develop logic models for district-wide family engagement efforts. This piece can be used to help determine which goals are realistic, develop a strategy to achieve those goals and chart progress.
W.K. Kellogg Foundation. (2004). Logic model development guide. Retrieved from http://www.wkkf.org/ knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx
This guide provides practical assistance to nonprofits engaged in evaluation. It aims to give staff of nonprofits and community members alike sufficient orientation to the underlying principles of “logic modeling” so that they can use this tool to enhance their program planning, implementation and dissemination activities.
Engaging Stakeholders
Albritton, E., Edmunds, M., Thomas, V., Petersen, D., Ferry, G., Brach, C., & Bergofsky, L. (n.d.). The national evaluation of the CHIPRA quality demonstration grant program, implementation guide number 1: Engaging stakeholders to improve the quality of children’s health care. Agency for Healthcare Research and Quality. Retrieved from http://www.ahrq.gov/policymakers/chipra/demoeval/what-we-learned/ implementation-guides/implementation-guide1/index.html
This guide was designed to help government officials and other program administrators engage and partner with stakeholders in initiatives to improve the quality of child health care.
Bourns, C. (2010). Do nothing about me without me: An action guide for engaging stakeholders. Grantmakers for Effective Organizations. Retrieved from http://docs.geofunders.org/?filename=do_nothing_about_ me_without_me.pdf
Page 228 | The Step-by-Step Guide to Evaluation
This guide, developed for nonprofit organizations, defines stakeholder engagement, makes the case that involving stakeholders leads to improved results, provides a variety of options for engaging stakeholders, and supplies examples of the positive impact stakeholder engagement has on grantmaking.
Preskill, H. & Jones, N. (2009). A practical guide for engaging stakeholders in developing evaluation questions. Retrieved from http://www.rwjf.org/content/dam/web-assets/2009/01/a-practical-guide- for-engaging-stakeholders-in-developing-evalua
This guide aims to assist evaluators and their clients in engaging stakeholders—those with a stake or interest in the program, policy, or initiative being evaluated. The guide should assist philanthropy, but also the field of evaluation more generally, as it seeks to increase the value and usefulness of evaluation.
U.S. Agency for International Development, FHI360, PEPFAR, PTA, AVAC. (2014). Stakeholder engagement toolkit quick guide. Retrieved from http://www.fhi360.org/sites/default/files/webpages/se-toolkit/ quick-guide.pdf
This guide was designed based on feedback from community liaison officers, community educators and others working in community outreach in research settings. Its purpose is to give users quick and easy access to modifiable (Microsoft Word or Excel) files of Stakeholder Engagement Toolkit tools with instruc- tions for using them.
Cultural Competence
Centers for Disease Control and Prevention. (2014). Practical strategies for culturally competent evaluation. Atlanta, GA: U.S. Department of Health and Human Services. Retrieved from http://www.cdc.gov/dhdsp/docs/cultural_competence_guide.pdf
This guide was developed as an introduction and resource for state partners to use to promote cultural competence in the evaluation of public health programs and initiatives. Designed for program staff and evaluators, this guide highlights the prominent role of culture in our work. It provides important strat- egies for approaching an evaluation with a critical cultural lens to ensure that evaluation efforts have cultural relevance and generate meaningful findings that stakeholders—individuals who are invested in the program or potentially affected by the evaluation—ultimately will value and use.
Throughout this guide, aspects of cultural competence in evaluation are discussed within the context of CDC’s Framework for Program Evaluation in Public Health to highlight opportunities for integrating cultural competence during each of the six steps of the evaluation process. A list of related resources and tools and an abbreviated version of this guide, titled Program Evaluation Tip Sheet: Integrating Cultural Competence into Evaluation, are available as an appendix (http://www.cdc.gov/dhdsp/docs/ cultural_competence_tip_sheet.pdf ).
The Step-by-Step Guide to Evaluation | page 229
Frierson, H.T., Hood, S. & Hughes, G.B. (2002). A guide to conducting culturally responsive evaluations. In The 2002 User-friendly Handbook for Evaluation (pp. 63-73). Arlington, VA: National Science Foundation. Retrieved from http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_5.pdf
This handbook, developed for evaluators and nonprofit organizations, reviews why evaluations must consider the cultural context within which programs occur, and provides strategies for culturally responsive evaluations.
Hood, S., Hopson, R. & Frierson, H.T. (Eds.) (2015). Continuing the journey to reposition culture and cultural context in evaluation theory and practice. Greenwich, CT: Information Age.
This book contains one of the largest collections of works on evaluation in indigenous contexts and settings to be found in a single edited volume. The authors attempt to answer questions about the attributes of culturally responsive evaluation and how evaluators should exhibit cultural competence in addition to their technical knowledge.
Inouye, T.E., Yu, H.C. & Adefuin, J. (2005). Commissioning multicultural evaluation: A foundation resource guide. Oakland, CA: Social Policy Research Associates. Retrieved from http://leadershiplearning.org/ system/files/multicult_eval_rpt.pdf
This resource guide was developed to help foundation executives, program staff, and other stakeholders to integrate a multicultural focus within their initiative and program evaluations. It synthesizes some of the best learning about multicultural evaluation from field experience.
Lee, K. (2007). The Importance of Culture in Evaluation: A Practical Guide for Evaluators. Denver, CO: Colorado Trust Foundation. Retrieved from http://www.communityscience.com/pubs/ crossculturalguide.r3.pdf
This guide written for evaluators focuses on three characteristics that influence interactions among people and between people and evaluators: culture, social identity and privilege or power. There are sample questions to help evaluators not make assumptions about people and instead, approach people in a respectful way to find out about the norms and traditions that shape their worldviews. This guide does not go into detail about evaluation methods or data collection.
Lee, K. (2009). The journey continues: Ensuring a cross-culturally competent evaluation. Denver, CO: Colorado Trust Foundation. Retrieved from http://www.coloradotrust.org/attachments/0001/0089/ ensurecrossculturalcompetent-evaluation.indd.pdf
This document provides fictional case studies that show how evaluation can be cultural responsive as well as sensitive to the people involved in the evaluation.
Page 230 | The Step-by-Step Guide to Evaluation
Racial Equity Lens
Bamberger, M. & Segone M. (2011). How to design and manage equity-focused evaluation. Retrieved from http://betterevaluation.org/resource/guide/design_manage_equity_focused_evaluation
The document also explains how to conduct and design equity-focused evaluations under real-world constraints.
Denzin, N.K., Lincoln, Y. S., & Smith, L.T. (2008). Handbook of critical and indigenous methodologies. Thousand Oaks, CA: SAGE Publications.
This handbook covers everything from the history of critical and indigenous theory and how it came to inform and impact qualitative research and indigenous peoples to the critical constructs themselves, including race and diversity, gender representation (queer theory, feminism), culture, and politics to the meaning of “critical” concepts within specific disciplines (critical psychology, critical communication/ mass communication, media studies, cultural studies, political economy, education, sociology, anthro- pology, history, etc.)—all in an effort to define emancipatory research and explore what critical qualita- tive research can do for social change and social justice.
Portland’s Partnership for Racial Equity. (2011). Racial equity strategy guide. Retrieved from https://www.portlandoregon.gov/oni/article/505494
This guide was created to build capacity within the city of Portland to achieve equity on a day-to-day basis. The guide was developed by engaging staff from key city bureaus to discuss how to successfully operationalize the goal of equity into their daily work and pursue intentional goals and measurable outcomes. This publication is meant to assist bureaus and decision-makers in developing and using effective tools, which inform the city’s day-to-day actions of policy-making, resource allocation, planning, program development and implementation, and evaluation.
Race and Social Justice Initiative. (2010). Racial equity toolkit for policies, programs, and budget. Retrieved from http://www.seattle.gov/documents/departments/rsji/racialequitytoolkit_final_august2012.pdf
This toolkit, written for Seattle government employees, can be adapted by nonprofit organizations for their evaluation. It lays out a process and a set of questions to guide the development, implementation, and evaluation of policies, initiatives, programs, and budget issues to address the impacts on racial equity.
Racial tools. (n.d.). Retrieved March 7, 2016 from http://www.racialequitytools.org/home This website is a good resource for individuals and community groups working for change in their
communities, specifically with issues relating to race and racial equity. The site is written for users who are new to evaluation, or perhaps those who give funds related to racial equity, but who are not yet clear on how to evaluate it. The site discusses how to apply a “racialized” perspective to evaluation, meaning using the ideas of racism, oppression, privilege and access to power as a lens through which evaluation questions are developed and results are analyzed. The site is organized around the typical stages of
The Step-by-Step Guide to Evaluation | page 231
evaluation: (A) Getting Ready: Defining Your Work; (B)Designing: Developing Evaluation Design and Plan; (C) Collecting Information: Finding and Collecting Data; (D) Analyzing: Examining and Interpreting Evaluation Information; (E) Sharing Findings: Sharing Outcome Information and Evaluation Findings; and (F) Reflecting: Ways Evaluation Findings Can Improve the Work.
Hiring an Evaluator
Community Toolbox. (n.d.). Choosing evaluators. Retrieved March 7, 2016 from http://ctb.ku.edu/en/ table-of-contents/evaluate/evaluation/choose-evaluators/main
This section from the Community Toolbox is about starting the process of setting up an evaluation, i.e., choosing the evaluators who will carry it out, and planning what it will look like.
Hoefer, R. (2012). Successful program evaluation: Hiring an evaluator. Amazon Digital Services [Kindle] Downloaded from Amazon.com.
This manuscript addresses one question that is very important to nonprofit managers and executive di- rectors “How to hire a program evaluator?” It covers topics such as “who is an evaluator, why a nonprofit needs an evaluator, interviewing an evaluator and mistakes that can get nonprofit organizations, its director and board, and the evaluator in very serious trouble with the federal government.”
Juvenile Justice Evaluation Center (n.d.). Hiring and working and with an evaluator. Retrieved March 7, 2016 from http://www.jrsa.org/pubs/juv-justice/evaluator.pdf
The purpose of this briefing is to provide information to juvenile justice program managers about how to go about hiring an evaluator. The briefing discusses how a qualified evaluator can assist a program manager in assessing her or his program’s performance, what characteristics to look for in hiring a qual- ified evaluator, and how to go about finding such a person.
Office of Minority Health. (n.d.). How to hire an evaluator. Retrieved March 3, 2016 from http://minority- health.hhs.gov/assets/pdf/checked/1/howtohireanevaluator.pdf
This publication offers guiding principles for hiring an evaluator that is appropriate for an organization’s project needs.
Using and Presenting Evaluation Findings
Baker, A. & Bruner, B. (n.d.). Evaluative thinking: Using evaluation findings. Cambridge, MA: Bruner Foundation. Retrieved from http://www.evaluativethinking.org/docs/evaluativethinking.bulletin.6.pdf
This document explains multiple ways to communicate and use evaluation findings. It highlights how results from surveys, interviews, observations, etc., can be incorporated into planning processes. Finally, the document provides guidelines to staff or external evaluators on completing a formal evaluation report.
Page 232 | The Step-by-Step Guide to Evaluation
Centers for Disease Control and Prevention. (2013). Evaluation reporting: A guide to help ensure use of evaluation findings. Retrieved from http://www.cdc.gov/dhdsp/docs/evaluation_reporting_guide.pdf
This guide is one in a series of Program Evaluation Guides developed by the CDC’s Division for Heart Disease and Stroke Prevention (DHDSP) to assist CDC grantees in evaluating heart disease and stroke prevention activities. While the guide is written for CDC grantees, nonprofit organizations will still find it useful, as it focuses on ensuring evaluation use through evaluation reporting and addresses the following topics: (1) key considerations for effectively reporting evaluation findings; (2) essential elements for evaluation reporting; (3) importance of dissemination and (4) tools and resources.
Emery, A. (n.d.). Ann’s blog: Equipping you to collect, analyze, and visualize data [Blog archives]. Retrieved from http://annkemery.com/blog/
This blog about data visualization is useful to nonprofit organizations, as the writer provides tips on presenting data in a pictorial or graphical format.
Evergreen, S. (n.d.). Evergreen data: Intentional reporting and data visualization [Blog archives]. Retrieved from http://stephanieevergreen.com/category/blog/
This blog about data visualization provides users with ways to make slopes in Excel and display data in reports.
Evergreen, S. (2013). Presenting data effectively: Communicating your findings for maximum impact. Thousand Oaks, CA: SAGE Publications.
This book focuses on the guiding principles of presenting data in evidence-based ways so that audiences are effectively engaged and researchers are better understood. The author draws on her extensive experience in the study of research reporting, interdisciplinary evaluation and data visualization, as well as from diverse interdisciplinary fields, including cognitive psychology, communications and graphic design, to extract tangible and practical data-reporting communication lessons and insights. She then demonstrates how to apply those principles to the design of data presentations to make it easier for the audience to understand, remember, and use the data.
Holm-Hansen, C. (2008). Communicating evaluation results. Wilder Research. Retrieved from http://www.wilder.org/wilder-research/publications/studies/program%20evaluation%20and%20 research%20tips/communicating%20evaluation%20results%20-%20tips%20for%20conducting%20 program%20evaluation%20issue%2014,%20fact%20sheet.pdf
This tip sheet provides basic options for organizing and analyzing quantitative and qualitative data. It also provides tips for writing reports.
The Canadian Institute of Chartered Accountants. (2011). Improved annual reporting by not-for-profit organizations. Retrieved from http://www.cica.ca/focus-on-practice-areas/reporting-and-capital- markets/item54324.pdf
The Step-by-Step Guide to Evaluation | page 233
This guide is designed to assist staff or volunteers responsible for preparing and approving the annual report of a not-for-profit organization. It provides guiding principles for reporting, key evaluation elements that should be addressed, and examples of report components from organizations of different sizes.
The Institute for Urban Research. (2012). Making data driven decisions. Retrieved from http://community- science.com/knowledge4equity/datadrivendecisionmakingworksheets.pdf
This presentation and supplementary piece educate users about the components of a data strategy: accessing data, analyzing data, and using data to make data driven decisions.
Torres, R.T. (2012). Drawing conclusions from data and presenting them to others. Torres Consulting Group. Retrieved from http://www.communityscience.com/knowledge4equity/DrawingConclusions FromData.pdf
This presentation teaches users the basics on interpreting graphs and charts of data related to health disparities, with the goal of presenting conclusions to others. This session also covers how communication is used as a strategy for ending health disparities.
Torres, R.T., Preskill, H.S. & Piontek, M.E. (2004). Evaluation strategies for communicating and reporting: Enhancing learning in organizations (Second edition). Thousand Oaks, CA: SAGE Publications.
This book includes worksheets and instructions for creating a detailed communicating and reporting plan based on audience needs and characteristics. It also covers advances in technology including website communications, web and videoconferencing and Internet chat rooms.
APPLIED EVALUATION RESOURCES BY TOPIC AREAS This section provides resources on targeted evaluation methods and approaches that are applied to specific bodies of work.
Advocacy and Policy
Beer, T., Ingargiola, P.S. & Beer, M.F. (2012). Advocacy and public policy grantmaking: Matching process to purpose. Denver, CO: The Colorado Trust. Retrieved from http://www.evaluationinnovation.org/sites/ default/files/advocacy public policy grantmaking.pdf
This report is intended for both funders who are new to advocacy funding, and those who have been at it for years. Using the information found within this report, funders can embark on an advocacy funding strategy understanding more clearly what to expect for all stakeholders.
Centers for Disease Control and Prevention. (n.d.). Brief 1: Overview of policy evaluation. Retrieved from https://www.cdc.gov/injury/pdfs/policy/brief%201-a.pdf
Page 234 | The Step-by-Step Guide to Evaluation
This brief provides users with an overview of policy evaluation, including a description of the evaluation framework used and a definition of policy. It also presents information about the general value and potential challenges of conducting policy evaluation.
Coffman, J. (2009). A user’s guide to advocacy evaluation planning. Harvard Family Research Project. Retrieved from http://www.hfrp.org/evaluation/publications-resources/a-user-s-guide-to-advocacy- evaluation-planning
This guide was developed for advocates, evaluators, and funders who want guidance on how to evaluate advocacy and policy change efforts. This guide is broken down into four steps: (1) identify how evaluation will be used, (2) map the strategy, (3) prioritize the components and (4) identify measures and methods. The guide recommends using data collection strategies, such as surveys, document review, observation polling, focus groups, and case studies.
Coffman, J. (n.d.). Monitoring and evaluating advocacy: companion to the advocacy toolkit. UNICEF. Re- trieved from http://www.unicef.org/cbsc/files/advocacy_toolkit_companion.pdf
This toolkit provides detailed steps, guidance and tools for developing and implementing an advocacy strategy. The toolkit also outlines eight fundamental areas that can help strengthen an office’s capacity for advocacy and covers several cross cutting aspects of advocacy including monitoring and evaluating advocacy.
Coffman, J. & Beer, T. (2015). The Advocacy Strategy Framework: A tool for articulating an advocacy theory of change. Retrieved from http://www.evaluationinnovation.org/sites/default/files/Adocacy%20 Strategy%20Framework.pdf
This brief offers a simple one-page tool for thinking about the theories of change that underlie public policy advocacy strategies. It first presents the tool and then offers six questions that advocates, and funders working with advocates, can work through to articulate better their theories of change.
Coffman, J. & Reed, E. (n.d.). Unique methods in advocacy evaluation. Retrieved from http://www.innonet. org/resources/files/unique_methods_brief.pdf
This brief describes four methods (Bellwether methodology, policymaker ratings, intense period debriefs, and system mapping) that were developed to respond to advocacy’s unique measurement challenges. All four methods have been tested in real-life evaluations. The brief is targeted to a more advanced evaluation audience.
Guthrie, K., Louie, J., David, T. & Foster, C.C. (2005). The challenge of assessing policy and advocacy activities: Strategies for a prospective evaluation approach. Blueprint Research & Design, Inc. (funded by and prepared for The California Endowment). Retrieved from http://www.arabellaadvisors.com/ wp-content/uploads/2012/03/challenge-policy-advocacy-activities.pdf
This paper presents a recommended approach to policy change evaluation and is written for evaluation experts.
The Step-by-Step Guide to Evaluation | page 235
Guthrie, K., Louie, J., David, T. & Foster, C. C. (2006). The challenge of assessing policy and advocacy activities: Part II - Moving from theory to practice. Blueprint Research & Design, Inc. (funded by and prepared for The California Endowment). Retrieved from http://www.arabellaadvisors.com/wp-content/ uploads/2012/03/challenge-assessing-policy-advocacy-activities.pdf
Drawing on interviews with evaluation experts and endowment stakeholders, as well as a literature review, this study identifies some of the key issues in evaluating work on policy change and advocacy. The authors then present a framework for monitoring progress, assessing impact and deriving lessons from this type of grantmaking.
Innovation Network. (n.d.). Pathfinder: A practical guide to advocacy evaluation. Retrieved from http://www.innonet.org/resources/files/pathfinder_advocate_web.pdf
This guide is an introduction to advocacy evaluation from the advocate’s perspective. The guide seeks to provide a sense of what is involved in advocacy evaluation. It also helps readers to know what to look for during an evaluation.
John S. and James L. Knight Foundation. (2011). IMPACT: A Practical Guide to Evaluation Community Information Projects. Retrieved from https://www.knightfoundation.org/media/uploads/publication_ pdfs/Impact-a-guide-to-Evaluating_Community_Info_Projects.pdf
This guide aims to help organizations collect useful information about the effectiveness and impact of their community information projects by highlighting aspects of the evaluation process that are unique, challenging, or critical in a community information context. It also describes and includes relevant and meaningful tools that assess community information projects.
Lynn, J. (2014). Assessing and Evaluating Change in Advocacy Fields. Spark Policy Institute. Retrieved from http://www.evaluationinnovation.org/sites/default/files/Spark-Evaluating_Change_In_Advocacy_ Fields.pdf
This white paper aims to bring together emerging ideas about how to assess advocacy fields and evaluate advocacy field building initiatives.
Mansfield, C. (2010). Monitoring & evaluation of advocacy campaigns literature review. Ecumenical Advocacy Alliance. Retrieved from http://actknowledge.org/resources/documents/monitoringandevaluationofad- vocacycampaignsliteraturereview.pdf.
This report summarizes a literature review that was undertaken by the Ecumenical Advocacy Alliance and its members in preparation for developing a monitoring and evaluation tool for advocacy work. The topics covered range from the basics of organizing and conducting an evaluation to theories of policy and social change. Advocacy strategies and the best practices for assessing progress on indicators that are appropriate for a given campaign are also discussed. Realizing the importance of social equity and cultural competency during evaluation is briefly mentioned.
Page 236 | The Step-by-Step Guide to Evaluation
Organizational Research Services. (2009). Ten considerations for advocacy evaluation planning: Lessons leared from Kids Count grantee experiences. Anne E. Casey Foundation. Retrieved from http://www.organizationalresearch.com/publicationsandresources/ten_considerations_for_ advocacy_evaluation_planning.pdf
This guide explores ways to think about evaluation of advocacy and policy work and presents a framework to name advocacy and policy outcomes as well as broad directions for evaluation of advocacy and policy efforts.
Pankaj, V., Athanasiades, K., & Emery, A. (2014). Coalition assessment: Approaches for measuring capacity and impact. Retrieved from http://www.innonet.org/resources/files/innonet-coalition-assessment.pdf
This resource guide provides practitioners and funders with insights into the coalition assessment process along with concrete examples and lessons learned from other studies.
Reisman, J., Gienapp, A., & Stachowiak, S. (2007). A Guide to Measuring Advocacy and Policy. Organizational Research Services for Annie E. Casey Foundation. Retrieved from http://www.aecf.org/m/ resourcedoc/aecf-aguidetomeasuringpolicyandadvocacy-2007.pdf
This guide helps grantmakers think and talk about measurement of advocacy and policy. The guide puts forth a framework for naming outcomes associated with advocacy and policy work as well as directions for evaluation design. The framework is intended to provide common ways to identify and talk about outcomes, providing philanthropic and nonprofit audiences an opportunity to react to, refine, and adopt the outcome categories presented. In addition, grantmakers can consider some key directions for evaluation design that include a broad range of methodologies, intensities, audiences, timeframes and purposes.
Reisman, J., Gienapp, A., & Stachowiak, S. (n.d.). A handbook of data collection tools: A companion to “A guide to measuring advocacy and policy.” Annie E. Casey Foundation. http://orsimpact.com/wp- content/uploads/2013/08/a_handbook_of_data_collection_tools.pdf
This guide provides perspectives on evaluation of advocacy and policy. It also provides users with examples of practical tools and processes for collecting useful information from policy and advocacy efforts.
Workforce Development and Employment
Austrian, Z. & Norton, J. (2002). What Works in Economic Development Practice? John S. and James L. Knight Foundation. Retrieved from http://cua6.urban.csuohio.edu/economicdevelopment/knight/what_works_ final.pdf
This document reviews evaluation literature relevant to economic development strategies. A “what works” matrix follows the review of the evaluation studies. The matrix highlights significant findings and key lessons learned from this work.
The Step-by-Step Guide to Evaluation | page 237
Center for Energy Workforce Development (n.d.). Workforce development evaluation toolkit. Retrieved from http://www.cewd.org/workforce/evaltoolkit.pdf
This toolkit is designed to evaluate workforce development efforts. The evaluation process described in this tool kit can be used as a framework for measuring any workforce strategy. Included in this publication are steps on how to use the toolkit, definitions of key terms, a sample survey, and examples of data.
Weigensberg, E., Schlecht, C., Laken, F., Goerge, Ro., Stagner, M., Ballard, P., & DeCoursey, J. (2012). Inside the Black Box: What Makes Workforce Development Programs Successful? Chapin Hall at the University of Chicago. Retrieved from http://www.chapinhall.org/sites/default/files/Inside%20the%20Black%20 Box_04_23_12_0.pdf
This document can be used as a reference by community organizations as it reports on a study that looks at several successful programs in Chicago. It attempt to understand the factors that may explain why the programs achieved different types of successes, and how these factors may be quantified or measured to help improve the system.
Financial Literacy and Family Asset-building
Anders, J., Graddy, S., Grieve, M., & Visser, D. (2011). Measuring outcomes of financial capability programs: Success Measures tools for practitioners. Success Measures. Retrieved from http://www.successmeasures.org/sites/all/files/financialcapabilityonlinefinaloct2011.pdf
This guide introduces new Success Measures® outcome indicators and data collection tools that can be used by nonprofit practitioners to measure the impact and effectiveness of an array of financial capability programs. The guide also describes how and why these tools were created.
The Government of Canada. (2010). Detailed Guide to Evaluating Financial Education Programmes. Organisation for Economic Cooperation and Development (OECD) International Network on Financial Education (INFE). Retrieved from http://www.oecd.org/finance/financial-education/49994090.pdf
This web document provides users with a better understanding of some key evaluation concepts and how they can be applied to financial education projects, programs and initiatives. The document also supplies information on how to design and implement an evaluation.
Jayaratne, K.S.U., Lyons, A.C., & Palmer, L. (n.d.). Financial education evaluation manual. National Endowment for Financial Education. Retrieved from http://toolkit.nefe.org/portals/0/toolkit-manual.pdf
This online database and companion manual (available in print or electronic format) is designed to help financial educators understand evaluation concepts and apply them in educational program evaluation. Specifically, the toolkit provides assistance to financial educators who are seeking help to evaluate and document the impact of their educational programs.
Page 238 | The Step-by-Step Guide to Evaluation
Ruskin, L., & Chiem, N. (2013). Implementing evaluation in financial coaching: An exploratory case study. Retrieved from https://www.earn.org/wp-content/uploads/2015/06/130920_earn_financial_coaching_ evaluation_case_study_vf.pdf
This report discusses some challenges that can arise when evaluating financial programs and provides solutions.
Early Childhood Education
California Department of Education. (2000). Handbook on assessment and evaluation in early childhood special education programs. Retrieved from http://www.cde.ca.gov/sp/se/fp/documents/ecassmnt.pdf.
This handbook is intended to provide users with a resource that presents quality criteria for best practices in program development, ideas, concepts, and evaluation in the context of the statutory requirements for early childhood special education programs.
Friese, S., King, C., & Tout, K. (2013). INQUIRE Data Toolkit. OPRE Report # 2013-58. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. Retrieved from http://www.acf.hhs.gov/sites/default/files/opre/inquire_data_ toolkit_final_dec_2013_submitted_1_8_13.pdf
This toolkit is designed to provide tools that can be used by staff to support effective data collection and the use of data to answer important policy and reporting questions through the use of common data elements.
Hepburn, K.S., Kaufmann, R.K., Perry, D.F., Allen, M.D., Brennan, E.M., & Green, B. L. (2007). Early childhood mental health consultation: An evaluation tool kit. Washington, DC: Georgetown University, Technical Assistance Center for Children’s Mental Health; Johns Hopkins University, Women’s and Children’s Health Policy Center; and Portland State University, Research and Training Center on Family Support and Children’s Mental Health. Retrieved from http://gucchd.georgetown.edu/ products/ecmhctoolkit.pdf
This toolkit is intended to increase the capacity for high-quality evaluation of early childhood mental health consultation (ECMHC) in community based settings. The toolkit also provides states, communities, programs and grant-funded projects that are developing or have developed early childhood mental health consultation programs, guidance, tools and resources that will assist them in designing and implementing program evaluations.
Lugo-Gil, J., Sattar, S., Ross, C., Boller, K., Tout, K, & Kirby, G. (2011). The Quality Rating and Improvement System (QRIS) evaluation toolkit. OPRE Report 2011-31. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation. Retrieved from https://www.acf.hhs.gov/sites/default/files/opre/qris_toolkit.pdf
This toolkit is intended to serve as an informational resource for state administrators, child care and early education practitioners, and other stakeholders on how Quality Rating and Improvement Systems (QRISs)
The Step-by-Step Guide to Evaluation | page 239
work. The authors also discuss why it is important to conduct evaluation of QRISs and on how to plan and design an evaluation of QRISs.
Slentz, K.L., Early, D.M., & McKenna, M. (2008). A guide to assessment in early childhood: Infancy to age eight. Washington State Office of Superintendent of Public Instruction. Retrieved from http://www.k12.wa.us/earlylearning/pubdocs/assessment_print.pdf
This guide is intended as a resource for designing professional development activities for program administrators and direct service staff responsible for gathering and interpreting assessment information.
UNICEF. (2012). A framework and tool box for monitoring and improving quality. Retrieved from http://www.unicef.org/ceecis/ecd_framework_part_ii_june3.pdf
The objective of this document is to provide a comprehensive framework with tools for monitoring and evaluating the quality of early childhood education services as a means to improving access to and avail- ability of such services.
K-12 and Higher Education
Baltimore City Public School System. (2003). Performance-based evaluation handbook. Retrieved from http://www.nctq.org/evaluation_handbook/34-07.pdf
This handbook provides teacher level employees with an overview of the benefits of evaluation for improving education. While the handbook is written for teacher level employees, it is still useful to nonprofit organizations seeking to evaluate their education programs.
Greene, J., Boyce, A., & Ahn J. A values-engaged, educative approach for evaluating education programs: A guidebook for practice. University of Illinois. Retrieved from http://comm.eval.org/higherlogic/system/ downloaddocumentfile.ashx?documentfilekey=75bc9c3b-b169-4529-b2d3-642056d95f35
This guidebook presents practical guidelines for evaluators of education programs. It presents these guidelines within a “values-engaged, educative” framework for evaluation. The guide is targeted toward a beginner level audience and guides the reader through a step-by-step lesson on designing a successful evaluation. Data collection methods are not discussed in this publication, but the benefits of evaluation are highlighted throughout the guidebook.
Kauerz, K., & Coffman, J. (2013). Framework for planning, implementing, and evaluating PreK-3rd grade approaches. Retrieved from http://depts.washington.edu/pthru3/PreK-3rd_Framework_11x17.pdf
This document offers a framework that is intended to be referenced and used over an extended period of time for reflection, self-evaluation, and improvement of PreK-3rd grade efforts. This framework helps to address key questions facing those who are developing PreK-3rd grade approaches in their school, districts and communities.
Page 240 | The Step-by-Step Guide to Evaluation
Opportunity Culture. (2012). Redesigning schools to reach every student with excellent teachers. Retrieved from http://opportunityculture.org/wp-content/uploads/2012/05/selection_development_evaluation_ toolkit-public_impact.pdf
This toolkit includes job descriptions, competencies, and companion tools that can be used to select, evaluate and develop teachers and staff.
Paulsen, C.A. & Dailey, D. (2002). A guide for education personnel: Evaluating a program or intervention. Elementary and Middle Schools Technical Assistance Center (EMSTAC). Washington, DC: American Institutes for Research. Retrieved from http://www.emstac.org/resources/eval.pdf
This guide provides school, district, and state personnel with an overview of the evaluation process. The guide describes what is entailed in an evaluation and issues to be aware of when planning one. It also provides steps to help evaluators get started in the planning process and identify areas where one may need assistance.
Rockwood School District. (2007). Rockwood School District program evaluation plan. Retrieved from http://www.rsdmo.org/dataquality/Program%20Evaluation%20Documents/Program%20Evaluation %20Plan.pdf
This evaluation plan document is intended to be used as a sample document by evaluators working with school district programs.
State University of New York. (2012). Guide for the Evaluation of Undergraduate Programs. Retrieved from http://www.oneonta.edu/academics/assessment/pdfs/ufsguidelines.pdf
This guide outlines how to proceed with an academic program evaluation. The document provides guidance, advice, and direction for every individual, department, governance, and administrator involved in the evaluation process and a set of useful references and highly relevant appendices.
University of North Carolina Chapel Hill. (2012). Instruction manual: Toolkit for evaluating K-12 International Outreach Programs. Retrieved from http://cgi.unc.edu/uploads/media_items/k-12- outreach-program-evaluation-manual.original.pdf
This manual is designed to help K-12 International Outreach Programs utilize evaluation to answer questions about their programs. Used in conjunction with the companion Evaluation Tool Kit on their website, the manual helps users develop basic competence and confidence in designing and implementing evaluations that address particular program needs.
Health
Centers for Disease Control and Prevention. Framework for program evaluation in public health. (MMWR 1999:48, No. RR-11). Retrieved from ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811.pdf
This document provides a framework to guide public health professionals in their use of program evaluation. The framework is a practical, non-prescriptive tool, designed to summarize and organize
The Step-by-Step Guide to Evaluation | page 241
essential elements of program evaluation. It comprises steps in program evaluation practice and standards for effective program evaluation.
Centers for Disease Control and Prevention. Introduction to program evaluation for public health programs: A self-study guide. Retrieved from http://www.cdc.gov/eval/guide/cdcevalmanual.pdf
This “how to” guide is intended to assist managers and staff of public, private, and community public health programs to plan, design, implement and use comprehensive evaluations in practical ways. The strategies presented in this manual will help ensure that evaluations meet the diverse needs of internal and external stakeholders. Such needs include (A) assessing and documenting program implementation, outcomes, efficiency and cost-effectiveness of activities and (B) taking action based on evaluation results to increase the impact of programs.
Cullen, A., Gilles, T., & Rosenthal, J. (2006). Evaluating community-based child health promotion pro- grams: A snapshot of strategies and methods. Retrieved from http://www.nashp.org/sites/default/files/ community_health_promotion.pdf
This report provides practical information for community groups and states developing evaluation components of community-based projects that focus on children’s health promotion. The report gives a snapshot of seven projects nationwide. It is not intended to provide a theoretical discussion of evaluation methods or to serve as a comprehensive review of evaluation programs.
Gage, A.J., Ali, D., & Suzuki, C. (2005). A Guide for Monitoring and Evaluating Child Health Programs. Retrieved from http://www.coregroup.org/storage/documents/Workingpapers/ms-05-15.pdf
This guide for international organizations provides a comprehensive listing of commonly used indica- tors for monitoring and evaluating child health programs in developing countries. The guide provides a succinct but thorough overview of monitoring and evaluation, describing program components (inputs, processes, outputs, and outcomes), evaluation approaches, indicators and data sources. Commonly used indicators and a description for each are listed for various child health topics, such as newborn health, immunization and mortality.
National Network for Oral Health Access. (2015). A user’s guide for implementation of interprofessional oral health core clinical competencies: Results of a pilot project. Retrieved from http://www.nnoha.org/ nnoha-content/uploads/2015/01/IPOHCCC-Users-Guide-Final_01-23-2015.pdf
This guide helps public health professionals in their use of program evaluation. This guide provides a structure, options and suggestions to help health centers develop programs to implement oral health competencies that integrate oral health care into primary health care, increasing access to oral health care and improving the oral health status of the populations the health centers serve.
Organizational Research Services. (2000). Community roots for oral health - Guidelines for successful coa- litions, supplement: Selecting and evaluating outcomes for oral health coalition efforts. Retrieved from http://www.doh.wa.gov/portals/1/documents/pubs/160-142-coalitions-apx4.pdf
Page 242 | The Step-by-Step Guide to Evaluation
This guide is intended to accompany the Community Roots for Oral health—Guidelines for Successful Coalitions and serves as a primer on selecting and evaluating outcomes for community-based oral health coalition efforts.
The Center for the Advancement of Community Based Public Health (2000). An evaluation framework for community health programs. Retrieved from http://prevention.sph.sc.edu/Documents/CENTERED%20 Eval_Framework.pdf
This document presents a framework that emphasizes program evaluation as a practical and ongoing process that involves program staff and community members along with evaluation experts. The overall goal of the framework is to help guide and inform the evaluation process.
Van Marris, B. & King, B. (2007). Evaluating health promotion programs. Centre for Health Promotion at the University of Toronto. Retrieved from http://www.blazesports.org/wp-content/uploads/2011/02/ THCU-Evaluation-Workbook.pdf
This document assists health promotion practitioners in the development and implementation of program evaluations and is applicable to all other evaluators. The workbook describes three main types of evaluation: formative, process and summative. Readers can use this document to develop an understanding of evaluation methods, data, and data collection tools such as interviews and questionnaires. Social equity is only briefly mentioned while the benefits of evaluation are discussed throughout the document.
World Health Organization, UNICEF, UC Davis, USAID, & IFPRI. (2007). Indicators for assessing infant and young child practices, part 1: Definitions. Retrieved from http://whqlibdoc.who.int/ publications/2008/9789241596664_eng.pdf
This report summarizes the discussions and consensus reached on eight core indicators and seven optional indicators for assessing infant and young child feeding practices that are population-based and can be derived from household survey data.
Poverty
Baker, J.L. (2000). Evaluating the Impact of Development Projects on Poverty. World Bank. Retrieved from http://siteresources.worldbank.org/INTISPMA/Resources/handbook.pdf
This guide provides project managers and policy analysts with the tools needed for evaluating project impact when working with individuals living in poverty. It is aimed at readers with a general knowledge of statistics. The publication focuses on the main components needed in planning impact evaluation. It discusses various data collection methods although no mention is made about social equity or cultural sensitivity. A large portion of the report focuses on the benefits of evaluation in regards to poverty reduction.
The Step-by-Step Guide to Evaluation | page 243
Ravallion, M., (1996) Issues in measuring and modeling poverty. Economic Journal, 106, 1328-44. This journal article discusses the empirical problems that confound measures of poverty. The authors
then provide recommendations to resolving this and other additional measures of poverty.
Youth Development
Harris, E. (2011). Afterschool evaluation 101: How to evaluate an expanded learning program. Harvard Family Research Project. Retrieved from http://www.k12.wa.us/21stcenturylearning/pubdocs/after- schoolevaluation101.pdf
This document is designed to help out-of-school time program directors who have little or no evaluation experience develop an evaluation strategy. The guide recommends using a variety of data collection methods, including case studies, document review, observation, tests and interviews. Evaluation approach and methods are clearly stated in this document, although no information on social equity or cultural sensitivity is included. Throughout this publication, the benefits of evaluation are frequently discussed.
Lorimer, J. (2006). Demystifying outcomes: Demonstrating results in youth development programs. Fund for the City of New York, Youth Development Institute. Retrieved from http://fred.fcny.org/ydipubs/pubs/ demystifyingoutcomes.pdf
This paper explains the importance of evaluation and presents strategies for identifying appropriate outcomes for youth development programs.
Yohalem, N. and Wilson-Ahlstrom, A. (with Fischer, S. and Shinn, M.). (2009). Measuring youth program quality: A guide to assessment tools (Second edition). Washington, DC: The Forum for Youth Investment. Retrieved from http://forumfyi.org/files/MeasuringYouthProgramQuality_2ndEd.pdf
This compendium provides useful guidance to practitioners, policy makers, and evaluators as to what options are available and what issues to consider when selecting and using a quality assessment tool. It focuses on the purpose and history, content structure and methodology, technical properties and user considerations for each of the instruments included, as well as a brief description of how they are being used in the field.
SECONDARY DATA SOURCES Juvenile Justice
Office of Juvenile Justice and Delinquency Prevention (OJJDP) Easy Access to Juvenile Populations. http://www.ojjdp.gov/ojstatbb/ezapop/default.asp This site provides access to national, state and county level population data detailed by age, sex, race,
and ethnicity. Users can create detailed population profiles for a single jurisdiction or create state com- parison or county comparison tables.
Page 244 | The Step-by-Step Guide to Evaluation
Statistical Briefing Book. https://www.ojjdp.gov/ojstatbb/ This site includes links to data on juveniles as victims, juveniles as offenders, juvenile justice system
structure and process, law enforcement and juvenile crime, juveniles in court, juveniles on probation, juveniles in corrections and juvenile reentry and aftercare.
The National Juvenile Court Data Archive. https://www.ojjdp.gov/ojstatbb/njcda/default.asp This site is a good source for statistics on juvenile court, available in number form on the Easy Access
page listed above, but the National Juvenile Court Data Archive provides the stat book for download in PDF, which is a little easier to digest. The data can also be downloaded for any state at http://www.ojjdp.gov/ojstatbb/njcda/default.asp.
The Juvenile Justice Information Exchange. http://jjie.org/ This site provides data and basic information on every juvenile justice system in the country.
U.S. Department of Education, Office of Civil Rights. http://ocrdata.ed.gov/ This site provides data collected on key education and civil rights issues in our nation’s public schools
for use by Office of Civil Rights in its enforcement and monitoring efforts regarding equal educational opportunity. The Civil Rights Dara Collection (CRDC) is also a tool for other department offices, federal agencies, policymakers, researchers, educators, school officials, and the public to analyze student equity and opportunity.
Early Childhood Education
Child Care and Early Education Research Connections.http://www.researchconnections.org/childcare/ welcome
This site offers a broad spectrum of research data on child care and early education and related policies. To make use of this data, the user needs statistical software and a good understanding of statistical methodology.
The Early Childhood Data Collaborative. http://www.ecedata.org/ This site provides tools and data resources to encourage state policy change. The site also provides a
national forum to support the development and use of coordinated state early childhood data collaborative data systems.
Education Resource Information Center (ERIC). http://eric.ed.gov/ This database provides access to information from journals included in the Current Index of Journals in
Education and Resources in Education Index.
National Center for Education Statistics, Early Childhood Longitudinal Program. https://nces.ed.gov/ecls/ This site provides national data on children’s status at birth and at various points thereafter; children’s
The Step-by-Step Guide to Evaluation | page 245
transitions to nonparental care, early education programs, and school; and children’s experiences and growth through the eighth grade. The site also provides data that can be used to analyze the relationships among a wide range of family, school, community, and individual variables with children’s development, early learning, and performance in school.
National Center for Education Statistics, Institute of Education Sciences’ DAS: Data Analysis System. http://nces.ed.gov/das/
This web-based data analysis software provides public access to mostly individual-level education survey data collected by the U.S. Department of Education. Users can build their own analysis tables and covariance matrices for regression analysis. With DAS 2.0, users are able to perform weighted least-squares and logistic regression analysis. Step-by-step tutorials are available for using these data.
National Center for Education Statistics, Institute of Education Sciences Peer Analysis System (PAS) and Dataset Cutting Tool (DCT). http://nces.ed.gov/ipeds/
This site provides an essential tool for performing institutional level comparisons and for creating specific data “slices” using the Integrated Postsecondary Data System (IPEDS). PAS provides a variety of analytical features for peer analysis. This includes the ability to create new calculated variables, to sort and rank schools based on the data items selected, and to view standard report templates. DCT allows users to download IPEDS data to use with software packages for analysis and comparisons. The DCT can also be used to obtain complete data files and codes or to create customized datasets according to your specifications. Users may select their schools of interest with the DCT or may upload a list previously created in PAS.
UNICEF. http://data.unicef.org/ecd/early-childhood-education This site acts as the main source of nationally representative and comparable data on attendance in early
childhood care and education programs. Users can download Microsoft Excel files containing data from the site.
K-12 and Higher Education
Data.Gov — Education http://www.data.gov/education/ This site includes multiple databases. Datasets cover education at all levels.
Education Resource Information Center (ERIC). http://eric.ed.gov/ This database provides access to information from journals included in the Current Index of Journals in
Education and Resources in Education Index.
National Center for Education Statistics. http://nces.ed.gov/datatools/ This site acts as the primary federal entity for collecting and analyzing data related to education. Under
the tab for Data & Tools, NCES offers custom datasets and tools, state/district profiles, and more.
Page 246 | The Step-by-Step Guide to Evaluation
U.S. Department of Education, Office of Civil Rights. http://ocrdata.ed.gov/ This site includes results from data on key education and civil rights issues in our nation’s public schools
for use by Office of Civil Rights in its enforcement and monitoring efforts regarding equal educational opportunity. The Civil Rights Data Collection is also a tool for other department offices, federal agencies, policymakers, researchers, educators, school officials, and the public who wish to access and analyze data on student equity and opportunity.
Health
CDC Behavioral Risk Factor Surveillance System (BRFSS). http://www.cdc.gov/brfss/data_documentation/
This resource offers information and links to the BRFSS data. The BRFSS collects health-related state data through telephone surveys of U.S. residents regarding their health-related risk behaviors, chronic health conditions and use of preventive services. BRFSS collects data in all 50 states as well as the District of Columbia and three U.S. territories. BRFSS is the largest continuously conducted health survey system in the world, interviewing over 400,000 adults each year.
CDC Wide-ranging Online Data for Epidemiologic Research (WONDER). http://wonder.cdc.gov This site is an easy-to-use, menu driven system that makes the information resources of the CDC
available to public health professionals and the public at large. The system allows users to access statistical research data published by CDC, as well as reference materials, reports and guidelines on health-related topics.
Community Health Status Indicators (CHSI). http://wwwn.cdc.gov/communityhealth This an interactive web application that produces health profiles for all 3,143 counties in the United
States. Each profile includes key indicators of health outcomes, which describe the population health status of a county and factors that have the potential to influence health outcomes, such as health care access and quality, health behaviors, social factors and the physical environment.
County Health Rankings. http://www.countyhealthrankings.org/ This website provides access to 50 state reports, ranking each county within the 50 states according to its
health outcomes and the multiple health factors that determine a county’s health.
Data.Gov — Health. https://www.data.gov/health/ This site includes links to multiple databases on health.
Data Set Directory of Social Determinants of Health at the Local Level. http://www.communityscience.com/ knowledge4equity/DataSetDirectory.pdf
This guide contains a list of existing data sets that exist and can be used to address determinants of health. The data are organized into 12 dimensions (categories) of the social environment—economy,
The Step-by-Step Guide to Evaluation | page 247
employment, education, political, environmental, housing, medical, governmental, public health, psy- chosocial, behavioral and transport.
FastStats. https://www.cdc.gov/nchs/fastats/ This site provides access to statistics on topics of public health importance and is organized
alphabetically. Links are provided to publications that include the statistics presented and to sources to more data and related web pages.
Food Environment Atlas. http://www.ers.usda.gov/foodatlas/ This site provides links to data on food choice and health. Users can create maps showing variation in a
single indicator across the United States, view all county-level indicators for a selected county and identify counties sharing the same degree of multiple indicators.
Health Data Interactive (HDI). www.cdc.gov/nchs/hdi.htm This web-based application provides access to pretabulated national data. The site provides interactive
access to a broad set of key public health statistics. The primary objective is to provide national estimates of public health measures cross tabulated by a common set of variables. HDI also aims to educate users about the data and data systems available from the National Center of Health Statistics.
Health Indicators Warehouse. http://www.healthindicators.gov/ This site provides access to high quality data that can be used to improve the users understanding of
community’s health status and health determinants. The purpose of the site is to (1) provide a single source for national, state, and community health indicators; (2) meet needs of multiple population health initiatives; (3) facilitate harmonization of indicators across initiatives and (4) link indicators with evi- dence-based interventions.
Health Resources and Services Administration’s (HRSA) Community Health Status Indicators (CHSI). http://www.communityhealth.hhs.gov/homepage.aspx?j=1
This site provides over 200 health indicators at the county level. Within this database, there are a range of summary tables. CHSI allows users to select a county and view a range of data published by different federal agencies. The site also suggests “peer counties” that have similar demographics.
ICPSR. https://www.icpsr.umich.edu/icpsrweb/icpsr/index.jsp This site maintains and provides access to a vast archive of social science data for research and instruc-
tion (over 8,000 discrete studies or surveys with more than 65,000 data sets), and offers training in analyzing quantitative data in order to facilitate effective data use. This is the best site to access both national and internal data related to health.
Page 248 | The Step-by-Step Guide to Evaluation
National Center for Health Statistics (NCHS) Summary of Available Data Sets. https://www.cdc.gov/nchs/ data/factsheets/factsheet_summary.htm
This site provides a list of current surveys and data collection systems used by the NCHS. Each source is accompanied with a description, data collection method, target sample size, disparity variables, fre- quency and FY 2012-2013 plans. The website organizes the surveys by population surveys, vital records, provider surveys, and telephone surveys.
Office on Women’s Health: Quick Health Data Online. http://www.healthstatus2020.com/owh/ This interactive web system provides reliable, easily accessible health data to help assess needs,
develop programs and inform policies.
The National Longitudinal Study of Adolescent to Adult Health (Add Health). http://www.cpc.unc.edu/ projects/addhealth
This site provides access to data on a longitudinal survey that began in 1994, which collects data on respondents’ social, economic, psychological, and physical well-being. The site also provides data on family, neighborhood, community, school, friendships, peer groups, and romantic relationships, providing unique opportunities to study how social environments and behaviors in adolescence are linked to health and achievement outcomes in young adulthood.
OTHER DATA-RELATED RESOURCES
American Community Survey (ACS). http://www.census.gov/acs/www/ This U.S. Census site provides data on an ongoing survey. Information from the survey generates data
that helps determine how more than $400 billion in federal and state funds are distributed each year. The U.S. Census Bureau sends the survey to about three million addresses every year.
American Fact Finder. http://factfinder2.census.gov/faces/nav/jsf/pages/index.xhtml This online tool allows users to search for “popular” facts on a city, county, or state level. These include
data on population, age, business, education, housing, income, origins and language, poverty, and veterans.
Annie E. Casey Foundation’s Kid Count Data Center. http://datacenter.kidscount.org/ This is a national and state-by-state effort to improve the well-being of children in the U.S. This program
aims to provoke local, state, and national discussions concerning ways to secure better futures for all children by providing high quality data and trend analysis.
Bureau of Labor Statistics (BLS). http://www.bls.gov/ The BLS is a data reporting agency for the U.S. government in the field of labor economics and
statistics. This unit collects, processes, analyzes, and disseminates essential statistical data to the
The Step-by-Step Guide to Evaluation | page 249
American public, the U.S. Congress, other federal agencies, state and local governments, business, and labor representatives. Data must meet a number of criteria, including relevance to current social and economic issues, timeliness in reflecting today’s rapidly changing economic conditions, accuracy and consistently high statistical quality, and impartiality in both subject matter and presentation.
Burns, J., Paul, D., & Paz, S. (2012). Participatory asset mapping. Advancement Project. Retrieved from http://www.communityscience.com/knowledge4equity/AssetMappingToolkit.pdf
This toolbox presents research concepts, methods, and tools through topical guides such as Commu- nity Research Participatory Asset Mapping. The focus is on community-based organizations that bring together community members to visualize and actualize research and its outcomes.
CDC Youth Behavioral Surveillance System (YRBSS). http://www.cdc.gov/healthyyouth/yrbs/index.htm This site monitors six types of health-risk behaviors that contribute to the leading causes of death and
disability among youth and adults. YRBSS also measures the prevalence of obesity and asthma among youth and young adults. Also included, is a national school-based survey conducted by CDC and state, territorial, tribal, and local surveys.
Census Data Federated Electronic Research, Review, Extraction, and Tabulation Tool (FERRETT). http://dataferrett.census.gov/
This data analysis and extraction tool offers recoding capabilities to customize federal, state, and local data depending on the desired search. Regardless of where the data reside, DataFerrett helps locate the data across the Internet.
Community Commons. http://www.communitycommons.org/ This is an interactive mapping, networking, and learning tool geared toward the broad-based healthy,
sustainable, and livable communities’ movement. Registered users have free access to a geographic information system (GIS) mapping tool that utilizes over 7,000 GIS data layers at state, county, zip code, block group, tract, and point-levels. The website also provides peer learning forums to help users interact with colleagues exploring similar interests and challenges.
Easy Access to the Census of Juveniles in Residential Placement: 1997-2013. http://www.ojjdp.gov/ojstatbb/ ezacjrp/
This site is updated almost annually although a few years ago, the site had some major gaps. However, the information there is incredibly useful for looking at number of youth in state facilities (a one-day count), both pre- and post-conviction. It also uses U.S. Census estimates and calculates a rate of commit- ment, making the data comparable from one state to another.
FBI Uniform Crime Reports. http://www.fbi.gov/about-us/cjis/ucr This site is useful for retrieving juvenile arrest rates, as the FBI uses a standard definition of juvenile as
anyone arrested under the age of 18. It makes the numbers from one region or state to the next more
Page 250 | The Step-by-Step Guide to Evaluation
comparable since the definition varies in some places. In this report, one can find year-by-year numbers. For example, Table 69 provides the same numbers every year, is fully downloadable, and looks at state arrest statistics for all offenses and offense categories for youth under 18.
Health, United States Annual Report. www.cdc.gov/nchs/hus.htm This document can be used by people who want to access data on health status and its determinants,
health care resources, health care utilization, and health insurance and expenditures. This report has two major components, a chart book that illustrates with text and figures major trends in health and 134 detailed trend tables.
Homeless Data Exchange (HRE). www.hudhdx.info This online tool is designed to allow the Continuum of Care programs for the homeless to submit data to
the U.S. Department of Housing and Urban Development for the Annual Housing Inventory Count, Homeless Point-in-Time Counts and the Annual Homelessness Assessment Report.
Juvenile Justice Geography, Policy, Practice & Statistics. http://www.jjgps.org/ This online repository provides visitors with a sweeping view of the juvenile justice landscape across
states and a place to make comparisons and chart changes. The site layers the most relevant national and state level statistics with information on state laws and practice and charts juvenile justice system change.
Mclean, J., Bell, J., & Rubin, V. Getting Equity Advocacy Results Guide. Policy Link. Retrieved from http://www.communityscience.com/knowledge4equity/GEARSnapshotBuildTheBase.PDF
This guide identifies essential components of successful equity advocacy for policy change.
National Data Archive on Child Abuse and Neglect. http://www.ndacan.cornell.edu/ This site provides statistics on the occurrence of child abuse and neglect across the U.S.
National Institute on Alcohol Abuse and Alcoholism (NIAAA). http://www.niaaa.nih.gov/research/ guidelines-and-resources/epidemiologic-data
This site provides alcohol-related trend data in the U.S. for apparent per capita alcohol consumption, liver cirrhosis mortality, alcohol-related morbidity among short-stay community hospital discharges, and underage drinking behavior. The data reference manuals provide detailed data tables on alcohol consumption and related conditions by demographic characteristics, including age, sex and race or ethnicity.
National Organization for Research at the University of Chicago—Neighborhoods and Families Making Connections Data. http://mcstudy.norc.org/data-access/
This site provides access to multiple databases related on various social issues. One needs to apply for access to the data.
The Step-by-Step Guide to Evaluation | page 251
Pew Research Center. http://www.pewresearch.org/topics/ This site provides access to multiple data bases related to diverse topics. Users can download data from
the site.
Promoting Health Equity. http://communityscience.com/knowledge4equity/cdcpromotinghealthequity.pdf This CDC resource provides results of a diverse forum of community participants who have experienced
developing, implementing and evaluating interventions to address conditions contributing to health inequalities.
Stories: Using information in community building and local policies. Retrieved from http://community- science.com/knowledge4equity/usinginformationincommunitybuildingandlocalpolicy.pdf
This guide presents cases from 18 cities where National Neighborhoods Indicators Partnerships (NNIP) partners have been working to achieve improvements in community conditions.
Wider Opportunities for Women, Economic Security Database. http://www.basiceconomicsecurity.org/ This site is about the Basic Economic Security Tables™ (BEST) Index and the Elder Economic Security
Standard™ Index (Elder Index) that measure the incomes workers and retired elders need to achieve economic security. The BEST and Elder Index present local expenses, savings requirements and economic security incomes by family type, and at the city, county and state levels. The BEST and Elder Index are comprehensive definitions of, and blueprints for, economic security. Database users can Find an index for a location and family type Compare their own families’ expenses to the local BEST Index Compare indexes or single expenses across locations and family types Download national, state, county and city index data Access additional information on economic security and the work supported by the BEST and
Elder Index
Page 252 | The Step-by-Step Guide to Evaluation
- Conclusion
- Glossary
- 1.1
