615 forum

AJPH LETTERS AND RESPONSES

WHAT IS PROGRAM EVALUATION?

We applaud the addition of the Methodssection to AJPH and are delighted that its inaugural article calls attention to the important role of program evaluation in public health.1 Roughly 40% of the American Evaluation Association’s 7100 members work in public health. As leaders of the association, we welcome the opportunity to engage our public health colleagues more widely, posing the following question—what is program evaluation?

The inaugural article suggests one answer, and it sparked lively discussion among our members. In the spirit of cross-disciplinary dialogue, we distill our understanding of their reaction: its definition of program evaluation is too narrow, and its framework relating program evaluation to select forms of inquiry is out of step with contemporary scholarly treatments of evaluation and interdisciplinary efforts to standardize practice.2–7

We recognize that the meanings of eval- uation broadly, and program evaluation specifically, are evolving and contested. However, both are considered broad domains of inquiry by evaluation scholars, standard- setting bodies, associations, and practitioners.

Within this view, impact evaluation, imple- mentation science, and comparative effec- tiveness research are subsets of program evaluation, itself a subset of evaluative inquiry. Program evaluation may serve many purposes, including those of program improvement; accountability and decision making; judg- ments of merit, worth, and significance; and ultimately promoting social welfare.

The article distinguishes program evalua- tion from other inquiry approaches by the type of knowledge it generates. The Joint Committee, a national interdisciplinary standard-bearer for evaluation, points out that striving for generalizable knowledge does not differentiate research from program evalua- tion.2 Some program evaluations seek gen- eralizable knowledge, others do not; it is not a fundamental imperative. Moreover, re- search and evaluation draw on the same pool of methods. Program evaluations may, for example, employ experimental designs just as research may be conducted without them. Neither the type of knowledge generated nor methods used are differentiating factors.

There are many perspectives on program evaluation, to be certain, but the community of evaluation scholars and practitioners de- scribe it as broad in scope, responsive to the purpose of inquiry, and focusedon promoting the public good. We believe the power to define program evaluation ultimately rests with this community. An essential purpose of AJPH is to help public health research and practice evolve by learning from within and outside the field. To that end, we hope to stimulate discussion on what program eval- uation is, what it should be, and why it matters in public health and beyond.

John Gargani, PhD Robin Lin Miller, PhD

ABOUT THE AUTHORS John Gargani is President of the American Evaluation Association and President of Gargani + Company, Inc., Berkeley, CA. Robin Lin Miller is with the Department of Psychology, Michigan State University, East Lansing, and also serves on the Board of Directors for the American Evaluation Association.

Correspondence should be sent to John Gargani, President, Gargani + Company, Inc., 2625 Alcatraz Avenue, Number 508, Berkeley, CA, 94705 (e-mail:[email protected]). Re- prints can be ordered at http://www.ajph.org by clicking the “Reprints” link.

This letter was accepted February 22, 2016. doi: 10.2105/AJPH.2016.303159

CONTRIBUTORS J. Gargani developed the original response. Both authors participated equally in its revision.

ACKNOWLEDGMENTS The authors received no funding for this work. They would like to thank Thomas Chapel, Chief Evaluation Officer at the Centers for Disease Control and Preven- tion, for his comments.

REFERENCES 1. Spiegelman D. Evaluating public health interventions: 1. Examples, definitions, and a personal note. Am J Public Health. 2016;106(1):70–73.

2. Yarbrough DB, Shulha LM, Hopson RK, Caruthers FA. The Program Evaluation Standards: A Guide for Evalu- ators and Evaluation Users. 3rd ed. Thousand Oaks, CA: SAGE Publications; 2011.

3. Shadish WR, Cook TD, Leviton LC. Foundations of Program Evaluation: Theories of Practice. Newbury Park, CA: SAGE Publications; 1990:19–35.

4. Chen HT. Practical Program Evaluation: Theory-Driven Evaluation and the Integrated Evaluation Perspective. 2nd ed. Los Angeles, CA: SAGE Publications; 2015:3–33.

5. Mertens DM, Wilson AT. Program Evaluation Theory and Practice: A Comprehensive Guide. New York, NY: Guilford Press; 2012:3–12.

6. Mark MM, Henry GT, Julnes G. Evaluation: An In- tegrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs. San Francisco, CA: Jossey-Bass; 2000.

7. Schwandt TA. Evaluation Foundations Revisited: Cul- tivating a Life of the Mind for Practice. Redwood City, CA: Stanford University Press; 2015.

SPIEGELMAN RESPONDS

Ithank Gargani for his remarks on my recentcommentary, launching a series “Evaluat- ing Public Health Interventions” in AJPH, which addresses critical methodological issues that arise in the course of evaluating public health interventions. In this series, I have1,2

and will consider study design and analysis, describe the latest and most innovative emerging methodologies, and provide an overview of best practices. In the first col- umn,1 the one that to which Gargani’s letter

Letters to the editor referring to a recent AJPH article are encouraged up to 3 months after the article’s appearance. By submitting a letter to the editor, the author gives permission for its publication in AJPH. Letters should not duplicate material being published or submitted elsewhere. The editors reserve the right to edit and abridge letters and to publish responses.

Text is limited to 400 words and 7 references. Submit online at www. editorialmanager.com/ajph. Queries should be addressed to the Editor-in- Chief, Alfredo Morabia, MD, PhD, at [email protected]. ◢

June 2016, Vol 106, No. 6 AJPH Gargani and Miller Letters and Responses e13

responds, I defined four overlapping focal areas of inquiry: implementation science, impact evaluation, comparative effectiveness research, and program evaluation. Based on my review of the literature defining “program evaluation,” it appeared that the goal of program evaluation is typically intended to be specific for the program being evaluated, rather than aspiring to broader generalizability.

Gargani disagrees and made a convincing case that program evaluation may also aim for generalizability beyond the index program. I thank him for providing further evidence that unifying methods for implementation science, impact evaluation, program evalua- tion, and comparative effectiveness research will be a useful exercise, and that the ex- ceptions to the unity of methods for these closely related disciplines will likely be rare. In future columns, I will be mindful to point out these exceptions when they occur.

Donna Spiegelman, ScD

ABOUT THE AUTHOR Donna Spiegelman is with the Departments of Epidemiology, Biostatistics, Nutrition, and Global Health, Harvard T. H. Chan School of Public Health, Boston, MA.

Correspondence should be sent to Donna Spiegelman, MS, ScD, Department of Epidemiology, Harvard T. H. Chan School of Public Health, 677 Huntington Ave, Boston, MA 02115 (e-mail: [email protected]). Re- prints can be ordered at http://www.ajph.org by clicking the “Reprints” link.

This letter was accepted March 1, 2016. doi: 10.2105/AJPH.2016.303171

REFERENCES 1. Spiegelman D. Evaluating public health interven- tions: 1. Examples, definitions, and a personal note. Am J Public Health. 2016;106(1):70–73.

2. Spiegelman D. Evaluating public health interven- tions: 2. Stepping up to routine public health evaluation with the stepped wedge design. Am J Public Health. 2016; 106(3):453–457.

AJPH LETTERS AND RESPONSES

e14 Letters and Responses Spiegelman AJPH June 2016, Vol 106, No. 6

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.