HSA4141.pdf

L. Michele Issel, PhD, RN Professor of PhD Program

University of North Carolina at Charlotte College of Health and Human Services

Charlotte, North Carolina

Rebecca Wells, PhD, MHSA Professor

The University of Texas School of Public Health

Houston, Texas

Health Program Planning and Evaluation A Practical, Systematic Approach for Community Health

FOURTH EDITION

World Headquarters Jones & Bartlett Learning 5 Wall Street Burlington, MA 01803 978-443-5000 [email protected] www.jblearning.com

Jones & Bartlett Learning books and products are available through most bookstores and online booksellers. To contact Jones & Bartlett Learning directly, call 800-832-0034, fax 978-443-8000, or visit our website, www.jblearning.com.

Substantial discounts on bulk quantities of Jones & Bartlett Learning publications are available to corporations, professional associations, and other qualified organizations. For details and specific discount information, contact the special sales department at Jones & Bartlett Learning via the above contact information or send an email to [email protected]

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning Company

All rights reserved. No part of the material protected by this copyright may be reproduced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the copyright owner.

The content, statements, views, and opinions herein are the sole expression of the respective authors and not that of Jones & Bartlett Learning, LLC. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not constitute or imply its endorsement or recommendation by Jones & Bartlett Learning, LLC and such reference shall not be used for advertising or product endorsement purposes. All trademarks displayed are the trademarks of the parties noted herein. Health Program Planning and Evaluation: A Practical, Systematic Approach for Community Health, Fourth Edition is an independent publication and has not been authorized, sponsored, or otherwise approved by the owners of the trademarks or service marks referenced in this product.

There may be images in this book that feature models; these models do not necessarily endorse, represent, or participate in the activities represented in the images. Any screenshots in this product are for educational and instructive purposes only. Any individuals and scenarios featured in the case studies throughout this product may be real or fictitious, but are used for instructional purposes only.

This publication is designed to provide accurate and authoritative information in regard to the Subject Matter covered. It is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional service. If legal advice or other expert assistance is required, the service of a competent professional person should be sought.

15842-7

Production Credits VP, Executive Publisher: David D. Cella Publisher: Michael Brown Associate Editor: Danielle Bessette Vendor Manager: Nora Menzi Senior Marketing Manager: Sophie Fleck Teague Manufacturing and Inventory Control Supervisor: Amy Bacus Composition and Project Management: S4Carlisle Publishing

Services

Cover Design: Scott Moden Director of Rights & Media: Joanna Gallant Rights & Media Specialist: Merideth Tumasz Media Development Editor: Shannon Sheehan Cover Image: © Lynne Nicholson/Shutterstock Printing and Binding: Edwards Brothers Malloy Cover Printing: Edwards Brothers Malloy

Library of Congress Cataloging-in-Publication Data Names: Issel, L. Michele, author. | Wells, Rebecca, 1966- author. Title: Health program planning and evaluation: a practical, systematic approach for community health/L. Michele Issel and Rebecca Wells. Description: Fourth edition. | Burlington, MA: Jones & Bartlett Learning, [2018] | Includes bibliographical references and index. Identifiers: LCCN 2017010386 | ISBN 9781284112115 (pbk.) Subjects: | MESH: Community Health Services—organization & administration | Program Development—methods | Health Planning—methods | Program Evaluation—methods | United States Classification: LCC RA394.9 | NLM WA 546 AA1 | DDC 362.12068—dc23 LC record available at https://lccn.loc.gov/2017010386

6048

Printed in the United States of America 21 20 19 18 17 10 9 8 7 6 5 4 3 2 1

iii

© Lynne Nicholson/Shutterstock

Contents List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

List of Exhibits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii

Preface to the Fourth Edition . . . . . . . . . . . . . . . . . . xix

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv

List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxvii

SECTION I The Context of Health Program Development 1

Chapter 1 Context of Health Program Development and Evaluation . . . . . . . . . . . . . . . 3

History and Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Concept of Health . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Health Programs, Projects, and Services . . . . . . 4

History of Health Program Planning and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Evaluation as a Profession . . . . . . . . . . . . . . . . . . . . . . . . . 8

Who Does Planning and Evaluations? . . . . . . .10

Roles of Evaluators . . . . . . . . . . . . . . . . . . . . . . . . . .10

Planning and Evaluation Cycle . . . . . . . . . . . . . . . . . . .11

Interdependent and Cyclic Nature of Planning and Evaluation . . . . . . . . . . . . . . .11

Using Evaluation Results as the Cyclical Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

Program Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . . .13

The Fuzzy Aspects of Planning . . . . . . . . . . . . . . . . . . .14

Paradoxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Uncertainty, Ambiguity, Risk, and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Introduction to the Types of Evaluation . . . . . . . . . .19

Mandated and Voluntary Evaluations . . . . . . .20

When Not to Evaluate . . . . . . . . . . . . . . . . . . . . . .21

The Public Health Pyramid . . . . . . . . . . . . . . . . . . . . . . .21

Use of the Public Health Pyramid in Program Planning and Evaluation . . . . . . . .23

The Public Health Pyramid as an Ecological Model . . . . . . . . . . . . . . . . . . . . .23

The Town of Layetteville in Bowe County . . . . . . . . .25

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

Chapter 2 Relevance of Diversity and Disparities to Health Programs . . . . . . . . . . . . . . . . . . 29

Health Disparities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30

Diversity and Health Disparities . . . . . . . . . . . . .32

Diversity and Health Programs . . . . . . . . . . . . . .33

Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33

Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

Influences of Sociocultural Diversity on Interventions . . . . . . . . . . . . . . . . . . . . . . . . .38

Influences of Biological Diversity on Interventions . . . . . . . . . . . . . . . . . . . . . . . . .39

Approaches to Developing Programs . . . . . . .39

Profession and Provider Diversity . . . . . . . . . . . .40

The Three Health Provider Sectors . . . . . . . . . .43

Diversity Within Healthcare Organizations and Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43

Organizational Culture . . . . . . . . . . . . . . . . . . . . . .44

Cultural Competency Continuum . . . . . . . . . . .44

Enhancing Cultural Competency . . . . . . . . . . .48

iv Contents

Types of Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . .75

Organizational Assessment . . . . . . . . . . . . . . . . .75

Marketing Assessment . . . . . . . . . . . . . . . . . . . . . .76

Needs Assessment . . . . . . . . . . . . . . . . . . . . . . . . .76

Community Health Assessment . . . . . . . . . . . . .77

Workforce Assessment . . . . . . . . . . . . . . . . . . . . . .77

Steps in Planning and Conducting the Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Form and Develop the Team . . . . . . . . . . . . . . . .78

Create a Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

Involve Community Members . . . . . . . . . . . . . .79

Define the Population . . . . . . . . . . . . . . . . . . . . . .80

Define the Problem to Be Assessed . . . . . . . . .81

Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81

Prioritize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

Make a Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

Implement and Continue . . . . . . . . . . . . . . . . . . .83

Anticipate Data-Related and Methodological Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .85

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87

Chapter 4 Characterizing and Defining the Health Problem . . . . . . . . . . . . . . . . . . . 91

Collecting Data From Multiple Sources . . . . . . . . . . .91

Public Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91

Primary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92

Observational Data . . . . . . . . . . . . . . . . . . . . . . . . .92

Archival Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

Proprietary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

Published Literature . . . . . . . . . . . . . . . . . . . . . . . .93

Data Beyond Street Lamp . . . . . . . . . . . . . . . . . . .93

Collecting Descriptive Data . . . . . . . . . . . . . . . . . . . . . .94

Magnitude of the Problem . . . . . . . . . . . . . . . . . .94

Dynamics Leading to the Problem . . . . . . . . . .94

Population Characteristics . . . . . . . . . . . . . . . . . .96

Attitudes and Behaviors . . . . . . . . . . . . . . . . . . . .96

Years of Life and Quality of Life . . . . . . . . . . . . . .96

Stakeholders and Coalitions . . . . . . . . . . . . . . . . . . . . .50

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54

SECTION II Defining the Health Problem 57

Chapter 3 Community Health Assessment for Program Planning . . . . . . . . 59

Defining Community . . . . . . . . . . . . . . . . . . . . . . . . . . . .59

Community as Context and Intended Recipient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60

Defining Terms: Based, Focused, and Driven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

Types of Needs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62

Types of Strengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63

Approaches to Planning . . . . . . . . . . . . . . . . . . . . . . . . .64

Incremental Approach . . . . . . . . . . . . . . . . . . . . . .64

Apolitical Approach . . . . . . . . . . . . . . . . . . . . . . . .66

Advocacy Approach . . . . . . . . . . . . . . . . . . . . . . . .66

Communication Action Approach . . . . . . . . . .67

Comprehensive Rational Approach . . . . . . . . .67

Strategic Planning Approach . . . . . . . . . . . . . . .68

Summary of Approaches . . . . . . . . . . . . . . . . . . .69

Models for Planning Public Health Programs . . . . .69

Mobilizing for Action through Planning and Partnership (MAPP) . . . . . . . . . . . . . . . . . .70

Community Health Improvement Process (CHIP) . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

Protocol for Assessing Community Excellence in Environmental Health (PACE-EH) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

In Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

Perspectives on Assessment . . . . . . . . . . . . . . . . . . . . .71

Epidemiological Perspective . . . . . . . . . . . . . . . .72

Public Health Perspective . . . . . . . . . . . . . . . . . . .74

Social Perspective . . . . . . . . . . . . . . . . . . . . . . . . . .74

Asset Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . .74

Rapid Perspective . . . . . . . . . . . . . . . . . . . . . . . . . .75

v Contents

Path to Program Outcomes and Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Components of the Effect Theory . . . . . . . . . 135

Matching Levels: Audience, Cause, Intervention, and Effects . . . . . . . . . . . . . . . 137

Generating the Effect Theory . . . . . . . . . . . . . . . . . . 138

Involve Key Stakeholders . . . . . . . . . . . . . . . . . 138

Draw Upon the Scientific Literature . . . . . . . 138

Diagram the Causal Chain of Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Check Against Assumptions . . . . . . . . . . . . . . 141

Functions of Program Theory . . . . . . . . . . . . . . . . . . 141

Provide Guidance . . . . . . . . . . . . . . . . . . . . . . . . . 141

Enable Explanations . . . . . . . . . . . . . . . . . . . . . . 142

Form a Basis for Communication . . . . . . . . . . 142

Make a Scientific Contribution . . . . . . . . . . . . 143

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

Discussion Questions and Activities . . . . . . . . . . . . 144

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Chapter 6 Program Objectives and Setting Targets . . . . . . . . 147

Program Goals and Objectives . . . . . . . . . . . . . . . . . 147

Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Foci of Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 148

Objectives and Indicators . . . . . . . . . . . . . . . . . 151

Good Goals and Objectives . . . . . . . . . . . . . . . 154

Using Data to Set Target Values . . . . . . . . . . . . . . . . 156

Decisional Framework for Setting Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

Stratification and Object Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Use of Logic Statements to Develop Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Options for Calculating Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Caveats to the Goal-Oriented Approach . . . . . . . 170

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

Discussion Questions and Activities . . . . . . . . . . . . 171

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

Statistics for Describing Health Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99

Descriptive Statistics . . . . . . . . . . . . . . . . . . . . . . 100

Geographic Information Systems: Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Small Numbers and Small Areas . . . . . . . . . . 101

Epidemiology Rates . . . . . . . . . . . . . . . . . . . . . . 102

Stating the Health Problem . . . . . . . . . . . . . . . . . . . . 102

Diagramming the Health Problem . . . . . . . . 102

Writing a Causal Theory of the Health Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Prioritizing Health Problems . . . . . . . . . . . . . . . . . . . 110

Nominal Group Technique . . . . . . . . . . . . . . . . 111

Basic Priority Rating System . . . . . . . . . . . . . . . 111

Propriety, Economics, Acceptability, Resources, and Legality (PEARL) Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Prioritizing Based on Importance and Changeability . . . . . . . . . . . . . . . . . . . . . 114

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Discussion Questions and Activities . . . . . . . . . . . . 117

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

SECTION III Health Program Development and Planning 121

Chapter 5 Program Theory and Interventions Revealed . . . . . . . . . . . . . . . . . . 123

Program Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Process Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Effect Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Finding and Identifying Interventions . . . . . 126

Types of Interventions . . . . . . . . . . . . . . . . . . . . 127

Specifying Intervention Administration and Dosage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

Interventions and Program Components . . . . 130

Characteristics of Good Interventions . . . . . 131

vi Contents

Budgeting as Part of Planning . . . . . . . . . . . . . . . . . . 204

Monetize and Compute Program Costs . . . . . 204

Budget for Start-Up and Evaluation Costs . . . 205

Break-Even Analysis . . . . . . . . . . . . . . . . . . . . . . . 205

Budget Justification . . . . . . . . . . . . . . . . . . . . . . 207

Budget as a Monitoring Tool . . . . . . . . . . . . . . . . . . . 209

Budget Variance . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Types of Cost Analyses . . . . . . . . . . . . . . . . . . . . 209

Information Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

Health Informatics Terminology . . . . . . . . . . . 214

Information Systems Considerations . . . . . . 214

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

Discussion Questions and Activities . . . . . . . . . . . . 217

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs . . . . . . . . . . . . . . . 219

Assessing the Implementation . . . . . . . . . . . . . . . . . 219

Implementation Documentation . . . . . . . . . 220

Implementation Assessment . . . . . . . . . . . . . 221

Implementation Evaluation . . . . . . . . . . . . . . . 221

Efficacy, Effectiveness, and Efficiency . . . . . . . . . . . 222

Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . 223

Quantifying Inputs to the Organizational Plan . . . . . . . . . . . . . . . . . . . 223

Human Resources . . . . . . . . . . . . . . . . . . . . . . . . 228

Physical Resources . . . . . . . . . . . . . . . . . . . . . . . . 229

Quantifying Outputs of the Organizational Plan . . . . . . . . . . . . . . . . . . . 230

Information Systems . . . . . . . . . . . . . . . . . . . . . . 230

Monetary Resources . . . . . . . . . . . . . . . . . . . . . . 230

Quantifying Inputs to the Services Utilization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

Participants and Recipients . . . . . . . . . . . . . . . 230

Intervention Delivery and Fidelity . . . . . . . . . 231

Quantifying Outputs of the Services Utilization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

Coverage as Program Reach . . . . . . . . . . . . . . 234

Participant-Related Issues . . . . . . . . . . . . . . . . . 238

Program Logistics . . . . . . . . . . . . . . . . . . . . . . . . . 240

SECTION IV Implementing and Monitoring the Health Program 173

Chapter 7 Process Theory for Program Implementation . . . . . . . . . . . 175

Organizational Plan Inputs . . . . . . . . . . . . . . . . . . . . . 175

Human Resources . . . . . . . . . . . . . . . . . . . . . . . . 177

Physical Resources . . . . . . . . . . . . . . . . . . . . . . . . 179

Transportation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

Informational Resources . . . . . . . . . . . . . . . . . . 180

Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

Managerial Resources . . . . . . . . . . . . . . . . . . . . 180

Fiscal Resources . . . . . . . . . . . . . . . . . . . . . . . . . . 182

Organizational Plan Outputs . . . . . . . . . . . . . . . . . . . 182

Time Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

Operations Manual . . . . . . . . . . . . . . . . . . . . . . . 182

Organizational Chart . . . . . . . . . . . . . . . . . . . . . . 184

Information System . . . . . . . . . . . . . . . . . . . . . . . 185

Inputs to Service Utilization Plan . . . . . . . . . . . . . . . 185

Social Marketing . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Eligibility Screening . . . . . . . . . . . . . . . . . . . . . . . 185

Queuing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

Intervention Delivery . . . . . . . . . . . . . . . . . . . . . 189

Services Utilization Plan Outputs . . . . . . . . . . . . . . . 191

Summary: Elements of Organizational and Services Utilization Plans . . . . . . . . . . . 192

Alternative Plan Formats . . . . . . . . . . . . . . . . . . . . . . . 192

Logic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

Business Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Discussion Questions and Activities . . . . . . . . . . . . 197

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

Chapter 8 Monitoring Implementation Through Budgets and Information Systems . . . . . . . 201

Budgets and Budgeting . . . . . . . . . . . . . . . . . . . . . . . 201

Budgeting Terminology . . . . . . . . . . . . . . . . . . . 202

vii Contents

Evaluation and Research . . . . . . . . . . . . . . . . . . 268

Rigor in Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 270

Variables from the Program Effect Theory . . . . . . 271

Outcome and Impact Dependent Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

Causal Factors as Independent Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

Antecedent, Moderating, and Mediating Factors as Variables . . . . . . . . . . 273

Measurement Considerations . . . . . . . . . . . . . . . . . . 275

Units of Observation . . . . . . . . . . . . . . . . . . . . . . 275

Types of Variables (Levels of Measurement) . . . . . . . . . . . . . . . . . . . . . . . 275

Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278

Sensitivity of Measures . . . . . . . . . . . . . . . . . . . . 278

Threats to Data Quality . . . . . . . . . . . . . . . . . . . . . . . . 279

Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

Reliability Concerns . . . . . . . . . . . . . . . . . . . . . . . 280

Validity of Measures . . . . . . . . . . . . . . . . . . . . . . 281

Contextual Considerations in Planning the Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

Evaluation Standards . . . . . . . . . . . . . . . . . . . . . 281

Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

Discussion Questions and Activities . . . . . . . . . . . . 284

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

Chapter 12 Choosing Designs for Effect Evaluations . . . . . . . . . 287

Evaluation Design Caveats . . . . . . . . . . . . . . . . . . . . . 288

Considerations in Choosing a Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

Using Designs Derived from Multiple Paradigms: An Example . . . . . . . . . . . . . . . . 294

Choosing the Evaluation Design . . . . . . . . . . . . . . . 294

Identifying Design Options . . . . . . . . . . . . . . . 294

Overview of the Decision Tree . . . . . . . . . . . . 295

Designs for Outcome Documentation . . . . 298

Designs for Outcome Assessment: Establishing Association . . . . . . . . . . . . . . . . 301

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Discussion Questions and Activities . . . . . . . . . . . . 242

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

Chapter 10 Program Quality and Fidelity: Managerial and Contextual Considerations . . . . . . . . . . . . 245

The Accountability Context . . . . . . . . . . . . . . . . . . . . 246

Program Accountability . . . . . . . . . . . . . . . . . . . 246

Professional Accountability . . . . . . . . . . . . . . . 246

Performance and Quality: Navigating the Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247

Quality Improvement Approaches . . . . . . . . 248

Quality Improvement Tools . . . . . . . . . . . . . . . 248

Relevance to Health Programs . . . . . . . . . . . . 251

Performance Measurement . . . . . . . . . . . . . . . 252

Informatics and Information Technology . . . .253

Creating Change for Quality and Fidelity . . . . . . . 255

Interpreting Implementation Data . . . . . . . . 255

Maintaining Program Process Quality and Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

Managing Group Processes for Quality and Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

When and What Not to Change . . . . . . . . . . . 259

Formative Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . 259

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 260

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

SECTION V Outcome and Impact Evaluation of Health Programs 263

Chapter 11 Planning the Intervention Effect Evaluations . . . . . . . . . 265

Developing the Evaluation Questions . . . . . . . . . . 266

Characteristics of the Right Question . . . . . 267

Outcome Documentation, Outcome Assessment, and Outcome Evaluation . . . 268

viii Contents

Issues with Quantifying Change from the Program . . . . . . . . . . . . . . . . . . . . . . 339

Relationship of Change to Intervention Effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

Clinical and Statistical Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

Across Levels of Analysis . . . . . . . . . . . . . . . . . . . . . . . 343

Statistical Answers to the Questions . . . . . . . . . . . 345

Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346

Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348

Association . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350

Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

Four Fallacies of Interpretation . . . . . . . . . . . . 353

Ecological Fallacy . . . . . . . . . . . . . . . . . . . . . . . . . 354

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356

Discussion Questions and Activities . . . . . . . . . . . . 356

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

Chapter 15 Qualitative Methods for Planning and Evaluation . . . . . . . . . . . . . . . 359

Qualitative Methods Throughout the Planning and Evaluation Cycle . . . . . . . . . . . . . . 359

Qualitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 360

Individual In-Depth Interview . . . . . . . . . . . . 361

Written Open-Ended Questions . . . . . . . . . . . 362

Focus Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364

Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 364

Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365

Innovative Methods . . . . . . . . . . . . . . . . . . . . . . 366

Scientific Rigor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368

Sampling for Qualitative Methods . . . . . . . . . . . . . 369

Analysis of Qualitative Data . . . . . . . . . . . . . . . . . . . . 372

Overview of Analytic Process . . . . . . . . . . . . . 372

Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374

Issues to Consider . . . . . . . . . . . . . . . . . . . . . . . . 374

Presentation of Findings . . . . . . . . . . . . . . . . . . . . . . . 375

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376

Designs for Outcome Evaluation: Establishing Causation . . . . . . . . . . . . . . . . . 307

Practical Issues with Experimental Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307

Designs and Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 312

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312

Chapter 13 Sampling Designs and Data Sources for Effect Evaluations . . . . . . . . . 315

Sampling Realities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Sample Construction . . . . . . . . . . . . . . . . . . . . . . . . . . 317

Hard-to-Reach Populations . . . . . . . . . . . . . . . 318

Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318

Calculating Response Rates . . . . . . . . . . . . . . . 319

Sampling for Effect Evaluations . . . . . . . . . . . . . . . . 322

Sampling for Outcome Assessment . . . . . . . 322

Sampling for Outcome Evaluation . . . . . . . . 324

Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . 324

Surveys and Questionnaires . . . . . . . . . . . . . . 325

Secondary Data . . . . . . . . . . . . . . . . . . . . . . . . . . 328

Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

Physical Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

Discussion Questions and Activities . . . . . . . . . . . . 330

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

Chapter 14 Quantitative Data Analysis and Interpretation . . . . . . . . . . . . 335

Data Entry and Management . . . . . . . . . . . . . . . . . . 335

Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Linked Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Sample Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338

Thinking About Change . . . . . . . . . . . . . . . . . . . . . . . 339

Change as a Difference Score . . . . . . . . . . . . . 339

ix Contents

Reporting Responsibly . . . . . . . . . . . . . . . . . . . . . . . . . 392

Report Writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392

Making Recommendations . . . . . . . . . . . . . . . 394

Misuse of Evaluations . . . . . . . . . . . . . . . . . . . . . 397

Responsible Contracts . . . . . . . . . . . . . . . . . . . . . . . . . 398

Organization–Evaluator Relationship . . . . . . 398

Health Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399

Responsible for Evaluation Quality . . . . . . . . . . . . . 400

Responsible for Dissemination . . . . . . . . . . . . . . . . . 401

Responsible for Current Practice . . . . . . . . . . . . . . . 402

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404

Discussion Questions and Activities . . . . . . . . . . . . 405

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .409

Discussion Questions and Activities . . . . . . . . . . . . 377

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

SECTION VI Additional Considerations for Evaluators 381

Chapter 16 Program Evaluators’ Responsibilities . . . . . . . . . . . 383

Ethical Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . 383

Ethics and Planning . . . . . . . . . . . . . . . . . . . . . . . 383

Institutional Review Board Approval and Informed Consent . . . . . . . . . . . . . . . . . 385

Ethics and Evaluation . . . . . . . . . . . . . . . . . . . . . 387

HIPAA and Evaluations . . . . . . . . . . . . . . . . . . . . 388

Responsible Spin of Data and Information . . . . . 389

Persuasion and Information . . . . . . . . . . . . . . . 389

Information and Sensemaking . . . . . . . . . . . . 391

xi

© Lynne Nicholson/Shutterstock

List of Figures Figure 1-1 The Planning and Evaluation Cycle

Figure 1-2 The Public Health Pyramid

Figure 1-3 The Pyramid as an Ecological Model

Figure 2-1 Effects of Diversity Throughout the Planning and Evaluation Cycle Stage in the Planning and Evaluation Cycle

Figure 3-1 Connections Among Program, Agency, and Community

Figure 3-2 Venn Diagram of Community- Based, Community-Focused, and Community-Driven

Figure 3-3 The Planning and Evaluation Cycle

Figure 4-1 Generic Model of a Theory of Causes

Figure 4-2 Diagram of Theory of Causes/ Determinants of Receiving Immunizations, as Contributing to Adult Immunization Rates, Using the Layetteville Example

Figure 4-3 Diagram of Theory of Causes/ Determinants for Deaths from Gunshot Wounds, as Contributing to Adolescent Death Rates, Using the Layetteville Example

Figure 4-4 Diagram of Theory of Causes/ Determinants for Neural Tube Defects, as Contributing to Rates of Congenital Anomalies, Using the Bowe County Example

Figure 4-5 Theory of Causes/Determinants with Elements of the BPRS Score: Size, Seriousness, and Interventions

Figure 5-1 Model of Program Theory

Figure 5-2 The Effect Theory Showing the Causal Theory Using Community Diagnosis Elements

Figure 5-3 Effect Theory Example: Effect Theory for Reducing the Rate of Congenital Anomalies

Figure 5-4 Two Roots of Program Failure

Figure 6-1 Using Elements of Program Theory as the Basis for Writing Program Objectives

Figure 6-2 Diagram Showing Relationship of Effect Theory Elements to Process and Outcome Objectives

Figure 6-3 Calculations of Options 1 Through 4 Using a Spreadsheet

Figure 6-4 Calculations of Options 5 Through 8 Using a Spreadsheet

Figure 6-5 Calculations of Options 9 and 10 Using a Spreadsheet

Figure 7-1 Amount of Effort Across the Life of a Health Program

Figure 7-2 Diagram of the Process Theory Elements Showing the Components of the Organizational Plan and Services Utilization Plan

Figure 7-3 Process Theory for Neural Tube Defects and Congenital Anomalies Health Problem

Figure 7-4 Effect and Process Theory for Neural Tube Defect Prevention Program

Figure 8-1 Relevance of Process Theory to Economic Evaluations

Figure 8-2 Information System Processes Throughout the Program Planning Cycle

Figure 9-1 Elements of the Process Theory Included in a Process Evaluation

Figure 9-2 Roots of Program Failure

xii List of Figures

Figure 12-2 Decision Tree for Choosing an Evaluation Design, Based on the Design’s Typical Use

Figure 12-3 Three Sources of Program Failure

Figure 13-1 Probability and Nonprobability Samples and Their Usage

Figure 14-1 Contributing Factors to the Total Amount of Change

Figure 14-2 Summary of the Three Decisions for Choosing an Analytic Approach

Figure 14-3 Five Ways That the Rate of Change Can Be Altered

Figure 16-1 Making Recommendations Related to the Organizational and Services Utilization Plans

Figure 16-2 Making Recommendations Related to the Program Theory

Figure 16-3 The Planning and Evaluation Cycle with Potential Points for Recommendations

Figure 9-3 Examples of Organizational Plan Inputs and Outputs That Can Be Measured

Figure 9-4 Examples of Services Utilization Inputs and Outputs That Can Be Measured

Figure 10-1 List of Quality Improvement Tools with Graphic Examples

Figure 11-1 Planning and Evaluation Cycle, with Effect Evaluation Highlights

Figure 11-2 Diagram of Net Effects to Which Measures Need to Be Sensitive

Figure 11-3 Using the Effect Theory to Identify Effect Evaluation Variables

Figure 11-4 Effect Theory of Reducing Congenital Anomalies Showing Variables

Figure 12-1 Relationship Between the Ability to Show Causality and the Costs and Complexity of the Design

xiii

© Lynne Nicholson/Shutterstock

Table 4-3 Global Leading Causes of Disability- Adjusted Life-Years (DALYs) and Years of Life Lost (YLL)

Table 4-4 Numerators and Denominators for Selected Epidemiological Rates Commonly Used in Community Health Assessments

Table 4-5 Existing Factors, Moderating Factors, Key Causal Factors, Mediating Factors, and Health Outcome and Impact for Five Health Problems in Layetteville and Bowe County

Table 4-6 Relationship of Problem Definition to Program Design and Evaluation

Table 4-7 Criteria for Rating Problems According to the BPRS

Table 4-8 Program Prioritization Based on the Importance and Changeability of the Health Problem

Table 4-9 Examples of Sources of Data for Prioritizing Health Problems at Each Level of the Public Health Pyramid

Table 4-10 Examples of Required Existing, Causal, and Moderating Factors Across the Pyramid

Table 5-1 Examples of Interventions by Type and Level of the Public Health Pyramid

Table 5-2 Comparison of Effect Theory, Espoused Theory, and Theory-in-Use

Table 5-3 Examples of Types of Theories Relevant to Developing Theory of Causative/Determinant Factors or Theory of Intervention Mechanisms by Four Health Domains

List of Tables Table 1-1 Comparison of Outcome-Focused,

Utilization-Focused, and Participatory Focused Evaluations

Table 1-2 Evaluation Standards Established by the Joint Commission on Standards for Educational Evaluation

Table 1-3 Fuzzy Aspects Throughout the Planning and Evaluation Cycle

Table 1-4 A Summary of the Healthy People 2020 Priority Areas

Table 2-1 Examples of Cultural Tailoring Throughout the Program Planning and Evaluation Cycle

Table 2-2 Indicators Used to Measure Race in Different Surveys

Table 2-3 Professional Diversity Among Health Professions

Table 2-4 Cultural Continuum with Examples of the Distinguishing Features of Each Stage

Table 3-1 Three Elements of Community, with Their Characteristics

Table 3-2 Summary of the Six Approaches to Planning, with Public Health Examples

Table 3-3 Comparison of Models Developed for Public Health Planning

Table 3-4 A Comparison of the Five Perspectives on Community Health and Needs Assessment

Table 4-1 Haddon’s Typology for Analyzing an Event, Modified for Use in Developing Health Promotion and Prevention Programs

Table 4-2 Quality-of-Life Acronyms and Definitions

xiv List of Tables

Table 5-4 Examples of Types of Theories Relevant to Developing the Organizational Plan and Services Utilization Plan Components of the Process Theory

Table 6-1 Aspects of Process Objectives as Related to Components of the Process Theory, Showing the TAAPS Elements

Table 6-2 Domains of Individual or Family Health Outcomes with Examples of Corresponding Indicators and Standardized Measures

Table 6-3 Bowe County Health Problems with Indicators, Health Outcomes, and Health Goals

Table 6-4 Effect Objectives Related to the Theory of Causal/Determinant Factors, Theory of the Intervention Mechanisms, and Theory of Outcome to Impact, Using Congenital Anomalies as an Example, Showing the TREW Elements

Table 6-5 Effect Objectives Related to the Theory of Causal/Determinant Factors, Theory of the Intervention Mechanisms, and Theory of Outcome to Impact, Using Adolescent Pregnancy as an Example, Showing the TREW Elements

Table 6-6 Matrix of Decision Options Based on Current Indicator Value, Population Trend of the Health Indicator, and Value of Long-Term Objective or Standard

Table 6-7 Framework for Target Setting: Interaction of Data Source Availability and Consistency of Information

Table 6-8 Summary of When to Use Each Option

Table 6-9 Range of Target Values Derived from Options 1 Through 10, Based on the Data from Figures 6-3 Through 6-5

Table 7-1 List of Health Professionals with a Summary of Typical Legal and Regulatory Considerations

Table 7-2 Relationship of Test Sensitivity and Specificity to Overinclusion and Underinclusion

Table 7-3 Examples of Partial- and Full-Coverage Programs by Level of the Public Health Pyramid

Table 7-4 Template for Tracking Services Utilization Outputs Using Example Interventions and Hypothetical Activities

Table 7-5 Hypothetical Logic Model of a Program for Reducing Congenital Anomalies

Table 7-6 Generic Elements of a Business Plan, with Their Purpose and Corresponding Element of the Process Theory and Logic Model

Table 8-1 Formulas Applied for Options A and B

Table 9-1 Methods of Collecting Process Evaluation Data

Table 9-2 Example of Measures of Inputs and Outputs of the Organizational Plan

Table 9-3 Examples of Measures of Inputs and Outputs of the Services Utilization Plan

Table 9-4 Matrix of Undercoverage, Ideal Coverage, and Overcoverage

Table 9-5 Examples of Process Evaluation Measures Across the Public Health Pyramid

Table 10-1 Types of Program Accountability, with Definitions and Examples of Process Evaluation Indicators

Table 10-2 Comparison of Improvement Methodologies and Program Process Evaluation

Table 10-3 Definitions of Terms Used in Performance Measurement

Table 10-4 Partial List of Existing Performance Measurement Systems Used by Healthcare Organizations, with Their Websites

Table 11-1 Three Levels of Intervention Effect Evaluations

Table 11-2 Differences Between Evaluation and Research

xvList of Tables

Table 11-3 Advantages and Disadvantages of Using Each Type of Variable

Table 11-4 Examples of Nominal, Ordinal, and Continuous Variables for Different Health Domains

Table 11-5 Example Time Line Showing the Sequence of Intervention and Evaluation Activities

Table 11-6 Summary of Evaluation Elements

Table 12-1 Contribution of Various Disciplines to Health Program Evaluation

Table 12-2 Summary of Main Designs and Their Use for Individual or Population-Level Evaluations

Table 12-3 Approaches to Minimizing Each of the Three Types of Program Failure

Table 13-1 Probability and Nonprobability Samples and Their Usage

Table 13-2 Comparison of Main Types of Samples with Regard to Implementation Ease, Degree of Representativeness, and Complexity of Sampling Frame

Table 13-3 Example of Data Sources for Each Health and Well-Being Domain

Table 13-4 Interaction of Response Bias and Variable Error

Table 14-1 Calculation of Effectiveness and Adequacy Indices: An Example

Table 14-2 Intervention Efficiency as a Relation of Effect Size and Causal Size

Table 14-3 Factors That Affect the Choice of a Statistical Test: Questions to Be Answered

Table 14-4 Analysis Procedures by Level of Intervention and Level of Analysis

Table 14-5 Commonly Used Parametric and Nonparametric Statistical Tests for Comparison, Association, and Prediction

Table 14-6 Main Types of Comparison Analyses Used by Level of Analysis and Assuming That the Variables Are at the Same Level of Measurement

Table 14-7 Main Types of Association Analyses Used by Level of Analysis, Assuming That Variables Are the Same Level of Measurement

Table 14-8 Example of Statistical Tests for Strength of Association by Level of Measurement, Using Layetteville Adolescent Antiviolence Program

Table 14-9 Examples of Statistical Tests by Evaluation Design and Level of Measurement, with Examples of Variables

Table 14-10 Main Types of Prediction Analyses Used by Level of Analysis, Assuming That Variables Are at the Same Level of Measurement

Table 15-1 Comparison of Qualitative Perspectives with Regard to the Basic Question Addressed and the Relevance to Health Program Planning and Evaluation

Table 15-2 Comparison of Major Qualitative Perspectives with Regard to the Method Used

Table 15-3 Summary of Key Benefits and Challenges to Using Qualitative Methods in Planning and Evaluation

Table 15-4 Sampling Considerations for Each of the Qualitative Methods Discussed

Table 15-5 Summary of Types of Sampling Strategies Used with Qualitative Designs

Table 15-6 Example of Interview Text with Final Coding

Table 15-7 Suggested Qualitative Methods by Pyramid Level and Planning Cycle

Table 16-1 Ethical Frameworks and Principles for Planning Health Programs

Table 16-2 Comparison of Types of IRB Reviews

Table 16-3 Eight Elements of Informed Consent, as Required in 45 CFR 46

Table 16-4 Effect of Rigor and Importance of Claims on Decision Making

Table 16-5 List of Ways to Make Graphs More Interpretable

Table 16-6 Examples of Dissemination Modes, Audiences, and Purposes

xvii

© Lynne Nicholson/Shutterstock

Exhibit 8-3 Break-Even Table Shows Number of Paying Students Needed to Break Even

Exhibit 8-4 Example of a Budget Showing Year-to-Date Variance

Exhibit 8-5 Types of Cost Analyses

Exhibit 9-1 Formulas for Measures of Coverage

Exhibit 9-2 Example of Narrative Background about Coverage and Dosage Measures

Exhibit 9-3 Examples of Coverage Measures Using an Excel Spreadsheet

Exhibit 9-4 Examples of Calculating Dosage for the Congenital Anomalies Prevention Program Using Excel

List of Exhibits Exhibit 2-1 Checklist to Facilitate Development

of Cultural and Linguistic Competence Within Healthcare Organizations

Exhibit 2-2 Checklist to Facilitate Cultural Competence in Community Engagement

Exhibit 7-1 Example of an Abbreviated Time Line for a Short-Term Health Program

Exhibit 7-2 Chapter Text Paragraph Rewritten at an Eighth-Grade Reading Level

Exhibit 8-1 Example of a Scenario Needing a Break-Even Analysis

Exhibit 8-2 Example of a Budget Used for a Break-Even Analysis for Bright Light on an Excel Spreadsheet

xix

© Lynne Nicholson/Shutterstock

the text are relevant to health administrators, medical social workers, nurses, nutritionists, pharmacists, public health professionals, physical and occupational therapists, and physicians.

This textbook grew from teaching experi- ences with both nurses and public health students and their need for direct application of the pro- gram planning and evaluation course content to their work and to their clients and communities. Today programs need to be provided through community-based healthcare settings to address broad public health issues and expand the individ- ual to population focus. The distinction between individual patient health and population health is a prerequisite for the thinking and planning—in terms of aggregates and full populations—by students from clinical backgrounds.

In most graduate health professions programs, students take a research methods course and a statistics course. Therefore, this evaluation text avoids duplicating content related to research methods and statistics while addressing and extending that content into health program de- velopment, implementation, and evaluation. In addition, because total quality management and related methodologies are widely used in healthcare organizations, areas of overlap between quality improvement methodologies and traditional program evaluation approaches are discussed. This includes ways that quality improvement methodologies complement program evaluations. Sometimes evaluations are appropriate; sometimes they are not. Enthusiasm for providing health programs and performing evaluation is tempered with thoughtful notes of caution in the hope that students will avoid potentially serious and costly program and evaluation mistakes.

Preface to the Fourth Edition The fourth edition of Health Program Planning and Evaluation has stayed true to the purpose and intent of the previous editions. This advanced- level text is written to address the needs of professionals from diverse health disciplines who find themselves responsible for developing, implementing, or evaluating health programs. The aim of the text is to assist health profes- sionals to become not only competent health program planners and evaluators but also savvy consumers of evaluation reports and prudent users of evaluation consultants. To that end, the text includes a variety of practical tools and concepts necessary to develop and evaluate health programs, presenting them in language understandable to both the practicing and novice health program planner and evaluator.

Health programs are conceptualized as encompassing a broad range of programmatic interventions that span the social-ecological range, from individual-level to population-level programs. Examples of programs cited through- out the text are specific yet broadly related to improving health and reflect the breadth of public health programs. The examples have been updated once again to reflect current best practices. Maintaining a public health focus provides an opportunity to demonstrate how health programs can target different levels of a population, different determinants of a health problem, and different strategies and interven- tions to address a health problem. In addition, examples of health programs and references are selected to pique the interests of the diverse students and practicing professionals who con- stitute multidisciplinary program teams. Thus, the content and examples presented throughout

xx Preface to the Fourth Edition

Articulating each of the component elements of the program theory sharpens the student’s awareness of what must be addressed to create an effective health program. One element of the program theory is the effect theory, which focuses on how the intervention results in the program effects. The effect theory had its genesis in the concepts of action and intervention hypotheses described by Rossi and Freeman; those concepts were dropped from later editions of their text. We believe these authors were onto something with their effort to elucidate the various path- ways leading from a problem to an effect of the program. Rossi and colleagues’ ideas have been updated with the language of moderating and mediating factors and an emphasis on the intervention mechanisms.

Throughout the current edition of this textbook, emphasis is given to the effect theory portion of the program theory. The effect theory describes relationships among health antecedents, causes of health problems, program interventions, and health effects. The hypotheses that comprise the effect theory need to be understood and ex- plicated to plan a successful health program and to evaluate the “right” elements of the program. The usefulness of the effect theory throughout the planning and evaluation cycle is highlighted throughout this text; for example, the model is used as a means of linking program theory to evaluation designs and data collection. The model becomes an educational tool by serving as an example of how the program theory is manifested throughout the stages of planning and evaluation, and by reinforcing the value of carefully articulating the causes of health problems and consequences of programmatic interventions. Students and novice program planners may have an intuitive sense of the connection between their actions and outcomes, but they may not know how to articulate those connections in ways that program stakeholders can readily grasp. The effect theory and the process theory—the other main element of the program theory—provide a basis from which to identify and describe these connections.

▸ Unique Features The Fourth Edition has retained the three unique features that distinguish this text from other program planning and evaluation textbooks: use of the public health pyramid, consistent use of a model of the program theory throughout the text, and role modeling of evidence-based practice.

The public health pyramid explains how health programs can be developed for individu- als, aggregates, populations, and service delivery systems. Use of the pyramid is also intended as a practical application of the social-ecological per- spective that acknowledges a multilevel approach to addressing health problems. The public health pyramid contains four levels: direct services to individuals; enabling services to aggregates; services provided to entire populations; and, at the base, infrastructure. In this textbook, the pyramid is used as an organizing structure to summarize the content of each chapter in the “Across the Pyramid” sections. In these sections, specific attention is paid to how key concepts in a given chapter might vary across each pyramid level. Summarizing the chapter content in this manner reinforces the perspective that enhancing health and well-being requires integrated efforts across the levels of the public health pyramid. Health program development and evaluation is relevant for programs intended for individuals, aggregates, populations, and service delivery systems, and this fact reinforces the need to tailor program plans and evaluation designs to the level at which the program is conceptualized. Using the pyramid also helps health professionals begin to value their own and others’ contributions within and across the levels and to transcend disciplinary boundaries.

The second unique feature of this text is that one conceptual model of program planning and evaluation is used throughout the text: the program theory. The program theory is like a curricular strand, connecting content across the chapters, and activities throughout the planning and evaluation cycle. The program theory, as a conceptual model, is composed of elements.

xxiPreface to the Fourth Edition

The third unique feature of this text is the intentional role modeling of evidence-based practice. Use of published, empirical evidence as the basis for practice—whether clinical practice or program planning practice—is the professional standard. Each chapter of this book contains substantive examples drawn from the published scientific health and health-related literature. Relying on the literature for examples of programs, evaluations, and issues is consistent with the espoused preference of using scientific evidence as the basis for making programmatic decisions. Each chapter offers multiple examples from the health sciences literature that substan- tiate the information presented in the chapter.

▸ Organization of the Book

The book is organized into six sections, each covering a major phase in the planning and eval- uation cycle. Chapter 1 introduces the fictitious city of Layetteville and the equally fictitious Bowe County. In subsequent chapters, chapter content is applied to the health problems of Layetteville and Bowe County so that students can learn how to use the material on an ongoing basis. In several chapters, the case study is used in the “Discussion Questions and Activities” section to provide students with an opportunity to practice applying the chapter content. In recognition of the availability of parts of the text in digital format, each use of the Layetteville case stands on its own in reference to the chapter’s content.

Section I explores the context in which health programs and evaluations occur. Chap- ter 1 begins with an overview of definitions of health, followed by a historical context. The public health pyramid is introduced and pre- sented as an ecological framework for thinking of health programs. An overview of community is provided and discussed as both the target and the context of health programs. The role of community members in health programs and

evaluations is introduced, and emphasis is given to community as a context and to strategies for community participation throughout the program development and evaluation process. Chapter 2 focuses on the role of diversity in the planning and evaluation cycle and its effects on the delivery and evaluation of health programs. Although a discussion of diversity-related issues could have been added to each chapter, the sensitive nature of this topic and its importance in ensuring a successful health program warranted it being covered early in the text and as a separate chapter. Cultural competence is discussed, particularly with regard to the organization providing the health program and with regard to the program staff members.

Section II contains two chapters that focus on the task of defining the health problem. Chapter 3 covers planning perspectives and the history of health program planning. Effective health program developers understand that approaches to planning are based on assump- tions. These assumptions are exemplified in six perspectives that provide points of reference for understanding diverse preferences for prioritizing health needs and expenditures and therefore for tailoring planning actions to fit the situation best. Chapter 3 also reviews perspectives on conducting a community needs assessment as foundational to decision making about the future health program. Essential steps involved in conducting a community health and needs assessment are outlined as well.

Chapter 4 expands on key elements of a community needs assessment, beginning with a review of the data collection methods appro- priate for a community needs assessment. This discussion is followed by a brief overview of key epidemiological statistics. Using those statistics and the data, the reader is guided through the process of developing a causal statement of the health problem. This causal statement, which includes the notion of moderating and mediating factors in the pathway from causes to problem, serves as the basis for the effect theory of the program. Once the causal statement has been

xxii Preface to the Fourth Edition

developed, prioritization of the problem is needed; four systems for prioritizing in a rational manner are reviewed in Chapter 4.

Following prioritization comes planning, beginning with the decision of how to address the health problem. In many ways, the two chapters in Section III form the heart of planning a successful health program. Unfortunately, students generally undervalue the importance of theory for selecting an effective intervention and of establishing target values for objectives. Chapter 5 explains what theory is and how it provides a cornerstone for programs and evaluations. More important, the concept of intervention is discussed in detail, with attention given to characteristics that make an intervention ideal, including attention to intervention dosage. Program theory is introduced in Chapter 5 as the basis for organizing ideas related to the selection and delivery of the interventions in conjunction. The effect theory element of the program theory is introduced and the components of the effect theory are explained. Because the effect theory is so central to having an effective program interven- tion and the subsequent program evaluation, it is discussed in conjunction with several examples from the Layetteville and Bowe County case.

Chapter 6 goes into detail about developing goals and objectives for the program, with particular attention devoted to articulating the interven- tions provided by the program. A step-by-step procedure is presented for deriving numerical targets for the objectives from existing data, which makes the numerical targets more defendable and programmatically realistic. We focus on distinguishing between process objectives and outcome objectives through the introduction of two mnemonics: TAAPS (Time frame, Amount of what Activities done by which Participants/ program Staff ) and TREW (Timeframe, what portion of Recipients experience what Extent of Which type of change).

Section IV deals with the task of imple- menting a health program. Chapter 7 provides an in-depth review of key elements that consti- tute the process theory element of the program theory—specifically, the organizational plan and services utilization plan. The distinction between

inputs and outputs of the process theory is high- lighted through examples and a comprehensive review of possible inputs and outputs. Budgeting for program operations is covered in this chapter as well. Chapter 8 is devoted entirely to fiscal data systems, including key aspects of budgeting, and informatics. Chapter 9 details how to evaluate the outputs of the organizational plan and the services utilization plan. The practical application of mea- sures of coverage is described, along with the need to connect the results of the process evaluation to programmatic changes. Program management for assuring a high-quality program that delivers the planned intervention is the focus of Chapter 10.

Section V contains chapters that are specific to conducting the effect evaluations. These chap- ters present both basic and advanced research methods from the perspective of a program effect evaluation. Here, students’ prior knowledge about research methods and statistics is brought together in the context of health program and services evaluation. Chapter 11 highlights the importance of refining the evaluation question and provides information on how to clarify the question with stakeholders. Earlier discussions about program theory are brought to bear on the development of the evaluation question. Key issues, such as data integrity and survey construction, are addressed with regard to the practicality of program evaluation. Chapter 12 takes a fresh approach to evaluation design by organizing the traditional experimental and quasi-experimental designs and epidemiological designs into three levels of program evaluation design based on the design complexity and purpose of the evaluation. The discussion of sampling in Chapter 13 retains the emphasis on practicality for program evalua- tion rather than taking a pure research approach. However, sample size and power are discussed because these factors have profound relevance to program evaluation. Chapter 14 reviews sta- tistical analysis of data, with special attention to variables from the effect theory and their level of measurement. The data analysis is linked to interpretation, and students are warned about potential flaws in how numbers are understood. Chapter 15 provides a review of qualitative designs

xxiiiPreface to the Fourth Edition

and methods, especially their use in health pro- gram development and evaluation.

The final section, Section VI, includes just one chapter. Chapter 16 discusses the use of evaluation results when making decisions about existing and future health programs. Practical and conceptual issues related to the ethics issues that program evaluators face are addressed. This chapter also reviews ways to assess the quality of evaluations and the professional responsibilities of evaluators.

Each chapter in the book concludes with a “Discussion Questions and Activities” section. The questions posed are intended to be provoc- ative and to generate critical thinking. At the graduate level, students need to be encouraged to engage in independent thinking and to foster their ability to provide rationales for decisions. The discussion questions are developed from this point of view. In the “Internet Resources” section, links are provided to websites that support the content of the chapter. These websites have been carefully chosen as stable and reliable sources.

▸ Additions to and Revisions in the Fourth Edition

The fourth edition of Health Program Planning and Evaluation represents continuous improve- ment, with corrections and updated references. Classical references and references that remain state of the art have been retained.

The Fourth Edition has retained the original intent—namely, to provide students with the ability to describe a working theory of how the intervention acts upon the causes of the health problem and leads to the desired health results. Some content has been condensed in order to allow enough room to describe current evaluation approaches adequately for both new and experi- enced practitioners. For instance, Chapter 1 now includes participatory evaluations in addition to outcome- and utilization-focused evaluations. In addition to disciplines traditionally recognized

in western medical care, Chapter 2 now includes acupuncture and massage therapy as examples of health professional diversity. Discussion of the nuances of cultural competency has been refined, in light of the continuing importance and challenges of this area. Community strengths have been given more attention in Chapter 3 in recognition of the powerful potential of shifting from a “deficit-based” to an “asset-based” perspec- tive on health planning. Chapter 4 now devotes greater attention to the health evaluation poten- tial of data from social media such as Facebook and Twitter, as well as geospatial data, including attendant concerns about privacy, and also notes implications of the increasingly prevalent public rankings of community health status. Examples of infrastructure-level interventions within the public health pyramid have been added in Chapter 5. Discussion of financial modeling options in Chapter 8 now includes simulation modeling, an exciting if also resource-intensive option to conducting real-world experiments, which are, of course, inevitably expensive themselves. Chapters 9 and 15 include emerging data collection techniques such as participant self-reports, video, photos, and audio recordings that may make public health evaluation more inclusive of the people such interventions seek to serve. Chap- ter 13 includes updates on surveying, reflecting the decreased numbers of people with land-line phones, long a mainstay of health evaluations. Options for online surveying have been updated in Chapter 14; given the rapid evolution of big data such as those available from social media, billing, and medical records, discussion of this topic has been updated in Chapter 13 as well. Finally, Chapter 16 now includes bioethics— the application of ethical and philosophical principles to medical decision making—as an increasingly salient component of responsible health evaluation.

In sum, we have worked hard to sustain this book’s conceptual and empirical rigor and currency in the Fourth Edition while maintaining accessibility for a range of health evaluators. Above all, we hope this book is useful to our readers’ vitally important efforts to improve health.

xxv

© Lynne Nicholson/Shutterstock

Acknowledgments We are indebted to the many people who supported and aided us in preparing this fourth edition of Health Program Planning and Evaluation: A Practical, Systematic Approach for Community Health. We remain grateful to the numerous students over the years who asked questions that revealed the typical sticking points in their acquiring and understanding of the concepts and content, as well as where new explanations were needed. Through their eyes we have learned there is no one way to explain a complex notion or process. Their interest and enthusiasm for planning and evaluating health programs was a great motivator for writing this book.

Several additional colleagues helped fine-tune this text. We are especially indebted to Arden

Handler at the School of Public Health, University of Illinois at Chicago, for taking time to contribute to this textbook. Her devotion to quality and clarity has added much to the richness of otherwise dry material. We remain deeply indebted to Deborah Rosenberg, also at the School of Public Health University of Illinois at Chicago, for sharing her innovative and quintessentially useful work on developing targets for program objectives. Special thanks as well to Joseph Chen, at the University of Texas School of Public Health, for his many contributions to updating the literature cited across many chapters and for his contribution on big data. Last, but not least, thanks to Mike Brown, publisher at Jones & Bartlett Learning, for his encouragement and patience over the years.

xxvii

© Lynne Nicholson/Shutterstock

DHHS U.S. Department of Health and Human Services

DSM-5 Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition

EBM Evidence-based medicine

EBP Evidence-based practice

EHR Electronic health record

EMR Electronic medical record

FTE Full-time equivalent

GAO U.S. Government Accountability Office

GNP Gross Product

GPRA Government Performance and Results Act

HEDIS Healthcare Effectiveness Data and Information Set

HIPAA Health Insurance Portability and Accountability Act

HIT Health information technology

HMOs Health maintenance organizations

HRQOL Health-related quality of life

HRSA Health Resources and Services Administration (part of DHHS)

i-APP Innovation–Adolescent Preventing Pregnancy (Program)

ICC Intraclass correlation

IRB Institutional review board

JCAHO Joint Commission on the Accreditation of Healthcare Organizations

MAPP Mobilizing for Action through Planning and Partnership

MBO Management by objectives

List of Acronyms ABCD Asset-based community

development

ACA Affordable Care Act

AEA American Evaluation Association

AHRQ Agency for Healthcare Research and Quality

ANOVA Analysis of variance

APHA American Public Health Association

BPRS Basic priority rating system

BRFSS Behavioral Risk Factor Surveillance System

BSC Balanced Score Card

CAHPS Consumer Assessment of Health Plans

CARF Commission on Accreditation of Rehabilitation Facilities

CAST-5 Capacity Assessment of Title-V

CBA Cost–benefit analysis

CBPR Community-based participatory research

CDC Centers for Disease Control and Prevention

CEA Cost-effectiveness analysis

CER Cost-effectiveness ratio

CFIR Consolidated Framework for Implementation Research

CFR Code of Federal Regulations

CHIP Community Health Improvement Process

CI Confidence interval

CPT Current Procedural Terminology

CQI Continuous quality improvement

CUA Cost–utility analysis

DALY Disability-adjusted life-year

xxviii List of Acronyms

PSA Public service announcement

QALY Quality-adjusted life-year

RAR Rapid assessment and response

RARE Rapid assessment and response and evaluation

RE-AIM Reach, Effectiveness, Adoption, Implementation, and Maintenance model

RR Relative risk

SAMHSA Substance Abuse and Mental Health Services Administration

SCHIP State Child Health Insurance Program

SES Socioeconomic status

SMART Specific, measurable, achievable, realistic, and time (objective)

TAAPS Time frame, Amount of what Activities done by which Participants/program Staff

TQM Total quality management

TREW Time frame, what portion of Recipients experience what Extent of Which type of change

UOS Units of service

WHO World Health Organization

WIC Special Supplemental Nutrition Program for Women, Infants, and Children

YHL Years of healthy life

YLL Years of life lost

YPLL Years of potential life lost

MCHB Maternal and Child Health Bureau (part of HRSA)

NACCHO National Association of City and County Health Officers

NAMI National Alliance on Mental Illness

NCHS National Center for Health Statistics

NCQA National Commission on Quality Assurance

NFPS National Family Planning Survey

NHANES National Health and Nutrition Examination Survey

NHIS National Health Interview Survey

NIH National Institutes of Health

NPHPS National Public Health Performance Standards

OHRP Office for Human Research Protections

OMB Office of Management and Budgeting

OR Odds ratio

PACE-EH Protocol for Assessing Excellence in Environmental Health

PAHO Pan American Health Organization

PDCA Plan-Do-Check-Act

PEARL Property, economic, acceptability, resource, legality system

PERT Program Evaluation and Review Technique

PPIP Putting Prevention into Prevention

PRECEDE Predisposing, Reinforcing, and Enabling Factors in Community Education Development and Evaluation (model)

SECTION I

The Context of Health Program Development

© Lynne Nicholson/Shutterstock

3

© Lynne Nicholson/Shutterstock

Context of Health Program Development and Evaluation

Health is not a state of being that can easily be achieved through isolated, uninformed, individualistic actions. Health of individ- uals, of families, and of populations is a state in which physical, mental, and social well-being are integrated to enable optimal functioning. From this perspective, achieving and maintaining health across a life span is a complex, complicated, intri- cate affair. For some, health is present irrespective of any special efforts or intention. For most of us, health requires, at a minimum, some level of attention and specific information. It is through health programs that attention is given focus and information is provided or made available, but that does not guarantee that the attention and information are translated into actions or behaviors needed to achieve health. Thus, those providing health programs, however large or small, need to understand both the processes whereby those in need of attention and health information can receive what is needed, and also the processes by which to learn from the experience of providing the health program.

The processes and effects of health pro- gram planning and evaluation are the subjects of this text. The discussion begins here with a brief overview of the historical context. This background sets the stage for appreciating the considerable number of publications on the topic of health program planning and evaluation, and for acknowledging the professionalization of evaluators. The use of the term processes to describe the actions involved in health program planning and evaluation is intended to denote action, cycles, and open-endedness. This chapter introduces the planning and evaluation cycle, and the interactions and iterative nature of this cycle are stressed throughout the text. Because health is an individual, aggregate, and population phenomenon, health programs need to be conceptualized across those levels. The public health pyramid, introduced in this chapter, is used throughout the text as a tool for conceptualizing and actualizing health programs for individuals, aggregates, and populations.

CHAPTER 1

4 Chapter 1 Context of Health Program Development and Evaluation

▸ History and Context An appropriate starting point for this text is reflecting on and understanding what “health” is, along with having a basic appreciation for the genesis of the fields of health program planning and evaluation. A foundation in these elements is key to becoming an evaluation professional.

Concept of Health To begin the health program planning and evaluation cycle requires first reflecting on the meaning of health. Both explicit and implicit meanings of health can dramatically influence what is considered the health problem and the subsequent direction of a program. The most widely accepted definition of health is that put forth by the World Health Organization (WHO), which for the first time defined health as more than the absence of illness and as the presence of well-being (WHO, 1947).

Since the publication of the WHO defini- tion, health has come to be viewed across the health professions as a holistic concept that encompasses the presence of physical, mental, developmental, social, and financial capabil- ities, assets, and balance. This idea does not preclude each health profession from having a particular aspect of health to which it primarily contributes. For example, a dentist contributes primarily to a patient’s oral health, knowing that the state of the patient’s teeth and gums has a direct relationship to his or her physical and social health. Thus the dentist might say that the health problem is caries. The term health problem is used, rather than illness, diagnosis, or pathology, in keeping with the holistic view that there can be problems, deficits, and pathologies in one component of health while the other components remain “healthy.” Using the term health problem also makes it easier to think about and plan health programs for aggregates of individuals. A community, a family, and a school can each have a health problem that is the focus of a health program intervention. The extent to which the health program planners have

a shared definition of health and have defined the scope of that definition influences the nature of the health program.

Health is a matter of concern for more than just health professionals. For many Americans, the concept of health is perceived as a right, along with civil rights and liberties. The right to health is often translated by the public and politicians into the perceived right to have or to access health care. This political aspect of health is the genesis of health policy at the local, federal, and international levels. The extent to which the political nature of health underlies the health problem of concern being programmatically addressed also influences the final nature of the health program.

Health Programs, Projects, and Services What distinguishes a program from a project or from a service can be difficult to explain, given the fluidity of language and terms. The term program is fairly generic but generally connotes a structured effort to provide a specific set of services or interventions. In contrast, a project often refers to a time-limited or experimental effort to provide a specific set of services or interventions through an organizational struc- ture. In the abstract, a service can be difficult to define but generally includes interaction between provider and client, an intangibility aspect to what is provided, and a nonpermanence or transitory nature to what is provided. Using this definition of service, it is easy to see that what is provided in a health program qualifies as a service, although it may not be a health service.

A health program is a totality of an organized structure designed for the provision of a fairly discrete health-focused intervention, where that intervention is designed for a specific target audience. By comparison, health services are the organizational structures through which providers interact with clients or patients to meet the needs or address the health problems of the clients or patients. Health programs, particularly

History and Context 5

in public health, tend to provide educational services, have a prevention focus, and deliver services that are aggregate or population-focused. In contrast, health services exist exclusively as direct services. Recognizing the distinction between health programs and health services is important for understanding the corresponding unique planning and evaluation needs of each.

History of Health Program Planning and Evaluation The history of planning health programs has a different lineage than that of program evaluation. Only relatively recently, in historical terms, have these lineages begun to overlap, with resulting synergies. Planning for health programs has the older history, if public health is consid- ered. Rosen (1993) argued that public health planning began approximately 4,000 years ago with planned cities in the Indus Valley that had covered sewers. Particularly since the Industrial Revolution, planning for the health of populations has progressed, and it is now considered a key characteristic of the discipline of public health.

Blum (1981) related planning to efforts undertaken on behalf of the public well-being to achieve deliberate or intended social change as well as providing a sense of direction and alternative modes of proceeding to influence social attitudes and actions. Others (Dever, 1980; Rohrer, 1996; Turnock, 2004) have similarly defined planning as an intentional effort to create something that has not occurred previously for the betterment of others and for the purpose of meeting desired goals. The purpose of planning is to ensure that a program has the best possible likelihood of being successful, defined in terms of being effective with the least possible resources. Planning encompasses a variety of activities undertaken to meet this purpose.

The quintessential example of planning is the development and use of the Healthy People goals. In 1979, Healthy People (U.S. Department of Health, Education, and Welfare [DHEW], 1979) was published as an outgrowth of the

need to establish an illness prevention agenda for the United States. The companion publica- tion, Promoting Health/Preventing Disease (U.S. Department of Health and Human Services [DHHS], 1980), marked the first time that goals and objectives regarding specific areas of the nation’s health were made explicit, with the expectation that these goals would be met by the year 1990. Healthy People became the framework for the development of state and local health promotion and disease prevention agendas. Since its initial publication, the U.S. goals for national health have been revised and published as Healthy People 2000 (DHHS, 1991), Healthy Communities 2000 (American Public Health Association [APHA], 1991), Healthy People 2010 (DHHS, 2000), and Healthy People 2020 (DHHS, 2011), with development of Healthy People 2030 underway. Other nations also set health status goals and international organizations, such as the World Health Organization (WHO) and Pan American Health Organization (PAHO), develop health goals applicable across nations.

The evolution of Healthy People goals also reflects the accelerating rate of emphasis on nationwide coordination of health promotion and disease prevention efforts and a reliance on sys- tematic planning to achieve this coordination. The development of the Healthy People publications also reflects the underlying assumption that planning is a rational activity that can lead to results. However, at the end of each 10-year cycle, many of the U.S. health objectives were not achieved, reflecting the potential for planning to fail. Given this failure potential, this text emphasizes techniques to help future planners of health programs to be more realistic in setting goals and less dependent upon a linear, rational approach to planning.

The Healthy People 1990 objectives were developed by academics and clinician experts in illness prevention and health promotion. In contrast, development of the goals and health problems listed in Healthy People 2010 and Healthy People 2020 incorporated ideas generated at public forums and through Internet commen- tary; these ideas later were revised and refined by expert panels before final publication of the

6 Chapter 1 Context of Health Program Development and Evaluation

as the basis for evaluation. Second-generation evaluations were predominantly descriptive. With the introduction in the 1960s of broad innovation and initiation of federal social service programs, including Medicare, Medicaid, and Head Start, the focus of evaluations shifted to establishing the merit and value of the programs. Because of the political issues surrounding these and similar federal programs, determining whether the social policies were having any effect on people become a priority. Programs needed to be judged on their merits and effectiveness. The U.S. General Accounting Office (GAO; now called the Government Accountability Office) had been established in 1921 for the purpose of studying the utilization of public finances, assist- ing Congress in decision making with regard to policy and funding, and evaluating government programs. The second-generation evaluation emphasis on quantifying effects was spurred, in part, by reports from the GAO that were based on the evaluations of federal programs.

Typically, the results of evaluations were not used in the “early” days of evaluating education and social programs. That is, federal health policy was not driven by whether evaluations showed the programs to be successful. Although the scientific rigor of evaluations improved, their usefulness remained minimal. Beginning in the 1980s, however, the third generation of evaluations—termed “the negotiation generation” or “the responsiveness generation”—began. During this generation, evaluators began to acknowledge that they were not autonomous and that their work needed to respond to the needs of those being evaluated. As a result of this awareness, several lineages have emerged. These lineages within the responsiveness generation account for the current diversity in types, emphases, and philosophies related to program evaluation.

One lineage is utilization-focused evaluation (Patton, 2012), in which the evaluator’s primary concern is with developing an evaluation that will be used by the stakeholders. Utilization-focused evaluations are built on the following premises (Patton, 1987): Concern for use of the evaluation pervades the evaluation from beginning to end;

objectives. Greater participation of the public during the planning stage of health programs has become the norm. In keeping with the emphasis on participation, the role and involvement of stakeholders are stressed at each stage of the planning and evaluation cycle.

The history of evaluation, from which the evaluation of health programs grew, is far shorter than the history of planning, beginning roughly in the early 1900s, but it is equally rich in important lessons for future health program evaluators. The first evaluations were done in the field of education, particularly as student assessment and evaluation of teaching strategies gained interest (Patton, 2008). Assessment of student scholastic achievement is a comparatively circumscribed outcome of an educational intervention. For this reason, early program evaluators came from the discipline of education, and it was from the fields of education and educational psychology that many methodological advances were made and statistics developed.

Guba and Lincoln (1987) summarized the history of evaluations by proposing generational milestones or characteristics that typify distinct generations. Later, Swenson (1991) built on their concept of generations by acknowledging that subsequent generations of evaluations will occur. Each generation incorporates the knowledge of early evaluations and extends that knowledge based on current broad cultural and political trends.

Guba and Lincoln (1987) called the first generation of evaluations in the early 1900s “the technical generation.” During this time, nascent scientific management, statistics, and research methodologies were used to test interventions. Currently, evaluations continue to incorporate the rationality of this generation by using activities that are systematic, science based, logical, and sequential. Rational approaches to evaluations focus on identifying the best-known interven- tion or strategy given the current knowledge, measuring quantifiable outcomes experienced by program participants, and deducing the degree of effect from the program.

The second generation, which lasted until the 1960s, focused on using goals and objectives

History and Context 7

evaluations done across similar programs. This trend in program evaluation parallels the trend in social science toward using meta-analysis of existing studies to better understand theorized relationships and the trend across the health professions toward establishing evidence-based practice guidelines. This new generation be- came possible because of a pervasive culture of evaluation in the health services and because of the availability of huge data sets for use in the meta-evaluations. An early example of the evaluation culture was the mandate from United Way, a major funder of community-based health programs, for their grantees to conduct outcome evaluations. To help grantees meet this mandate, United Way published a user-friendly manual (United Way of America, 1996) that could be used by nonprofessionals in the development of basic program evaluations. More broadly, the culture of evaluation can be seen in the explicit requirement of federal agencies that fund community-based health programs that

evaluations are aimed at the interests and needs of the users; users of the evaluation must be in- vested in the decisions regarding the evaluation; and a variety of community, organizational, political, resource, and scientific factors affect the utilization of evaluations. Utilization-focused evaluation differs from evaluations that are focused exclusively on outcomes

Another lineage is participatory evaluation (Whitmore, 1998), in which the evaluation is merely guided by the expert and is actually gen- erated by and conducted by those invested in the health problem. A participatory or empowerment approach invites a wide range of stakeholders into the activity of planning and evaluation, providing those participants with the skills and knowledge to contribute substantively to the activities and fostering their sense of ownership of the product (TABLE 1-1).

The fourth generation of evaluation, which emerged in the mid-1990s, seems to be meta-evaluation, that is, the evaluation of

TABLE 1-1 Comparison of Outcome-Focused, Utilization-Focused, and Participatory Focused Evaluations

Outcome-Focused Evaluations

Utilization-Focused Evaluations

Participatory Focused

Evaluations

Purpose Show program effect Get stakeholders to use evaluation-findings for decisions regarding program improvements and future program development

Involve the stakeholders in designing programs and evaluations, and utilizing findings

Audience Funders, researchers, other external audience

Program people (internal audience), funders

Those directly concerned with the health problem and program

Method Research methods, external evaluators (usually)

Research methods, participatory

Research methods as implemented by the stakeholders

8 Chapter 1 Context of Health Program Development and Evaluation

serves evaluators primarily in the United States. Several counterparts to the AEA exist, such as the Society for Evaluation in the United King- dom and the Australian Evaluation Society. The establishment of these professional orga- nizations, whose members are evaluators, and the presence of health-related sections within these organizations demonstrate the existence of a field of expertise and of specialized knowl- edge regarding the evaluation of health-related programs.

As the field of evaluation has evolved, so have the number and diversity of approaches that can guide the development of evaluations. Currently, 23 different approaches to evaluation have been identified, falling into 3 major groups (Stufflebeam  & Coryn, 2014). One group of evaluations is oriented toward questions and methods such as objectives-based studies and experimental evaluations. The second group of evaluations is oriented toward improvements and accountability and includes consumer-oriented and accreditation approaches. The third group of evaluations includes those that have a social agenda or advocacy approach, such as respon- sive evaluations, democratic evaluations, and utilization-focused evaluation. They also acknowl- edge pseudo-evaluations and quasi-evaluations as distinct groups, reflecting the continuing evolution of the field of evaluation.

Several concepts are common across the types of evaluations—namely, pluralism of values, stakeholder constructions, fairness and equity regarding stakeholders, the merit and worth of the evaluation, a negotiated process and outcomes, and full collaboration. These concepts have been formalized into the standards for evaluations that were established by the Joint Commission on Standards for Educational Evaluation in 1975 (American Evaluation Association, 2011). Currently, this Joint Commission includes many organizations in its membership, such as the American Evaluation Association and the American Educational Research Association.

The five standards of evaluation established by the American Evaluation Association are utility, feasibility, propriety, accuracy, and evaluation

such programs include evaluations conducted by local evaluators.

Most people have an intuitive sense of what evaluation is. The purpose of evaluation can be to measure the effects of a program against the goals set for it and thus to contribute to subsequent decision making about the program (Weiss, 1972). Alternatively, evaluation can be defined as “the use of social research methods to systematically investigate the effectiveness of social intervention programs in ways that are adapted to their political and organizational environments and are designed to inform social action to improve social conditions” (Rossi, Lipsey, & Freeman, 2004 , p. 16). Others (Herman, Morris, & Fitz-Gibbon, 1987) have defined evaluation as judging how well policies and procedures are working or as assessing the quality of a program. These definitions of evaluation all remain relevant.

Inherently these definitions of evaluation have an element of being judged against some criteria. This implicit understanding of evaluation leads those involved with the health program to feel as though they will be judged or found not to meet those criteria and will subsequently experience some form of repercussions. They may fear that they as individuals or as a program will be labeled a failure, unsuccessful, or inadequate. Such feel- ings must be acknowledged and addressed early in the planning cycle. Throughout the planning and evaluation cycle, program planners have numerous opportunities to engage and involve program staff and stakeholders in the evaluation process. Taking advantage of these opportuni- ties goes a long way in alleviating the concerns of program staff and stakeholders about the judgmental quality of the program evaluation.

▸ Evaluation as a Profession

A major development in the field of evaluation has been the professionalization of evaluators. The American Evaluation Association (AEA)

Evaluation as a Profession 9

and values held by professional evaluators and deserve attention in health program evaluations. The existence and acceptance of standards truly indicates the professionalism of evaluators.

Achieving these standards requires that those involved in the program planning and evaluation have experience in at least one aspect of planning or evaluation, whether that is experience with the health problem; experience with epidemiological, social, or behavioral science research methods; or skill in facilitating processes that involve diverse constituents, capabilities, and interests. Program planning and evaluation can be done in innumerable ways, with no single “right way.” This degree of freedom and flexibility can feel uncomfortable for some people. As with any skill or activity, until they have experience, program planners and evaluators may feel intimidated by the size of the task or by the experience of others involved. To become a professional evaluator, therefore, requires a degree of willingness to learn, to grow, and to be flexible.

accountability (TABLE 1-2; American Evaluation Association, 2011).

The utility standard specifies that an evalu- ation must be useful to those who requested the evaluation. A useful evaluation shows ways to make improvements to the intervention, increase the efficiency of the program, or enhance the possibility of garnering financial support for the program. The feasibility standard denotes that the ideal may not be practical. Evaluations that are highly complex or costly will not be done by small programs with limited capabili- ties and resources. Propriety is the ethical and politically correct component of the standards. Evaluations can invade privacy or be harmful to either program participants or program staff members. The propriety standard also holds evaluators accountable for upholding all of the other standards. Accuracy is essential and is achieved through the elements that constitute scientific rigor. These established and accepted standards for evaluations reflect current norms

TABLE 1-2 Evaluation Standards Established by the Joint Commission on Standards for Educational Evaluation

Standard Description

Utility To increase the extent to which program stakeholders find evaluation processes and products valuable in meeting their needs.

Feasibility To increase evaluation effectiveness and efficiency.

Propriety To support what is proper, fair, legal, right, and just in evaluations.

Accuracy To increase the dependability and truthfulness of evaluation representations, propositions, and findings, especially those that support interpretations and judgments about quality.

Evaluation accountability

To encourage adequate documentation of evaluations and a meta-evaluative perspective focused on improvement and accountability for evaluation processes and products.

Data from American Evaluation Association (2012).

10 Chapter 1 Context of Health Program Development and Evaluation

organizations and public health agencies can be integral to achieving well-functioning programs.

External evaluators can bring a fresh perspective and a way of thinking that gener- ates alternatives not currently in the agencies’ repertoire of approaches to the health problem and program evaluation. Compared to internal evaluators, external evaluators are less likely to be biased in favor of one approach—unless, of course, they were chosen for their expertise in a particular area, which would naturally bias their perspective to some extent. External pro- gram planners and evaluators, however, can be expensive consultants. Some organizations that specialize in health program evaluations serve as one category of external evaluator. These research firms receive contracts to evaluate health program initiatives and conduct national evaluations that require sophisticated method- ology and considerable resources.

The question of who does evaluations also can be answered by looking at who funds health program evaluations. From this perspective, org- anizations that do evaluations as a component of their business are the answer to the question, Who does evaluations? Although most funding agencies prefer to fund health programs rather than stand-alone program evaluations, some exceptions exist. For example, the Agency for Healthcare Research and Quality (AHRQ) funds health services research about the quality of medical care, which is essentially effect evalu- ation research. Other federal agencies, such as the National Institutes of Health and the bureaus within the Department of Health and Human Services, fund evaluation research of pilot health programs. However, the funding priorities of these federal agencies change to be consistent with federal health policy. This is a reminder that organizations funding and conducting health program evaluations evolve over time.

Roles of Evaluators Evaluators may be required to take on various roles, given that they are professionals involved in a process

Who Does Planning and Evaluations? Many different types of health professionals and social scientists can be involved in health program planning and evaluation. At the out- set of program planning and evaluation, some trepidation revolves around who ought to be the planners and evaluators. In a sense, almost anyone with an interest and a willingness to be an active participant in the planning or evalua- tion process could be involved, including health professionals, businesspersons, paraprofessionals, and advocates or activists.

Planners and evaluators may be employees of the organization about to undertake the ac- tivity, or they may be external consultants hired to assist in all phases or just a specific phase of the planning and evaluation cycle. Internal and external planners and evaluators each have their advantages and disadvantages. Regardless of whether an internal or external evaluator is used, professional stakes and allegiances ought to be acknowledged and understood as factors that can affect the decision making.

Planners and evaluators from within the org- anization are susceptible to biases, consciously or not, in favor of the program or some aspect of the program, particularly if their involvement can positively affect their work. On the positive side, internal planners and evaluators are more likely to have insider knowledge of organizational factors that can be utilized or may have a positive effect on the delivery and success of the health program. Internal evaluators may experience divided loyalties, such as between the program and their job, between the program staff members and other staff, or between the proposed program or evaluation and their view of what would be better.

A source of internal evaluators can be members of quality improvement teams, par- ticularly if they have received any training in program development or evaluation as they relate to quality improvement. The use of total quality management (TQM), continuous quality improvement (CQI), and other quality improvement methodologies by healthcare

Planning and Evaluation Cycle 11

(FIGURE 1-1) and that the activities occur more or less in stages or sets of activities. The stages are cyclical to the extent that the end of one program or stage flows almost seamlessly into the next program or planning activity. The activities are interdependent to the extent that the learning, insights, and ideas that result at one stage are likely to influence the available information and thus the decision making and actions of another stage. Interdependence of activities and stages ideally result from information and data feedback loops that connect the stages.

Naturally, not all of the possible interactions among program planning, implementation, and evaluation are shown in Figure 1-1. In reality, the cyclical or interactive nature of health pro- gram planning and evaluation exists in varying degrees. In the ideal, interactions, feedback loops, and reiterations of process would be reflected throughout this text. For the sake of clarity, however, the cycle is presented in a linear fashion in the text, with steps and sequences covered in an orderly fashion across the progression of chapters. This pedagogical approach belies the true messiness of health program planning and program evaluation. Because the planning and evaluation cycle is susceptible to and affected by external influences, to be successful as a program planner or evaluator requires a substantial degree of flexibility and creativity in recovering from these influences.

The cycle begins with a trigger event, such as awareness of a health problem; a periodic strategic planning effort; a process required by a stakeholder, such as a 5-year strategic planning process or a  grant renewal; or newly available funds for a health program. An indirect trigger for planning could be information generated from an evaluation that reveals either the failure of a health program, extraordinary success of the program, or the need for additional programs. The trigger might also be a news media exposé or legal action. For those seeking to initiate the planning process, getting the attention of influential individuals requires having access to them, packaging the message about the need for planning in ways that are immediately attractive,

that very likely involves others. For example, as the evaluation takes on a sociopolitical process, the evaluators become mediators and change agents. If the evaluation is a learning–teaching process, evaluators become both teacher and student of the stakeholders. To the extent that the evaluation is a process that creates a new reality for stakeholders, program staff members, and program participants, evaluators are reality shapers. Sometimes the evaluation may have an unpredictable outcome; at such times, evaluators are human instruments that gauge what is occurring and analyze events. Ideally, evaluations are a collaborative process, and evaluators act as collaborators with the stake- holders, program staff members, and program participants. If the evaluation takes the form of a case study, the evaluators may become illustrators, historians, and storytellers.

These are but a few examples of how the roles of the professional program evaluator evolve and emerge from the situation at hand. The individual’s role in the planning and evaluation activities may not be clear at the time that the project is started. Roles will develop and evolve as the planning and evaluation activities progress.

▸ Planning and Evaluation Cycle

Although planning and evaluation are commonly described in a linear sequential manner, they actually constitute a cyclical process. In this section, the cycle is described along with an emphasis on factors that enhance and detract from that process being effective.

Interdependent and Cyclic Nature of Planning and Evaluation A major premise running through the current thinking about programs and evaluation is that the activities constituting program planning and program evaluation are cyclical and interdependent

12 Chapter 1 Context of Health Program Development and Evaluation

and their solutions are prioritized. The planning phase includes developing the program theory, which explicates the connection between what is done and the intended effects of the program. Another component of the planning phase includes assessment of organizational and infrastructure resources for implementing the program, such as garnering resources to implement and sustain the program. Yet another major component of program planning is setting goals and objectives that are derived from the program theory.

After the resources necessary to implement the program have been secured and the activities that make up the program intervention have been explicated, the program can be implemented. The logistics of implementation include marketing

and demonstrating the salience of the issue. Thus, to get a specific health problem or issue “on the table,” activists can use the salient events to get the attention of influential individuals. The impor- tance of having a salient trigger event is to serve as a reminder that key individuals mentally sort through and choose among competing attention getters. This trigger event or situation leads to the collection of data about the health problem, the characteristics of the people affected, and their perceptions of the health problem. These data, along with additional data on available resources, constitute a community needs and assets assessment.

Based on the data from the needs assess- ment, program development begins. Problems

FIGURE 1-1 The Planning and Evaluation Cycle

Health program planning

Priorities established

Health status changes

Findings from the evaluation

Intervention effect

Findings from the evaluation

Program implementation of process and effect

theories

Participant/recipient health outcome and

impact

Assessment of community needs

and assets

Program and evaluation planning

Process theory and effect

theory delineated

Evaluation design and methodology

Statement of the health problems

Effects evaluation

Process evaluation

Planning and Evaluation Cycle 13

of an evaluation depends on the extent to which questions that need to be answered are, in fact, answered. Naturally, different stakeholder groups that are likely to use evaluation findings will be concerned with different questions.

Funding organizations, whether federal agencies or private foundations, constitute one stakeholder group. Funders may use process evaluations for program accountability and effect evaluations for determining the success of broad initiatives and individual program effec- tiveness. Project directors and managers, another stakeholder group, use both process and effect evaluation findings as a basis for seeking further funding as well as for making improvements to the health program. The program staff members, another stakeholder group, are likely to use both the process and the effect evaluation as a vali- dation of their efforts and as a justification for their feelings about their success with program participants or recipients. Scholars and health professionals constitute another stakeholder group that accesses the findings of effect evaluations through the professional literature. Members of this group are likely to use effect evaluations as the basis for generating new theories about what is effective in addressing a particular health problem and why it is effective.

Policy makers are yet another stakeholder group that uses both published literature and final program reports regarding process and effect evaluation findings when formulating health policy and making decisions about program resource allocation. Community action groups, community members, and program participants and recipients form another group of stake- holders. This stakeholder group is most likely to advocate for a community health assessment and to use process evaluation results as a basis for seeking additional resources or to hold the program accountable.

Program Life Cycle Feedback loops contribute to the overall de- velopment and evolution of a health program, giving it a life cycle. In the early stages of an idea

the program to the target audience, training and managing program personnel, and delivering or providing the intervention as planned. During implementation of the program, it is critical to conduct an evaluation of the extent to which the program is provided as planned; this is the process evaluation. The data and findings from the process evaluation are key feedback items in the planning and evaluation cycle, and they can and ought to lead to revisions in the program delivery.

Ultimately, the health program ought to have an effect on the health of the individual program participants or on the recipients of the program intervention if provided to the community or a population. The evaluation can be an outcome evaluation of immediate and closely causally linked programmatic effects or an impact evaluation of more temporally and causally distal program- matic effects. Both types of evaluations provide information to the health program planners for use in subsequent program planning. Evaluation of the effects of the program provides data and information that can be used to alter the program intervention. These findings can also be used in subsequent assessments of the need for future or other health programs.

The model used throughout this text as a framework (Figure 1-1) generically represents the steps and processes. It is one of many pos- sible ways to characterize the planning and evaluation cycle. As a generic representation, the planning and evaluation cycle model used in this text includes the essential elements, but it cannot provide detailed instructions on the “whens” and “hows” because each situation will be slightly different.

Using Evaluation Results as the Cyclical Link Before embarking on either a process or an effect evaluation, it is important to consider who will use the results because, in being used, evaluation results are perpetuating the program planning and evaluation cycle. The usefulness

14 Chapter 1 Context of Health Program Development and Evaluation

dying patients (Kaur, 2000). As its advocates saw the need for reimbursement for the service, they began systematically to control what was done and who was “admitted” to hospice. Once evaluations of these hospice programs began to yield findings that demonstrated their positive benefits, they became the model for more wide- spread programs that were implemented in local agencies or by new hospice organizations. As hospice programs became accepted as a standard of care for the dying, the hospice programs became standard, institutionalized services for the organization. Today the availability and use of hospice services for terminally ill patients are accepted as standard practice, and most larger healthcare organizations or systems have established a hospice program. The evolution of hospice is but one example of how an idea for a “better” or “needed” pro- gram can gradually become widely available as routine care.

▸ The Fuzzy Aspects of Planning

We like to think of planning as a rational, linear process, with few ambiguities and only the rare dispute. Unfortunately, this is not the reality of health program planning. Many paradoxes inherently exist in planning as well as implicit assumptions, ambiguities, and the potential for conflict. In addition, it is important to be familiar with the key ethical principles that underlie the decision making that is part of planning.

Paradoxes Several paradoxes pervade health planning (Porter, 2011), which may or may not be resolv- able. Those involved can hold assumptions about planning that complicate the act of planning, whether for health systems or programs. Being aware of the paradoxes and assumptions can, however, help program planners understand possible sources of frustration.

for a health program, the program may begin as a pilot. At this stage, program development occurs and involves use of literature and needs assessment data (Scheirer, 2012). The program may not rely on any existing format or theory, so simple trial and error is used to determine whether it is feasible as a program. It is likely to be small and somewhat experimental because a similar type of program has not been developed or previously attempted. As the program matures, it may evolve into a model program. A model program has interventions that are formalized, or explicit, with protocols that standardize the intervention, and the program is delivered under conditions that are controlled by the program staff members and developers. Model programs can be difficult to sustain over time because of the need to follow the protocols. Evaluations of programs at this stage focus on identifying and documenting the effects and efficacy of the pro- gram (Scheirer, 2014). Successful model programs become institutionalized within the organization as an ongoing part of the services provided. Suc- cessful programs can be institutionalized across a number of organizations in a community to gain wide acceptance as standard practice, with the establishment of an expectation that a “good” agency will provide the program. At this last stage, the health program has become institutionalized within health services. Evaluations tend to focus on quality and performance improvements, as well as sustainability. The last life cycle stage is the dissemination and replication of programs shown to be effective.

Regardless of the stage in a program’s life cycle, the major planning and evaluation stages of community assessment and evaluation are carried out. The precise nature and purpose of each activity vary slightly as the program matures. Being aware of the stage of the program being implemented can help tailor the community assessment and evaluation.

This life cycle of a health program is reflected in the evolution of hospice care. Hospice—care for the dying in a home and family setting—began in London in 1967 as a grassroots service that entailed trial and error about how to manage

The Fuzzy Aspects of Planning 15

and communitywide mandates, does not take into account cultural trends or preferences.

Another paradox is that those in need ideally, but rarely, trigger the planning of health programs; rather, health professionals initiate the process. This paradox addresses the issue of who knows best and who has the best ideas for how to resolve the “real” problem. The perspective held by health professionals often does not reflect broader, more common health social values (Reinke & Hall, 1988), including the values possessed by those individuals with the “problem.” Because persons in need of health programs are most likely to know what will work for them, community and stakeholder participation becomes not just crucial but, in many instances, is actually mandated by funding agencies. This paradox also calls into question the role of health professionals in developing health programs. Their normative perspective and scientific knowledge need to be weighed against individuals’ choices that may have caused the health problem.

A corollary to the paradox dealing with the sources of the best ideas is the notion that poli- ticians tend to prefer immediate and permanent cures, whereas health planners prefer long-term, strategic, and less visible interventions (Reinke & Hall, 1988). Generally, people want to be cured of existing problems rather than to think probabi- listically about preventing problems that may or may not occur in the future. As a consequence, the prevention and long-term solutions that seem obvious to public health practitioners can conflict with the solutions identified by those with the “problem.”

One reason that the best solutions might come from those with the problem is that health professionals can be perceived as blaming those with the health problem for their problem. Blum (1981), for example, identified the practice of “blaming the victim” as a threat to effective planning. When a woman who experiences domestic violence is said to be “asking for it,” the victim is being blamed. During the planning process, blaming the victim can be implicitly and rather subtly manifested in group settings

One paradox is that planning is shaped by the same forces that created the problems that planning is supposed to correct. Put simply, the healthcare, sociopolitical, and cultural factors that contributed to the health problem or condition are very likely to be same factors that affect the health planning process. The interwoven relationship of health and other aspects of life affects health planning. For example, housing, employment, and social justice affect many health conditions that stimulate planning. This paradox implies that health planning itself is also affected by housing, employment, and social justice.

Another paradox is that the “good” of indi- viduals and society experiencing the prosperity associated with health and well-being is “bad” to the extent that this prosperity also produces ill health. Prosperity in our modern world has its own associated health risks, such as higher cholesterol levels, increased stress, increased risk of cardiovascular disease, and increased levels of environmental pollutants. Also, as one group prospers, other groups often become dispropor- tionately worse off. So, to the extent that health program planning promotes the prosperity of a society or a group of individuals, health issues for others will arise that require health program planning.

A third paradox is that what may be eas- ier and more effective may be less acceptable. A good example of this paradox stems from decisions about active and passive protective interventions. Active protection and passive protection are both approaches to risk reduc- tion and health promotion. Active protection requires that individuals actively participate in reducing their risks—for example, through diet changes or the use of motorcycle helmets. Passive protection occurs when individuals are protected by virtue of some factor other than their behavior—for example, water fluoridation and mandates for smoke-free workplaces. For many health programs, passive protection in the form of health policy or health regulations may be more effective and efficient. However, ethical and political issues can arise when the emphasis on passive protection, through laws

16 Chapter 1 Context of Health Program Development and Evaluation

health problem. The assumption of possibilities further presumes that the resources available, whether human or otherwise, are sufficient for the task and are suitable to address the health problem. The assumption of adequate capacity and knowledge is actually tested through the process of planning.

A companion assumption is that planning leads to the allocation of resources needed to address the health problem. This assumption is challenged by the reality that four groups of stakeholders have interests in the decision making regarding health resources (Sloan & Conover, 1996) and each group exists in all pro- gram planning. Those with the health problem and who are members of the target audience for the health program are one group. Another group of stakeholders is health payers, such as insurance companies and local, federal, and philanthropic funding agencies. The third group is individual healthcare providers and healthcare organizations and networks. Last, the general public is a stakeholder group because it is affected by how resources are allocated for health programs. This list of stakeholder groups highlights the variety of motives each group has for being involved in health program planning, such as personal gain, visibility for an organization, or acquisition of resources associated with the program.

Another assumption about those involved is that they share similar views on how to plan health programs. During the planning process, their points of view and cultural perspectives will likely come into contrast. Hoch (1994) suggested that planners need to know what is relevant and important for the problem at hand. Planners can believe in one set of community purposes and values yet still recognize the validity and merit of competing purposes. He argues that effective planning requires tolerance, freedom, and fairness and that technical and political values are two bases from which to give planning advice. In other words, stakeholders involved in the planning process need to be guided into appreciating and perhaps applying a variety of perspectives about planning.

through interpretation of data about needs, thereby affecting decisions related to those needs. Having the attitude that “the victim is to blame” can also create conflict and tension among those involved in the planning process, especially if the “victims” are included as stakeholders. The activities for which the victim is being blamed need to be reframed in terms of the causes of those activities or behaviors.

Yet another paradox is the fact that planning is intended to be successful; no one plans to fail. Because of the bias throughout the program planning cycle in favor of succeeding, unantic- ipated consequences may not be investigated or recognized. The unanticipated consequences of one action can lead to the need for other health decisions that were in themselves unintended (Patrick & Erickson, 1993). To overcome this paradox, brainstorming and thinking creatively at key points in the planning process ought to be fostered and appreciated.

A final paradox of planning, not included on Reinke and Hall’s (1988) list, is that most planning is for making changes, not for creating stability. Yet once a change has been achieved, whether in an individual’s health status or a community’s rates of health problems, the achievement needs to be maintained. Many health programs and health improvement initiatives are designed to be accomplished within a limited time frame, with little or no attention to what happens af- ter the program is completed. To address this paradox requires that planning anticipate the conclusion of a health program and include a plan for sustaining the gains achieved.

Assumptions Assumptions also influence the effectiveness of planning. The first and primary assumption underlying all planning processes is that a solu- tion, remedy, or appropriate intervention can be identified or developed and provided. Without this assumption, planning would be pointless. It is fundamentally an optimistic assumption about the capacity of the planners, the stakehold- ers, and the state of the science to address the

The Fuzzy Aspects of Planning 17

Uncertainty is the unknown likelihood of a possible outcome. Rice, O’Connor, and Pierantozzi (2008) have identified four types of uncertainty: types and amount of resources, technological, market receptivity to the product, and organizational. Each of these uncertainties is present in planning health programs. Ambiguity is doubt about a course of action stemming from awareness that known and unknown factors exist that can decrease the possibility of certainty. In this sense, ambiguity results in uncertainty. Both uncertainty and ambiguity pervade the planning process because it is impossible to know and estimate the effect of all relevant factors—from all possible causes of the health problem, to all possible health effects from program interventions, to all possible acts and intentions of individuals. A rational approach to planning presumes that all relevant factors can be completely accounted for by anticipating the effect of a program, but our experiences as humans tell us otherwise.

Ambiguity is the characteristic of not having a clear or single meaning. Change, or the possibility of change, is a possible source of ambiguity. When ambiguity is ignored, the resulting differences in interpretation can lead to confusion and conflict among stakeholders and planners, among planners and those with the health problem, and among those with var- ious health problems vying for resources. The conflict, whether subtle and friendly or openly hostile, detracts from the planning process by requiring time and personnel resources to address and resolve the conflict. Nonetheless, openly and constructively addressing the am- biguity and any associated conflict can lead to innovations in the program.

Risk is the perceived possibility or uncertain probability of an adverse outcome in a given situation. Health planners need to be aware of the community’s perception and interpretation of probabilities as they relate to health and illness. Risk is not just about taking chances (e.g., bungee jumping or having unprotected sex) but is also about uncertainty and ambiguity (as is the case with estimates of cure rates and projections about future health conditions).

Each stakeholder group assumes that there are limited resources to be allocated for addressing the health problem and is receptive or respon- sive to a different set of strategies for allocating health resources. The resulting conflicts among the stakeholders for the limited resources apply whether they are allocating resources across the healthcare system or among programs for specific health problems. Limited  resources, whether real or not, raise ethical questions of what to do when possible gains from needed health programs or policies are likely to be small, especially when the health program addresses serious health problems.

It is interesting that, the assumption of limited resources parallels the paradox that planning occurs around what is limited rather than what is abundant. Rarely is there a discussion of the abundant or unlimited resources available for health planning. Particularly in the United States, we have an amazing abundance of volunteer hours and interest and of advocacy groups and energy, and recently retired equipment that may be appropriate in some situations. Such resources, while not glamorous or constituting a substantial entry on a balance sheet, deserve to be acknowledged in the planning process.

Another assumption about the planning process is that it occurs in an orderly fashion and that a rational approach is best. To understand the implications of this assumption, one must first acknowledge that four key elements are inherent in planning: uncertainty, ambiguity, risk, and control. The presence of each of these elements contradicts the assumption of a rational approach, and each generates its own paradoxes.

Uncertainty, Ambiguity, Risk, and Control Despite the orderly approach implied by use of the term planning, this process is affected by the limits of both scientific rationality and the usefulness of data to cope with the uncertainties, ambiguities, and risks being addressed by the planning process (see TABLE 1-3).

18 Chapter 1 Context of Health Program Development and Evaluation

the target audience provides planners with a basis from which to be flexible and speculative.

Control, as in being in charge of or man- aging, is a natural reaction to the presence of ambiguity, conflict, and risk. It can take the form of directing attention and allocating resources or of exerting dominance over others. Control

Risk is pervasive and inherent throughout the planning process in terms of deciding who to involve and how, which planning approach to use, which intervention to use, and in estimating which health problem deserves attention. The importance of understanding risk as an element both of the program planning process and of

TABLE 1-3 Fuzzy Aspects Throughout the Planning and Evaluation Cycle

Stages in the Planning and Evaluation Cycle

Community Assessment Planning Implementation

Effect Evaluation

Uncertainty Unknown likelihood of finding key health determinants

Unknown likelihood of selecting an effective intervention, unknown likelihood of the intervention being effective

Unknown likelihood of the intervention being provided as designed and planned

Unknown likelihood of intervention being effective

Ambiguity Unclear about who is being assessed or why

Unclear about the process, who is leading planning process, or what it is intended to accomplish

Unclear about the boundaries of the program, who ought to participate, or who ought to deliver the program

Unclear about meaning of the evaluation results

Risk Unknown possibility of the assessment causing harm

Unknown possibility of planning touching on politically sensitive issues

Unknown possibility of the intervention having an adverse effect on participants

Unknown possibility of adverse effect from the evaluation design, or from misinterpretation of the findings

Control Directing the process of gathering and interpreting data about the health problem

Directing the decisions about the program

Directing the manner in which the program is provided

Directing the process of data collection, analysis and interpretation

Introduction to the Types of Evaluation 19

the overall program theory developed during the planning stage. The process theory delineates the logistical activities, resources, and interventions needed to achieve the health change in program participants or recipients. Information from the process evaluation is used to plan, revise, or improve the program.

The third type of evaluation seeks to determine the effect of the program—in other words, to demonstrate or identify the program’s effect on those who participated in the program. Effect evaluations answer a key question: Did the program make a difference? The effect theory component of the program theory is used as the basis for designing this evaluation. Evaluators seek to use the most rigorous and robust designs, methods, and statistics possible and feasible when conducting an effect evaluation. Findings from effect evaluations are used to revise the program and may be used in subsequent initial program planning activities. Effect evaluations may be referred to as outcome or impact evaluations, terms which seem to be used interchangeably in the literature. For clarity, outcome evaluations focus on the more immediate effects of the program, whereas impact evaluations may have a more long-term focus. Program planners and evaluators must be vigilant with regard to how they and others are using terms and should clarify meanings and address misconceptions or misunderstandings.

A fourth type of evaluation focuses on efficiency and the costs associated with the pro- gram. Cost evaluations encompass a variety of more specific cost-related evaluations—namely, cost-effectiveness evaluations, cost–benefit evaluations, and cost–utility evaluations. For the most part, cost evaluations are done by re- searchers because cost–benefit and cost–utility evaluations, in particular, require expertise in economics. Nonetheless, small-scale and simpli- fied cost-effectiveness evaluations can be done if good cost accounting has been maintained by the program and a more sophisticated outcome or impact evaluation has been conducted. The similarities and differences among these three types of cost studies are reviewed in greater detail

remains a key element of management. In other words, addressing the ambiguity, uncertainty, and risk that might have been the trigger for the planning process requires less—not more— control. Those who preside over and influence the planning process are often thought of as having control over solutions to the health problem or condition. They do not. Instead, effective guid- ance of the planning process limits the amount of control exerted by any one stakeholder and addresses the anxiety that often accompanies the lack of control.

▸ Introduction to the Types of Evaluation

Several major types of activities are classified as evaluations. Each type of activity requires a specific focus, purpose, and set of skills. The types of evaluations are introduced here as an overview of the field of planning and evaluation.

Community needs assessment (also known as community health assessment) is a type of eval- uation that is performed to collect data about the health problems of a particular group. The data collected for this purpose are then used to tailor the health program to the needs and distinctive characteristics of that group. A community needs assessment is a major component of program planning because it is, done at an early stage in the program planning and evaluation cycle. In addition, the regular completion of community assessments may be required. For example, many states do 5-year planning of programs based on state needs assessments.

Another type of evaluation begins at the same time that the program starts. Process evalu- ations focus on the degree to which the program has been implemented as planned and on the quality of the program implementation. Process evaluations are known by a variety of terms, such as monitoring evaluations, depending on their focus and characteristics. The underlying framework for designing a process evaluation comes from the process theory component of

20 Chapter 1 Context of Health Program Development and Evaluation

are usually contrasted with formative evaluations. The term formative evaluation is used to refer to program assessments that are performed early in the implementation of the program and used to make changes to the program. Formative evaluations might include elements of process evaluation and preliminary effect evaluations.

Mandated and Voluntary Evaluations Evaluations are not spontaneous events. Rather, they are either mandated or voluntary. A mandate to evaluate a program is always linked in some way to the funding agencies, whether a govern- mental body or a foundation. If an evaluation is mandated, then the contract for receiving the program funding will include language specifying the parameters and time line for the mandated evaluation. The mandate for an evaluation may specify whether the evaluation will be done by project staff members or external evaluators, or both. For example, the State Child Health Insurance Program (SCHIP), created in 1998, is a federally funded and mandated program to expand insurance coverage to children just above the federal poverty level. Congress has the authority to mandate evaluations of federal programs and did just that with the SCHIP. Mandated evaluations of SCHIP include an overall evaluation study by Wooldridge and associates from the Urban Institute (2003), and an evaluation specifically focused on outcomes for children with special healthcare needs (Zickafoose, Smith, & Dye, 2015).

Other evaluations may be linked to ac- creditation that is required for reimbursement of services provided, making them de facto mandated evaluations. For example, to receive accreditation from the Joint Commission, a health services organization must collect data over time on patient outcomes. These data are then used to develop ongoing quality improvement efforts. A similar process exists for mental health agencies. The Commission on Accreditation of Rehabilitation Facilities (CARF) requires that

in the text so that program planners can be, at minimum, savvy consumers of published reports of cost evaluations. Because cost evaluations are performed late in the planning and evaluation cycle, their results are not likely to be available in time to make program improvements or re- visions. Instead, such evaluations are generally used during subsequent planning stages to gather information for prioritizing program options.

Comprehensive evaluations, the fifth type of evaluation, involve analyzing needs assessment data, process evaluation data, effect evaluation data, and cost evaluation data as a set of data. Given the resources needed to integrate analysis of various types of data to draw conclusions about the effectiveness and efficiency of the program, comprehensive evaluations are relatively uncommon. A sixth type of evaluation is a meta-evaluation. A meta-evaluation is done by combining the findings from previous outcome evaluations of various programs for the same health problem. The pur- pose of a meta-evaluation is to gain insights into which of the various programmatic approaches has had the most effect and to determine the maximum effect that a particular programmatic approach has had on the health problem. This type of evaluation relies on the availability of existing information about evaluations and on the use of a specific set of methodological and statistical procedures. For these reasons, meta-evaluations are less likely to be done by program personnel; instead, they are generally carried out by evaluation researchers. Meta-evaluations that are published are extremely useful in program planning because they indicate which programmatic interventions are more likely to succeed in having an effect on the participants. Published meta-evaluations can also be valuable in influencing health policy and health funding decisions.

Summative evaluations, in the strictest sense, are done at the conclusion of a program to provide a conclusive statement regarding pro- gram effects. Unfortunately, the term summative evaluation is sometimes used to refer to either an outcome or impact evaluation, adding even more confusion to the evaluation terminology and vernacular language. Summative evaluations

The Public Health Pyramid 21

as techniques for designing and conducting both program process and effect evaluations have improved, and the expectation is that even mandated evaluations will be useful in some way. Nonetheless, it remains critical to consider how to conduct evaluations legitimately, rigorously, inexpensively, and fairly. In addition, if the AEA standards of utility, feasibility, propriety, and accuracy cannot be met, it is not wise to conduct an evaluation (Patton, 2008).

Interests and the degree of influence held by stakeholders can change. Such changes affect not only how the evaluation is conceptualized but also whether evaluation findings are used. In addition, the priorities and responsibilities of the organizations and agencies providing the program can change during the course of delivering the program, which can then lead to changes in the program implementation that have not been taken into account by the evaluation. For example, if withdrawal of resources leads to a shortened or streamlined evaluation, subsequent findings may indicate a failure of the program intervention. However, it will remain unclear whether the apparently ineffective intervention was due to the design of the program or the design of the evaluation. In addition, unanticipated problems in delivering the program interventions and the evaluation will always exist. Even rigorously designed evaluations face challenges in the real world stemming from staff turnover, potential participants’ noninvolvement in the program, bad weather, or any of a host of other factors that might hamper achieving the original evaluation design. Stakeholders will need to understand that the evaluator attempted to address challenges as they arose if they are to have confidence in the evaluation findings.

▸ The Public Health Pyramid

Pyramids tend to be easy to understand and work well to capture tiered concepts. For these reasons, pyramids have been used to depict the

provider organizations conduct a self-evaluation as an early step in the accreditation process. These accreditation-related evaluations apply predominantly to direct care providers rather than to specific programs.

Completely voluntary evaluations are initi- ated, planned, and completed by the project staff members in an effort to make improvements. However, given the relatively low reward from, and cost associated with, doing an evaluation when it is not required, these evaluations are likely to be small with low scientific rigor. Pro- grams that engage voluntarily in evaluations may have good intentions, but they often lack the skills and knowledge required to conduct an appropriate evaluation.

When Not to Evaluate Situations and circumstances that are not amenable to conducting an evaluation do exist, despite a request or the requirement for having an evaluation. Specifically, it is not advisable to attempt an evaluation under the following four circumstances: when there are no questions about the program, when the program has no clear direction, when stakeholders cannot agree on the program objectives, and when there is not enough money to conduct a sound evaluation (Patton, 2008). In addition to these situations, Weiss (1972) recognized that sometimes eval- uations are requested and conducted for less than legitimate purposes, namely, to postpone program or policy decisions, thereby avoiding the responsibility of making the program or policy decision; to make a program look good as a public relations effort; or to fulfill program grant requirements. As these lists suggest, those engaged in program planning and evaluation need to be purposeful in what is done and should be aware that external forces can influence the planning and evaluation processes.

Since Weiss made her observation in 1972, funders have begun to require program process and effect evaluations, and conducting these evaluations to meet that requirement is consid- ered quite legitimate. This change has occurred

22 Chapter 1 Context of Health Program Development and Evaluation

mental health drop-in centers, hospice programs, financial assistance programs that provide trans- portation to medical care, community-based case management for patients with acquired immune deficiency syndrome (AIDS), low-income hous- ing, nutrition education programs provided by schools, and workplace child care centers. As this list of programs demonstrates, the services at this level may directly or indirectly contribute to the health of individuals, families, and communities and are provided to aggregates. Enabling services can also be thought of as addressing some of the consequences of social determinants of health.

The next, more encompassing level of the public health pyramid is population-based services. At the population level of the pyramid, services are delivered to an entire population, such as all persons residing in a city, state, or country. Examples of population services include immu- nization programs for all children in a county, newborn screening for all infants born in a state, food safety inspections carried out under the auspices of state regulations, workplace safety programs, nutrition labeling on food, and the Medicaid program for pregnant women whose incomes fall below the federal poverty guidelines. As this list reflects, the distinction between an aggregate and a population can be blurry. Programs at this level typically are intended to reach an entire population, sometimes without the conscious involvement of individuals. In this sense, individuals receive a population-based health program, such as water fluoridation, rather than participating in the program, as they would in a smoking-cessation class. Inter- ventions and programs aimed at changing the socioeconomic context within which populations live would be included at this population level of the pyramid. Such programs are directed at changing one or more social determinants of health. Population-level programs contribute to the health of individuals and, cumulatively, to the health status of the population.

Supporting the pyramid at its base is the infrastructure of the healthcare system and the public health system. The health services at the other pyramid levels would not be possible

tiered nature of primary healthcare, secondary healthcare, and tertiary healthcare services (U.S. Public Health Service, 1994), the inverse relationship of effort needed and health impact of different interventions (Frieden, 2010), and nutrition recommendations (Gil, Ruiz-Lopez, Fernandez-Gonzalez, & de Victoria, 2014).

The public health pyramid is divided into four sections (FIGURE 1-2). The top, or the first, section of the pyramid contains direct healthcare services, such as medical care, psychological counseling, hospital care, and pharmacy services. At this level of the pyramid, programs are delivered to individ- uals, whether patients, clients, or even students. Generally, programs at the direct services level have a direct, and often relatively immediate, effect on individual participants in the health program. Direct services of these types appear at the tip of the pyramid to reflect that, overall, the smallest proportion of a population receives them. These interventions, according to the Health Impact Pyramid (Frieden, 2010), require considerable effort, with minimal population effects.

At the second level of the pyramid are en- abling services, which are those health and social services that support or enhance the health of aggregates. Aggregates are used to distinguish between individuals and populations; they are groups of individuals who share a defining char- acteristic, such as mental illness or a terminal disease. Examples of enabling services include

FIGURE 1-2 The Public Health Pyramid

Direct healthcare services

Enabling services

Population-based services

Infrastructure services

The Public Health Pyramid 23

of the program with meeting the needs of the broadest number of people with a given need. Reaching the same number of persons with a direct services program as with a population services program poses additional expense and logistic challenges.

The pyramid also serves as a reminder that stakeholder alignments and allegiances may be specific to a level of the pyramid. For example, a school health program (an enabling-level program) has a different set of constituents and concerned stakeholders than a highway safety program (a population-level program). The savvy program planner considers not only the potential program participants at each level of the pyramid but also the stakeholders who are likely to make themselves known during the planning process.

The public health pyramid has particular relevance for public health agencies concerned with addressing the three core functions of public health (Institute of Medicine, 1988): assessment, assurance, and policy. These core functions are evident, in varying forms, at each level of the pyramid. Similarly, the pyramid can be applied to the strategic plans of organizations in the private healthcare sector. For optimal health program planning, each health program being developed or implemented ought to be considered in terms of its relationship to services, programs, and health needs at other levels of the pyramid. For all these reasons, the public health pyramid is used throughout this text as a framework for summarizing specific issues and applications of chapter content to each level of the pyramid and to identify and discuss potential or real issues related to the topic of the chapter.

The Public Health Pyramid as an Ecological Model Individual behavior and health are now under- stood to be influenced by the social and physical environment of individuals. This recognition is reflected in the growing use of the ecological approach to health services and public health

unless there were skilled, knowledgeable health professionals; laws and regulations pertinent to the health of the people; quality assurance and improvement programs; leadership and managerial oversight; health planning and program evaluation; information systems; and technological resources. The planning and evaluation of health programs at the direct, enabling, and population services levels is itself a component of the infrastructure; these are infrastructure activities. In addition, planning programs to address problems of the infrastructure, as well as to evaluate the infra- structure itself, are needed to keep the health and public health system infrastructure strong, stable, and supportive of the myriad of health programs.

Use of the Public Health Pyramid in Program Planning and Evaluation Health programs exist across the pyramid levels, and evaluations of these programs are needed. However, at each level of the pyramid, certain issues unique to that level must be addressed in developing health programs. Accordingly, the types of health professionals and the types of expertise needed vary by pyramid level, reinforcing the need to match program, participants, and pro- viders appropriately. Similarly, each level of the pyramid is characterized by unique challenges for evaluating programs. For this reason, the public health pyramid, as a framework, helps illuminate those differences, issues, and challenges, as well as to reinforces that health programs are needed across the pyramid levels if the Healthy People 2020 goals and objectives are to be achieved.

In a more general sense, the public health pyramid provides reminders that various aggre- gates of potential audiences exist for any health problem and program and that health programs are needed across the pyramid. Depending on the health discipline and the environment in which the planning is being done, direct service programs may be the natural or only inclination. The public health pyramid, however, provides a framework for balancing the level

24 Chapter 1 Context of Health Program Development and Evaluation

Because it distinguishes and recognizes the importance of enabling and population services, the public health pyramid can be integrated with an ecological view of health and health problems. If one were to look down on the pyramid from above, the levels would appear as concentric circles (FIGURE 1-3)—direct services for individ- uals nested within enabling services for families, aggregates, and neighborhoods, which are in turn nested within population services for all residents of cities, states, or countries. This is similar to individuals being nested within the enabling environment of their family, workplace setting, or neighborhood, all of which are nested within

programs. The ecological approach, which stems from systems theory applied to individuals and families (Bronfenbrenner, 1970, 1989), postulates that individuals can be influenced by factors in their immediate social and physical environment. This perspective has been expanded into the social determinants perspective in public health, which has wide acceptance (Frieden, 2010). The individ- ual is viewed as a member of an intimate social network, usually a family, which is a member of a larger social network, such as a neighborhood or community. The way in which individuals are nested within these social networks has conse- quences for the health of the individual.

FIGURE 1-3 The Pyramid as an Ecological Model

S cience, theory, practice, programs, planning, structu

re, p olic

ies , re

so ur

ce

s, e

va lu

at io

n

Pu blic

hea lth an

d private health infrastructure

Populations

F am

ilie

s, a

gg reg

ates, neighborhoods, com

m unities

Individuals

Across the Pyramid 25

or patients—that is, on developing programs that are provided to those individuals and on assessing the extent to which those programs make a difference in the health of the individ- uals who receive the health program. Health is defined in individual terms, and program effects are measured as individual changes. From this level of the public health pyramid, community is most likely viewed as the context affecting individual health.

At the enabling services level, health program planning and evaluation focus on the needs of aggregates of individuals and on the services that the aggregate needs to maintain health or make health improvements. Enabling services are often social, educational, or human services that have an indirect effect on health, thus warranting their inclusion in planning health programs. Health continues to be defined and measured as an individual characteristic to the extent that enabling services are provided to individual members of the aggregate. However, program planning and evaluation focus not on individuals but rather on the aggregate as a unit. At this level of the pyramid, community can be either the aggregate that is targeted for a health program or the context in which the aggregate functions and lives. How community is viewed depends on the health problem being addressed.

At the population-based services level, health program planning and evaluation focus on the needs of all members of a population. At this level of the pyramid, health programs are, at a minimum, population driven, meaning that data collected in regard to the health of the population drive the decisions about the health program. This approach results in programs that are population focused and, ideally (but not necessarily), population based. It is worth noting that population-focused programs tend to have a health promotion or health maintenance focus rather than a focus on treatment of illnesses. At a population level, health is defined in terms of population statistics, such as mortality and morbidity rates. In this regard, the Healthy People 2020 objectives (TABLE 1-4) are predominantly at

the population environment of factors such as social norms and economic and political envi- ronments. The infrastructure of the healthcare system and public health system is the foundation and supporting environment for promoting health and preventing illnesses and diseases.

The end of the chapter presents a summary of challenges or issues related to applying the chapter content to each level of the pyramid. This feature reinforces the message that each level of the pyramid has value and importance to health program planning and evaluation. In addition, certain unique challenges are specific to each level of the pyramid. The chapter summary by levels offers an opportunity to acknowledge and address the issues related to the levels.

▸ The Town of Layetteville in Bowe County

As an aid to understanding and assimilating the content covered, examples from the literature are provided throughout this book. In addition, chapters include application of content to a hy- pothetical town (Layetteville) in an imaginary county (Bowe County). Based on a fictional community needs assessment, subsequent prior- itization leads to the identification of five health problems as foci for health program planning. These health problems are used throughout the text as opportunities to demonstrate application of the chapter content. Also, some discussion questions and activities use Layetteville and Bowe County as opportunities for the reader to practice applying the chapter content. While the town and county are fictitious, the health problems around which the program planning and evaluation occur are very real and relevant.

▸ Across the Pyramid At the direct services level, health program plan- ning and evaluation focus on individual clients

26 Chapter 1 Context of Health Program Development and Evaluation

health program identify which Healthy People 2020 objectives are being addressed. To the extent that health planners and evaluators are familiar with these objectives, they will be better able to design appropriate programs and then to argue in favor of the relevance of each of those programs. At the infrastructure level, health can be defined in terms of the individual workers in the healthcare sector (an aggregate). More to the point, because program planning and evaluation are infrastructure activities, it is actually at the infrastructure level that the decisions are made on the definition of health to be used in the program. Similarly, the way that community is viewed is determined at the infrastructure level.

the population level of the public health pyramid. Community is more likely to be the population targeted by the health program.

At the infrastructure level, health program planning and evaluation are infrastructure activities of both the public health system and the healthcare system. Infrastructure includes organizational management, acquisition of resources, and development of health policy. A significant document reflecting health policy is Healthy People 2020, which outlines the goals and objectives for the health of the people of the United States. These national objectives are considered when setting priorities and are used by many federal and nongovernmental funding agencies, which often require that a

TABLE 1-4 A Summary of the Healthy People 2020 Priority Areas

1. Access to quality health services 2. Adolescent health 3. Arthritis, osteoporosis, and chronic back

conditions 4. Blood disorders and blood safety 5. Cancer 6. Chronic kidney disease 7. Dementias, including Alzheimer’s 8. Diabetes 9. Disability and secondary conditions

10. Early and middle childhood 11. Educational and community-based

programs 12. Environmental health 13. Family planning 14. Food safety 15. Genomics 16. Global health 17. Health communication and health

information technology 18. Healthcare-associated infections 19. Health-related quality of life and well-being 20. Hearing and other sensory or

communication disorders

21. Heart disease and stroke 22. Human immunodeficiency virus

infection (HIV ) 23. Immunization and infectious diseases 24. Injury and violence prevention 25. Lesbian, gay, bisexual, and transgender

health 26. Maternal, infant, and child health 27. Medical product safety 28. Mental health and mental disorders 29. Nutrition and weight status 30. Occupational safety and health 31. Older adults 32. Oral health 33. Physical activity 34. Preparedness 35. Public health infrastructure 36. Respiratory diseases 37. Sexually transmitted diseases 38. Sleep health 39. Social determinants of health 40. Substance abuse 41. Tobacco use 42. Vision

Source: Department of Health and Human Services. Retrieved August 20, 2016, from www.cdc.gov/nchs/healthy_people/hp2020/hp2020_topic _areas.htm.

27References

▸ Discussion Questions 1. When and under what conditions

might it be advisable not to conduct an evaluation?

2. Oral health is a major health problem, especially for children living in poverty. Describe how an oral health program developed at each level of the public health pyramid would differ and how the considerations would differ.

3. Conduct a literature search using words such as “planning,” “evaluation,” “pro- gram,” and a health condition of interest to you. Which journals publish articles about health program planning and health program evaluations? What are the cur- rent trends in the field as reflected in the published literature that you reviewed?

4. Review the program planning and evalu- ation cycle in Figure 1-1. Using the litera- ture you found for Discussion Question 3, where does most of the literature fall in that cycle?

5. Access and review the material in the fol- lowing document and compare it with the perspective given in this chapter: Centers for Disease Control and Preven- tion. (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Report, 48(RR-11): i–41. Retrieved August 31, 2012, from www .cdc.gov/mmwr/ preview/mmwrhtml /rr4811a1.htm.

▸ Internet Resources American Evaluation Association (AEA) This international, professional organization of evaluators is devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. The AEA website (www.eval.org/) includes links to professional groups, foundations,

online publications, and other resources related to evaluation.

Centers for Disease Control and Prevention Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Report, 48(RR-11): i–41. Retrieved August 31, 2012, from www.cdc.gov/mmwr/preview/mmwrhtml /rr4811a1.htm. This online textbook describes the steps involved in conducting an evaluation.

Evaluation Center of Western Michigan University This organization focuses on advancing the theory and practice of program, personnel, and student/constituent evaluation, as applied primarily to education and human services. Its website (http://www.wmich.edu/evaluation) has links and a set of checklists that can be used to improve the quality of an evaluation project.

The Evaluation Exchange Harvard Family Research Project’s evaluation periodical, The Evaluation Exchange, addresses current issues facing program evaluators of all levels, with articles written by the most prominent evaluators in the field. Designed as an ongoing discussion among evaluators, program practitioners, funders, and policy makers, The Evaluation Exchange highlights innovative methods and approaches to evalu- ation, emerging trends in evaluation practice, and practical applications of evaluation theory. It goes out to its subscribers free of charge four times per year. It can be accessed via the Internet at http://www.hfrp.org/evaluation /the-evaluation-exchange.

References American Evaluation Association. (2011). The program

evaluation standards: Summary form. Retrieved August 31, 2016, from http://www.eval.org/p/cm/ld/fid=103

American Public Health Association (APHA). (1991). Healthy communities 2000: Model standards. Washington, DC: Author.

28 Chapter 1 Context of Health Program Development and Evaluation

Sloan, F. A., & Conover, C. J. (1996). The use of cost– effectiveness/cost–benefit analysis in actual decision making: Current status and prospects. In F. A. Sloan (Ed.), Valuing health care: Costs, benefits, and effective- ness of pharmaceuticals and other medical technologies (pp. 207–232). Cambridge, UK: Cambridge University Press.

Stufflebeam, D. L., & Coryn, C. L. S. (2014). Evaluation theory, models and applications (2nd ed.). San Francisco, CA: Jossey-Bass.

Swenson, M. M. (1991). Using fourth-generation evaluation. Evaluation and Health Professions, 14(1), 79–87.

Turnock, B. (2004). Public health: What it is and how it works (3rd ed.). Sudbury, MA: Jones & Bartlett.

United Way of America. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: Author.

U.S. Department of Health and Human Services (DHHS). (1980). Promoting health/preventing disease: Objectives for the nation. Washington, DC: Author.

U.S. Department of Health and Human Services (DHHS). (1991). Healthy People 2000: National health promotion and disease prevention objectives. Publication No. (PHS) 91-50212. Washington, DC: Author.

U.S. Department of Health and Human Services (DHHS). (2000). Healthy People 2010: Understanding and improving health (2nd ed.). Washington, DC: U.S. Government Printing Office.

U.S. Department of Health and Human Services (DHHS). (2011). Healthy People 2020. Retrieved November 9, 2011, from www.healthypeople.gov/2020/default.aspx

U.S. Department of Health, Education, and Welfare (DHEW). (1979). Healthy people: The surgeon general’s report on health promotion and disease prevention (DHEW, PHS Publication No.  79-55071). Washington, DC: Author.

U.S. Public Health Service. (1994). For a healthy nation: Return on investment in public health. Washington, DC: Author.

Weiss, C. (1972). Evaluation. Englewood Cliffs, NJ: Prentice Hall.

Whitmore, E. (Ed.). (1998). Understanding and practicing participatory evaluation: New directions for evaluation. San Francisco, CA: Jossey-Bass.

Wooldridge, J., Hill, I., Harrington, M., Kenney, G. M., Hawkes, C., & Haley, J. M. (2003). Interim evaluation report: Congressionally mandated evaluation of the State Children’s Health Insurance Program. Washington, DC: Urban Institute. Retrieved August 31, 2011, from www .urban.org/expert.cfm?ID=JudithWooldridge.

World Health Organization. (1947). Constitution of the World Health Organization. Chronicle of World Health Organization, 1, 29–43.

Zickafoose, J. S., Smith, K. V., & Dye, C. (2015). Children with special health care needs in CHIP: Access, use, and child and family outcomes. Academic Pediatrics, 15(3), S85–S92. http://dx.doi.org/10.1016/j.acap.2015.02.001

Blum, H. L. (1981). Planning for health: Generics for the eighties (2nd ed.). New York, NY: Human Sciences Press.

Bronfenbrenner, U. (1970). Two worlds of childhood. New York, NY: Russell Sage Foundation.

Bronfenbrenner, U. (1989). Ecological systems theory. Annals of Child Development, 16, 187–249.

Dever, G. E. (1980). Community health analysis: A holistic approach. Germantown, MD: Aspen.

Frieden, T. R. (2010). A framework for public health action: The health impact pyramid. American Journal of Public Health, 100, 590–595.

Gil, A., Ruiz-Lopez, M. D., Fernandez-Gonzalez, M., & de Victoria, E. M. (2014). The FINUT Health Lifestyles Guide: Beyond the food pyramid. Advances in Nutrition, 5, 358S–367S.

Guba, E. G., & Lincoln, Y. S. (1987). Fourth generation evaluation. In D. J. Palumbo (Ed.), The politics of program evaluation (pp. 202–204). Newbury Park, CA: Sage.

Herman, J. L., Morris, L. L., & Fitz-Gibbon, C. T. (1987). Evaluators’ handbook. Newbury Park, CA: Sage.

Hoch, C. (1994). What planners do: Power, politics and persuasion. Chicago, IL: Planners Press.

Institute of Medicine, National Academy of Sciences. (1988). The future of public health. Washington, DC: National Academy Press.

Kaur, J. (2000). Palliative care and hospice programs. Mayo Clinic Proceedings, 75, 181–184.

Patrick, D. L., & Erickson, P. (1993). Health status and health policy: Allocating resources to health care. Oxford, UK: Oxford University Press.

Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.

Patton, M. Q. (2008). Utilization-focused evaluation: Sage. Patton, M. Q. (2012). Essentials of utilization-focused eval-

uation. Thousand Oaks, CA: Sage. Porter, L. (2011). Informality, the commons and the paradoxes

for planning: Concepts and debates for informality and planning. Planning Theory and Practice, 12(1), 115–153.

Reinke, W. A., & Hall, T. L. (1988). Political aspects of planning. In W. A. Reinke (Ed.), Health planning for effective management (pp. 75–85). New York, NY: Oxford University Press.

Rice, M. P., O’Connor, G. C., & Pierantozzi, R. (2008). Imple- menting a learning plan to counter project uncertainty. Sloan Management Review, 29(2), 54–62.

Rohrer, J. (1996). Planning for community-oriented health systems. Washington, DC: American Public Health Association.

Rosen, G. (1993). A history of public health (expanded ed.). Baltimore, MD: Johns Hopkins University Press.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Newbury Park, CA: Sage.

Scheirer, M. A. (2012). Planning evaluation through the program life cycle. American Journal of Evaluation, 33(2), 263–294. doi:10.1177/1098214011434609

29

Relevance of Diversity and Disparities to Health Programs

The health status of individuals and populations is influenced by biological processes and by lifestyle behaviors and circumstances. The intersection of biology, lifestyle, and environmental circumstances leads to disparities in health status, with some groups having lower morbidity and mortality rates than other groups. At the root of health disparities is diversity in biological characteristics, as well as in social, cultural, ethnic, linguistic, and economic characteristics of individuals and populations.

In the late 1990s, President Bill Clinton put race, racism, and ethnic diversity on the public agenda. As a consequence, federal agencies, including the National Institutes of Health (NIH) and the Department of Health and Human Services (DHHS), began explic- itly to fund research into understanding and eliminating racial and ethnic disparities in health status. Private foundations and other agencies funding health programs followed suit by requiring grantees to state explicitly how each program contributes to reducing racial and ethnic health disparities. The high level of attention given to health disparities means

that program planners and evaluators must appreciate the sources of disparity—notably diversity, understand what the key aspects of diversity are and how those aspects are relevant to health programs, and know which strategies can be used to address diversity so that the health program will be successful. This chapter begins to address these issues.

A current urban legend exemplifies the influence of culture on healthcare decisions and the importance of having culturally competent staff. A woman from Africa was in labor with her first child in a U.S. hospital. Her labor was not progressing, and the physician wanted to deliver the baby by cesarean section in an effort to minimize the potential brain damage that was likely to result from a vaginal delivery. The woman and her husband refused the surgery, opting for a difficult vaginal delivery. The couple explained that they needed to make their decision based on what their life would be like when they returned to Africa. In their home village, a woman with a history of a cesarean section would be in grave danger if she were to have another baby because of the lack of surgical services for delivery in her home village. The life and health of the woman

CHAPTER 2

© Lynne Nicholson/Shutterstock

30 Chapter 2 Relevance of Diversity and Disparities to Health Programs

were paramount. The child would be loved and cared for by the entire village, even if it were cognitively impaired from the difficult delivery. Whether the story is true has been lost in the telling. Regardless of its veracity, it highlights the influence of cultural values and norms on behavior and demonstrates the vital roles that culture, diversity, and life circumstances play in health discussions.

The topic of diversity is addressed early in this text because of its relevance throughout the planning and evaluation cycle (FIGURE 2-1). Diversity is relevant with regard to assessment of the health disparities to be addressed. It also affects the intervention choice and delivery, a

component of which is the issue of diversity of health providers. TABLE 2-1 provides examples of considerations that need to be weighed throughout the health program planning and evaluation cycle. The culture of the healthcare organization and the cultural competency of the program staff are directly related to the ability to tailor programs culturally, as is the formation of coalitions.

▸ Health Disparities Health disparities and health inequities are terms denoting important differences in health status

FIGURE 2-1 Effects of Diversity Throughout the Planning and Evaluation Cycle Stage in the Planning and Evaluation Cycle

Cultural influence on priorities

Changes in health disparities

Interpretation of evaluation findings

Diversity affects intervention effect

Culturally tailored process theory and

effect theory

Culturally sensitive evaluation design

and methods

Diversity and health disparity

Health program planning

Findings from the evaluation

Program implementation of process and effect

theories

Participant/recipient health outcomes and

impact

Assessment of community needs

and assets

Program and evaluation planning

Effects evaluation

Process evaluation

Health Disparities 31

among socioeconomic, racial, and ethnic groups. Disparities in health care are defined as differences by race or ethnicity in access to or the quality of health care that are not due to the health or clinical needs or preferences of the person. The intuitive understanding belies the challenges in defining heath disparities in a way that addresses the complexity of the problem (Braveman, 2006).

Well-documented health disparities exist. For example, blacks have nearly twice the rates of low birthweight infants and infant mortality as whites (National Center for Health Statistics [NCHS], 2016). Unintentional injury mortalities for American Indian children ages 1 to 4 years (11.7 per 100,000) is higher than for black (10.7 per 100,000) or non-Hispanic whites (7.2 per 100,000) (Hearon, 2016). Disparities also exist for

chronic illnesses: American Indians and Alaska Natives are 1.7 times more likely to have diabe- tes than non-Hispanic whites of a similar age, a notable improvement in narrowing that disparity since 2000 (NCHS, 2016). Black women have higher mortality rates from breast cancer than any other racial or ethnic group in the United States (National Cancer Institute, 2014). Some health status disparities might be explained by disparities in physical activity and consumption of fruits and vegetables (Gavin et al., 2011). These are a few examples of health disparities that could be addressed by individual practitioners but are perhaps more appropriately targeted by health programs across the public health pyramid.

The causes of health disparities remain the subject of research, but current theories regarding

TABLE 2-1 Examples of Cultural Tailoring Throughout the Program Planning and Evaluation Cycle

Stage in the Planning and Evaluation Cycle Examples of Tailoring for Cultural and Ethnic Diversity

Community needs assessment

Definitions of health and illness; willingness to reveal needs or wants; self-definition in terms of culture, race, or ethnicity; health disparities; experience of disparities in access to or quality of health care

Program theory and development

Identification of contributing and determinant factors of the health disparities; role of discrimination and culturally bound health behaviors in the disparities; culturally acceptable and appropriate interventions

Process or program implementation

Culturally and ethnically adjusted program objective targets; cultural, racial, and ethnic representations and appropriateness of materials developed or chosen, such as visual representations, colors used, language or languages, location, media used, modality of distribution, and enticement used

Program intervention delivery

Type of intervention; length of time participants receive intervention (e.g., session length); amount of intervention (e.g., number of sessions)

Program effect evaluation Language or languages of survey questionnaires; culturally appropriate enticements to participate; access to culturally and ethnically equivalent control groups

32 Chapter 2 Relevance of Diversity and Disparities to Health Programs

Assigning a label to a culture is less import- ant than seeking information about unique or distinct culturally bound patterns of behavior that have health implications. For example, it is not as important to be able to identify a person as being from Hopi culture versus Navajo culture as it is to ask about daily consumption of meats and fresh vegetables and the ways in which those foods are prepared, and to understand the historical context that has influenced changes in dietary patterns. Culture, as the sharing of similar beliefs, values, and norms, contributes to a sense of unity among the members of the culture. The cultural cohesion and sense of belonging to a cultural group is a powerful force in creating conflicts as well as in creating opportunities. Both the Hopi and the Navajo have strong cultural identities that present an opportunity for health program planners to build that cultural identity into a program. The strong cultural identity can also create conflicts, however, between program planners and people from the Hopi or Navajo nations if the program is perceived as threatening their culture or being inconsistent with their cultural beliefs.

The relationship between culture and illness is recognized as having distinct manifestations, especially in mental health. The American Psychiatric Association’s Diagnostic and Statis- tical Manual of Mental Disorders, Fifth Edition (DSM-5), has replaced the diagnostic category of “culture-bound syndromes” (Tseng, 2006) with “cultural concepts of distress” (Ventriglio, Ayonrinde, & Bhugra, 2016). Cultural-concepts of distress is a broad classification that considers the ways that cultural groups experience and frame symptoms, ways of communicating the distress, and explanations of the symptoms. The new diagnostic category stems from a growing understanding that illness is, at least in part, socially and culturally constructed. The interac- tion of culture and illness extends into physical illnesses: A study of pain found different levels of pain tolerance between Italians, Swedes, and Saudis (Al-Harthy, Ohrback, Michelotti, & List, 2016), suggesting the possible influence of cultural

health disparities posit that they have multiple, interactive (i.e., not mutually exclusive) causes that are biological, socioeconomic, and cultural in nature. For instance, among black mothers in particular, the biological effect of heightened levels of cortisol due to perceived discrimination has been associated with adverse birth outcomes (e.g., preterm birth, low birthweight). Among American Indians in the Southwest, the prev- alence of diabetes was minimal until the 20th century, when water restrictions transformed an agricultural society into one dependent on government subsidies, largely comprised of processed commodities (Satterfield, DeBruyn, Francis, & Allen, 2014). The interactive causes of health disparities either can be primary targets for health programs or can constitute a contextual environment for the health program. In either case, at the heart of addressing health disparities in a practical manner and develop- ing successful health programs lies the need to understand the relationship of diversity to health disparities.

Diversity and Health Disparities Diversity, in the context of health, refers to the numerous ways in which individuals and groups differ in their beliefs, behaviors, values, backgrounds, preferences, and biology. Diversity is most often described in terms of language, culture, ethnicity, and race. Each of these aspects, along with biological diversity within the human population, has health implications.

Culture is a learned set of beliefs, values, and norms that are shared by a group of people; it is a design for how to live (Spector, 1991). As a set of behavioral norms or expectations, cultural beliefs influence a wide range of behaviors, including dietary choices, hygiene practices, sexual practices, and illness behaviors. Through such behaviors, culture has an effect on health and therefore is relevant to health programs. Cultures can be difficult to define and distinguish, particularly when subcultures rather than the dominant culture are the target of a program.

Measurement 33

and 13% were refugees (Mossaad, 2016). By 2060, the population of the United States will be 43% non-Hispanic white, 29% Hispanic, 13% African American, and 9% Asian and Pacific Islander (Colby & Ortman, 2014).

Diversity and Health Programs As this very brief introduction to health dis- parities suggests, the extent of diversity within a target population can have various effects on how health programs are developed and provided. Diversity has particular implications for measurement done during planning and evaluating the health program; for the design and implementation of the health program intervention; and for the healthcare organi- zation and program itself, including cultural competency and coalition formation. Each of these is addressed in some detail in the sections that follow.

▸ Measurement Measurement occurs throughout the planning and evaluation cycle. Measurement of health status and of factors contributing to the health problem occurs during the community needs assessment phase. Program delivery and partic- ipation measurement occurs during the process evaluation phase. Measurement of program effects occurs during program evaluation. At each of these points in the planning and eval- uation cycle, diversity in the target audience and in program participants or recipients has ramifications regarding what is measured, which data are collected, and how data are collected.

The first consideration is always the pur- pose of measuring an aspect of diversity. This purpose is paramount in deciding how diver- sity will be measured. Imagine that in a Bowe County community assessment, an atheist born in Layetteville and a Muslim born in a neigh- boring town were grouped into the same ethnic category. Stated in this way, it seems strange to

factors. The message is that diversity in culture is related to diversity in illness manifestations and responses to illnesses (Edwards, Moric, Husfeldt, Buvanendran, & Ivankovich, 2005; Hastie, Riley, & Fillingim, 2005), even within ethnic subgroups (Tan et  al., 2008). Such differences can only be identified through cross-cultural comparisons and astute observations. If program planners lack direct personal knowledge of the culture, they will need to rely on key informants and published reports of cultural influences on illness manifestations that are specific to the target audience.

Diversity also exists with regard to the economic well-being of individuals, as measured through socioeconomic status (SES). The relationship between SES and a wide variety of health status indicators has a long, well-documented history (Kosa & Zola, 1975; Polednak, 1997). Research continues to unravel the relationships among education, income, and health. For example, Russell and colleagues (2016) found that mothers with less than high school education had 1.7 times increase in the likelihood of having a child with attention deficit-hyperactivity disorder and that coming from a low income family increased that risk to more 2.2 times. The correlation between SES and health status applies both across racial groups and within racial groups. For example, Bruce and colleagues (2010) found that, among African Americans, those with a higher SES had a lower risk of chronic kidney disease. Other research has found that the interaction among gender, income, and race/ethnicity has varying effects on self-rated health (Brown, Richardson, Hargrove, & Thomas, 2016). The fact that indi- viduals in lower SES groups, regardless of other characteristics, have poorer health suggests that health programs may need to target specific SES groups, not just specific cultural, racial, or ethnic groups.

The attention given to cultural and ethnic diversity is driven, in part, by the numbers. In 2014, over one million individuals newly immi- grated to the United States. Of those, 13% were from Mexico, 8% from India, 8% from China,

34 Chapter 2 Relevance of Diversity and Disparities to Health Programs

nationality by birth or birthplace of the parents is sometimes used as an indicator of culture and ethnicity. Many countries have multiple ethnic groups, however, making it problematic to equate nationality with ethnicity or culture. Thus, if nationality is measured, another measure, such as primary language, may be needed to have a more accurate measure of ethnicity and culture.

The following example demonstrates the importance of carefully choosing indicators of diversity, such as measures of ethnicity or culture, for planning health programs. In one neighborhood of Chicago, a large percentage of the residents belong to a specific sect of Judaism. In this neighborhood, the food stores are kosher, the women’s clothing is consistent with their religion, and friendships are built around syn- agogue membership. Less than half a mile away is another neighborhood with a large percentage of residents with ties to the Indian subcontinent. In this neighborhood, the food stores stock food for their cuisine, the women wear the traditional sari, and the social structure is built around the dominance of the male head of the household. The health statistics for the Jewish neighborhood are relatively good, but the health statistics for the East Indian neighborhood reveal women’s health problems due to high rates of domestic violence and chronic illness related to alcoholism. Unless the data from the two neighborhoods are separated, the health statistics for the area as a whole will mask some of the women’s health problems and understate the males’ health problems related to alcoholism.

This description of two actual neighboring ethnic groups shows the extent to which program planners need to be familiar not only with the data but, more important, with the community characteristics. These characteristics include the cultural beliefs of the residents and the degree to which ethnic and religious diversity coexist rather than overlap. Having this level of understanding about the cultural and ethnic diversity of a community facilitates appropriate interpretation of community health status data.

Race has long been considered a physical characteristic. From a biological perspective, race

assign these two individuals the same ethnicity. But grouping these individuals together makes sense if the purpose of the assessment is to have data on Mexican immigrant culture. Given that ethnicity denotes a set of religious, racial, national, linguistic, or cultural characteristics that define a group, the ethnicity measure in this community needs assessment was based on religion as Catholic or not and on birth- place as Bowe County or not. Thus, non–Bowe County–born Catholics were assigned a Mex- ican ethnicity. This example was intentionally contrived to demonstrate the importance of purpose in developing indicators of diversity and the profound effect the variables used have on the indicator and subsequent findings.

Culture is often implicit, tacit, and not expressed as a distinct factor, making it difficult to measure. In addition, because a dominant culture exists at a societal level, measures of culture are less useful in health programs than indicators of more discrete, smaller subpop- ulations, such as those that might be defined by ethnicity or nationality. For these reasons, ethnicity is used as a proxy for cultural identity. Typically, ethnicity is measured with a single item; however, using a valid and reliable mea- sure of ethnicity is key to having good data for planning and evaluating health programs. The extent of language diversity and religious diversity makes constructing a comprehensive measure of ethnicity very difficult. For example, the large number of religions, religious sects, and churches listed in the U.S. military’s Ministry Team Handbook reflects wide religious diversity, with each having specific dietary practices; clothing; health practices; religious practices; and birth, marriage, and death rituals. Health researchers are attempting to understand the relationship between health status and main- stream religious beliefs and practices (Baetz & Toews, 2009; Masters & Spielmans, 2007; Krause, Emmons, & Ironson, 2015).

Nationality, which identifies the place of birth of the individual or the parents, is a more straightforward measure. Because cultural iden- tity and ethnicity can be difficult to measure,

Measurement 35

transmitted health conditions, it has some med- ical value. The categories used to measure race remain somewhat inconsistent across key national health data sources, as do indicators of ethnicity (TABLE 2-2). Use of race and ethnicity indicators deserves careful attention (Bhopal, 2006).

has historically been associated with specific genetic diseases, including sickle cell anemia, thalassemia, and some forms of lactose intolerance. Race has also been used as a proxy measure of culture, ethnicity, and SES. To the extent that race can be used as a risk factor for specific genetically

TABLE 2-2 Indicators Used to Measure Race in Different Surveys

Category Used

2010 U.S.

Census1

2003 U. S. Standard

Certificate of Live Birth2

2015–2016– NHANES3

2012 National Hospital

Ambulatory Care Survey4

Race

White X X X

Non-Hispanic white X

Non-Hispanic black X

Black, African American X X X

American Indian or Alaska Native X X X X

Other Pacific Islander X

Native Hawaiian or Pacific Islander X X X

Asian Indian X

Asian or Pacific Islander (API) X X

Asian X

Ethnicity

Hispanic or Latino X X X X

Mexican, Mexican American, Chicano

X X

(continues)

36 Chapter 2 Relevance of Diversity and Disparities to Health Programs

Category Used

2010 U.S.

Census1

2003 U. S. Standard

Certificate of Live Birth2

2015–2016– NHANES3

2012 National Hospital

Ambulatory Care Survey4

Puerto Rican X X X

Cuban X X

Other Spanish, Hispanic, Latino X

Other Spanish or Hispanic X

Chinese X

Filipino X

Japanese X

Korean X

Vietnamese X

Other Asian X

Guamanian or Chamorro X X

Samoan X X

Other Pacific Islander X

Central American, South American, other Hispanic or Latino

X

Other race X X X

Sources: 1 U.S. Census. Retrieved June 30, 2016, from http://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid =DEC_10_DP_DPDP1&src=pt. 2 U.S. Standard Certificate of Live Birth. Retrieved June 30, 2016, from http://www.cdc.gov/nchs/nvss/births.htm. 3 National Health and Nutrition Examination Survey (NHANES), Screener Module 1. Retrieved June 30, 2016, from http://www.cdc.gov/nchs/nhanes /nhanes2015-2016/questionnaires15_16.htm. 4 Survey Content for the National Ambulatory Medical Care Survey and Hospital Ambulatory Medical Care Survey. [Revised Nov. 2012] Retrieved June 30, 2016, from http://www.cdc.gov/nchs/ahcd/ahcd_questionnaires.htm.

TABLE 2-2 Indicators Used to Measure Race in Different Surveys (continued )

Measurement 37

The back-translated version was considerably different from the original English version.

In trying to understand what had happened, the researchers found that two factors had come into play. First, the lack of a future tense in some Chinese languages made it impossible to translate directly the English items that asked about the future actions of the respondent. Second, the questionnaire had been designed to measure the degree of individual and group functioning based on the American value of individualism. Thus, the questionnaire was difficult to translate both linguistically because of the future tenses and conceptually because of the individualist versus collectivist values of the two cultures.

This example hints at the potential complexity of using a survey questionnaire designed for one culture with a second culture. It also highlights the potential ethnocentrism involved in thinking that what is valued in the American culture—in this example, individualism—would be relevant in other cultures. This translation story helps explain why such extensive publications exist for the SF-12, a 12-item measure of overall health that is one of the most widely translated health questionnaires and now is also available with only 8 items. The various publications document the SF-12’s psychometric properties when translated and used in different countries, demonstrating that even a widely used and thoroughly researched questionnaire requires a considerable amount of work to ensure that it is culturally and linguistically appropriate with each culture.

Cultural diversity also affects the interpre- tation of findings based on the data collected. Stakeholders involved in the health program who come from different backgrounds and cultures will often hold different values and ideas. Their culturally based interpretations may be quite different from the interpretations of health professionals, who have their own professional culture.

Culture influences how meaning is attributed to findings and how data are collected for program evaluation. For example, a violence prevention

Researchers have made progress in iden- tifying genetic markers for diseases that are more specific than self-reported race. As tests for genetic markers have become more widely available and affordable, race, as currently mea- sured, has lost some medical value. It is easy to imagine a future when the current self-report measures of race will no longer be medically relevant. Until the future arrives, however, race will continue to be used as an indicator in plan- ning health promotion and disease prevention programs.

The cultural and ethnic background of program participants affects the development or choice of questionnaires as well as the interpretation of results. In the development of scientifically sound and rigorous data collection tools, the language and culture of the intended respondents must be considered. To ensure that a questionnaire is culturally and linguistically appropriate and understood requires that the questionnaire be translated from the primary language into the second language and then translated back into the primary language. The back-translated version is then compared with the original version of the questionnaire to determine the accuracy of the translation. In addition, the translation in each direction ought to be done with input from several fully bilingual experts in the content of the questionnaire. Translation of the words is not sufficient; both the ideas embodied in the questionnaire and the wording of each item need to be translated (Epstein, Santo, & Guillemin, 2015; Willis & Zahnd, 2007). The questionnaire needs to be culturally equivalent so that the ideas and the expressions are the same, not just the words.

The following is an example of the mea- surement challenges that program planners and evaluators could face. A questionnaire developed in the United States in English regarding group functioning in a work unit was chosen for use with Taiwanese employees. The questionnaire was translated into Chinese by three Taiwan- ese researchers and then translated back into English by three other Taiwanese researchers.

38 Chapter 2 Relevance of Diversity and Disparities to Health Programs

program measured the program’s effectiveness in terms of the lack of gang tags spray-painted on walls in a neighborhood that was next to a city park. When residents of the neighborhood were presented with the findings, they interpreted the findings in a skeptical manner. They explained that, for them, the lack of gang tags did not mean the lack of gangs, just that they no longer knew where the gang boundaries were and therefore where it was safe to go, including whether it was safe to go to the park for exercise. This actual example exemplifies both the powerful influences of culture on interpreting data and the value of involving stakeholders in any data interpretation.

▸ Interventions Program interventions are the actions done intentionally to have a direct effect on program participants or recipients. The interventions used in health programs must be tailored to the intended audience if the program is to be suc- cessful in achieving the desired health effects. The choice of interventions and manner of intervention delivery ought to be based on both the sociocultural diversity of the target audience and the biological diversity within the target audience. Three approaches are evident in how culture is addressed during the devel- opment of program interventions. In addition, the diversity of the health professionals and health sectors plays a role in the effectiveness of program interventions.

Influences of Sociocultural Diversity on Interventions Fisher, Burnet, Huang, Chin, and Cagney (2007) conducted a literature review of interventions focused on culture as means of improving health. They argue that cultural leverage is a strategy used to improve the “health of racial and ethnic communities by using their cultural practices,

products, philosophies, or environments as vehicles that facilitate behavior change” (p. 245) of individuals and healthcare providers. Cultural leverage, therefore, encompasses culturally tailoring interventions to specific ethnic or cultural groups as well as culturally targeting specific ethnic or cultural groups. In addition, the interventions developed for cultural lever- age are culturally competent (as discussed later in this chapter). The key point is that there is increasing emphasis on, and more sophisticated approaches to, addressing culture in ways that are appropriate and beneficial to improving health and decreasing health disparities.

Understanding how to tailor the program given cultural differences begins with having or collecting information about differences across and within cultural groups. Navarro, Wilson, Berger, and Taylor (1997), in providing Native American students with a program to prevent alcohol and substance abuse, found that tribal differences and conflicting religious themes among tribes were important to individuals participating in the program. This is not surprising given that more than 500 Native American languages exist, each associated with a different tribal culture. Among low-income, urban, African American women, Beckjord and Klassen (2008) found variations in cultural values such that the women with more traditional values were less likely to seek and receive breast cancer screening. Sometimes, however, cultural tailoring may not be necessary. For example, Bond et al. (2016) found similar knowledge and perceptions about human pap- illomavirus (HPV) among African Americans, Caucasians, and Hispanics living in the same town in South Carolina. A health information campaign in this area intended to increase HPV vaccination thus would not need to culturally tailored with regard to race or ethnicity.

A common practice is to incorporate faith into health programs and to design programs that are church or parish based. In a review of the literature on church-based health promotion programs, Campbell and colleagues (2007) found that this approach is effective for African

Interventions 39

examples of how diversity can affect program interventions.

Influences of Biological Diversity on Interventions For some health conditions, physiological responses may vary by race, gender, or age, which in turn affects decisions about the type and intensity of interventions used in the health program. Gener- ational differences in values, norms, beliefs, and health problems all contribute to diversity. From the perspective of health program planning, age distribution is an important factor in reaching the intended audiences of a program. Gender and sexual orientation are other dimensions of physical diversity that have ramifications for program development. Disability—whether physical, mental, or developmental—is another dimension of diversity but is less often mentioned. Nonetheless, it may be extremely relevant for some health programs.

The distribution of physical characteristics within a population or community influences decisions during health program planning and later during program evaluation. Take age as an example. Imagine that the Bowe County Board wants to increase the physical activity of all county residents. The age distribution across the com- munity and within its towns will affect the nature and content of the countywide media messages. Messages that relate to the physical abilities of the elderly will need to be quite different from messages that address the physical abilities of adolescents. Similar considerations would be needed for the other types of physical diversity.

Approaches to Developing Programs Various perspectives exist in regard to explain- ing patterns of health behavioral differences by culture, ethnicity, and race. Kim, McLeod, and Shantzis (1992) suggest that three approaches are used in health-related programs: cultural content

Americans. The practice of collaborating with spiritual leaders and basing health programs in places of worship is likely to continue as an approach to reducing disparities. Developing faith-based programs may or may not require first understanding the health disparity in terms of religion, but it certainly does require understanding ways to collaborate effectively with church leadership and members for the delivery of a health program.

Interventions may result in immediate or permanent changes, but most health behavior interventions are intended to change behav- iors that must be sustained over time. Culture can affect whether behaviors are sustained. Potentially, one type of culturally tailored intervention might be needed to initiate change and another type of culturally appropriate and tailored intervention might be needed to maintain the change or program effects. Also, program outcome objectives need to be culturally appropriate, with correspondingly appropriate target levels.

Sociocultural influences on intervention may emerge in unanticipated ways, such as through program participants themselves. For example, program participants bring their cul- ture to the program in ways that can affect the intervention and its effectiveness. Higginson (1998) studied adolescent mothers in a high school program to examine their competitive culture, concluding that it was shaped by the mothers’ social class, age, and race. Their competitive culture also pervaded the beliefs and norms of the health program in which they participated. These adolescents socialized new program participants into the competitive culture, thereby creating a “program culture.”

One way to understand the culture of a program is to analyze the ceremonies and rituals associated with it. Intentionally building ceremo- nies and rituals into programs can be important for high-risk groups whose members need the sense of belonging that comes with having a program culture. This can help to retain them as active program participants. These are some

40 Chapter 2 Relevance of Diversity and Disparities to Health Programs

of the target population ought to assess the degree of cultural conflict. Program planners need to address the immediate, root causes of the cul- tural conflict if they are to develop appropriate interventions for the health program.

Profession and Provider Diversity Health program planning and evaluation draw on the expertise of individuals from a multitude of health disciplines, including medicine, nurs- ing, pharmacy, social work, nutrition, physical therapy, and dentistry, as well as social science disciplines, including health education, health psychology, social demography, and medical sociology. Each discipline has its own specialized knowledge, values, and professional norms. Successful planning, implementation, and eval- uation of health programs require working on teams that bring together the strengths of the various professions and that respect the differ- ent educational backgrounds of team members (TABLE 2-3). Each health discipline speaks a slightly different professional language, holds different beliefs about how to identify and address health problems, and adopts a different perspective on what constitutes a health outcome. To tap into the wealth of information and experience available through professional diversity requires that the team develop a common language and shared goals for the health program.

Health professionals do not reflect the diver- sity profile of the population of the United States in terms of cultural, racial, and ethnic diversity. For example, African Americans accounted for 10.7% of registered nurses in 2010, yet African Americans made up 13.6% of the overall popu- lation. Similarly, 5.4% of registered nurses were Hispanic, compared to 15.5% of the total popu- lation (U.S. Department of Health and Human Services [U.S. DHHS], 2015). This same pattern of underrepresentation of minorities exists across all health professions. The ensuing lack of racial and ethnic diversity among health professionals creates a cultural gap between professionals and patients, clients, and program participants. The extent of the cultural gap between planners and

approaches, cultural integration approaches, and cultural conflict approaches.

In the cultural content approach, cultural backgrounds and norms are viewed as leading to behaviors and illnesses. For example, Kleinman (1980), a medical anthropologist, explains that illness is cultural in that sickness and symptoms are saturated with specific meaning and are given patterns of human behavior. The notion that illness is cultural, and not just biological, affects the degree to which individuals accept professional explanations of health and illness.

Cultural integration approaches to devel- oping health programs focus on acculturation. Acculturation, the adoption and assimilation of another culture, affects behavior in that the less dominant group takes on behaviors of the dominant group. When planning programs, planners need to consider the degree of acculturation because it affects health beliefs and behaviors. Behavior is also affected when individuals identify with more than one culture to varying degrees so that bicultural individuals have health beliefs and behaviors that are a blend of the dominant and less dominant cultures. When targeting groups or individuals who identify with more than one culture, planners need to understand their health beliefs and behaviors as a “new” culture. This is particularly relevant for health programs target- ing immigrants or first-generation U.S. citizens.

Cultural conflict approaches underscore conflict as the genesis of behaviors. Several areas of potential cultural conflict exist. One area stems from the generation gap, which leads to family conflict and unhealthy behaviors and illnesses. Differences between the role expec- tations of different cultures are another source of cultural conflict and unhealthy behaviors. Racism, oppression, and lack of political power lead to alienation and identity conflict, and sub- sequently to unhealthy behaviors and illnesses. From a psychological perspective, individuals who are experiencing these kinds of conflicts are more likely to experience stress and there- fore have less attention and energy to engage in health-promoting behaviors or may be less receptive to making change. Thus an assessment

Interventions 41

TA B

LE 2

-3 P

ro fe

ss io

na l D

iv er

sit y A

m on

g He

al th

P ro

fe ss

io ns

H ea

lt h

D is

ci pl

in e

Av er

ag e

Ed uc

at io

n Pr

im ar

y Fo

cu s

Li ce

ns ur

e/ Ce

rt if

ic at

e Pr

og ra

m m

at ic

Co

nt ri

bu ti

on

Es ti

m at

ed

N um

be r i

n U

ni te

d St

at es

A cu

p u

n ct

u ris

t M

in im

u m

3

ye ar

s o

f A

si an

m ed

ic in

e ed

u ca

ti o

n

R es

to re

b al

an ce

b et

w ee

n t

h e

p h

ys ic

al , e

m o

ti o

n al

a n

d s

p ir

it u

al

as p

ec ts

o f t

h e

p er

so n

Li ce

n su

re

b as

ed o

n

ce rt

ifi ca

ti o

n

A lt

er n

at iv

e ap

p ro

ac h

t o

u

n d

er st

an d

in g

h ea

lt h

a n

d

ill n

es s

b as

ed o

n o

ri en

ta l

m ed

ic in

e

27 ,8

35 in

2 01

23

C o

m m

u n

it y

h ea

lt h

w o

rk er

H ig

h s

ch o

o l o

r b

ac ca

la u

re at

e In

fo rm

al e

d u

ca ti

o n

, a d

vo ca

cy ,

as si

st an

ce o

f c o

m m

u n

it y

m em

b er

s

Va ri

es b

y st

at e

fo r

ce rt

ifi ca

ti o

n Sh

ar es

e th

n ic

, l in

g u

is ti

c,

so ci

o ec

o n

o m

ic s

ta tu

s,

an d

li fe

e xp

er ie

n ce

s w

it h

co

m m

u n

it y

m em

b er

s

48 ,1

30 in

2 01

51

D en

ti st

ry D

en ta

l- d

o ct

o ra

te

(3 –4

y ea

rs a

ft er

b

ac ca

la u

re at

e)

D ia

g n

o si

s an

d t

re at

m en

t o

f co

n d

it io

n s

o f t

h e

te et

h a

n d

g u

m s

Li ce

n su

re O

ra l h

ea lt

h k

n o

w le

d g

e 19

5, 72

2 in

2 01

52

D ie

ti ti

an ,

n u

tr it

io n

is t

Ba cc

al au

re at

e D

ie ta

ry a

n d

n u

tr it

io n

al e

le m

en ts

n

ec es

sa ry

fo r

h ea

lt h

Li ce

n su

re a

n d

ce

rt ifi

ca te

N u

tr it

io n

al k

n o

w le

d g

e,

in flu

en ce

o f n

u tr

it io

n o

n

h ea

lt h

59 ,7

40 in

2 01

51

H ea

lt h

ad

m in

is tr

at io

n M

as te

r’s d

eg re

e Le

ad er

sh ip

a n

d m

an ag

em en

t o

f h

ea lt

h ca

re o

rg an

iz at

io n

s C

er ti

fic at

e M

an ag

em en

t an

d

ad m

in is

tr at

io n

16 6,

99 0

in 2

01 51

H ea

lt h

ed

u ca

ti o

n Ba

cc al

au re

at e

D ev

el o

p m

en t

an d

d el

iv er

y o

f m at

er ia

ls a

n d

c u

rr ic

u lu

m

d es

ig n

ed t

o im

p ar

t h

ea lt

h

kn o

w le

d g

e an

d c

h an

g e

b eh

av io

r

C er

ti fic

at e

So ci

al a

n d

b eh

av io

ra l

kn o

w le

d g

e 57

,5 70

in 2

01 51

(c on

tin u

es )

42 Chapter 2 Relevance of Diversity and Disparities to Health Programs

TA B

LE 2

-3 P

ro fe

ss io

na l D

iv er

sit y A

m on

g He

al th

P ro

fe ss

io ns

(c on

tin ue

d)

H ea

lt h

D is

ci pl

in e

Av er

ag e

Ed uc

at io

n Pr

im ar

y Fo

cu s

Li ce

ns ur

e/ Ce

rt if

ic at

e Pr

og ra

m m

at ic

Co

nt ri

bu ti

on

Es ti

m at

ed

N um

be r i

n U

ni te

d St

at es

M as

sa g

e th

er ap

is t

C er

ti fic

at e

M as

sa g

e o

f s o

ft t

is su

es a

n d

jo in

ts Li

ce n

su re

va

ri es

b y

st at

e,

ce rt

ifi ca

te

92 ,0

90 in

2 01

51

M ed

ic in

e M

ed ic

al d

oc to

ra te

(4 y

ea rs

a ft

er

b ac

ca la

u re

at e)

, p

lu s

re si

d en

cy o

f 3–

4 ye

ar s

D iff

er en

ti al

d ia

g n

o si

s an

d

tr ea

tm en

t o

f i lln

es se

s Li

ce n

su re

a n

d

ce rt

ifi ca

te M

ed ic

al , p

at h

o lo

g ic

al , a

n d

tr

ea tm

en t

kn o

w le

d g

e 62

1, 06

6 in

2 01

51

N u

rs in

g Ba

cc al

au re

at e

(a ss

o ci

at e

d eg

re e

m in

im u

m

re q

u ire

m en

t)

Pr o

m o

ti o

n o

f h ea

lt h

a n

d w

el l-

b ei

n g

b as

ed o

n s

ci en

ti fic

kn

o w

le d

g e

Li ce

n su

re a

n d

ce

rt ifi

ca te

In te

g ra

ti o

n o

f b eh

av io

ra l

an d

m ed

ic al

k n

o w

le d

g e

2, 74

5, 91

0 in

20

15 1

Ph ys

ic al

th

er ap

y Ba

cc al

au re

at e

R es

to ra

ti o

n a

n d

m ai

n te

n an

ce

o f b

o d

y st

re n

g th

a n

d fl

ex ib

ili ty

fo

r th

e p

u rp

o se

o f m

ax im

iz in

g

p h

ys ic

al c

ap ab

ili ti

es

Li ce

n su

re Fo

cu s

o n

e n

h an

ci n

g

ca p

ab ili

ty w

it h

in li

m it

at io

n s

20 9,

69 0

in 2

01 51

So ci

al w

o rk

M as

te r’s

d eg

re e

A d

d re

ss b

as ic

n ee

d s;

h el

p p

eo p

le

m an

ag e

en vi

ro n

m en

ta l f

o rc

es t

h at

cr

ea te

p ro

b le

m s

in li

vi n

g

Li ce

n su

re Fo

cu s

o n

fa m

ily a

n d

p

sy ch

o lo

g ic

al fa

ct o

rs 61

9, 31

0 in

2 01

51

So ur

ce s:

1 C om

m un

ity H

ea lth

W or

ke rs

N at

io na

l W or

kf or

ce S

ur ve

y. Re

tri ev

ed S

ep te

m be

r 1 1,

2 01

2, fr

om h

tt p:

// bh

pr .h

rs a.

go v/

he al

th w

or kf

or ce

/r ep

or ts

/c hw

st ud

y2 00

7. pd

f; 2

Bu re

au o

f L ab

or S

ta tis

tic s (

BL S)

. ( 20

11 ).

Oc cu

pa tio

na l O

ut lo

ok H

an db

oo k

(O OH

), 20

10 –2

01 1

Ed iti

on . R

et rie

ve d

Se pt

em be

r 1 1,

2 01

2, fr

om w

w w

.b ls.

go v/

oc o/

ho m

e. ht

m ;

3 D at

a no

t a va

ila bl

e on

th es

e he

al th

d isc

ip lin

es .

b B ur

ea u

of La

bo r S

ta tis

tic s (

BL S)

. ( 20

11 ).

Oc cu

pa tio

na l O

ut lo

ok H

an db

oo k

(O OH

), 20

10 -2

01 1

Ed iti

on . R

et rie

ve d

Se pt

em be

r 7 , 2

01 1,

fr om

h tt

p: //

w w

w .b

ls. go

v/ oc

o/ ho

m e.

ht

Diversity Within Healthcare Organizations and Programs 43

this sector—curanderos, espiritistas, santeros, singers, shamans, and root-workers, among others. Evidence of the presence of folk healers can be found when visiting neighborhoods that are ethnically isolated or that maintain folkloric traditions. This would be case with enclaves of recent immigrants. Individuals may consult folk healers while also receiving modern or Western health care. The theories of illnesses and diseases that are the basis of folk health practices can conflict with allopathic theories and thus may diminish the effectiveness of interventions based on an allopathic frame of reference. The role of folk healers in community health behaviors and in addressing health problems can be central for some health programs, especially those targeting individuals who have maintained “the old ways.”

The third (and largest) sector of health pro- viders is the popular or lay sector, consisting of family and friends. Undoubtedly, most of us talk to a family member or friend about our illness before seeking either professional or folk health care. This sector is the most relied upon, from receiving the latest news disseminated through the mass or social media to getting a mother’s recipe for chicken soup. Health information is spread through the lay sector through social networks, making it a powerful factor in influencing health knowledge and behavior. Health programs that seek to change social norms or population-level behaviors are essentially seeking to change the lay healthcare sector.

▸ Diversity Within Healthcare Organizations and Programs

From a systems theory perspective, an organi- zation that is internally diverse will be better able to respond to externally diverse needs and demands. This concept has been formalized into the concept of requisite variety (Weick, 1979). The concept of requisite variety suggests that

a health program’s target audience contributes to a reduced understanding of the target audi- ence, a greater need to become informed about the target audience, and—potentially—tensions between the planners and advocates for or from the target audience. The more comprehensive the health program and the greater the cultural diversity of the target population, the greater the need to have parallel diversity among those planning, providing, and receiving the program.

The Three Health Provider Sectors From an anthropological perspective, the effects of health provider diversity (or lack thereof) can be understood by considering the three sectors of the health–illness system from which individuals seek help when experiencing illness (Kleinman, 1980). Each sector has direct implications for planning, implementing, and evaluating health programs.

One sector consists of allopathic, naturo- pathic, and other formally trained health pro- fessionals who make up the medical healthcare system. Professionals from this sector have legally sanctioned practice parameters. Tradi- tional, Western medical professions include, among others, physicians, nurses, pharmacists, psychologists and physical therapists, whereas complementary health professionals include, for example, naturopathic physicians, chiroprac- tors, acupuncturists, homeopathy practitioners, licensed massage therapists, and mind–body therapists. The notion of health program plan- ning falls within this sector, as do the methods and knowledge about health program planning and evaluation. In addition, the preponderance of health programs are designed in accordance with theories and knowledge generated from this sector.

A second sector from which individuals might seek help is the folk healthcare sector, which comprises nonprofessional, secular, or sacred healers who have not received formal education but who are very likely to have received training through some type of apprenticeship. A wide variety of traditional healers makes up

44 Chapter 2 Relevance of Diversity and Disparities to Health Programs

within an organization as the variation in the social and cultural identities of people existing together. For organizations, diversity provides added value because it increases respect, improves problem solving, increases creativity and ideas, increases organizational flexibility, improves the quality of employees, and improves marketing strategies. Diversity within organizations does not just create benefits, however; it also poses challenges for managing and enhancing that diversity.

An essential element contributing to a healthcare organization’s cultural competency is its ability to engage in self-assessment of its cultural competency. Understanding the cultural competency continuum puts the self-assessment into perspective.

Cultural Competency Continuum Accompanying the emphasis on diversity and health disparities is the emphasis on cultural competency, the extent to which individuals are able to live or work in a culture other than their own. Cultural competency, by its very nature, has shades of less and more that extend along a continuum (Cross, Bazron, Dennis, & Issacs, 1989; Orlandi, 1992; TABLE 2-4), an idea that has gained wide acceptance (e.g., Knibb-Lamouche, 2012). It is possible for health professionals and program staff members to reside at different points along the continuum, depending on a variety of factors, such as the specific circumstances and the individuals’ experiences with cultures other than their own. While the prevailing norm and politically correct stance is to be as culturally sensitive and as competent as possi- ble, acceptance of different values and beliefs can be difficult, particularly those of cultures that are dramatically different from one’s own.

Cultural Destructiveness At the least tolerant end of the continuum is cultural destructiveness (Orlandi, 1992), which includes a set of attitudes and practices that explicitly promote one culture over another based on the notion of one culture being superior to the

healthcare organizations with a culturally diverse and culturally competent workforce are better suited to provide services that meet culturally diverse health needs. The need for requisite variety is a fundamental reason for having a culturally and ethnically diverse health professions sector. The need for a diverse workforce was recognized in a report to the Bureau of Health Professions, within the Health Resources and Services Admin- istration (U.S. DHHS, 2015), especially to benefit underserved and minority populations.

Organizational Culture Many different types of organizations offer health programs, including state or local health agencies, for-profit acute care networks, nonprofit community-based agencies, and academic institu- tions. Each organization has a unique set of values, norms, and beliefs that are collectively held by its members and that are passed on to new employees; this constitutes the organizational culture (Deal & Kennedy, 1982; Schein, 1995). Well-known exam- ples of organizational culture are the norms about starting meetings on time and the willingness to help other employees accomplish tasks.

Program managers need to be sensitive to the degree of fit between the organizational culture and the goals of the health program. Not all good ideas for programs are good for the organization. A good match or fit between the organization’s view of its mission and philosophy—in other words, its beliefs and values—and the purpose of the health program may be important to the success of the health program in terms of financial, personnel, and other organizational support. In a similar vein, the integration and sustainability of a program within an organization are affected by organizational culture.

Another implication of organizational culture for program managers is that staff members with work experience hold some of the values and norms of their prior organizational culture. These values and norms can be shaped; in other words, new employees need to become acculturated into the new organization, a process that begins with their initial orientation. Cox (2001), an expert on multicultural organizations, defined diversity

Diversity Within Healthcare Organizations and Programs 45

TA B

LE 2

-4 C

ul tu

ra l C

on tin

uu m

w ith

E xa

m pl

es o

f t he

D ist

in gu

ish in

g Fe

at ur

es o

f E ac

h St

ag e

Cu lt

ur al

De

st ru

ct iv

en es

s Cu

lt ur

al

In ca

pa ci

ty Cu

lt ur

al

Bl in

dn es

s Cu

lt ur

al

Op en

ne ss

Cu lt

ur al

Co

m pe

te nc

e Cu

lt ur

al

Pr of

ic ie

nc y

A tt

it u

d e

t o

w a

rd

o th

e r

cu lt

u re

s H

o st

ili ty

D is

lik e,

s ep

ar at

e b

u t

eq u

al A

m b

iv al

en ce

, tr

ea t

al l a

lik e

C u

ri o

u s,

c u

lt u

ra l

aw ar

en es

s R

es p

ec t

an d

to

le ra

n ce

, c u

lt u

ra l

se n

si ti

vi ty

Fu lly

co

m fo

rt ab

le ,

cu lt

u ra

l at

tu n

em en

t

K n

o w

le d

g e

o f

o th

e r

cu lt

u re

s A

ct iv

e av

o id

an ce

o

f k n

o w

le d

g e

N o

n e

Li tt

le o

r n

o n

e So

m e

Fa ir

am o

u n

t Ex

te n

si ve

D e

g re

e o

f in

te g

ra ti

o n

a

cr o

ss c

u lt

u re

s

N o

n e

N o

n e

N o

n e

C o

n te

m p

la ti

o n

o f

p o

te n

ti al

b en

ef it

s o

f i n

te g

ra ti

o n

So m

e in

te g

ra ti

o n

, so

m e

el em

en ts

o

f m u

lt ic

u lt

u ra

l in

te g

ra ti

o n

Ex te

n si

ve

in te

g ra

ti o

n ,

fu lly

m

u lt

ic u

lt u

ra l,

fu si

o n

o f

cu lt

u re

s

Im p

li ca

ti o

n s

fo r

h e

a lt

h p

ro g

ra m

o

f p

a rt

ic ip

a n

ts

a t

e a

ch s

ta g

e

Pr o

g ra

m s

ad d

re ss

co

n se

q u

en ce

s o

f c u

lt u

ra l

d es

tr u

ct iv

en es

s

N ee

d t

o h

av e

p ro

g ra

m s

p ro

vi d

ed t

o

se p

ar at

e g

ro u

p s

If h

av e

m u

lt ic

u lt

u ra

l el

em en

ts , m

ay

n ee

d t

o ju

st ify

an

d e

xp la

in

C an

p ro

vi d

e p

ro g

ra m

t o

p

ar ti

ci p

an ts

fr

o m

m u

lt ip

le

cu lt

u re

s b

u t

w ill

n

ee d

t o

p ro

vi d

e in

fo rm

at io

n a

n d

ro

le m

o d

el in

g o

f co

m p

et en

ce

C an

p ro

vi d

e p

ro g

ra m

t o

p

ar ti

ci p

an ts

fr o

m

m u

lt ip

le c

u lt

u re

s w

it h

m in

im al

ad

ju st

m en

ts

C an

p ro

vi d

e m

u lt

ili n

g u

al ,

m u

lt ic

u lt

u ra

l in

te rv

en ti

o n

s

in o

n e

p ro

g ra

m

46 Chapter 2 Relevance of Diversity and Disparities to Health Programs

other. The attitude of superiority of one’s culture over the inferior culture stems from the notion of the other being different or distasteful. Often physical (visible) characteristics, especially race, gender, sexual orientation, and age, are used as the basis for cultural destructiveness. Although staff members of a health program are not likely to be at this end of the continuum, health programs might be needed by and planned for individuals with attitudes reflective of cultural destructiveness. In fact, many of the global conflicts that lead to humanitarian crises and refugees have their roots in cultural destructiveness. International health programs are likely to deal directly with the conse- quences of cultural destructiveness. For programs within the United States, program planners will need to have an “insider” understanding of factors that would make the health program acceptable to culturally destructive groups.

Cultural Incapacity Individuals at the next step, cultural incapacity, also promote one culture over another, albeit more implicitly than individuals at the cultural destructiveness stage. Cultural incapacity is manifested in the doctrine of “separate but equal,” with the accompanying segregation and discrimination. In the United States, both cultural incapacity and cultural destructiveness have been made illegal through constitutional, federal, and various state statutes.

Cultural Blindness Cultural blindness is a perspective of being unbi- ased, such that people are viewed as being alike and consequently are treated alike. At this point, the definition of “alike” is based on the dominant culture, giving cultural blindness ethnocentric overtones. Historically, health programs sought and delivered universal solutions without regard to different communication patterns of different cultures (Airhihenbuwa, 1994). Treating everyone in an unbiased manner would seem to be a reasonable premise for a health program. Cultural blindness, however, does not lead to effective programs.

One explanation for this phenomenon, taken from educational psychology, centers on the role of the dominant culture. Boekaerts (1998) suggests that because culture affects self-constructs, it also affects key features of how individuals learn and process information. As a result, what may be an effective learning environment for members of the dominant culture may not be effective for members of the less-dominant culture, who are being treated like members of the dominant culture. This theory implies that health programs, especially those with education or learning components that are based on a cultural blindness perspective, are not likely to be effective for individuals who are not from the dominant culture.

Another way of thinking about the conse- quences of cultural blindness is by acknowledging its failure to recognize that ideas and concepts are not the same across cultures due to the dif- ferences in self-constructs and learning. From this perspective, the earlier discussion of the need to translate concepts used in questionnaires is another example of how to overcome cultural blindness and its potential consequences for health program planning and evaluation.

Cultural Openness Cultural openness is the attitude of being recep- tive to a different culture and to active learning about other cultures. Although other cultures are valued and some knowledge of other cultures exists, cultural openness does not include any integration of cultures or cross-pollination of cultural ideas. In this regard, cultural openness is similar to cultural awareness. Each culture is valued and understood as separate and distinct.

An example of being culturally open is someone from a dominant white culture going to a local Native American powwow or to an inner-city black evangelical church service sim- ply to observe what happens. Cultural openness in health programs would be evident in having minority representation on community or advisory boards for the health program, using consultants with expertise in cultural awareness, and providing

Diversity Within Healthcare Organizations and Programs 47

knowing.” Her work has influenced thinking across health disciplines (e.g., Jackson & Samuels, 2011). Hoskins’s principles are notably developed for members of the dominant culture, with the implicit expectation that the member of the dominant culture needs to become culturally competent. In other words, it is incumbent upon the member of the dominant culture to strive for cultural competence. These principles also reveal that cultural competence, as a set of behaviors, may be difficult to attain or maintain over time.

The Lewin Group (2001), writing in a report for the Health Resources and Services Adminis- tration (HRSA) on cultural competence, listed nine domains for measuring cultural competence for healthcare organizations: values and attitudes of mutual respect and regard; cultural sensitivity; communication; policies and procedures (i.e., hiring staff members who reflect the linguistic and cultural diversity of the community); training and staff development; facility characteristics; capacity and infrastructure (i.e., cultural appro- priateness of the physical environment, materials, and resources; use of posters and brochures with representatives from different races and ethnicities); intervention and treatment features; and community and consumer involvement and participation in decision making, and monitor- ing and evaluation of research. These domains continue to be relevant and used as the basis for organizational self-assessments.

This list of domains hints at the corresponding amount of work needed to achieve and maintain a culturally competent organization and workforce. These same domains clearly apply to programs.

Cultural Proficiency At the most culturally capable end of the cultural competency continuum is cultural proficiency, which involves proactively seeking knowledge and information about other cultures, as well as educating others about other cultures. Cultural proficiency, as with any end point on a con- tinuum, is difficult to achieve and may not be sustained for a long period of time. Those rare individuals who can move seamlessly among

cultural sensitivity training for staff members. Such culturally open practices increase the likelihood that the health program will be culturally appro- priate, but they do not ensure its appropriateness. To ensure that the health program is culturally appropriate requires actively seeking information and integrating that information into the design, delivery, and evaluation of the health program. This process requires cultural competence.

Cultural Competence Cultural competence encompasses not only demonstrating respect for other cultures but also actively seeking advice and consultation from members of the less dominant cultural group about what is culturally appropriate from their perspective. Acting in a culturally competent manner requires various skills that one needs to acquire intentionally. These skills are more specific than listening and being respectful. Continuing with the Native American example, if a tribal healer is consulted and included as a full member in the planning team for a health program intended for members of his tribe, then the health planning team is exhibiting culturally competent behaviors, especially if the healer’s approach to healing is included in the program. Generally, cultural competence is understood as an individual characteristic of providers. For example, in a study of medical clinics, Paez, Allen, Carson, and Cooper (2007) found that more culturally competent provider behavior was associated with the clinic having more nonwhite staff members and more culturally adapted patient education materials.

One challenge to understanding what constitutes cultural competence is that other terms may be used to describe it, such as “cul- tural sensitivity” and “cultural attunement.” Both sensitivity and attunement can be viewed as elements of cultural competence. Hoskins (1999) has proposed five principles of cultural attunement: acknowledging the pain of oppres- sion by the dominant culture, engaging in acts of humility, acting with reverence, engaging in mutuality, and coming from a place of “not

48 Chapter 2 Relevance of Diversity and Disparities to Health Programs

through several strategies other than hiring consultants or sending staff members for cultural competency training. Cox (2001) has stressed that to have a diverse, friendly organization, workplace, or program requires making system- wide changes, affecting everything from hiring policies to the physical structure of the workplace, that are aligned with valuing and respecting the diversity of personnel.

For example, before making plans for orga- nizational system changes, an organizational or program self-assessment of cultural competency is warranted. A variety of assessment tools have been developed and validated for assessing health- care employees (e.g., Loftin, Hartin, Branson, & Reyes, 2013). In addition, the National Center for Cultural Competence (Cohen & Goode, 1999) has developed a simple checklist (EXHIBIT 2-1) for use by program planners as well as by other individuals who have roles in shaping policy

cultures, be accepted in those cultures, and act as ambassadors of multiple cultures would be considered culturally proficient.

Being multicultural—that is, fully accepting and integrating two or more sets of cultural values and beliefs—is a manifestation of cultural proficiency. Multiculturalism in an organiza- tion or program (Cox, 1991) is the extent to which different cultures are fully integrated. It is manifested in programs that integrate folk or professional practitioners and treatment options, have predominantly bicultural staff, celebrate holidays important to cultural groups involved in the program, and synthesize different cultural beliefs into the program plan and implementation.

Enhancing Cultural Competency Program managers can enhance cultural sensitivity, cultural awareness, and cultural competencies

EXHIBIT 2-1 Checklist to Facilitate Development of Cultural and Linguistic Competence Within Healthcare Organizations

Does the healthcare organization, primary healthcare system, or program have:

■ A mission statement that articulates its principles, rationale, and values for culturally and linguistically competent healthcare service delivery?

■ Policies and procedures for staff recruitment, hiring, and retention that will achieve the goal of a diverse and culturally competent workforce?

■ Position descriptions and personnel performance measures that include skill sets related to linguistic competence?

■ Policies and resources to support ongoing professional development and in-service training (at all levels) related to linguistic competence?

■ Policies, procedures, and fiscal planning to ensure the provision of translation and interpretation services? ■ Policies and procedures regarding the translation of patient consent forms, educational materials,

and other information in formats that meet the literacy needs of patients? ■ Policies and procedures to evaluate the quality and appropriateness of interpretation and translation

services? ■ Policies and procedures to periodically evaluate consumer and personnel satisfaction with interpretation

and translation services that are provided? ■ Policies and resources that support community outreach initiatives to persons with limited English

proficiency? ■ Policies and procedures to periodically review the current and emergent demographic trends for

the geographic area served in order to determine interpretation and translation services needs?

Included with permission of the Georgetown University National Center for Cultural Competence, Georgetown University Center for Child & Human Development, Georgetown University Medical Center.

Diversity Within Healthcare Organizations and Programs 49

cultural competency. Ignoring the difficulties inherent in having diversity can lead to further problems; therefore, the challenges inherent in moving an organization, a program, or an individual toward cultural competency need to be acknowledged and addressed in a forthright yet sensitive manner.

Another strategy for enhancing the cul- tural competency of program personnel is to make diversity visible. This effort might include displaying posters or cultural artifacts. It may also include making available to staff members professional journals with a health and culture focus, such as American Indian Culture and Research Journal, Ethnicity and Disease, International Journal of Intercultural Relations, Journal of Black Psychology, Journal of Cross-Cultural Psychology, Journal of Health Care for the Poor and Underserved, and Journal of Multicultural Counseling and Development. The high visibility of diversity in the workplace becomes a symbol that reflects the organiza- tional culture of valuing and respecting cultural diversity.

Rea, Martin, & Wright (2002) offer realistic suggestions for addressing cultural issues in the workplace. They recommended avoiding open conflicts over cultural issues, especially given that no one “right way” exists. They also recommend working through informal commu- nication channels when cultural issues need to be addressed or to achieve changes in organi- zational culture. Another realistic suggestion is for managers to focus on reinforcing those new behaviors that promote cultural competency and sensitivity. Acknowledging that individuals have personalities and thus rotating staff members to other work units or programs may be the best approach in some situations. The positive aspect of this last suggestion recognizes a hard truth: When a fit between the program and staff members does not exist, both parties may benefit from a change in the relationship. The trick to addressing this type of situation in a culturally and legally competent manner is for both parties to understand the issue as one of fit and not as a personal judgment.

at the federal, state, or local levels. Using this checklist can help determine which areas are in need of attention (Goode, Jones, & Mason, 2002), with actions subsequently being taken to enhance the cultural competency of staff members and the program as a whole.

Enhancing the cultural competency of program staff members begins with recruiting minorities that mirror the composition of potential program participants. Staffing pattern includes having a diverse workforce as a venue for staff members to learn from each culture—that is, from other staff members. The beliefs, values, and goals of individual staff members need to be congruent with those of the organization, which results in the second factor, fit between an individual and the organization is well accepted as an appro- priate criterion for hiring decisions (Cable & Judge, 1997; McCulloch & Turban, 2007) and may entail sensitive hiring decisions. Program personnel must believe in achieving cultural competency for all program staff members, not just themselves or other staff.

Cultural competency training sessions ought to be designed to overcome learner resistance and avoid creating stereotypes (Boutin-Foster, Foster, & Konopasek, 2008). One strategy to use with individual program staff members is to make it acceptable to ask questions about cultural beliefs, practices, and norms so that staff members can acquire the information necessary to become more culturally competent. Program personnel need to be able to express both their comfort and their discomfort with other cultures as a step toward receiving whatever information or counseling is needed to overcome the discom- fort. Out of respect, cultural labels ought to be avoided, using instead objective descriptors or names of individuals.

Not all staff members will be equally accepting and competent with all other cul- tures, depending on their cultural background. Some cultures are more accepting and seeking of new experiences than others. Being alert to cultural differences within program staff members is an important step toward develop- ing and ensuring organizational and program

50 Chapter 2 Relevance of Diversity and Disparities to Health Programs

characteristics were associated with greater coalition effectiveness: use of formal rules and procedures, an inclusive leadership style, par- ticipation by members, a diverse membership, collaboration with agencies, and group cohesion.

The process of forming a coalition follows commonsense, deceptively simple steps. At the core of a coalition is attention to group process, as the following discussion suggests. The initial step in forming a coalition is to identify potential coalition members who are either individual stakeholders or representatives of organizations with a potential stake in the healthcare pro- gram. Naturally, the potential members ought to reflect the diversity being addressed by the health program.

An early step is the task of articulating the common goal for the coalition. Coalitions are more likely to succeed if they have a defined goal with specific tasks that can be realistically accomplished with minimal expense. As coalition members, funding priorities, and leadership changes, and as time passes, the goal for which the coalition was established will need to be reiterated as a sounding board for decisions and directions. It is also worth noting that coalitions have a life cycle, which may begin with a programmatic focus but evolve to have a policy focus (Hill et al., 2007).

Also early in the formation of the coalition, program staff members must build credibility and trust both within the coalition and with stakeholders in its work. It takes time to build trusting and credible relationships, which are inevitably tested over time. Credibility and trust are extremely difficult to recover if lost. The credibility and trustworthiness of organizers are especially important considerations when work- ing with culturally and ethnically diverse groups whose members have had negative experiences with coalitions or health programs in the past.

Rose (2000) suggested two strategies for building relationships in the coalition. One approach is to adopt issues of the coalition members as issues for the coalition. This strat- egy would be feasible when issues overlap—say, housing affordability and health programs for the homeless. The other strategy is to promote

▸ Stakeholders and Coalitions

Another key approach to achieving requisite variety is through the inclusion of diverse stakeholders in the process of planning and evaluating the health program, which is often accomplished through the development of coalitions. Several federal agencies, such as the Office of Minority Health and the Centers for Disease Control and Prevention (CDC), and private foundations, such as the W. K. Kellogg Foundation and the Robert Wood Johnson Foundation, have funding priorities related to health disparities that require programs to engage in coalition development, often in the form of community engagement. The emphasis on developing coalitions parallels the emphasis on health disparities and diversity. Coalitions, partnerships, alliances, consortia, and collaborative linkages are some of the structural forms that result when stakeholders, interested parties, members of the target audience, and pro- fessionals with expertise agree to work together toward the common goals of community and health improvements for common constituents. The term coalition is used as the umbrella term for such agreements.

Coalitions, in whatever form, can be viewed as potentially having power and being power brokers (Braithwaite, Taylor, & Austin, 2000). Underlying the emphasis on coalition initiatives is growing evidence that collaboration among stakeholders is key to ensuring effective com- munity involvement and to decreasing health disparities. For example, coalitions have proved effective in reducing the number of uninsured children (Stevens, Rice, & Cousineau, 2007). Coalitions for health programs may be developed for a variety of reasons, such as creating a power base from which to gain attention for the health problem or resources to address the problem, or to achieve long-term sustainability of the health program. To achieve this purpose, of course, the coalition must be effective (i.e., successful). In a review of the literature, Zakocs and Edwards (2006) found some evidence that the following

Across the Pyramid 51

“program” for “organization”—as a means of gauging the cultural competency of the health program to engage the community in health program development.

▸ Across the Pyramid At the direct services level of the public health pyramid, disparities are seen as affecting indi- viduals and their health status. As individuals from diverse cultures, ethnicities, races, and SES backgrounds interact with health professionals and the health program staff, the training in cultural sensitivity and competency is put into practice. If the professionals and staff members have not received or integrated this knowledge into their practice, the potential for continued healthcare disparities is present.

Health programs designed for the direct services level of the pyramid will need to verify that the interventions included in the program match the culture, language, and norms of the

honest dialogue, in which members can be frank without feeling threatened by retribution for ideas. Complementing this strategy is the adoption of a policy of “agree to disagree.” This ground rule for interactions tends to foster cooperation as well as trust. Rose reminded us that humor is a very effective tool for unifying members and for relieving tensions. It is always healthy to laugh at situations, to find the bright side, and to be amused. This need transcends cultures, despite cultural differences in what makes something humorous.

Throughout the process of forming and working with a coalition, attention to cultural competency is crucial. One aspect of being culturally competent involves conducting a self-assessment that assesses the values and principles that govern participation in coalitions. The National Center for Cultural Competence has developed a checklist that can be used to assess cultural competency in community engagement (Goode, 2001; EXHIBIT 2-2). The health program planners could use this tool—after substituting

EXHIBIT 2-2 Checklist to Facilitate Cultural Competence in Community Engagement

Does the healthcare organization, primary healthcare system, or program have:

■ A mission that values communities as essential allies in achieving its overall goals? ■ A policy and structures that delineate community and consumer participation in planning,

implementing, and evaluating the delivery of services and supports? ■ A policy that facilitates employment and the exchange of goods and services from local communities? ■ A policy and structures that provide a mechanism for the provision of fiscal resources and in-kind

contributions to community partners, agencies, or organizations? ■ Position descriptions and personnel performance measures that include areas of knowledge and

skill sets related to community engagement? ■ A policy, structures, and resources for in-service training, continuing education, and professional

development that increase capacity for collaboration and partnerships within culturally and linguistically diverse communities?

■ A policy that supports the use of diverse communication modalities and technologies for sharing information with communities?

■ A policy and structures to periodically review current and emergent demographic trends? ■ Community partners who are representative of the diverse population in the geographic or service

area? ■ Ways to identify new collaborators and potential opportunities for community engagement? ■ A policy, structures, and resources to support community engagement in languages other than English?

Included with permission of the Georgetown University National Center for Cultural Competence, Georgetown University Center for Child & Human Development, Georgetown University Medical Center.

52 Chapter 2 Relevance of Diversity and Disparities to Health Programs

will vary from minimal (e.g., in an immunization campaign) to none (e.g., in a media campaign). Thus, issues of cultural competency for program staff members are lessened.

The need for the intervention itself to reflect cultural competency remains at the population-based service level. Health programs targeted at pop- ulations face the challenge of deciding whether to make the program generically acceptable for most members of the population or whether to develop different versions of the intervention tailored to known, culturally distinct subpop- ulations or aggregates. This challenge, while similar to the need for flexibility in direct services programs, is complicated by the inability to tailor the intervention during a program encounter.

With regard to measurement, most data collected at the population-based services level of the public health pyramid will be on such a scale that simple, generic data collection methods will be needed. This will result in having data that offer less detail but cover more program recipients. Unlike programs at the direct services or enabling services levels, a population-based program may not be able to gather data on actual program recipients. This fact creates a situation in which program planners may need to work more closely with the organizations and agencies responsible for collecting population-level data to ensure that the measures employed are as relevant to the program as possible.

At the infrastructure level, personnel diver- sity, organizational culture, and program culture all play roles in program planning and delivery. Overall, diversity and disparities are visible through their effects on existing and new health policy and priorities and on organizational processes and culture. Interpersonal interactions among program planners, staff members, stakeholders, and policy makers are the focus of efforts to address health disparities and cultural issues. Programs at the infrastructure level aim to change the cultural competency of the workforce and the capacity of the workforce to address health disparities and cultural diversity.

As with programs for the other levels of the pyramid, interventions implemented at the

program recipients. It may also be necessary for the health program to be designed so that the intervention can be culturally, ethnically, and linguistically tailored to those participating in the program at the moment. In terms of measurement considerations at this level of the pyramid, the direct interaction with program participants allows for needs assessment, program process, and program effect data to be collected from individuals, through either quantitative ques- tionnaires or qualitative interviews.

At the enabling services level of the pyramid, disparities are seen as they affect aggregates and families. Diversity is manifested in subcultures or enclave ethnicity, as well as in the larger cultural context. The interpersonal interaction between the program staff members and the program recipients remains an essential element of services at this level. As a consequence, the cultural competency of individual program staff members continues to be important as they implement the program interventions.

The interventions provided as enabling services will need to be tailored to the specific sociocul- tural characteristics and preferences of the target aggregate. For example, an existing enabling service may be planned for a new target audience. This endeavor would result in fairly specific changes, modifications, or additions to the existing program in an effort to make it culturally and linguistically acceptable to the new target audience. In terms of measurement, data are likely to be collected from individuals, allowing for tailoring the data collection to the characteristics of the aggregate.

At the population-based services level of the public health pyramid, disparities within a population are revealed through the collection of data related to that population, such as vital statistics and healthcare utilization. For all practical purposes, disparities are most easily identified by examining differences within a population, although they can also be identified within large aggregates, such as schools. Because health programs designed for the population level of the pyramid are delivered or provided to the population, interpersonal interaction between program staff members and program recipients

53Internet Resources

either of the cultural competency self- assessments included in this chapter (Exhibits 2-1 and 2-2). What surprised you about taking the self-assessment? Which recommendations would you make based on the results of the self- assessment?

3. Identify one health-related questionnaire that has been used with more than one cultural or linguistic group. Discuss the adequacy of the linguistic and conceptual translations of the questionnaire.

4. List four health programs in your com- munity. Are they supported by coali- tions? What is the composition of each coalition? Does there appear to be a relationship between coalition diversity and health program success?

▸ Internet Resources Centers for Disease Control and Prevention The Centers for Disease Control and Prevention’s National Center for Health Statistics is a good resource for health statistics. It allows users to manipulate data on a specific health indicator by variables such as race or income. Find it at www.cdc.gov/nchs/hdi.htm.

National Center on Cultural Competence Georgetown University’s National Center on Cultural Competence (NCCC) website (http:// nccc.georgetown.edu/) has a wealth of resources related to cultural competency. The mission of the NCCC is to increase the capacity of health and mental health programs to design, implement, and evaluate culturally and linguistically competent service delivery systems.

Bureau of Primary Health Care The Bureau of Primary Health Care, which is part of the Health Resources and Services Administration (HRSA), has a website devoted

infrastructure level need to be tailored to the sociocultural characteristics of the target audi- ence within the infrastructure. In addition, they need to address the professional diversity that exists within the infrastructure, within specific healthcare organizations or agencies, and within the healthcare system as a whole.

With regard to measurement at the infra- structure level, the availability of individual data versus aggregate data will depend on the nature of the health program. Health programs provided to groups of workers, such as cultural competency training, make it possible to mea- sure specific attributes of program participants. Health program interventions designed to change health policy are not amenable to direct data collection but would rely on population-level data, especially for program effects.

One other infrastructure issue that warrants mentioning is the legal implications of diversity. For example, the Americans with Disabilities Act (ADA) of 1990 requires that planning for programs take into account issues of accessibility for disabled persons. Another legal issue relates to antidiscrimination laws, which affect both the management of program personnel and the process by which program participants are recruited and accepted into the health program. State laws and local ordinances regarding same-sex marriage and civil unions may also affect reimbursement for programs, responses to survey questions about marriage, and recruitment of family members into programs. All these factors influence the planning and evaluation of the health program and therefore fall within the purview of the infrastructure level of the public health pyramid.

▸ Discussion Questions 1. Discuss the ways in which the linguis-

tic diversity of a target audience affects programs being planned at each level of the public health pyramid.

2. Think of a specific health program pro- vided by a specific healthcare organization with which you are familiar. Complete

54 Chapter 2 Relevance of Diversity and Disparities to Health Programs

Bhopal, R. (2006). Race and ethnicity: Responsible use from epidemiological and public health perspectives. Journal of Law, Medicine, and Ethics, 34, 500–507.

Boekaerts, M. (1998). Do culturally rooted self-construals affect students’ conceptualization of control over learning? Educational Psychologist, 33, 88–108.

Bond, S. M., Cartmell, K. B., Lopez, C. M., Ford, M. E., Brandt, H. M., Gore, E. I., & Alberg, A. J. (2016). Racial and ethnic group knowledge, perceptions and behav- iors about HPV, HPV vaccination, and cervical cancer among adolescent females. Journal of Pediatric and Adolescent Gynocology. doi:http://dx.doi.org/10.1016 /j.jpag.2016.02.005

Boutin-Foster, C., Foster, J. C., & Konopasek, L. (2008). Viewpoint: Physician, know thyself: The professional culture of medicine as a framework for teaching cultural competence. Academic Medicine, 83(1), 106–111.

Braithwaite, R. L., Taylor, S. E., & Austin, J. N. (2000). Build- ing health coalitions in the black community. Thousand Oaks, CA: Sage.

Braveman, P. (2006). Health disparities and health equity: Concepts and measurement. Annual Review of Public Health, 27, 167–94.

Brown, T. H., Richardson, L. J., Hargrove, T. W., & Thomas, C. S. (2016). Using multiple-heirarchy stratification and life course approaches to understand health inequalities: The intersecting consequences of race, gender, SES, and age. Journal of Health and Social Behavior, 57, 200–222.

Bruce, M. A., Beech, B. M., Crook, E. D., Sims, M., Wyatt, S. B., Flessner, M. F., et al. (2010). Association of socioeco- nomic status and CKD among African Americans: The Jackson Heart Study. American Journal of Kidney Diseases, 55(6), 1001–1008.

Cable, D. M., & Judge, T. A. (1997). Interviewers’ perceptions of person–organization fit and organizational selection. Journal of Applied Psychology, 82, 546–561.

Campbell, M. K., Hudson, M. A., Resnicow, K., Blakeney, N., Paxton, A., & Baskin, M. (2007). Church-based health promotion interventions: Evidence and lessons learned. Annual Review of Public Health, 28, 213–234.

Cohen, E., & Goode, T. (1999). Policy brief 1: Rationale for cultural competence in primary health care. Washington, DC: National Center for Cultural Competence, Georgetown University Child Development Center. Retrieved July 27, 2016, from http://nccc.georgetown.edu/information /organizations.html

Colby, S. L., &.Ortman, J. M. (2014). Projections of the size and composition of the U.S. population: 2014 to 2060, Current Population Reports, P25-1143. Washington, DC: U.S. Census Bureau.

Cox, T. (1991). The multicultural organization. Academy of Management Executive, 5, 34–47.

Cox, T. (2001). Creating the multicultural organization. San Francisco, CA: Jossey-Bass.

Cross, T. L., Bazron, B. J., Dennis, K. W., & Isaacs, M. R. (1989). Towards a culturally competent system of care:

to creating centers of excellence. The relevant document is entitled Transforming the Face of Health Professions Through Cultural and Linguistic Competence Education: The Role of the HRSA Centers of Excellence; it can be found at www .hrsa.gov/culturalcompetence/cultcompedu.pdf.

Public Health Services, Office of Minority Health The 2013 document entitled National Standards for Culturally and Linguistically Appropriate Ser- vices in Health Care, by the Office of Minority Health, sets out the standards for linguistically appropriate health care. It and related resources can be found at https://www.thinkculturalhealth .hhs.gov/Content/clas.asp.

Medical Anthropology This medical anthropology website includes a page of culture-specific syndromes, with some explanation and maps. Visit it at http://anthro .palomar.edu/medical/med_4.htm.

The Joint Commission The Joint Commission. (2010). Advancing effective communication, cultural competence, and patient and family-centered care: A roadmap for hospitals. Oakbrook Terrace, IL: Author. Although this workbook is written for hospitals, the content provides a comprehensive overview of the considerations involved in effectively communicating. It can be found at https://www .jointcommission.org/roadmap_for_hospitals/.

References Airhihenbuwa, C. O. (1994). Health promotion and the

discourse on culture: Implications for empowerment. Health Education Quarterly, 21, 345–353.

Al-Harthy, M., Ohrbach, R., Michelotti, A., & List, T. (2016). The effect of culture on pain sensitivity. Journal of Oral Rehabilitation, 43(2), 81–88. doi:10.1111/joor.12346

Baetz, M., & Toews, J. (2009). Clinical implications of research on religion, spirituality, and mental health. Canadian Journal of Psychiatry, 54(5), 292–301.

Beckjord, E. B., & Klassen, A. C. (2008). Cultural values and secondary prevention of breast cancer in African American women. Cancer Control, 15, 63–71.

55References

communities: Some practical considerations. In M. A. Orlandi, R. Weston, & L. G. Epstein (Eds.), Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethical/ racial communities (pp. 203–260). [DHHS Publication No. (ADM) 92–188]. Washington, DC: U.S. Government Printing Office.

Kleinman, A. (1980). Patients and healers in the context of culture. Berkeley, CA: University of California Press.

Knibb-Lamouche, J. (2012). Culture as a social determinant of health: Examples from native communities. Institute on Medicine: Roundtable on the Promotion of Health Equity and the Elimination of Health Disparities, Seattle.

Kosa, J., & Zola, I. K. (1975). Poverty and health: A socio- logical analysis (rev. ed.). Cambridge, MA: Harvard University Press.

Krause, N., Emmons, R., & Ironson, G. (2015). Benevolent images of god, gratitude, and physical health status. Jour- nal of Religion & Health, 54(4), 1503–1519. doi:10.1007 /s10943-015-0063

Lewin Group. (2001). Health Resources and Services Ad- ministration study on measuring cultural competence in health care delivery settings: A review of the literature. Retrieved July 27, 2016, from https://archive.org/stream /healthresourcess00lewi#page/n0/mode/2up

Loftin, C., Hartin, V., Branson, M., & Reyes, H. (2013). Measures of cultural competence in nurses: An integra- tive review. The Scientific World Journal, Volume 2013. Retrieved from http://dx.doi.org/10.1155/2013/289101

Masters, K. S., & Spielmans, G. I. (2007). Prayer and health: Review, meta-analysis, and research agenda. Journal of Behavioral Medicine, 30, 329–338.

McCulloch, M. C., & Turban, D. B. (2007). Using person– organization fit to select employees for high-turnover jobs. International Journal of Selection and Assessment, 15, 63.

Mossaad, N. (2016). U.S., Lawful permanent residents: 2014. Annual flow report. Washington, DC: Office of Immi- gration Statistics, Department of Homeland Security.

National Cancer Institute. (2014). A snapshot of breast cancer. Retrieved June 30, 2016, from http://www.cancer.gov /research/progress/snapshots/breast

National Center for Health Statistics [NCHS]. (2016). Na- tional Center for Health Statistics. Health, United States, 2015: With Special Feature on Racial and Ethnic Health Disparities. Hyattsville, MD.

Navarro, J., Wilson, S., Berger, L. R., & Taylor, T. (1997). Substance abuse and spirituality: A program for Native American students. American Journal of Health Behav- ior, 21, 3–11.

Orlandi, M. A. (1992). Defining cultural competence: An organizing framework. In M. A. Orlandi, R. Weston, & L. G. Epstein (Eds.), Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethical/racial communities (pp. 293–299). [DHHS Publication. No. (ADM) 92–1884]. Washington, DC: U.S. Government Printing Office.

A monograph on effective services for minority children who are severely emotionally disturbed: Volume 1. Washington, DC: Georgetown University Child Development Center.

Deal, T. E., & Kennedy, A. A. (1982). Corporate cultures: The rites and rituals of corporate life. Reading, MA: Addison-Wesley.

Edwards, R. R., Moric, M., Husfeldt, B., Buvanendran, A., & Ivankovich, O. (2005). Ethnic similarities and differ- ences in the chronic pain experience: A comparison of African American, Hispanic, and white patients. Pain Medicine, 6, 88–98.

Epstein, J., Santo, R. M., & Guillemin, F. (2015). A review of guidelines for cross-cultural adaptation of ques- tionnaires could not bring out a consensus. Journal of Clinical Epidemiology, 68(4), 435–441. doi:10.1016 /j.jclinepi.2014.11.021

Fisher, T. L., Burnet, D. L., Huang, E. S., Chin, M. H., & Cagney, K. A. (2007). Cultural leverage interventions using culture to narrow racial disparities in health care. Medical Care Research and Review, 64, 243s–282s.

Gavin, A. R., Melville, J. L., Rue, T., Guo, Y., Dina, K. T., & Katon, W. J. (2011). Racial differences in the prevalence of antenatal depression. General Hospital Psychiatry, 33(2), 87–93.

Goode, T. (2001). Policy brief 4: Engaging communities to realize the vision of one hundred percent access and zero health disparities: A culturally competent approach. Washington, DC: National Center for Cultural Compe- tence, Georgetown University Child Development Center. Retrieved July 8, 2016, from http://nccc.georgetown .edu/information/organizations.html

Goode, T., Jones, W., & Mason, J. (2002). A guide to planning and implementing cultural competence organization self-assessment. Washington, DC: National Center for Cultural Competence, Georgetown University Child Development Center.

Hastie, B. A., Riley, J. L., & Fillingim, R. B. (2005). Ethnic differences and responses to pain in healthy young adults. Pain Medicine, 6, 61–71.

Hearon, M. (2016). Deaths: Leading causes for 2014. National Vital Statistics Report. Hyattsville, MD.

Higginson, J. G. (1998). Competitive parenting: The culture of teen mothers. Journal of Marriage and Family, 60, 135–149.

Hill, A., De Zapien, J. G., Staten, L. K., McClelland, D. J., Garza, R., Moore-Monroy, M., et  al. (2007). From program to policy: Expanding the role of community coalitions. Preventing Chronic Disease, 4, A103.

Hoskins, M. L. (1999). Worlds apart and lives together: Developing cultural attunement. Child and Youth Care Forum, 28, 73–84.

Jackson, K. F., & Samuels, G. M. (2011). Multiracial competence in social work: Recommendations for culturally attuned work with multiracial people. Social Work, 56, 235–245.

Kim, S., McLeod, J. H., & Shantzis, C. (1992). Cultural com- petence for evaluators working with Asian-American

56 Chapter 2 Relevance of Diversity and Disparities to Health Programs

coalitions pursuing universal coverage for children. American Journal of Public Health, 97, 738–743.

Tan, E., Lim, Y., Teo, Y., Goh, R., Law, H., & Sia, A. T. (2008, September). Ethnic differences in pain perception and patient-controlled analgesia usage for postoperative pain. The Journal of Pain, 9(9), 849–855.

Tseng, W. (2006). From peculiar psychiatric disorders through culture-bound syndromes to culture-related specific syndromes. Transcultural Psychiatry, 43, 554–576.

U.S. Department of Health and Human Services, Health Resources and Services Administration. (2015). Sex, race, and ethnic diversity of U.S. health occupations (2010–2012), Rockville, MD: National Center for Health Workforce Analysis. Retrieved July 8, 2016, from http:// bhpr.hrsa.gov/healthworkforce/index.html

Ventriglio, A., Ayonrinde, O. & Bhugra, D. (2016). Rele- vance of culture-bound syndromes in the 21st century. Psychiatry and Clinical Neuroscience, 70, 3–6.

Weick, K. (1979). Social psychology of organizing (2nd ed.). Reading, MA: Addison-Wesley.

Willis, G., & Zahnd, E. (2007). Questionnaire design from a cross-cultural perspective: An empirical investigation of Koreans and non-Koreans. Journal of Health Care for the Poor and Underserved, 18(4S), 197–217.

Zakocs, R. C., & Edwards, E. M. (2006). What explains community coalition effectiveness? A review of the literature. American Journal of Preventative Medicine, 30, 351–361.

Paez, K. A., Allen, J. K., Carson, K. A., & Cooper, L. A. (2007). Provider and clinic cultural competence in a primary care setting. Social Science and Medicine, 66, 1204–1216.

Polednak, A. P. (1997). Segregation, poverty, and mortality in urban African-Americans. Oxford, UK: Oxford University Press.

Rea, J., Martin, C., & Wright, K. (2002). Using person-centered supports to change the culture of large intermediate care facilities. Person-Centered Planning: Research, Practice and Future Directions. Baltimore: Paul H Brookes.

Rose, F. (2000). Coalitions across the class divide: Lessons from the labor, peace and environmental movements. Ithaca, NY: Cornell University Press.

Russell, A. E., Ford, T., Williams, R., & Russell, G. (2016). The association between socioeconomic disadvantage and attention deficit/hyperactivity disorder (ADHD): A systematic review. Child Psychiatry & Human Devel- opment, 47, 440–458.

Satterfield, D., DeBruyn, L., Francis, C., & Allen, A. (2014). A stream is always giving life: Communities reclaim native science and traditional ways to prevent diabetes and promote health. American Indian Culture and Research Journal, 38(1), 157–190.

Schein, V. E. (1995). Working from the margins: Voices of mothers in poverty. Ithaca, NY: ILR Press, Cornell University Press.

Spector, R. E. (1991). Cultural diversity in health and illness (3rd ed.). Norwalk, CT: Appleton & Lang.

Stevens, G. D., Rice, K., & Cousineau, M. R. (2007). Children’s health initiatives in California: The experiences of local

SECTION II

Defining the Health Problem

© Lynne Nicholson/Shutterstock

59

Community Health Assessment for Program Planning

This chapter begins with a definition of community and types of needs. It then provides an overview of different approaches to planning. Then types of needs assessment are reviewed, with full acknowledgment that, as with many aspects of program planning and evaluation, this field is an area of specialization. The various models that underlie an assessment are reviewed, as are the prototypical types of assessment. The distinctions made among the models and the types of assessment do become blurred in practice. Nonetheless, having a clear sense of what could be involved allows for more thoughtful and intentional decisions on how best to design and carry out the community health assessment.

▸ Defining Community The use of the ecological model and the public health pyramid leads naturally to considering community and its role in health program planning and evaluation. Ambiguity about the

meaning of community can lead to conflicts and confusion throughout the planning and evaluation cycle. Thus, it is worth considering what is and is not a community, as a prelude to clarify thinking and as a foundation for better planning of health programs. A community encompasses people and some form of proximity or place that enables interaction, and that inter- action leads to shared values or culture (Bell & Newby, 1971; TABLE 3-1). A defining characteristic of a community is a group of people with the potential for interaction. Without the potential for interaction, sharing of values and norms cannot occur. Interactions can be virtual as well as person-to-person. To the extent that a large number of interacting individuals share values and culture, a community can exist. Of course, virtual communities that exist via electronic media extend the traditional, anthropological notions of community that grew from the study of tribes and villages.

Another defining characteristic of a community is that its members have shared values and norms of behavior. The prerequisite that a community

CHAPTER 3

© Lynne Nicholson/Shutterstock

60 Chapter 3 Community Health Assessment for Program Planning

have commonly held values precludes, in many instances, a census tract, a ZIP code, a telephone area code, a consortium of health agencies, or a catchment area for a health service from being a community in the more pure sense. In contrast, active members of a church or residents of a small and homogeneous neighborhood might be a community.

This distinction between a convenient geo- graphic designation and an actual community is important for planning how to have partici- pation by community members in the planning process. Nilsen (2006) reminds us that the major problem with defining a community based on geographic location is that the geography does not ensure that a sense of community exists. Shared values form the basis for the cultural unity of a community, which in turn is the basis for the perception of being connected and belonging to a community. From that sense of belonging stems the subsequent behaviors that might be attributed to members of a particular community.

The concept of a “sense of community” has been studied and found to include aspects of membership, influence over what occurs within the community, shared values and needs fulfillment, and a shared emotional connection. Perhaps not surprisingly, the breadth of this list of the elements means that community and its associated emotional and cultural components create challenges for health program planning and evaluation. Individuals’ sense of commu- nity varies according to their race/ethnicity

and income (Coffman & BeLue, 2009; Kono, Tadaka, Kanaya, Dai, Itoi, & Imamatsu, 2012; Belue, Taylor-Richardson, Lin, McClellan, & Hargreaves, 2006). To achieve participatory planning, it may first be necessary to understand the local variations in sense of community. Regardless of the size of the population, the sense of community can be a key factor in gaining support for the program and for maximizing the health effects of some programs.

Community as Context and Intended Recipient Program planners must clarify the purpose for which “the community” is being delineated. A community, as a unit of individuals with some degree of cultural cohesion, can be both the target of a health program and the context in which a health program is provided and evaluated.

When a health program targets a community for the intervention, the health program may intend some or all of the community members to receive the health program. Thus, to establish the size of the health problem within that com- munity, planners must delineate the community boundaries. In this sense, a community is akin to a population. Community assessments or community needs assessments are processes by which planners seek to understand and describe the health problem more fully. The word community is used whether or not a community

TABLE 3-1 Three Elements of Community, with Their Characteristics

Element Characteristics of the Community Elements

People Values, beliefs, behaviors, size, membership, demographic characteristics, social and economic status, sense of power or influence, sense of belonging

Place Geography, boundaries, housing, industry, air, water, land, virtual presence

Interaction Interpersonal communication, family based, education related, religion based, politically related, recreationally focused, virtual communication

Defining Community 61

or a population is being assessed. However, if an actual community—in contrast to a population or an aggregate—is the focus, planners must clarify the boundaries of the community membership if they are truly to understand the specific health and social conditions within that community.

The community may be viewed as the con- text of the program in two different ways. First, the notion of community embodies a myriad of sociopolitical and economic factors that can influence the program plan and implementation. It may be possible only to acknowledge, artic- ulate, and take into account these influences as contextual to the program (if resources will not address these influences directly). Second, from a different perspective, community members who are invited to participate in the planning process become an immediate, intimate context of the program intervention.

The concepts of community as target and community as context are not mutually exclusive. An example helps demonstrate their interactive nature. At an elementary school in Toronto, stu- dents from fifth and sixth grades, with assistance from pediatric residents, developed a 20-page health magazine for their classmates (Porepa et al., 2016). They were a community as target to the extent that all the children at the school were the intended recipients of the health magazine and were assessed as a unit to identify health topics. They were community as context to the extent that their level of knowledge, norms, social structure, and school structure were influences on developing the magazine.

In this case, to address the contextual influences, the program initiators included and promoted participation by members of the university community in planning the wellness program. In so doing, they overcame institutional barriers and mobilized resources for the program. The synergies achieved by involving community members became possible because the university community was understood not as a single thing or a simple geographic location but rather as a group of individuals. In other words, to involve “the community” in program planning requires having influential, energetic, devoted actual

individuals who serve as representatives of or who come from that group actively participate in planning activities.

Collaboration between health program planners and formal or informal leaders from a community, and the participation of those community leaders in the development and evaluation of a health program are increasingly valued. In fact, funding agencies often mandate collaboration as a prerequisite to being considered for funding. Collaboration creates interaction, which can intensify the sense of community and promote synergy among the community representatives, the agency sponsoring the health program, and the program staff (FIGURE 3-1). These interactions and influences move in two directions, with ideas and energy flowing toward the health program, and results and respect flowing from the health program.

Defining Terms: Based, Focused, and Driven Three terms related to community need to be defined in any discussion of community health planning: community based, community focused, and community driven.

Health program planning

and evaluation

Health services agency or organization

Community at large: target audience, stakeholders

FIGURE 3-1 Connections Among Program, Agency, and Community

62 Chapter 3 Community Health Assessment for Program Planning

easy to imagine a community-based city health clinic that is moderately community-focused and not at all community-driven.

Third, the designations of “based,” “focused,” and “driven” can also apply to families, popula- tions, or other aggregates. Thus, family-based programs might be provided to individuals, but within the theoretical or physical presence of families, family-focused programs would be designed to enhance the family as a unit, and family-driven programs would be the result of a group of families advocating for or demanding the programs.

These distinctions can be important in terms of describing health programs and conceptual- izing the nature of a program. Whether a health program is based or focused on a unit ought to flow from an understanding of the health problem and the best strategy for addressing it.

▸ Types of Needs Four types of needs (Bradshaw, 1972) ought to be considered in a needs assessment. This classic typology of needs continues to be useful in planning the community assessment. Under- standing the characteristics of each type of need as it relates to the population of interest is critical to successful health program planning.

Expressed need is the problem revealed through health care–seeking behavior. In other words, expressed need is manifested as the demand for services and the market behavior of the target audience. Measures of expressed need include the number of people who request services, the types of services sought, and utilization rates.

Normative need is a lack, deficit, inadequacy, or excess as defined by experts and health pro- fessionals, usually based on a scientific notion of what ought to be or what the ideal is from a health perspective. A norm or normal value is used as the gauge for determining if a need exists. For example, a community with an infant mortality rate above the national average would have a normative need related to causes of infant mortality. At an individual level, having a body

Community-based is an adjective describing where a program or service is provided. A health program is community-based if it is delivered at locations considered within the boundaries of the community rather than at a centralized location outside the community boundaries. Generally, this understanding translates into a program being delivered in local churches, schools, recreation centers, local clinics, or libraries.

Community-focused refers to the way in which the program is designed. Health programs that seek to affect the community as a whole, as a unit, are best described as community-focused. A community-focused program may seek to change the norms or behaviors of the members of a com- munity that contribute to the health problem, or it may seek to reach all members of the community.

In contrast, health programs that result from the involvement of community members and their preferences and needs are referred to as community-driven. A community-driven program has its genesis—that is, its design and implementation—in the involvement, persistence, and passion of key representatives of or members from the community.

Three points are worth noting about these qualifiers. First, the terms based, focused, and driven are not mutually exclusive (FIGURE 3-2). In other words, a program can be all three: based in, focused on, and driven by a community.

Second, the degree to which a program is community-based, community-focused, or community-driven can vary. Each can vary independently of the other. For example, it is

FIGURE 3-2 Venn Diagram of Community-Based, Community-Focused, and Community-Driven

Community Based

Community Focused

Community Driven

Types of Strengths 63

Confident in their understanding of what was needed to improve the health of the neigh- borhood, the group of planners approached the residents with a plan to establish a primary care clinic in the neighborhood that would pro- vide prenatal care and diabetes management. Bluntly, the residents indicated that they were not interested: They wanted a community swimming pool in their neighborhood. They felt a strong need for recreation, a community meeting place, and an equal opportunity to engage in the healthy behavior of swimming as an alternative to the gang activities that were contributing to the shootings. Only after those conducting the assessment agreed to address the community’s perceived need for a community swimming pool did the residents then consider how to address the normative, comparative, and relative needs that had been identified by the health professionals.

▸ Types of Strengths A counterpoint to types of needs is types of community strengths that exist as resources available for use. Briefly, these three community strengths—community competence, community resilience and community assets—remind us to acquire a balanced view and understanding of communities. These concepts help focus the assessment on the glass as half full.

Community competence has long been considered a key strength (Cottrell, 1976; Goep- pinger, Lassiter, & Wilcox, 1982). Community competence is the process whereby a community is able to identify problems and take actions to address those problems, and it is increased by community organizing (Denham, Quinn, & Gamble, 1998; Norris, Stevens, Pfefferbaum, Wyche, & Pfefferbaum, 2008). Greater com- munity competence has been associated with both better health of the community and larger amounts of social capital (Lochner, Kawachi, & Kennedy, 1999) Community resilience, the ability of a community to withstand and recover from adverse situations or conditions (Plough et al., 2013),

mass index (BMI) greater than 29.9 indicates, normatively, a need for weight reduction. Given that the health professional is an outside observer, normative need reflects norms through the eyes of an observer.

The third type of need is the perceived or felt need, which is the sense of lack as experienced by the target audience. Perceived needs are demon- strated in what members of the target audience say that they say they want, and in their stated deficits and inadequacies. For example, parents in a community may demand a new school based on their perception that their children have too far to travel to go to school. Perceived need is the view through the eyes of the person having the experience.

The relative or comparative need is the identified gap or deficit as identified through a contrast between advantaged and disadvantaged groups. Relative need entails a comparison that demonstrates a difference that is interpreted as one group having a need relative to the other group. Most health disparities are stated as relative need. For example, the black infant mortality rate is twice that of the white infant mortality rate. Health professionals or community members can choose the point of comparison, but each is likely to choose a different point of comparison. The relative needs thus would also be different.

The ways in which the interaction of these needs plays out in day-to-day situations can be seen in the experiences of a group of health planners who conducted a community needs assessment in a fictitious town called Layetteville. In conducting their assessment, they identified a neighborhood as having a relative need. The planners found that the neighborhood had higher rates of adolescent pregnancy, deaths due to gunshot injuries, birth defects, and diabetes than other neighborhoods in the city and that these rates were two to three times higher than those set out in the Healthy People 2020 objec- tives. When the group explored the healthcare utilization patterns of the neighborhood resi- dents, they found that the residents rarely used a primary healthcare clinic; instead, they used the local emergency department.

64 Chapter 3 Community Health Assessment for Program Planning

program priorities. Each of the six different perspectives on planning provides a different lens through which the program planning process can be viewed, understood, and then used to address the health problem. Beneveniste (1989) identified the processes underlying each approach. These approaches remain reflective of current approaches (Oliveira & Pinho, 2010; Guyadeen & Seasons, 2016). Each approach has various typical planning activities (TABLE 3-2). None of the approaches is inherently better or worse than any of the others. Rather, the purpose of becoming aware of all six approaches is to select an approach that matches the situation and to use the strengths of the approaches to arrive at an optimal process for developing a health program and its evaluation.

Incremental Approach The incremental approach to planning does not attempt to address the problem in any particular context or across a specific time span. Instead, the approach is one of addressing immediate concerns and, to some extent, having faith that the small, rather disconnected plans and actions will have a cumulative effect on the problem. Incrementalism, by its very nature, focuses only on the immediate, without attempting to see the big picture or implement a long- term plan.

In the very early days of the human immu- nodeficiency virus, acquired immunodeficiency syndrome (HIV/AIDS) epidemic, before the causative virus had been identified or named, the only health planning options were incre- mental: shut down bath houses, use infectious disease precautions, and seek funding to study the health problem. These actions were isolated, disjointed efforts, but under the circumstances, the incremental approach was the only available option. Advocates for infected individuals had not yet emerged, nor was there a scientific basis for making rational, apolitical decisions.

As this example points out, incrementalism, while not the most effective planning approach, may be the only option in some circumstances.

has been largely applied to disasters and pre- paredness. Community resilience encompasses community capital, along with social, economic, and institutional factors (Ostadtaghizadeh, Ardalan, Paton, Jabbari, & Khankeh, 2015). Resilience contributes to the overall strength of a community. In contrast, community assets embody a breadth of tangible and intangible building blocks that exist as resources within the community and community members. Schools can be a community asset (Caan, Cassidy, Coverdate, Ha, Nicholson, & Rao, 2014). In one study of Chicago, in a 62-square-mile area, over 8,000 assets were identified across 23 sectors, such as food, trade services, and religious worship (Lindau et al., 2016).

▸ Approaches to Planning With the basic understanding of needs and strengths, the health program planner next needs to engage more intentionally in planning. Planning, in the context of the planning and evaluation cycle, is the process of gathering information and making the subsequent decision regarding how something (e.g., the community assessment or the health program) will be done. A variety of definitions of “planning” have been suggested (Hoch, 1994; Alexander, 1992), but all share the elements of using a rational approach, making change, and using a democratic or participatory process. In terms of programs, planning is the set of activities in which key individuals define a set of desired improvements, develop an approach to achieve those desired improvements, and establish a means to measure the attainment of those desired improvements.

The wide range of different methods for planning can be categorized broadly into six approaches. An awareness of the various approaches used to accomplish planning helps planners interpret events and guides others through the planning process. Each approach provides a basis for assessing possible points of contention or agreement regarding health and

Approaches to Planning 65

TABLE 3-2 Summary of the Six Approaches to Planning, with Public Health Examples

Approach Underlying

Assumptions Consequences of Use Public Health

Examples

Incremental Not feasible to do more than small portions at a time; the parts are greater than the whole

Plans may fail because of unforeseen or unaccounted for factors

Specific programs implemented that reflect discrete, categorical funding despite potential overlap or existing similar programs

Apolitical Options are known; makes the problem technical; the means to the ends are known; can anticipate all caveats

Experts may not accurately speak for others; media attention is likely to focus on the spokesperson rather than the issue

Evidence-based practice

Advocacy An external expert can accurately speak for those with less power

An increased sense of confidence and an increased ability to solve one’s own problems; potential for conflict

Activists for marginalized groups and environmental issues

Communicative action

Language is powerful; those with the problem have the capability to enact a solution

Takes considerable time and effort to implement; likely to have more dissent to overcome; results in an encompassing, intertwined set of actions

Community coalitions that take on a program or become nonprofit organizations

Comprehensive rational

System feedback loops are contextual and can be known; rational choices are preferred

Lacks flexibility to respond to emerging issues; a costly process to arrive at a plan

Community-focused initiatives

Strategic planning

Can anticipate and predict the future; stability is more pervasive than change

Circumscribed focus may miss opportunities or emergent needs

Healthy People series; states’ 2- to 5-year plans; Title V 2-year plans

In addition, when resources are limited, incre- mentalism can lead to small gains related to immediate problems. The major disadvantage of incrementalism is that the myriad small

planning efforts may lead to conflicting plans, confusing programs, programs or services that are not integrated, or personnel redundancy or mismatch with the “new” program.

66 Chapter 3 Community Health Assessment for Program Planning

planning activities. Beneveniste (1989) described advocacy planning as a bottom-up form of comprehensive rational planning. Planners using the advocacy approach, however, would be likely to speak for or on behalf of those with the health problem.

For example, experts in environmental hazards may testify before city or county elected officials to plead the case for people living in areas where hazardous waste is generated or stored. When those who live in the area are unaware of a problem but an expert is nevertheless safe- guarding their best interest, advocacy planning is occurring.

As Guerin, Allotey, Elmi, & Baho (2006) discovered, advocacy is personal because of the intense involvement between the advocate and those receiving advocacy. They also see political advocacy as a final stage of community-based participatory research. This perception is similar to the findings of Hill and colleagues (2007), who noted that community planning groups that begin with a focus on program development often evolve into policy planning and advocacy. These reports suggest that an advocacy approach to planning is more likely when the group has been established long enough to acquire an in-depth understanding of the health problem and a deep appreciation that an advocacy planning approach is needed to address the health problem further.

The advantages of the advocacy approach are most readily evident in situations in which clients or citizens are not empowered, for what- ever reasons, to convey their own preferences or concerns. In such situations, having an advocate may be the only option for planning a needed health program. The disadvantages are that the clients or citizens may not agree with the opinions or views of the advocate. In fact, their “advocate” may not even be representing those he or she claims to speak for because this individual implicitly holds a normative view of the needs of the clients or citizens. The advocacy approach also implicitly entails some degree of conflict or confrontation, which may have negative repercussions over the long term. Social irresponsibility arises when the solution ignores important social or cultural

Apolitical Approach The apolitical approach to planning relies solely on technical knowledge to arrive at a solution and assumes that technical knowledge makes it possible to achieve compromises among those involved in the health problem and the planning process. In a sense, the apolitical approach is fundamentally a problem-solving approach that relies on current knowledge about the problem and known alternatives to address the problem. The name of this approach reflects the fact that its focus on the technical aspects ignores the political aspects inherent in any problem. This approach is implicitly a gold standard of planning, particularly when those involved in the planning process are more technically inclined and focused.

To the extent that the apolitical approach relies on objective information for decision making, the application of evidence-based practice (EBP) guidelines can be viewed as essentially apolitical approaches to planning. In EBP, guidelines for practice by individual practitioners and complex health promotion programs are developed solely based on the best available scientific knowledge, without consideration of the context of the prac- tice or the preferences of those experiencing the health problem. When planners use EBP guide- lines as the basis for health program planning without taking other factors into account, they are engaged in apolitical planning.

One criticism of the apolitical problem-solving approach is that it does not account for interper- sonal dynamics and possible struggles for control (Forester, 1993). This approach also neglects cultural issues involving the potential program participants and program staff members, which can be substantial stumbling blocks in applying the health programs. Nonetheless, the apolitical approach has the advantage of being—or at least providing—the appearance of being logical and rational and of specifying solutions with the documented highest efficacy.

Advocacy Approach The advocacy perspective on planning focuses on the client and mandates citizen participation in

Approaches to Planning 67

Comprehensive Rational Approach The comprehensive rational perspective on planning is fundamentally a systems approach. It involves analyzing the problem by drawing on ideas from systems theory—namely, feed- back loops, input and output, systems, and subsystems. The systematic, logical sequence of thought processes and actions employed explains why this approach is termed “rational.” Assumptions are made that the factors affecting the problem (the elements of the systems that contribute to the goals) are knowable and that virtually all contingencies can be anticipated. In this sense, this perspective is rational and logical. The approach is comprehensive in the sense that planners can take into account those contingencies and peripheral influences.

In the comprehensive rational approach, the planners set goals, identify alternatives, implement programs, and monitor results. This approach is clearly the dominant perspec- tive of this text and of most courses in planning and evaluation. Health program planning— particularly of national initiatives—often reflects the effort to use a comprehensive rational approach. In fact, one of the planning principles outlined by Reinke and Hall (1988) is to be as objective as possible, given the context, and to use rationality rather than status or position as much as possible as a basis for power.

One benefit of this approach is that it facilitates obtaining information from stakeholders who might otherwise be reluctant to share informa- tion because it diffuses authority in favor of an information and rational base. The comprehensive rational approach allows planners to address issues faced by the entire system rather than just by subsystems; in this respect, it resembles quality improvement methodologies. Another benefit of the comprehensive rational approach is that it yields more information for decision making than does an incremental approach.

The comprehensive rational approach nonetheless has flaws. Forester (1993), for example, critiqued the cybernetic (systems) perspective that underlies this approach for

factors (Blum, 1982). Strong advocates and users of the apolitical approach to planning may be prone to this pitfall simply because a scientific basis may not take into account social realities and needs.

Communication Action Approach Communication action planning is concerned with the distribution of power and communi- cation. From this perspective, those involved in the planning make efforts to empower those with the problem through sharing of information. Whereas the advocacy approach does not enable those with the problem to participate as equals with the “experts” in the planning process, the communication action approach is predicated on making those with the problem equals in the planning process. According to Forester (1993), this perspective leads planners to think of planning as shaping attention, changing beliefs, gaining consent, and engendering trust and understanding among those involved.

One of the many examples of this approach to planning is evident in mental health. The National Alliance on Mental Illness (NAMI) is a nonprofit organization whose purpose is to support those with mental illness, and it is run by individuals with mental illness. Individuals with mental health problems are taught and guided in the process of developing small-scale, community-based programs and services for those with mental illnesses. This approach exemplifies the communication action approach.

A major advantage of the communication action approach is that members of the target audience gain skills, knowledge, and confidence in addressing their own problems. However, the health planner who is involved in critical planning needs to have a set of skills that are different from those needed to do rational or incremental planning. Also, because time and effort are needed to enable those with the prob- lem to participate fully in the planning process, planning may proceed more slowly. The time needed to implement communication action lengthens the planning time line such that it is not useful in emergencies.

68 Chapter 3 Community Health Assessment for Program Planning

process. At one planning meeting, the chief executive officer of one of the community health agencies asked what his organization could do, what was being asked of his organization, and how his organization would benefit from participating in the health programs that would result from the planning process. His questions reflect implicit thinking about how to place his organization strategically within the field of contenders for grant money. His participation in the planning process also implied that his organization’s mission was compatible with the general direction being taken to address the health problem.

The strategic planning approach has some clear advantages: It takes into account the context, whether competition or policy, and it has a slightly longer-term focus. Most strategic planning scholars recommend a time frame of approximately 5 years when planning because it typically takes that long to make strategic changes in programs and services. Strategic planning is a rational model or systematic approach to decision making, so decision points can be quantified, weighted, and sequenced, and then programmed into computer software that shows which option is the “best.”

Despite the capability to quantify mathemat- ically the decision-making process, identifying the best option does not guarantee that the best decision option or program plan will be adopted. Human beings are irrational, with biases in how they think about probabilities and possibilities (Tversky & Kahneman, 1974). These human characteristics are usually not quantified in the decision models, but they are powerful forces in interpreting information and then shaping the actions taken. This irrationality can lead to situations in which the broad goals developed during the planning process are not, in fact, acted upon. Another disadvantage of the stra- tegic planning approach is its lack of flexibility to respond to newly emerging environmental opportunities or threats (Egger, 1999). In addi- tion, strategic planning, if done properly, is time and resource intensive.

its failure to take into account the norms and values of individuals either involved in the planning process or affected by the planning process. Beneveniste (1989) acknowledged that the comprehensive rational approach separates planners from the political realities of the health situation. Forester (1993) also pointed out that the systems approach assumes that the means and the ends are known, which may not be the case. Unsubstantiated claims about the relationship of means (actions, interventions, process) to ends (effects, outcomes) adversely affect planning. The idea that planners know best in terms of which means are optimal reveals a normative perspective on planning that may be unacceptable to stakeholders with less expertise.

Strategic Planning Approach The strategic planning perspective focuses on the organization and its ability to accomplish its mission in a fiscally responsible manner. While this approach is rarely used to address specific health conditions, it is particularly applicable to the infrastructure level of the public health pyramid. Through strategic plan- ning, resources needed to address the health problem are identified and considered in terms of the mission of the organization. This widely used approach often affects program choices, as in healthcare organizations whose services center on one area of health, such as Planned Parenthood or the March of Dimes, or on one aggregate, such as the Boys and Girls Club or the American Association of Retired Persons. To some extent, the national goals and objec- tives set forth in Healthy People 2020 are also examples of strategic planning.

This approach, when used by one member of a planning team, can affect the planning of the whole team. For example, the Centers for Disease Control and Prevention (CDC) awarded a large city a grant for a 1-year planning phase of a major community-based health promotion initiative. The funded agency involved numer- ous local community agencies in the planning

Models for Planning Public Health Programs 69

is important when assessing constraints for implementing the health program and devel- oping the organizational resources.

▸ Models for Planning Public Health Programs

Although planning is generally described as a linear process, program planning is a cyclical activity, with recursive events requiring additional or revised courses of action for the life of a health program. Advances in defining planning in health care parallel advances in models for use by prac- titioners. Beginning in the mid-1980s, the CDC began to develop and promote methodologies for systematic approaches to health planning for those working in public health. These models are important for their structured approach to planning health programs and for synthesizing the knowledge available at the time about health and program planning. Internet resources related to Mobilizing for Action through Planning and Partnership (MAPP) and Protocol for Assessing Community Excellence in Environmental Health (PACE-EH) are listed at the end of this chapter.

Two events in the United States spurred devel- opment of a new type of planning: the September 2001 attacks and the 2005 Katrina hurricane disaster in the Gulf Coast, particularly in New Orleans. In the years following these events, greater emphasis was placed on preparedness planning. According to the U.S. Department of Homeland Security, planning for disasters ought to result in “plans that incorporate an accurate threat analysis and risk assessment and ensure that capabilities required to prevent, protect against, respond to, and recover from all-hazards events are available when and where they are needed” (U.S. Department of Homeland Security, 2007, p. 6). Although this definition is specific to disasters, the notion that planning results in a plan that takes into account data from an assessment can be generalized to health program planning efforts at the local and systems levels.

Summary of Approaches To some extent, all the approaches to health planning discussed in this section are likely to occur during the planning phase of addressing a health problem. One example is the efforts to increase adult immunization in Bowe County. The need to track immunizations necessitated the involvement of both state health officials and representatives from large health maintenance organizations (HMOs) in the planning process. A lobbyist used advocacy to gain passage of state legislation supportive of financing immu- nizations across all age groups, particularly for underinsured individuals from minority groups. The choice of vaccines was an apolitical decision based on research indicating which combinations were likely to result in the highest levels of protection for the population. Senior citizens were involved as informed consumers in shaping policy and tracking procedures, so communication action was also part of the planning process.

It is quite unlikely that all the individuals involved in the immunization planning efforts recognized the mixture of approaches being used in this community-based effort. Had they been aware of the approaches being used by various constituents, additional strategies could have been developed that would have made the planning process more effective, efficient, and palatable to parents, providers, and policy makers alike. Undeniably, a blend of approaches is typically needed, particularly in health program planning that aims to address more recalcitrant or population-based health problems.

Each of the six approaches to health plan- ning represents a way to identify a problem, identify options, and make a choice—the classic definition of decision making. In other words, planning is decision making. From the perspective of an organization or agency engaged in health planning, health-planning activities can be framed in terms of managerial and organizational decision making. This fact

70 Chapter 3 Community Health Assessment for Program Planning

of data, involvement and engagement of the community, and priority setting. CHIP con- tinues to be used by both state and local health governments (Oconto County Department of Health and Human Services, 2008; Wisconsin Department of Health Services, 2010).

Protocol for Assessing Community Excellence in Environmental Health (PACE-EH) The Protocol for Assessing Community Excellence in Environmental Health (PACE-EH), developed by CDC and NACCHO, focuses the community assessment on evaluation of environmental health conditions (CDC, n.d.). Thus, PACE-EH is a community environmental health assessment tool. The data gathered via this tool are then used to identify populations at risk and set priorities. A key feature of PACE-EH is the emphasis on health equity and social justice. As with the other public health planning models, PACE-EH focuses on building relationships with constituents and sharing the power and responsibilities with the community. PACE-EH has been used by county governments to improve the environment and thereby the health of their citizens.

In March 2006, a summit was held to address the barriers to the use of PACE-EH and the applications of this tool. The summary of the summit (Hubbard, 2006) highlighted the fact that more than 60 communities had used PACE-EH, primarily with a focus on improving air and water quality. Given the overall importance of air and water quality to individual and population health, using PACE-EH can be an effective approach to planning at local, county, and state levels. CDC continues to support use of PACE-EH, citing its contribution to many of the essential public health services (CDC, 2011; CDC, 2014).

In Summary This brief overview of the models that have been used in planning public health programs highlights the ongoing evolution of thinking in this area.

Mobilizing for Action through Planning and Partnership (MAPP) Mobilizing for Action through Planning and Part- nership (MAPP) is a strategic planning tool that helps public health leaders facilitate community prioritization of public health issues and identify resources for addressing them. The first phase of MAPP is to mobilize community members and organizations under the leadership of public health agencies. The second phase is to generate a shared vision and common values that provide a framework for long-range planning. The third step of MAPP involves conducting four assessments of four areas: community strengths, the local public health system, community health status, and the forces of change. The final step is implementation. MAPP materials can be ordered from the MAPP website, which can be accessed through the National Association of County and City Health Officials (NACCHO) website (NACCHO, n.d.).

Application of the MAPP process at a county level is possible and can result in a user-friendly, public document outlining a long-term health improvement plan at both the city and county levels. Since its release, MAPP has been established as a standard approach to planning and is widely used by local health departments (NACCHO, 2011).

Community Health Improvement Process (CHIP) CHIP, developed in 1997 (Durch, Bailey, & Stoto, 1997), incorporates organizational performance monitoring and community assets, followed by strategic planning, implementation, and evaluation. Like the other models, it involves two phases: (1) problem identification and prioritization and (2) implementation. CHIP was the first model that linked the community assessment and planning to the performance measures of Healthy People 2000 and the Healthy Communities 2000 model standards. The principles and recommended processes for conducting CHIP are similar to those of MAPP and APEXPH—specifically, collection

Perspectives on Assessment 71

Evident in this evolution is the development of tools for designing health promotion programs, particularly those focused at the community or population level. Each model has a slightly dif- ferent strength and emphasis (TABLE 3-3). Many of these materials are designed for public health agencies and leaders as well as for community members. It is also worth noting that the content of the materials and the underlying philosophi- cal perspectives are applicable to other types of health agencies that provide health programs to across the public health pyramid.

▸ Perspectives on Assessment

In the planning and evaluation model (FIGURE 3-3), the transition from planning to assessment occurs somewhat iteratively, with planning needs and approaches being informed by the data about needs and strengths. In turn, the assessment becomes directed by the planning efforts. The goal of a needs assessment is to guide and inform decisions related to problem prioritization and program development. Basically, a needs assessment is a procedure used to collect data that describe the needs and strengths of a specific group, community,

TABLE 3-3 Comparison of Models Developed for Public Health Planning

MAPP CHIP PACE-EH

Developers CDC and NACCHO NACCHO CDC and NACCHO

Appropriate for which unit/level

Broadly, community health at city, county, and state levels

Local health departments and county level

City, county, and state levels

Distinguishing emphasis

Strategic planning, community involvement in planning process

Performance of local health departments, data used for planning

Environmental health, legal advocacy on environmental issues

or population. To simplify the language, needs assessment is the term used in this chapter to encompass broadly both the deficit and the asset perspectives. Also, the term intended audience is used as a way to denote those for whom a program is intended. Only if the program is targeted at a true population is the term intended population used. The intended audiences can be a group (a relatively small set of individuals who inter- act), a community, a neighborhood, an aggregate (a set of individuals who share one characteristic in common, such as a school or a health condition), or a complete population. One of the first tasks in planning and conducting a needs assessment is to determine who is likely to make up the target audience or target population and in what larger unit they are situated.

Five types of models exist for conducting a needs assessment: the epidemiological model, the public health model, the social model, the asset model, and the rapid perspective model. Each one has its own intellectual perspective, as well as both advantages and disadvantages (TABLE 3-4). The distinctions among these types have been accentuated for the purpose of highlighting the possible different emphases. In practice, a model may be chosen but supplemented with elements of other models in accordance with resources and the purpose of the assessment.

72 Chapter 3 Community Health Assessment for Program Planning

FIGURE 3-3 The Planning and Evaluation Cycle

External trigger event or

opportunity

Health program planning

Assessment of community needs and assets

Program development and evaluation planning

Effect evaluation implementation

and results

Participant–recipient outcomes and impacts

Process evaluation implementation

and results

Program process theory implementation

Epidemiological Perspective The epidemiological perspective on needs assessment focuses on quantifying health problems, using national data sets, and applying epidemiological methods and statistics. This model seeks to answer questions with an epidemiological focus, such as, “What is the magnitude of the problem?” or “What illness and disease trends are evident?” or “What patterns of selectivity are exhibited in the distribution of the problem?” Other questions stemming from this perspective would be, “Is the problem preventable?,” “How treatable is the problem?,” and “What is currently being done?” As these questions suggest, epidemiological models

often include a focus on identifying hazards, risks, and precursors to the health problem.

Examples of tools used in the epidemiolog- ical model are disease and death registries and national probability sample surveys such as the National Health Information Survey (NHIS) and the National Health and Nutrition Examination Survey (NHANES). An advantage of epidemi- ological models is that they provide data for assigning relative weights to the seriousness of a health problem, the importance of that health problem, and its prevalence. However, these models do not provide a breadth of data that might also be key in prioritizing health problems.

Perspectives on Assessment 73

TA B

LE 3

-4 A

C om

pa ris

on o

f t he

Fi ve

P er

sp ec

tiv es

o n

Co m

m un

ity H

ea lth

a nd

N ee

ds A

ss es

sm en

t

E pi

de m

io lo

gi ca

l Pe

rs pe

ct iv

e Pu

bl ic

H ea

lt h

Pe rs

pe ct

iv e

So ci

al P

er sp

ec ti

ve As

se t

Pe rs

pe ct

iv e

Ra pi

d Pe

rs pe

ct iv

e

P o

p u

la ti

o n

a

ss e

ss e

d Po

p u

la ti

o n

s St

at e

an d

co

m m

u n

it ie

s o

r co

u n

ti es

Po p

u la

ti o

n s,

s el

ec te

d

ag g

re g

at es

C o

m m

u n

it y,

n

ei g

h b

o rh

o o

d s

C o

m m

u n

it y,

n

ei g

h b

o rh

o o

d s

D a

ta s

o u

rc e

s R

eg is

tr ie

s, n

at io

n al

p

ro b

ab ili

ty s

am p

le s

u rv

ey s,

ex

is ti

n g

n at

io n

al d

at ab

as es

St at

e an

d lo

ca l

ag en

ci es

, v it

al

re co

rd s

In d

iv id

u al

s u

rv ey

s,

n at

io n

al s

u rv

ey s

R o

st er

s o

f ag

en ci

es , f

o cu

s g

ro u

p s,

m ap

s

W in

d sh

ie ld

s u

rv ey

s,

ex is

ti n

g d

at a,

in te

rv ie

w s

E x

a m

p le

s N

at io

n al

H ea

lt h

In te

rv ie

w

Su rv

ey (N

H IS

), H

ea lt

h ca

re

C o

st U

ti liz

at io

n P

ro fil

e (H

C U

P)

M A

PP , C

H IP

, PA

C E-

EH U

.S . C

en su

s, B

u re

au o

f La

b o

r St

at is

ti cs

A ss

et s-

Ba se

d

C o

m m

u n

it y

D ev

el o

p m

en t

In st

it u

te

R ap

id A

ss es

sm en

t an

d

R es

p o

n se

(R A

R) , R

ap id

A

ss es

sm en

t an

d R

es p

o n

se

an d

E va

lu at

io n

(R A

RE )

Ty p

e s

o f

n e

e d

s a

ss e

ss e

d

N o

rm at

iv e,

e xp

re ss

ed ,

an d

re la

ti ve

n ee

d s

ca n

b e

es ti

m at

ed

N o

rm at

iv e

an d

re

la ti

ve n

ee d

s ca

n

b e

es ti

m at

ed

R el

at iv

e n

ee d

s ca

n b

e es

ti m

at ed

, p er

ce iv

ed

n ee

d s

ar e

d ire

ct ly

d

et er

m in

ed

Pe rc

ei ve

d n

ee d

s,

p er

ce iv

ed

st re

n g

th s

N o

rm at

iv e

an d

p er

ce iv

ed

n ee

d s

A d

v a

n ta

g e

s St

at is

ti ca

lly s

o u

n d

a n

d

g en

er al

iz ab

le fi

n d

in g

s A

d m

in is

tr at

iv el

y so

u n

d ; i

n cl

u d

es

fo cu

s o

n

co n

st it

u en

t co

n ce

rn s

St at

is ti

ca lly

s o

u n

d ;

p ro

vi d

es in

fo rm

at io

n

o n

fa ct

o rs

co

n tr

ib u

ti n

g t

o

h ea

lt h

p ro

b le

m

Ex is

ti n

g

re so

u rc

es a

re

id en

ti fie

d

Q u

ic kl

y co

m p

le te

d a

n d

p

ro vi

d es

b as

ic in

fo rm

at io

n

D is

ad va

n ta

g es

N o

in fo

rm at

io n

o n

p

er ce

iv ed

n ee

d s;

lo ca

l va

ri at

io n

s m

ay n

o t

b e

ca p

tu re

d o

r d

es cr

ib ed

R el

ie s

o n

o th

er d

at a

so u

rc es

; p er

ce iv

ed

n ee

d s

ar e

n o

t d

ire ct

ly d

et er

m in

ed

D o

es n

o t

d ire

ct ly

m

ea su

re t

h e

ex te

n t

o f t

h e

h ea

lt h

p

ro b

le m

D o

es n

o t

m ea

su re

t h

e ex

te n

t o

f t h

e h

ea lt

h p

ro b

le m

D o

es n

o t

m ea

su re

t h

e ex

te n

t o

f t h

e h

ea lt

h

p ro

b le

m ; m

ay m

is s

p ro

b le

m s

o r

ca u

se s

74 Chapter 3 Community Health Assessment for Program Planning

collecting data regarding social characteristics, such as income, and on collecting data from specific aggregates about specific social and economic topics, as is done by the U.S. Census. The social approach reflects a social determinant perspec- tive on health and healthcare needs. In health care, planning based solely on social indicators is considered incomplete. Without the health indicators, the community needs assessment is incomplete. Nevertheless, assessments that include social ecological data do generate crucial infor- mation that helps identify antecedents or prior conditions leading to the health problems.

Asset Perspective A fourth perspective on needs assessment is based on asset models, which focus on the strengths, assets, abilities, and resources that exist and are available rather than on the needs, deficits, lacks, and gaps between the healthy and ill. Assessments from this perspective seek to answer questions such as, “Which social and health resources exist within the community experiencing the health problem?” or “What do community members view as strengths and resources within their community?” Another question might be, “To what extent are the resources mobilized or able to be mobilized to address health problems?”

Asset-based community development (ABCD) has been recognized as potentially useful for creating healthy communities for children (Baker et al., 2007) and assessing mental health care resources in Ethiopia (Selamu et al., 2015). Indeed, asset models represent an important counterbalance to deficit models (Morgan & Ziglio, 2007) in efforts to overcome barriers to improving community health. Inherent in the asset perspective is a focus on the collective resources of individuals, particularly in the form of their social networks. In this regard, the asset perspective is aligned with the emphasis on social capital. An asset assessment could reasonably include an inventory of the social capital within the community being assessed for health problems.

Public Health Perspective Public health perspective focuses on quantifying health problems for the purpose of prioritizing the identified health problems being addressed with limited resources. From this perspective, the questions reflect an interest in the health of a population. Needs assessment questions from this perspective might be, “What is the seriousness of the problem?” or “What is the distribution of the problem?” or “What factors are contributing to the problem?” Other questions relate to the prioritization aspect, such as, “What is the per- ceived importance of the problem?” or “What resources are available to address the problem?”

Public health approaches to needs assessment typically rely on existing data and epidemiological data. Specific tools or models are used in the public health approach to needs assessment, such as MAPP and PACE-EH. These models provide a framework for determining which types of data ought to be collected as part of the needs assessment. There is considerable overlap between public health models and the epidemiological approach to needs assessment. Although these models are fairly comprehensive, they are somewhat weak in terms of their ability to account for the sociocultural aspects of health.

Social Perspective The social perspective on needs assessment focuses on quantifying characteristics that contribute to the sociocultural, economic, and political context that may affect the health of individuals. This approach leads to questions focused on social ecological determinants of health. For example, a social perspective question might be, “What is the relationship among health problems and social characteristics?” or “Which social trends are evident in health behaviors?” Other questions stemming from this approach are, “What is the relationship between the use of social and health resources and the problem?” and “How have social and health policies affected the magnitude, distribution, or trends of the problem?” Key char- acteristics of a social perspective are a focus on

Types of Assessments 75

disasters such as Hurricane Katrina (Rogers, Guerra, Suchdev, & Chapman, 2006).

Stimson, Fitch, and Don Des Poznyak (2006), in a review of RAR studies conducted in several countries, found that RAR was effective in linking the assessment findings to the devel- opment and implementation of new or modified interventions.

▸ Types of Assessments Before initiating a full-scale needs assessment, planners and planning groups need to be familiar with assessment as a process. This process begins by studying available data in order to gain a working knowledge of the community and the prevalent health problems. The familiarization assessment, as it is called, is a starting point from which to consider whether more data are needed and whether to proceed with conducting a larger-scale needs assessment. It is possible that a local agency has already done a needs assessment that might be adequate for the task at hand. Thus, becoming familiar with the community, the health prob- lem, and existing assessments can save time and effort during the planning process. If it appears that an assessment is needed or is mandated by a regulating body, five types of assessments listed in Table 3-4 may need to be performed. Again, the distinctions are accentuated for the purpose of highlighting the possibilities, but the distinctions are blurred in practice.

Organizational Assessment An organizational assessment determines the strengths, weaknesses, opportunities, and threats to the organization providing the health program. While an organizational assessment can be thought of as part of the logistics planning for a health program, it is critical to have a good sense of the organizational willingness and capabilities to provide a program to address the health problem under consideration before planning proceeds. In the PRECEDE-PROCEED model (Green & Kreuter, 2005), which is widely used by health educators, an organizational assessment is viewed

The asset perspective on community assessment seeks to identify and then build on the capabilities of a community and thus resolve health issues. Although the asset models have some appeal, especially to community stakehold- ers, gathering asset data can be challenging. No generally accepted set of asset indicators exists. And rarely does asset information exist at the time of the assessment, making data collection necessary. These disadvantages contribute to asset models being less widely used as a sole approach to needs assessment, and they are poorly integrated into the more widely used models of needs assessment.

Rapid Perspective In some circumstances, expediency is needed or desirable when conducting an assessment. Over the past 20 years, a consensus has emerged that a rigorous, reliable approach to rapidly assess- ing the health of a population or community is both possible and of value. Essentially, the rapid assessment approach uses multiple methods— such as focus groups, existing data, interviews, and mapping—to involve the community in rapidly developing and implementing needed health interventions. A rapid assessment seeks to answer the key question, “What are the most immediate and pressing needs that can be addressed with readily available resources?” As this question implies, the focus is on obtaining a quick response rather than ensuring the depth or breadth of the assessment.

Various terms are used as names for this type of assessment: rapid assessment and response (RAR) and rapid assessment and response and evaluation (RARE). These names denote the process by which an assessment is rapidly conducted and used to develop interventions. The RARE model consists of community participation, use of multi- ple methods and triangulation, and evaluation of both short- and long-term outcomes (Kamineni, Turk, Wilson, Satyanarayana, & Chauhan, 2011; McNall & Foster-Fishman, 2007). These basic steps are also part of the RAR model, which has been used as part of the response to natural

76 Chapter 3 Community Health Assessment for Program Planning

program?” or “Which other health organizations are providing what types of services?”

Typical market analyses, such as those conducted in businesses, differ from marketing assessments for health programs in several ways. In health programs, marketing concepts are adapted to reflect the programs’ social and behavioral inter- vention focus. In addition, the price and packaging aspects that are addressed in marketing assess- ments play different roles in health programs.

Key data to be collected in a marketing assess- ment deal with competitive programs (available community resources) and overall interest in the intended program. Incorporating elements of the marketing assessment into assessment activities minimizes lost opportunities to collect key data and helps to provide a more complete assessment of the conditions that might affect both services utilization and health outcomes.

Needs Assessment A needs assessment, in the more narrowly defined, traditional sense, is a means by which to determine the gaps, lacks, and wants relative to a defined population and to define a specific health problem. Data from a needs assessment are used to identify health problems or conditions that can be addressed by a health program. In this way, the assessment serves as a starting point for planning a program, such as was done by Norris and colleagues (2011) and then subsequently for planning, imple- menting, and evaluating the program (Palmer, Samson, Batra, Triantis, & Mullan, 2011).

A needs assessment provides health-related information that enables planners to gauge the priorities to be given to specific health problems, and it helps identify the trade-offs that inevitably arise in addressing one health problem rather than another. Often a needs assessment is done to answer the question, “What health problems exist, and to what extent?” Fundamentally, a needs assessment identifies normative needs. A needs assessment can be roughly equated to the “community health status assessment” in MAPP.

Typically a needs assessment is problem- oriented; thus, it tends to begin with a stated health

as a key component in planning health educational interventions. The organizational assessment seeks to answer the key question: “What is the capability and willingness of the organization to provide the health program?” The breadth of the organizational assessment is akin to the infrastructure assessment addressed in MAPP.

Data for an organizational assessment are gathered from members of the organization as well as from existing organizational records and documents. These data help determine the organizational feasibility of developing and implementing a health program—that is, whether adequate and appropriate resources are available and whether the health program fits with the organization’s mission and goals. One key aspect of the organizational assessment is the assessment of human resources within the organization, with particular attention being paid to their ability to meet the needs identified in the community needs assessment. This type of assessment can also identify changes needed within the organization as a prerequisite to providing the health program. In this way, the organizational assessment provides critical information for developing internal strategies to ensure the success of the health program.

The extent to which an organizational assess- ment is critical to the success of a health program is reflected in the organizational capacity study carried out by Roberts-Gray, Gingiss, and Boerm (2007). Their measures of organizational capacity predicted both the quantity of implementation activities and the quality of the implementation of a schoolwide intervention.

Marketing Assessment Just as understanding the needs and assets of a target audience is key in program planning, it is equally important to understand the extent to which the target audience would be interested in the health program. The data for a market assessment can be collected at the same time as the needs assessment data are gathered. This type of assessment seeks to answer questions such as, “What will draw the target audience into the

Steps in Planning and Conducting the Assessment 77

At the infrastructure level of the public health pyramid, however, workforce assessments are particularly germane. A workforce assessment seeks to answer the question, “Which human resources exist at which level of expertise to address the health needs?” A workforce assessment examines the current competencies among the workforce, trends and drivers of change related to the quantity and quality of the workforce, and building scenarios to understand the potential size of the gap between projected needs and the projected available workforce.

Across the health professions, workforce assessments conducted by scholars as rigorous research projects have revealed a dire situation for health care. In the foreseeable future, predictions of shortages continue for personnel in various health-related fields, such as nursing (American Association of Colleges of Nursing [AACN], 2014), occupational health (Lin, Zhang, & Dixon, 2015), and medicine (Dall, West, Chakrabrth, & Iacobucci, 2016). These predictions make it nec- essary that. before undertaking the development of any health program, a local assessment be conducted to identify the current and future workforces that will be tapped into to support the anticipated program. After all, there is no point in developing a great program on paper if it will not be possible to hire health profession- als with the qualifications needed to make the program a success in the real world. Just as the organizational assessment is critical to deter- mining the support of potential programs, so the workforce assessment is critical to determining the feasibility of attracting program personnel.

▸ Steps in Planning and Conducting the Assessment

No one right way exits for conducting a community health assessment. Nevertheless, certain basic steps exist across any approach to community health assessment. First, involve community

problem about which more information is wanted or needed. Another facet of needs assessment is the delineation of the community as a subsystem deserving of specific assessment. Sometimes more detailed information is needed about one aspect of a community, and a needs assessment focused on the community can provide such information.

Community Health Assessment A community health assessment is used to estab- lish the magnitude of selected health problems in a selected community, neighborhood, or other designated locality relative to the strengths and resources within that community, and to determine the priority the community gives to addressing the health problem. A community health assessment casts a broad net, encompassing all aspects of the community. It examines health and human service resources and assets, as well as the health problem and other community weaknesses. This type of assessment seeks to answer the question, “What are the key health problems, and which resources are available to address those health problems?” In this sense, a community health assessment encompasses and integrates each of the four assessment models described previously. From this integrative perspective, the chapter provides details on conducting a community health assessment.

The Affordable Care Act (ACA) includes a mandate for nonprofit healthcare organizations to conduct a community health needs assessment. According to the regulations, community is defined geographically, but it must also include the target population served by the healthcare organization. The healthcare organization must implement a strategy to address the gaps identified through the community health needs assessment. A greater emphasis on community health needs assessment is likely to generate new approaches to meeting needs as well as potentially new awareness of the community health problems.

Workforce Assessment A workforce assessment is not commonly thought of as part of a community health assessment.

78 Chapter 3 Community Health Assessment for Program Planning

while also recognizing that an optimal size for a task force or work group is 5 to 7 persons.

One “law” of groups points out the chal- lenges in working collectively: There will always be one person who is not a team player. Given this likelihood, knowing the strengths and weaknesses of the individuals involved helps to understand why each individual has been selected as a member. Having such information helps ensure that a planning group possesses a balance of strengths that will contribute to an efficient and effectively functioning group. Attention to the composition of the planning group also ensures that a breadth of knowledge and concerns is represented while eliminating disruptive individuals. Reinke and Hall (1988) also remind us that it is critical to have trained, skilled, and knowledgeable planning staff. Simi- larly, a growing consensus suggests that successful planning processes begin with developing the planning group’s awareness, concern, and skills to address the problem at hand.

Different types of public health planning groups exist, but the most common type is a consortium. A consortium is a quasi-temporary body that is formed for a specific programmatic purpose and that has an independent sponsor, broad representation, and experts as members. Consortia are popular means to increase involve- ment of community members and to address (implicitly) the paradox of professionals not having the “right” solution.

Overall, the literature on participation in decision making reveals a pattern in favor of involving those affected by the decision in the decision-making process (e.g., Kegler, Painter, Twiss, Aronson, & Norton, 2009; Quick & Feldman, 2011). For this reason, the involve- ment of stakeholders throughout the planning, implementation, and evaluation cycle is highly recommended. It is also important to educate those involved in the planning process, using a communication action approach (as discussed earlier in this chapter). When those individuals who will be affected by the decision are involved in making it, their resistance to the change is likely to be diminished and they will begin to

members in the development and execution of the community assessment. Second, define the community or population to be assessed, followed by making decisions about which data to collect regarding the nature of the health problem, such as the magnitude of the problem, precur- sors to the health problem, and demographic and behavioral characteristics. Third, collect these data using a variety of data sources and approaches. Once the data have been collected, the assessment and planning team must analyze the data using statistical procedures to arrive at statistical statements about the health problems in the population. Based on these data and the statistics obtained via their analysis, the last step seeks to develop a summary statement of the need or the problem that ties together the antecedent and causal factors of the health problem, along with the asset factors that counter the existence of the health problem.

Form and Develop the Team Planning is a collective activity. The individuals involved at the various stages of the planning cycle easily influence the directions and decisions made. A key strategy for achieving successful planning is to have a visible, powerful sponsor. Given that politics of one form or another are inherently part of the planning process, having a backer who is recognized, respected, and influential becomes an essential element to successfully planning and implementing a health program.

Many factors influence the selection of the planning team members, including legal considerations (i.e., antidiscrimination laws and municipal mandates for advisory boards), the reasons for wanting to participate, the level and type of expertise that an individual can contribute to the process, the amount and type of resources that an individual can contribute to the process, the person’s status as a current or potential user or client of the health program, and the person’s role as an advocate for a group likely to be affected by the health program. Group size is another consideration, with groups of 10 to 15 being acceptable for the planning process

Steps in Planning and Conducting the Assessment 79

stakeholders can identify and to which they can devote attention and energy. A vision frames information for the stakeholders and helps identify economic assumptions that may affect the overall health program.

Part of the process of creating a vision of the final “product” is reaching a consensus on how to arrive at that final ideal. In this regard, one element of creating a vision is deciding on a system for prioritizing both problems and pos- sible solutions to the highest-priority problems. How decisions are made—whether by voting, consensus, or complex algorithm—should be one of the first decisions of the planning group.

Involve Community Members Before starting a community health assessment, the planners will ideally devote time to devel- oping a strategy for involving members of the community to be assessed. The rationale behind involvement of those likely to be targeted by a program stems from a philosophy of empowerment as well as a practical concern with stakeholder and consumer reactions to the data. From the philosophical perspective of empowerment, involvement by community members enhances both their capacity to assist in the assessment and their ownership of the data gathered and results produced by the assessment. This theme of involvement is carried throughout the phases of planning and evaluating a health program.

From a practical perspective, involving those likely to be affected by the assessment has immediate and direct consequences for how the community health assessment evolves. The involvement of community members can even shape the questions addressed by the needs assessment. Of course, the strong views and bias of any one group can also become evident during the planning of the assessment. Program planners can use these revelations to begin to anticipate how those views might influence the interpretation of the data. By involving commu- nity stakeholder groups in the community health assessment, planners can uncover, acknowledge, and hopefully address their concerns.

“own” the program or plan, although participa- tion does not guarantee ownership (Goodman, Steckler, Hoover, & Schwartz, 1993).

The last consideration in the team development stage is the selection of a leader for the planning group. Duhl (2000) argues that many types of leaders may be necessary for an effective planning process. A leader can emerge or be appointed based on his or her capability to function as an educator, a doer, or even a social entrepreneur. At any point in the planning process, or even during oversight of the program implementation, different individuals may be better suited to play a leadership role. Recognizing the fluidity of the leadership situation and acting on that recognition are both healthy and useful.

What may be less fully articulated, espe- cially during the earliest stages, is the degree of formalization of leader selection. In other words, the planning group needs an acknowledged and standard process for designating a legitimate leader. This process can and does vary, ranging from the ad hoc emergence of a natural leader to the election of an individual from a slate of candi- dates according to formalized bylaws. Regardless of where along that continuum the group wants to be, the key will be to have an articulated and accepted process that facilitates the planning process rather than hindering team members’ creativity and commitment. Ideally, each member of the planning group needs to be actively involved in ensuring that the process is open and agreed upon.

Create a Vision The first step in planning, according to the American Planning Association, is to create a vision. Development of a vision is also one of the first steps in the MAPP models of assessment.

The nature of the mission statement appears to have a bearing on the success of healthcare organizations (Bart, 2007). For example, Healthy People 2020 outlines a national vision of elim- inating racial and ethnic disparities in health and increasing the quality of life. Whatever the trigger event, health program planners must create a vision with which existing and future

80 Chapter 3 Community Health Assessment for Program Planning

influenced by who is doing the assessment. That “who” can be defined geographically, enabling the population of interest to be delineated by a site, such as workplace, location of residence, or school. Using locality as the defining charac- teristic is common, and ZIP code areas, census tracts, community areas, or legal boundaries are often used to define who is assessed. A state health department or a state health program will focus on the state population, whereas a small, local, nonprofit agency is likely to focus only on individuals who are potential customers. For example, the Traditional Indian Alliance in Tucson serves only Tucson’s Native American population. Not surprisingly, its needs assessment was very limited in terms of both geography and population segment (Evaneshko, 1999). In contrast, a United Way organization in a large metropolitan area will assess the health and social needs of the population in its catchment area.

Using highly specific parameters to define “who” allows the assessment to be more focused and detailed. For example, Dickerson, Smith, Ahn, and Ory (2011) used the needs assessment data from a seven-county region of Texas to focus on the needs of individuals with diabetes. The data from these authors’ work-site assessment enabled them to develop work-site health pro- motion programs that addressed both work-site and other health risks. This example shows that defining “who” based on a narrowly defined location may be a convenient means to access an aggregate, obtain detailed information, and tailor a health program very specifically.

In program planning and evaluation, the term intended audience refers to those for whom the program or intervention is designed and intended—in other words, those who are targeted by the program. The term target population is used if the program is intended for an entire population rather than a subpopulation. Put simply, the target audience includes all potential participants. Those who actually receive the program or intervention are referred to as the recipient audience. Think- ing about this distinction between targets and recipients helps clarify who ought to be included in the community health assessment: Basically,

Involving community members is rarely an easy venture. Numerous barriers to their involve- ment must be overcome: time constraints on busy individuals, competing interests for available time, parking problems, limited accessibility of the meeting location, lack of awareness of the opportunity for involvement, feelings of inad- equacy or insecurity about being involved, and lack of day care for members’ children. There is no one best way to increase involvement of community members. Instead, multiple strat- egies are needed, and the ones used are likely to evolve as the community health assessment proceeds. In addition to strategies that specifically address barriers to community participation, other strategies to increase involvement can include obtaining names of key individuals from agency personnel, providing food as an incentive, providing informal training or skills related to being involved, having specific tasks in which individuals can be involved, or having regularly scheduled meeting dates and times.

Sometimes, however, it may not be wise to focus on involving community members: when severe time constraints on completing the community health assessment exist, when severe fiscal constraints limit the scope of the assessment, when profound allegiances might affect the quality of interactions among commu- nity members, or when insufficient leadership skills exist to initiate and sustain community involvement. At other times, community mem- bers simply must be involved in the community health assessment: when there is a mandate from a funding agency for community involvement, when doing so will reduce the perception of being excluded, when insiders’ connections and perceptions are needed to ensure a complete community health assessment, or when the goal is to have the community take responsibility for sustained implementation of the health program.

Define the Population Delineating who is to be assessed is an important early step in conducting a community health assessment. The question of who to assess is often

Steps in Planning and Conducting the Assessment 81

statewide needs assessment every 5 years as part of the states’ application for Title V block grant funding. Similarly, local jurisdictions may request a needs assessment as part of a strategic plan to be more responsive to changing health and social needs of their constituents and to address challenges created by budgetary constraints. If township or county officials face pressure to address the health or social problems of a particular group, such as adolescents, immigrants, or the disabled, those officials may request a community assessment to substantiate or discredit the needs of the group. This may be one way that the political aspect of program planning is played out.

For larger health agencies, organizations, or jurisdictions, a community health assessment might be conducted in response to calls for grant proposals that specify health problems or conditions as a high priority for being addressed. For example, the CDC has funded health pro- motion programs designed to reduce racial and ethnic disparities with regard to diabetes, cardiovascular diseases, infant mortality, HIV/ AIDS, and immunizations. An agency wishing to compete for these CDC funds needed to identify within its community the specific racial or ethnic disparities as well as needs and assets relative to one of those five health problems.

Investigate During the investigation phase of planning, data are gathered that will be used first to prioritize health problems and then to prioritize possible programmatic solutions. Generally, data rele- vant to planning health programs come from community assessment, population preferences, previous program evaluations, and research on possible interventions. The importance and possible scope of data collection carried out through community assessment are very critical factors that are addressed in subsequent chapters.

Two elements of the nonlinear nature of health program planning are worth introducing during this stage of planning. One is the need to focus on future considerations—specifically, interventions—even before the program direction

both groups should be included, and the target audience encompasses the potential recipients.

The parameters used to distinguish indi- viduals for whom the program or intervention is intended from individuals for whom it is not intended become the boundaries of the target population. The target audience is usually some portion of the population at risk—that is, those individuals who have some social, physical, or other condition that increases their likelihood of an undesirable health problem or state. The term at high risk is usually reserved for those individuals with the highest probability of having an undesirable health state or outcome.

When conducting a community health assess- ment, planners must recognize that the boundaries of the target audience are likely to change with the collection and analysis of the data. For example, imagine that when a community health assessment is begun, an entire neighborhood or community area is viewed as the target audience. As epide- miological data and asset data are analyzed and interpreted, the planners may realize that only the black elderly residents or white adolescent residents or working mothers are at high risk for a health problem that can be addressed by the organization. This evolution of “who” from the broad boundaries to a more refined definition of the target audience is what ought to occur as a result of the community health assessment.

Define the Problem to Be Assessed Just as the “who” of a community health assessment evolves with the collection of data and synthesis into information, so, too the “what” is likely to evolve as the assessment process unfolds.

Community health assessments are undertaken to address a purpose—and that purpose is never an altruistic desire to identify the breadth of the health problems that exist within a community. More likely, the community health assessment will be performed in response to a specific mandate. For example, the federal Maternal and Child Health Bureau mandates that all states conduct a

82 Chapter 3 Community Health Assessment for Program Planning

strongly held beliefs about what is scientifically the right thing to do.

An additional benefit of focusing on inter- ventions is that this tactic helps avoid an undue emphasis on needs assessment and data collec- tion. Goodman et al. (1993) found that planning groups have a tendency to “frontload” the planning cycle, by devoting considerable time and effort to collecting risk and health problem data and conducting data analysis. Focusing on identifying realistic interventions balances the early planning stages with the later stages of implementation.

Prioritize During the prioritization stage, data gathered during the community health assessment, along with the information on preferences and interven- tions, are integrated into a decision about what to address and how. During the establishment of priorities, the planning group is likely to face conflicts stemming from the group members’ different philosophies about how to establish priorities. As discussed earlier with regard to ethics, no single ethical approach to prioritizing issues is inherently right or wrong. Centering the team’s debates on what ought to be the guiding philosophical and ethical framework for making prioritization decisions thus can be an important step toward building consensus, trust, and mutual respect among stakeholders. Once these underlying principles have been agreed upon, a systematic, quantitative approach can be applied to determine health priorities. These priority-establishing techniques are covered in more detail in subsequent chapters.

Make a Decision Inevitably, the priority ranking of health problems will not be acceptable to some stakeholders. As a consequence, the rankings may need to be revised by seeking stakeholder input or until a consensus is gained. Such an activity reflects the reality of blending the rational and political approaches to program planning. These decisions regarding which health problems to address serve

has been decided. The other is the need to be aware of the willingness of key individuals to support the planning and program process and to understand the quantification of health problems in terms of quality of life.

Interventions are actions that are done intentionally to have a direct effect on the health problem or condition. This broad definition of intervention includes medical treatments, phar- macological treatments, behavioral treatments, and health policy development, as well as educa- tion and skill enhancement, social support, and financial aid. In the healthcare realm, research resulting in EBP can be used to determine the effectiveness of potential interventions. Of par- ticular concern are the sensitivity of the health problem to the intervention and the specificity of the intervention with regard to the health problem it addresses. Another consideration in determining the effectiveness of an intervention is the theoretical or conceptual logic underlying the way in which the intervention alters the health problem or condition. This issue is discussed in more detail in subsequent chapters.

Solutions, whether programmatic interven- tions or other ideas, often exist even before the problem is formally identified. Proponents of the solution might, for example, be waiting for a win- dow of opportunity for “their” idea to be applied. Although having ready-made solutions available can certainly be helpful, too many individuals are inclined to jump on a particular solution bandwagon before the planning process has fully explicated the problem and all of its potential solutions. To the extent that any intervention or solution is well suited to a clearly defined problem, the planning process is effective.

According to Blum (1982), one factor that adversely affects planning is wishful thinking. In other words, solutions are sometimes based on idealistic and overly optimistic hopes rather than on scientific knowledge. This factor leads to the failure to examine the range of possible effective interventions or solutions to the prob- lem. A key to avoiding this pitfall is the use of evidence-based medicine (EBM) and EBP, even though facts may not convince individuals with

Anticipate Data-Related and Methodological Issues 83

cycle and program implementation. If these foci are maintained, then feedback loops will develop more quickly, new triggers to additional planning activities will be perceived, and the evolution of involvement of stakeholders will be more rapid.

▸ Anticipate Data-Related and Methodological Issues

Data-related issues associated with conducting a needs assessment are well recognized and documented (Altschuld & Witkin, 2000; Lee, Altschuld, & White, 2007). Such issues must be addressed to enhance the quality of the data collected, thereby improving the accuracy of the community health assessment. Several types of methodological issues are discussed here in the context of conducting a community health assessment rather than within the more typical research framework.

First, when one is attempting to uncover what is occurring, there is a temptation to ask those experiencing the problem to provide information about the problem. As mentioned earlier, the trouble with this approach is that those receiving services may be systematically different from those not receiving services. Also, this approach is unlikely to uncover latent needs, meaning that some needs may not be manifested in an easily recognizable form. In the earlier example near the beginning of the chapter of the community that wanted a swimming pool, a latent need was to have an inequity addressed as manifested in community members’ perceived need for recreational opportunities.

Another methodological problem is that asking potential consumers of the program about their needs has the potential to bias the answer. In other words, when asked about their needs, community members may take the opportunity to express all kinds of frustrations, wants, and needs. In addition, asking about needs, problems, and deficits does not allow for understanding the

as the starting point for program development and then implementation.

Decisions about which health problems to address can fail for two major reasons. First, the organizational norms and institutionalized objec- tives may support conflicting priorities regarding health problems, limiting which interventions are acceptable. Second, the experts conducting the community health needs assessment may be biased, which will shape their findings. In other words, data from the community health assessment that are made available for planning may reflect the views of those who conducted the assessment rather than revealing the full scope of what exists as both strengths and problems in the community.

Once a health problem or condition has been chosen as the focus of a health program or service, a detailed implementation plan needs to be developed, along with a plan for conducting the evaluation. Planners should be aware that once a health problem has been identified, the composition of the planning group is likely to change. Members with vested interests will remain part of the team, while those with little expertise or interest in the chosen priority will fade away. At this point in the cycle, it may be important to revisit the group’s composition and address why potential key stakeholders should become involved.

Implement and Continue The planning cycle is complete after one full iteration, when the program is implemented, monitored to determine the extent of the imple- mentation, and assessed for its effectiveness. For some health programs, implementation includes a termination phase or phase-out period, as hap- pens with health programs funded for a limited time. Evaluation, whether of immediate effects or long-term outcomes, provides a basis for further program planning and completes one cycle.

Throughout the planning process, multiple foci are useful as evaluation end points: an epi- demiological focus on the characteristics of the health problem; a scientific focus on identifying the best possible, feasible programmatic inter- ventions; and a managerial focus on the planning

84 Chapter 3 Community Health Assessment for Program Planning

sampling strategies being possible. In terms of conducting a needs assessment, the sampling decision depends on the degree to which indi- viduals providing community health assessment data must be representative of the entire target population. The epidemiological and social approaches to assessment favor strategies that include individuals who look as much as possible like the target audience. However, if primary data are being collected, developing and employing strategies to achieve representativeness of the sample can be very difficult and costly. This issue is especially critical with hard-to-reach populations, such as intravenous drug users, emotionally abused spouses, or women who have experienced a perinatal loss. Less expensive but also less scientifically rigorous sampling strategies are certainly possible. The key decision to be made is how important it is to describe the population with a high degree of accuracy, based on data from less than the entire target population or target audience.

Overarching concerns are the cultural appro- priateness of the data collection methods and the cultural competence of the data collectors and interpreters. Culture, language, and ethnicity all influence the responses of individuals to survey questions. The match between measurement approaches and the conceptualization of the neighborhood, for example, has subsequent implications for program intervention devel- opment (Nicotera, 2007).

Another key issue is the need to have community-level indicators—that is, data about the community, rather than data about individu- als, that are then aggregated by community. For example, daily intake of fat is an individual-level indicator, and an average of percentage of daily intake of fat based on sampling of all residents in the community is still an individual-level indicator. The percentage of grocery store shelf space allocated to low-fat foods, by contrast, is a community-level indicator. The percentage of workers at a work site who smoke is an individual-level indicator, but the number of antismoking posters or announcements at the work site is a work site–level measure.

community’s assets, strengths, potential, resources, and capabilities. Thus, data collection methods are best designed to enable the collection of data that would fall on both sides of the equation.

Community health assessments can take as long as a year to accomplish, particularly if the assessment is comprehensive in scope and involves community members in the process. Unfortunately, time constraints are a reality that can heavily influence both the quality and the quantity of data collected as part of the assessment. Realistic strategies and designs for collecting data must match the time lines; otherwise, only partial data will be collected and will most likely be imbalanced in nature, leading to faulty conclusions.

In addition, the measures used to collect data must adhere to scientifically rigorous standards. Most important, the instruments used must have both validity and reliability. Validity is the degree to which that instrument measures what it is intended to measure. Reliability is the degree to which the instrument will yield the same results with different samples. Epidemiological mea- sures, such as mortality, have high validity; death is rarely misdiagnosed. By contrast, the underlying causes of death as reported on death certificates are prone to both validity and reliability problems. The validity problems stem from conceptual issues of whether the cause of death ought to be the immediate cause or the underlying cause. The reliability problems relate to how each death certificate is completed and coded. Similarly, other epidemiological measures, such as adequacy of prenatal care, have been questioned with regard to validity and reliability. In terms of conducting a community health assessment, the point being made here is that no data are perfect, and these imperfections can lead to inaccurate numbers and hence faulty program planning decisions. Given this potential for error, planners should openly discuss the limits of the data and take reasonable scientific steps to obtain the best data possible.

The issue of determining from whom to collect the community health assessment data will always be important. This is a sampling problem. Sampling is a science, with numerous

85Discussion Questions

community-based or social services interventions. Assessments at this level hence would focus on describing the social context of those individuals with the health problem or condition, as well as the community subsystem in terms of local infra- structure capabilities and human services agencies.

At the population services level of the public health pyramid, health problems are viewed across a population. As a consequence, assessments at this level are likely to be epidemiological in approach, with attention being paid to describing the magnitude of various health problems or conditions. At the same time, social sciences approaches to assessment, using population data on social indicators, may provide valuable information about contributing and antecedent factors to the health problems and conditions.

At the infrastructure level of the public health pyramid, the concerns relate to the capabilities of the organization or the health delivery system to address the health problems or conditions at the direct, enabling, and population services levels of the pyramid. In the more ideal sense, the com- munity health assessment is most appropriate for this level because it encompasses understanding the health problems and conditions within the social context of the target population as well as identifying the assets that are available to address those health problems and conditions. In addition, the organizational assessment fits at this pyramid level because it focuses on identifying the resources, capabilities, and mission currently available. The findings of the organizational assessment, when considered in conjunction with the find- ings of the community health assessment, ought to establish a solid foundation for garnering resources and planning health programs at the corresponding optimal level of the pyramid.

▸ Discussion Questions 1. Select one of the perspectives on assess-

ment. In what way does that perspec- tive change, alter, or influence each step in the process of performing a needs assessment?

Very few ready sources of community-level measures or indicators of aggregates such as work sites or communities are available. Rather, it takes creativity, working with the community members, and careful consideration to develop communi- ty-level indicators and then to collect data reliably. But this extra work is worth the effort: For many of the health problems targeted by health promo- tion or disease prevention programs, what exists in the community will be extremely important as a component of assessing the antecedents and causal factors to the health problems.

In summary, there are five “principles” of collecting data for a community health assessment. One, collect data from more than one source; in other words, use multiple methods and multiple sources, and be multicultural. Two, involve mem- bers of the community in the design, collection, and interpretation of the community health data: Be inclusive and be empowering. Three, give full disclosure and then get informed consent from individuals from whom data are being collected: Be forthright, be honest, and be safe. Four, go beyond the available and collect data from unlikely but enlightening sources: Be creative, be inventive, and be open. Five, be as scientifically rigorous as time and other resources allow: Be scholarly, be interdisciplinary, and be systematic.

▸ Across the Pyramid At the direct services level of the public health pyramid, health problems and conditions are viewed as individual problems that are best addressed by individual practitioners. Thus, at this level, assessments are of a focused type. A needs assessment is likely to concentrate on describing the magnitude of a specific medical problem. In addition, such an assessment would describe a subsystem of the community—namely, the diagnostic and treatment capabilities of the direct services providers within that community.

At the enabling services level of the public health pyramid, health problems and conditions are viewed as individual problems that are the direct results of nonindividual factors and that require

86 Chapter 3 Community Health Assessment for Program Planning

assessment, and program evaluation (evaluation framework, funders, developing evaluation plan, and more).

Group Dynamics and Community Building This website offers a different perspective on thinking about communities, with a wealth of links to resources on community and team building. Find it at www.community4me.com.

National Center for Health Statistics (NCHS) The NCHS website (www.cdc.gov/nchs/about. htm) has information on accessing and using existing national data sets.

Institute for Policy Research: ABCD For information on the ABCD approach, check out the website www.northwestern.edu/ipr/abcd. html, maintained by the Institute for Policy Research at Northwestern University.

PRECEDE-PROCEED Model The PRECEDE-PROCEED model can be viewed at the personal webpage of Lawrence Green, its author: www.lgreen.net/precede.htm.

MAPP NACCHO’s link to MAPP is found at http://www .naccho.org/programs/public-health-infrastructure /mapp.

National Association of County and City Health Officials: PACE-EH NACCHO’s webpage has links to PACE-EH resources and support at http://www.naccho.org /programs/environmental-health/assessment-tools /pace-eh/resources

Other PACE-EH Sources You can look at the toolbox developed by New Mexico at this site: www.naccho.org/topics

2. Why is each type of assessment relevant to health program planning?

3. Consider the strengths of and the dif- ferences among the MAPP, PACE-EH, and CHIP models. Under which cir- cumstances would you choose which approach? Access the websites for these programs (see the Internet Resources list), and compare the planning mod- els in terms of how they demonstrate the planning approaches identified by Forester (1993).

4. Do an online search about community assessments. One suggestion is to begin with state health departments or the Community Tool Box (described below under Internet Resources). Which per- spective on assessment is reflected in the context of the Internet site? Which of the steps described in this chapter received more or less emphasis by the authors of the Internet site? What im- plications does that shift in emphasis have for the problem statement?

5. Discuss the relevance of each perspec- tive for developing programs at each level of the public health pyramid. What effects might choosing one perspective have on the level of the program subse- quently developed based on its assess- ment results?

6. A health program planning committee wants to address various health problems by developing programs for each health problem at all four levels of the public health pyramid. What would be some implications of this perspective on setting priorities and on the nature of the subse- quent community health assessment?

▸ Internet Resources Community Tool Box The Community Tool Box, found at http://ctb .ku.edu/en, has gained wide recognition for its information on community building, community

87References

health through risk reduction (pp. 19–36). New York, NY: Macmillan.

Bradshaw, J. (1972). The concept of social need. New Society, 30, 640–643.

Caan, W., Cassidy, J., Coverdate, G., Ha, M-A., Nicholson, W., & Rao, M. (2015). The value of using schools as community assets for health. Public Health, 129, 3–16. Retrieved from http://dx.doi.org/10.1016/j.puhe.2014.10.006

Centers for Disease Control and Prevention: https://www .cdc.gov/.

Centers for Disease Control and Prevention. (2011). Com- munity Environmental Health Assessment (CEHA). Retrieved September 21, 2016, from www.cdc.gov /nceh/ehs/ceha/

Centers for Disease Control and Prevention. (2014). Improv- ing environmental public health services: Performance to meet community standards. National Center for Environmental health. Retrieved September 21, 2016, from https://www.cdc.gov/nceh/ehs/envphps/docs /improving-eph-serv-perf-comm.pdf.

Coffman, D. L., & BeLue, R. (2009). Disparities in sense of community: True race differences or differential items functioning? Journal of Community Psychology, 37(5), 547–558.

Cottrell, L. S. (1976). The competent community. In B. H. Kaplan, A. Leighton, & R. Wilson (Eds.), Further explorations in social psychiatry (pp. 195–209). New York, NY: Basic Books.

Dall, T., West, T., Chakrabrth, R., & Iacobucci, W. (2016). The complexities of physician supply and demand: Projections from 2014 to 2025. Retrieved October 11, 2016, from https://www.aamc.org/download/458082 /data/2016_complexities_of_supply_and_demand _projections.pdf

Denham, A., Quinn, S. C., & Gamble, D. (1998). Community organizing for health promotion in the rural South: An exploration of community competence. Family and Community Health, 21, 1–21.

Dickerson, J. B., Smith, M. L., Ahn, S., & Ory, M. G. (2011). Associations between health care factors and self-reported health status among individuals with diabetes: Results from a community assessment. Journal of Community Health, 36(2), 332–341.

Duhl, L. S. (2000). A short history and some acknowledge- ments. Public Health Reports, 115, 116–117.

Durch, J. S., Bailey, L. A., & Stoto, M. A. (Eds.). (1997). Improving health in the community: A role for per- formance monitoring IOM. The National Academies Press. Retrieved September 7, 2012, from www.nap .edu/readingroom/books/improving/.

Egger, E. (1999). Old ways of planning, thinking won’t work in today’s volatile health care industry. Health Care Strategic Management, 17(9), 18–19.

Evaneshko, V. (1999). Mental health needs assessment of Tucson’s urban Native American population. American Indian and Alaska Native Mental Health Research, 8(3), 41–61.

/demonstration/PACE/upload/NMCommunity EnvHealthAss.pdf.

At the county level, Multnomah, Oregon has created a website describing its efforts. You can find it at www.pace-eh.org/index.shtml; find its publication at www.pace-eh.org/documents /pilot_nne.pdf.

Another example of PACE-EH being used by a county government is available at http://www .delawarehealth.org/content.cfm?article=pace-eh.

Disaster Preparedness Examples of ethics in disaster preparedness planning in the United States can be found at http://ethics.missouri.edu/docs/Health_ Ethics_Considerations.pdf for Missouri, and at http://pandemic.wisconsin.gov/docview .asp?docid=14447 for Wisconsin.

References Alexander, E. R. (1992). Approaches to planning: Introducing

current planning theories, concepts, and issues (2nd ed.). Philadelphia: PA: Gordon and Breach.

Altschuld, J. W., & Witkin, B. R. (2000). From needs assess- ment to action: Translating needs into solution strategies. Thousand Oaks, CA: Sage.

American Association of Colleges of Nursing. (2014, July 15). Nursing shortage: Fact sheet. Retrieved September 21, 2016, from www.aacn.nche.edu/media-relations /fact-sheets/nursing-shortage.

Baker, I. R., Dennison, B. A., Boyer, P. S., Sellers, K. F., Russo, T. J., & Sherwood, N. A. (2007). An asset-based community initiative to reduce television viewing in New York state. Preventive medicine, 44(5), 437–441.

Bart, C. (2007). A comparative analysis of mission statement content in secular and faith-based hospitals. Journal of Intellectual Capital, 8(4), 682–694.

Bell, C., & Newby, H. (1971). Community studies: An intro- duction to the sociology of the local community. London, England: George Allen & Unwin.

BeLue, R., Taylor-Richardson, K. D., Lin, J. M., McClellan, L., & Hargreaves, M. K. (2006). Racial disparities in sense of community and health status: Implications in community-based participatory interventions target- ing chronic disease in African-Americans. Journal of Ambulatory Care Management, 29, 112–124.

Beneviste, G. (1989). Mastering the politics of planning: Crafting credible plans and policies that make a difference. San Francisco, CA: Jossey-Bass.

Blum, H. L. (1982). Social perspective on risk reduction. In M. M. Farber & A. M. Reinhart (Eds.), Promoting

88 Chapter 3 Community Health Assessment for Program Planning

Lindsay, A. C., Sussner, K. M., Greaney, M. L., & Peterson, K. E. (2008). Influence of social context on eating, physical activity, and sedentary behaviors of Latina mothers and their preschool-age children. Health Education and Behavior, 36(1), 81–96.

Lochner, K., Kawachi, I., & Kennedy, B. P. (1999). Social capital: A guide to its measurement. Health and Place, 5, 259–270.

McNall, M., & Foster-Fishman, P. G. (2007). Methods of rapid evaluation, assessment, and appraisal. American Journal of Evaluation, 28, 151–168.

Morgan, A., & Ziglio, E. (2007). Revitalising the evidence base for public health: An assets model. Promotion and Education, 2(Suppl.), 17–22.

National Association of County and City Health Officials website: www.naccho.org/.

National Association of County and City Health Officials. (2011). MAPP network: MAPP in the news. Retrieved September 7, 2012, from http://mappnetwork.naccho. org/page/mapp-in-the-news.

Nicotera, N. (2007). Measuring neighborhood: A conun- drum for human services researchers and practitioners. American Journal of Community Psychology, 40, 26–51.

Nilsen, P. (2006). The theory of community based health and safety programs: A critical examination. Injury Prevention, 12, 140–145.

Norris, F. H., Stevens, S. P., Pfefferbaum, B., Wyche, K. F., & Pfefferbaum, R. L. (2008). Community resilience as a metaphor, theory, set of capacities, and strategy for disaster readiness. American Journal of Community Psychology, 41, 127–150.

Norris, S. L., McNally, T. K., Zhang, X., Burda, B., Chan, B., Chow- dhury, F. M., et al. (2011). Published norms underestimate the health-related quality of life among persons with type 2 diabetes. Journal of Clinical Epidemiology, 64(4), 358–365.

Oconto County Department of Health and Human Services. (2008). Community health improvement plan. Retrieved September 7, 2012, from www.co.oconto.wi.us/i_oconto /d/whole_plan.pdf.

Oliveira, V., & Pinho, P. (2010). Evaluation in urban plan- ning: Advances and perspectives. Journal of Planning Literature, 24(4), 343–361.

Ostadtaghizadeh, A., Ardalan, A., Paton, D., Jabbari, H., & Khankeh, H. R. (2015, April 8). Community disaster resilience: A systematic review on assessment models and tools. PLoS Currents Disasters. (1st ed.). doi:10.1371 /currents.dis.f224ef8efbdfcf1d508dd0de4d8210ed

Palmer, R. C., Samson, R., Batra, A., Triantis, M., & Mullan, I. D. (2011). Breast cancer screening practices of safety net clinics: Results of a needs assessment study. BMC Women’s Health, 11, 6.

Plough, A., Fielding, J. E., Chandra, A., Williams, M., Eisenman, D., Wells, K. B., . . . Maganz, A. (2013). Building community disaster resilience: Perspectives from a large urban county health department. American Journal of Public Health, 103, 1190–1197. doi:10.2105 /AJPH.2013.301268

Forester, J. (1993). Critical theory, public policy and planning practice: Toward a critical pragmatism. New York, NY: State University of New York Press.

Goeppinger, J., Lassiter, P. G., & Wilcox, B. (1982). Community health is community competence. Nursing Outlook, 30, 464–467.

Goodman, R. M., Steckler, A., Hoover, S., & Schwartz, R. (1993). A critique of contemporary community health promotion approaches: Based on a qualitative review of six programs in Maine. American Journal of Health Promotion, 7, 208–220.

Green, L. W., & Kreuter, M. W. (2005). Health promotion planning: An educational and environmental approach (4th ed.). New York, NY: McGraw-Hill.

Guerin, P. B., Allotey, P., Elmi, F. H., & Baho, S. (2006). Ad- vocacy as a means to an end: assisting refugee women to take control of their reproductive health needs. Women & Health, 43(4), 7-25.

Guyadden, D., & Seasons, M. (2016). Plan evaluation: Challenges and directions for future research. Planning Practice & Research, 31(2), 215–228. doi:10.1080/026 97459.2015.1081335

Hill, L. G., Maucione, K., & Hood, B. K. (2007). A focused approach to assessing program fidelity. Prevention Science, 8(1), 25-34.

Hoch, C. (1994). What planners do: Power, politics and persuasion. Chicago, IL: Planners Press.

Hubbard, B. (2006). Working to Build Healthy Communities: Community Environmental Health Assessments Using PACE EH. Journal of Environmental Health, 69(3), 32-33.

Kamineni, V. V., Turk, T., Wilson, N., Satyanarayana, S., & Chauhan, L. S. (2011). A rapid assessment and response approach to review and enhance advocacy: Communi- cation and social mobilisation for tuberculosis control in Odisha state, India. BMC Women’s Health, 11, 463.

Kegler, M. C., Painter, J. E., Twiss, J. M., Aronson, R., & Norton, B. L. (2009). Evaluation findings on community partici- pation in the California Healthy Cities and Communities Program. Health Promotion International, 24(4), 300–310.

Kono, A., Tadaka, E., Kanaya, Y., Dai, Y., Itoi, W., & Imamatsu, Y. (2012). Development of a community commitment scale with cross-sectional survey validation for pre- venting social isolation in older Japanese people. BMC Public Health, 12(1), 903. doi:10.1186/1471-2458-12-903

Lee, Y., Altschuld, J. W., & White, J. L. (2007). Problems in needs assessment data: Discrepancy analysis. Evaluation and Program Planning, 30, 258–266.

Lin, V., Zhang, X., & Dixon, P. (2015). Occupational therapy workforce in the United States: Forecasting nationwide shortages. PM&R, 7, 946–954. doi:10.1016/j.pmrj.2015 .02.012

Lindau, S. T., Vickery, K. D., Choi, H., Makelarski, J., Matthews, A., & Davis. M. (2016). A community-powered, asset-based approach to intersectoral urban health system planning in Chicago. American Journal of Public Health, 106, 10, 1872–1878.

89References

Selamu, M., Asher, L., Hanlon, C., Medhin, G., Hailemariam, M., Patel, V., Thornicroft, G & Fekadu, A. (2015). Beyond the biomedical: Community resources for mental health care in rural Ethiopia. PLoS ONE, 10(5): e0126666. doi:10.1371/journal.pone.0126666

Stimson, G. V., Fitch, C. J., & Don Des Poznyak, V. (2006). Rapid assessment and response studies of injection drug use: Knowledge gain, capacity building, and intervention development in a multisite study. American Journal of Public Health, 96, 288–295.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 18(4157), 1124–1131.

U.S. Department of Homeland Security. (2007, September). National preparedness guidelines. Retrieved September 7, 2012, from www.fema.gov/pdf/emergency/nrf /National_Preparedness_Guidelines.pdf

Wisconsin Department of Health Services. (2010). Community Health Improvement Process & Plans (CHIPPs). Retrieved September 7, 2012, from www.dhs.wisconsin.gov/chip/

Porepa, M., Chan, M., Huber, J., Lam, C. G., Au, H., & Birken, C. S. (2016). Creating a student-led health magazine with an urban, multicultural, resource-restricted elementary school: Approach, process and impact. Paediatrics & Child Health (1205-7088), 21(3), 119–122.

Quick, K. S., & Feldman, M. S. (2011). Distinguishing par- ticipation and inclusion. Journal of Planning Education and Research, 31(3), 272–290.

Reinke, W. A., & Hall, T. L. (1988). Political aspects of planning. In W. A. Reinke (Ed.), Health planning for effective management (pp. 75–85). New York, NY: Oxford University Press.

Roberts-Gray, C., Gingiss, P. M., & Boerm, M. (2007). Evaluating school capacity to implement new programs. Evaluation and Program Planning, 30, 247–257.

Rogers, N., Guerra, F., Suchdev, P. S., & Chapman, A. S. (2006). Rapid assessment of health needs and resettlement plans among Hurricane Katrina evacuees—San Anto- nio, Texas. Morbidity and Mortality Weekly Report, 55, 242–244.

91

Characterizing and Defining the Health Problem

In this chapter, the focus on the community health assessment phase of the planning cycle continues with a review of data collection options. This is followed by a discussion of the basic epide- miological analyses often used in community health assessment. After program planners have identified health problems through the assessment process, those health problems need to be summarized into statements and diagrams that facilitate both prioritization and the subsequent detailed program planning. A statement of the health problem can be developed in many ways and take a variety of forms. This chapter introduces an approach to developing a health problem statement, which is then used throughout this text as a means of linking the elements of the planning and evaluation cycle. The last phase of the assessment process is to prioritize the problems, preferably through a systematic and intentional process.

▸ Collecting Data From Multiple Sources

Numerous types and sources of data are used in a community health assessment. Each has the

potential to contribute to an understanding of the parameters of the health problem or condition. However, each has limitations and caveats that need to be considered.

Public Data Data are called public data if they are available without a fee, are accessible by anyone, and have no personal identifiers. Public data primarily include national surveys, vital statistics, and census social indicators, which are gathered through the national surveys administered by the National Center for Health Statistics (NCHS) of the Centers for Disease Control and Preven- tion (CDC), such as the National Health and Nutrition Examination Survey (NHANES) and the National Health Interview Survey (NHIS). Secondary analyses of these public data sets can be used to extend data available for the community health assessment. Such national data have been helpful in making decisions about specific health problems, targeting populations, identifying barriers, and influencing health policy. They can also be used to create synthetic estimates, as explained later in this chapter.

CHAPTER 4

© Lynne Nicholson/Shutterstock

92 Chapter 4 Characterizing and Defining the Health Problem

Public data also include more and more often what is available via various and numerous social media outlets. Posts at public Facebook pages, images from photograph-sharing apps, and tweets from Twitter accounts can provide insights into cultural preferences, existing health information, emerging problems, and community resources. For example, Ramanadhan, Mendez, Roa, and Viswanath (2013) found that a substan- tial portion of community-based organizations use Facebook and Twitter to promote their respective organizations and their activities. Social media are used increasingly as the basis for developing visuals of the social networks among community organizations (e.g., Jung & Valero, 2016), which provide insights into the community power structure. Google Maps, as public data, can be useful in identifying com- mercial, religious, and educational resources, as well as public transportation routes. Such data complement and enrich the community assess- ment, especially when associated with geographic data distribution of health problems. Maps of the geographic distribution of health problems, as might be found at the website of a county or state health department, are yet another type of informative public data. Caveats to using public data exist. One caveat is that many of these data, especially the large national survey data sets, were collected years ago. The age of the data may influence their usefulness in understanding or identifying a current health problem. Another caveat is that social media, as public data, may not reflect an overall trend or sentiment, only that of a few vocal individuals. Although social media data tend to be quite current, its repre- sentativeness needs to be carefully considered.

Primary Data Primary data are those collected by the commu- nity assessment or evaluation team, specifically to illuminate a need of interest. A wide variety of methods can be used to collect primary data, including interviews, surveys, community forums, focus groups, and interviews with key informants and service providers.

Three key points must be kept in mind when collecting primary data for a community health assessment. First, data from participants in a program are rarely used as the sole source of data for a community health assessment. Although the program participants can provide valuable insights into the perceived needs of the intended audience, that information must be considered in light of the fact that the par- ticipants are already in the program. This fact alone makes them potentially dissimilar to those persons targeted by the program. Second, rigor is required to obtain valid, reliable, trustworthy data. In most cases, only minor modifications to rigorous designs are needed for conducting a community health assessment. Third, primary data can, of course, be collected from members of the target audience, but providers can also provide valuable insights into the needs of the target population. Although data from providers are useful in identifying specific service needs of the target population, this information must be viewed as revealing the normative needs only. That is, providers are notorious for holding views of what is needed that differ from the views held by their clients. Naturally occurring discrepancies between providers’ normative assessment of a problem and the clients’ perceived needs can pose a particular challenge for the health pro- gram planning. Making program development decisions based only on provider data is likely to result in programs that are not attractive to the intended audience.

Observational Data Unobtrusive (Webb, Campbell, Schwartz, & Sechrest, 2000) or nonreactive (Webb, Campbell, Schwartz, Sechrest, & Grove, 1981) measures are also sources of data and are particularly relevant to community characteristics. These types of data constitute what is collected during a windshield or walking survey, as explained in the Community Toolbox (http://ctb.ku.edu/en). For example, walking around a neighborhood and observing how many blocks contain abandoned buildings or storefront churches is an unobtrusive

Collecting Data From Multiple Sources 93

measure. Counting the number of liquor bottles in a garbage can, counting the number of bill- board advertisements for unhealthy behaviors, estimating the ratio of bars and pubs to banks, watching the interactions among residents in a local bakery, and collecting local community newspapers are all examples of data collection of the least invasive nature. Each of these examples provides clues to the character, strengths, and problems in the community as a whole. The use of unobtrusive measures is inexpensive and can provide interesting data about what health problems exist and what may be contributing to those problems.

Archival Data Archival data, as existing but not current, include newspaper articles, reports, and other types of agency records. On a local level, clinics, agencies, and nonprofit organizations may have data appro- priate for a community health assessment they may be conducting. Archival data can provide the historical or cultural context of a problem, as well as information about past demand or need for a problem or historical information on previous program participants. One limitation to archival material is that the data may not include key information that is sought or may not be complete. Another potential problem with archival data is that the extent to which the data were initially collected accurately is unknown. These factors influence the data’s overall usefulness and trustworthiness.

Proprietary Data Another possible source of data is proprietary data—specifically, data that are owned by an organization and that can be purchased for use. For example, the American Hospital Associa- tion, American Medical Association, and health insurance companies own databases about their members that can contain information needed for a comprehensive community health assessment. Generally, medical record data are proprietary. Like archival data and public

data, the information that can be gained from proprietary data is limited to what has already been collected.

Published Literature The published scientific literature is a good source of information, particularly for determining relative and normative needs. In other words, information may be available that allows for comparative statements about the health status in reference to other groups, professional health standards, or national goals. This inexpensive, reliable source of information should not be overlooked as a source of data when doing a community needs assessment.

Data Beyond Street Lamp It can be important to collect data from sources that are not readily available. This practice is called “going beyond the streetlamp,” which derives its name from a little story.

One night, a man lost his keys. He be- gan to look for them, crawling around on his hands and knees beneath a street lamp. Before long, a stranger stopped and asked the man what he was doing on his hands and knees. He replied that he was looking for his keys. The stranger offered to help and asked where he had lost his keys. The man replied, “Over there,” pointing to a dark area down the block just out- side the bar. So the stranger asked, “Then why are you looking over here?” To which the man replied, “Be- cause there is more light over here.”

The point of this story is that the informa- tion you need may not be the same as the data to which you already have access: You need to go beyond the streetlamp. Some of the sources of data just described are available under the streetlamp, whereas others are not readily avail- able and require primary data collection. What determines the extent to which data need to be

94 Chapter 4 Characterizing and Defining the Health Problem

The magnitude of a problem is also conveyed through measures such as rates and proportions. In epidemiological terms, these measures are a matter of numerators and denominators. The denominator is generally the total number in the population or the total number in the population that is potentially at risk. The numerator is gen- erally the number of individuals who have the health problem or condition or who are actually found to be at risk. Using these basic numbers, a wide variety of commonly defined rates and proportions can be developed related to health. Increasingly, the rates and proportions for various health problems are available online at the websites for local and state health departments and federal agencies, such as the National Center for Health Statistics, which is housed within the CDC.

The ability to obtain accurate rates and proportions depends in part on the quality of the tests used to identify cases. Ideal tests have both high sensitivity (the extent to which there are no false negatives) and high specificity (the extent to which there are no false positives). Sensitivity and specificity are often used in reference to medical tests, such as occult blood tests, mammography, or urine tests for cocaine use, but they are also important characteristics of psychological and behavioral measures, such as the CES-D scale, which measures the level of depression in an individual (Radloff, 1977), and the SF-12 health survey, which measures overall health and functioning (Farivar, Cunningham, & Hays, 2007). The sensitivity and specificity of medical tests and of psychological or behav- ioral measures determine the extent to which a condition is accurately identified, which in turn influences the estimated incidence or prevalence rates for a given condition or behavior. In this way, sensitivity and specificity affect the accuracy of an estimated magnitude of a health problem or condition within a population.

Dynamics Leading to the Problem Another category of data is information about the precursors of the health problem or condition.

collected from beyond the streetlamp are factors such as time constraints, fiscal resources, level of expertise, and endorsement or expectations of those who will be using the community health assessment.

▸ Collecting Descriptive Data

To understand the health problem and formulate a definition of the health problem or condition, it is necessary to collect data. Baker and Reinke (1988) suggest that, from an epidemiological perspective, four categories of information need to be collected as a prelude to health planning: the magnitude of the problem, the precursors of the problem, population characteristics, and attitudes and behaviors. These four categories provide a useful framework for organizing a community health assessment, especially when they are expanded to include elements from the public health, social, and asset perspectives.

Magnitude of the Problem One category of information needed is the mag- nitude of the problem. The magnitude can be described in terms of the extent of the disease or health condition, the acute or chronic nature of the problem, and the intensity of the problem.

The extent of the health problem is described in terms of incidence and prevalence. The inci- dence is the rate at which new cases occur. The prevalence is the extent to which cases currently exist in a population. Incidence and prevalence, although typically used in reference to disease conditions, can be used to think about behav- iors as well. For example, the number of new smokers among a defined group of adolescents (incidence) and the percentage of that same adolescent population that is currently smoking (prevalence) provide information that can be used to determine whether smoking is a problem of sufficient magnitude to warrant attention in the program planning effort.

Collecting Descriptive Data 95

Such information is necessary in order later to tailor interventions to the specific precursors of the health problem.

From an epidemiological perspective, the precursors to a health problem are understood in terms of agent, host, and environment. Baker and Haddon (1974), in studying childhood injuries, developed a model of factors associated with a health problem—namely, the human, physical, environmental, and sociocultural factors. TABLE 4-1 is based on this model, albeit with the addition of the healthcare system as another element in analyzing the health problem or condition. Each cell in the table contains a definition of what might go into that cell, along with a few examples. For any single health problem that is the focus of a needs assessment, data can be placed into the

As the community health assessment progresses, the planning group uses the data collected to generate lists of factors, conditions, situations, and events that in some way contribute to the health problem coming into existence and being observable. We broadly call the factors social determinants of health. All these factors are precursors or antecedents to the health problem. In addition, those factors, conditions, situations, and events that mediate, potentiate, or suppress the expression of the health problem may be uncovered during an assessment. While much may be known about a health problem from the scientific literature, the community health assessment is done to elucidate specific precursors that are unique to a locality, whether a neighborhood or a state, or to a target population.

TABLE 4-1 Haddon’s Typology for Analyzing an Event, Modified for Use in Developing Health Promotion and Prevention Programs

Agent Factors

Human Factors

Physical Environment

Sociocultural Environment

Health System Environment

Pre-event Latency Genetic makeup, motivation, knowledge

Proximity, transportation, availability of agent (e.g., alcohol or drugs)

Norms, policy and laws, cultural beliefs about causes, family dynamics

Accessibility, availability, acceptability

Event (behavior)

Virulence, addictiveness, difficulty of behavior

Susceptibility, vulnerability, hardiness, reaction

Force Peer pressure Iatrogenic factors, treatments

Post- event

Resistance to treatment

Motivation, resilience, time for recovery

Proximity, availability of agent (e.g., alcohol or drugs)

Meaning of event, attribution of causality, sick role

Resources and services, treatment options, emergency response

Modified from Haddon, W., Jr. (1972). A logical framework for categorizing highway safety phenomena and activity. Journal of Trauma, 12, 193–207. Cited in D. C. Grossman. (2000), The history of injury control and the epidemiology of child and adolescent injuries. The Future of Children, a publication of the David and Lucile Packard Foundation., 10(1), 23–52.

96 Chapter 4 Characterizing and Defining the Health Problem

problems. Still, other attitudes and behaviors have a more direct, causal relationship with health problems. Distrust in medical providers and a failure to obtain preventive health services lead directly to severe morbidity conditions in some populations. Accordingly, attitudes toward health promotion and disease prevention behaviors as well as attitudes toward healthcare services and providers must be considered in order to have a comprehensive data set for a community health assessment.

Years of Life and Quality of Life A number of measures have been developed to account not only for deaths but also the quality of years lived with an illness and the number of those years. TABLE 4-2 summarizes these measures, whose definitions are drawn from a variety of sources. These measures are widely used to assess the value of new treat- ments and medication, which also makes them applicable to planning health programs. These burden-of-disease measures can also be used as part of a community needs assessment, albeit only if the population being assessed is sufficiently large to have stable statistics. Thus, measures of burden of disease might be used for larger states within the United States or regions of the United States. In the United  States, national surveys, surveillance registries, and hospital discharge data are major sources that can be used to calculate burden-of-disease measures (Murry, 2013). These measures also are used in the economic evaluations of programs, particularly in cost–benefit analyses. Data on the negative consequences of health problems, beyond the familiar mortality rates, can be quite influential during the problem prioritization processes.

Definitions of quality of life include the notions of a perceived overall state of well-being across various domains such as sociocultural relationships and physical functioning or in relation to goals and expectations. Often quality of life is measured as it pertains to a particular illness or disease process—usually a chronic disease such as asthma

cells in Table 4-1, thereby giving an overview and a preliminary analysis of precursors to the health problem or condition. This format is especially useful for infectious diseases and injuries. The Haddon (1972) model reveals the complexity of data that might need to be analyzed to understand the health problem or condition fully.

Population Characteristics Population characteristics data, the third cate- gory, relate mainly to the social model of needs assessments. Obtaining this information involves collecting data on characteristics, such as distribu- tion of age categories, income levels, educational levels, and occupation distribution within a community. If the “who” has been narrowly defined in terms of location, the population character- istics uncovered through this effort can be very specific. For example, if the assessment focuses on prison inmates, then their characteristics— such as types of crime committed, length of time incarcerated, or race—can become part of the population characteristics data collected for the community (prison) health assessment.

Attitudes and Behaviors The fourth category of information concerns the attitudes and behaviors of the population being assessed, with particular attention being paid to the attitudes and behaviors of the target audience. Data about attitudes and behaviors help complete or flesh out the description of the factors related to a health problem.

Some attitudes and behaviors may be antecedents to health problems or conditions. For example, culturally held beliefs about illnesses, illness prevention, and treatments, as well as beliefs concerning appropriate health behaviors and the sick role, may all be important to understanding the health problem. Other lifestyle behaviors contribute to the existence of health problems. For example, secondhand smoke contributes to childhood asthma, whereas regular aerobic exercise contributes to reduced numbers of health

Collecting Descriptive Data 97

length of life as affected by that health condition is what becomes important. In other words, the assessment must consider the quality of the life as lived with the health condition. The measures known as quality-adjusted life-years (QALYs) and disability-adjusted life-years (DALYs) were developed specifically to give a numeric value to the quality of years of life (Table 4-2). These composite scores are used with populations and therefore have the advantage of being indifferent to individual preferences. Because the number of years for which quality can be adjusted is naturally shorter for older persons, however, QALYs and DALYs mathematically discriminate against the elderly. Nonetheless, the use of DALYs reveals the extent to which diseases affect the years of life (TABLE 4-3).

or arthritis. Although each person has a sense of what constitutes quality of life, its measurement is complex; hence the plethora of quality-of-life measures that are available (Preedy & Watson, 2009). An issue with quality-of-life measures, as Kaplan (1996) stressed, is that quality of life is multidimensional, so the measures must address the relative importance of the many dimensions of quality of life. The choice of which measure to use in health program planning depends on the resources available for the assessment phase, the sophistication of the planning team, and the role played by the rational approach in the plan- ning process.

The length of life is as important as its qual- ity. Because individuals may live with the same health condition for varying lengths of time, the

TABLE 4-2 Quality-of-Life Acronyms and Definitions

Acronym Spelled-out Form Definition

QALYs Quality-adjusted life-years Number of years of life expected at a given level of health and well-being.

DALYs Disability-adjusted life-years Number of years of life lost from living with a level of morbidity or disability.

YLL Years of life lost Number of years a person is estimated to have remained alive if the disease experienced had not occurred.

YPLL Years of potential life lost A measure of the impact of disease or injury in a population that calculates years of life lost before a specific age (often age 65 or age 75). This approach assigns additional value to deaths that occur at earlier ages.

HYE Healthy years equivalent Number of years in perfect health that are considered equivalent to a particular health state or health profile.

YHL Years of healthy life Number of healthy years of life lived or achieved, adjusted for level of health status.

98 Chapter 4 Characterizing and Defining the Health Problem

TABLE 4-3 Global Leading Causes of Disability-Adjusted Life-Years (DALYs) and Years of Life Lost (YLL)

Rank DALY Causes % of DALYs DALYs per 100,000

1 Ischaemic heart disease 6.0 2,342

2 Lower respiratory infections 5.4 2,076

3 Stroke 5.2 1,998

4 Preterm birth complications 3.9 1,515

5 Diarrheal diseases 3.6 1,409

6 Chronic obstructive pulmonary disease 3.4 1,306

7 HIV/AIDS 3.4 1,299

8 Road injury 2.9 1,113

9 Unipolar depressive disorders 2.8 1,081

10 Birth asphyxia and birth trauma 2.7 1,054

Total % for top 10 DALY causes 39.2%

Rank YLL Causes % of YLL YLL per 100,000

population

1 Ischaemic heart disease 7.8 2,212

2 Lower respiratory infections 7.2 2,042

3 Stroke 6.8 1,933

4 Preterm birth complications 5.2 1,472

5 Diarrheal diseases 4.6 1,290

6 HIV/AIDS 4.4 1,234

7 Birth asphyxia and birth trauma 3.4 965

8 Road injury 3.2 916

Statistics for Describing Health Problems 99

For example, if surveys were conducted or if data were abstracted from medical records, those data need to be analyzed. This section briefly reviews statistical approaches, with particular attention to epidemiological considerations relevant to community health assessment. More details about quantitative data analysis are provided elsewhere in this text as well as in statistical textbooks. The intention of this section is to relate what is learned in statistics and epidemiology to the community health assessment.

Given that most community health assess- ments involve some population-based data, it is worth reviewing basic epidemiological techniques here. More complete and in-depth presenta- tions are available in traditional epidemiology textbooks, such as those by Friss and Sellers (2014) or Fos (2011). For a direct application of epidemiology to community health assessment, Dever’s (1980) book is a classic. However, the more recent publication by Dever (2006) also cover basic epidemiological techniques but from the point of view of healthcare executives planning for population health. Health program planners would do well to have at least one of these texts on their bookshelves for quick reference. With the widespread availability of computer spreadsheet and database programs, the calculation of most statistics is less a mat- ter of doing the math and more a matter of understanding which numbers to use and how to make sense of the numbers generated by the software.

A slightly different perspective is based on the number of years of life that are lost due to a health condition. Years of life lost (YLL) reveals the number of years lost at the end of life because of a health condition. The shortened life span could be due to acute or chronic health problems, chronic environmental conditions, or injuries. Years of potential life lost (YPLL) is a similar measure but indicates the number of years of life lost at the beginning of life, such as the shortening of life caused by neonatal sepsis or childhood drowning.

All of these quality-of-life and life-year measures are particularly useful in public health assessments. They assist program planners in deciding which health condition warrants health promotion or disease prevention programs, particularly when resources are severely limited. Unfortunately, these measures are difficult to calculate and exist for nations or very specific populations. Therefore, it is challenging to use these measures in local community assessments.

▸ Statistics for Describing Health Problems

Data collected from the community health assessment need to be analyzed and interpreted, particularly the primary data that were collected.

Rank YLL Causes % of YLL YLL per 100,000

population

9 Chronic obstructive pulmonary disease 3.1 871

10 Malaria 2.5 718

Total % of YLL from top 10 causes 48.2%

Sources: World Health Organization. (2014). Summary: YLL (000s) by cause, in WHO Regions (a), estimates for 2004. Retrieved September 11, 2012, from www.who.int/healthinfo/global_burden_disease/risk_factors/en/; World Health Organization. (2004). The Global Burden of Disease 2004 Update. Accessed September 11, 2012, from www.who.int/healthinfo/global_burden_disease/GBD_report_2004update_full.pdf.

100 Chapter 4 Characterizing and Defining the Health Problem

Relative risk ranges from 0.0 to infinity: the larger the relative risk, the greater the chance of developing the health problem with exposure. Similarly, the odds ratio ranges from 0.0 to infinity. Odds ratios from 0.0 to 1.0 indicate a protective effect, whereas odd ratios greater than 1.0 indicate an increased likelihood of having the health problem. The larger the odds ratio, the more likely one is to have the health problem. Although the odds ratio can range to infinity, in practice it rarely exceeds 10. Relative risk compares two cumulative incidences, thereby providing a direct comparison of the probabilities. This makes the relative risk measure preferable to the odds ratio (Handler, Rosenberg, Kennelly, & Monahan, 1998). The odds ratio does not use the population in the denominator, making it less accurate than the relative risk. However, when the health problem is rare, the odds ratio begins to approximate the relative risk.

Both the relative risk and the odds ratio are used widely in epidemiology and thus are likely available for use in community health assessments. Both convey information about the comparative influence of factors or expo- sure variables on health outcomes. Having this information available then allows planners to prioritize which causal or exposure factors to address in a health program.

Population Parameters The confidence interval (CI) indicates the upper and lower range of values between which the value for the true population is likely to fall. It helps to understand the likelihood that the score or mean value for a health condition derived from a sam- ple is similar to the value in the true population. Confidence intervals, like standard deviations, provide a level of assurance about whether the mean or score value for the variable reflects the value for the whole population. For example, if a score is within the CI, then the value falls within a range that is reflective of the larger population. However, if the value falls outside the CI, then that score can be viewed as being important because it is different from the value in the general population.

Descriptive Statistics Descriptive statistics—the fundamentals of statistics—are a family of statistics that portray the distribution of values for a single variable. These statistics provide an amazing wealth of information but are often underappreciated for their ability to communicate important information simply.

The simplest descriptive statistic is the frequency, or count, of occurrences. Based on the frequency, two other informative types of descriptive statistics can be calculated. One type measures central tendency, such as the mean (average), whereas the other type measures dispersion as variance and standard deviation. Standard deviation is related to the range of val- ues in the data and thus indicates the dispersion of the data. Remember that 68.3% of data are contained within one standard deviation, 95.5% within two standard deviations, and 99.7% within three standard deviations.

Descriptive statistics are easy to calculate with a calculator or with spreadsheet software, such as Excel. They are often presented in the form of graphical displays of frequency such as bar graphs. A bar graph of frequencies provides a rough picture of the distribution, thereby revealing visually whether the data approximate the normal curve.

Odds Ratio and Relative Risk Two statistical tests that help estimate the likeli- hood of having or getting a given health problem are the odds ratio (OR) and the relative risk (RR). The odds ratio is calculated as the odds of having the health problem if exposed divided by the odds of having the problem if not exposed. The relative risk is calculated as the cumulative incidence in the exposed population divided by the cumulative incidence in the unexposed population. In conducting a community health assessment, planners mostly obtain the odds ratio and relative risk from published studies because having data on exposure usually requires epidemiological research.

Statistics for Describing Health Problems 101

mapping of health problems or population characteristics. As mapping software has become more accessible, the usual map-based display of the distribution of health problems or conditions can be done more readily at any level, such as by state, county, census tract, ZIP code, or street address. Mapping at very specific levels of geography provides an extremely refined picture of what is where. Geographic mapping technology can accommodate a variety of types of data, including survey data; social media data; and address data, such as that of community assets. The integrated use of these data lead to a useful and insightful statistical analysis of the intersection of needs or problems and existing resources. For example, Cebrecos et al. (2016) investigated the association between food envi- ronments and physical activity environments to identify neighbourhoods with both healthy food outlets and walkable streets. Another option is to overlay rates of chronic health conditions with location of health services resources, as can be done at a CDC website (http://www.cdc.gov /dhdsp/maps/atlas/index.htm). Such a map helps visualize that the highest rates of chronic health conditions are found in geographic areas with the lowest density of health services. Mapping health problems and suspected factors in the causal path provides very engaging information and can be crucial in reaching a consensus or attracting the attention of key stakeholders.

Advanced spatial analytics can provide quantitative information, including statistical evidence of association between location and other variables, and statistical difference in rates relative to adjoining geographies. As the geographic analysis software becomes more mainstream, the statistical analysis and inter- pretations will become more familiar to health program planners and evaluators.

Small Numbers and Small Areas Small numbers are a big problem, whether one is looking at epidemiological data or social data. In particular, those conducting the community assessment and health planners for rural areas face

Confidence intervals play a valuable role during the community health assessment by focusing attention on values that are unusual and thus merit attention. They also provide a clue about relative need because values that fall outside the CI are “abnormal” relative to the population. Naturally, the reverse of this can also be true: A value falling within the CI can be a cause for concern.

Tests of Significance A test of significance is done to assess whether the probability is high or low that the statistical result can be accepted as being true. The test of significance is the same across the types of statistics used, and the interpretation of signifi- cance is the same. The first step is to set the alpha level, which is the probability of rejecting a null hypothesis, when in fact it is true. Commonly, the alpha is set at 5%, and the null hypothesis is stated as there will be no difference between groups. Using a table of critical values for the type of statistical test used, one can then determine whether the statistical result was above or below the cutoff for the alpha.

A critical issue for program planners in analyzing community health assessment data might be to determine whether the difference between two communities or two groups is just a random variation or whether the difference is sufficiently large to suggest something else is contributing to their difference. If the two groups are compared, the p-value gives the probability of falsely claiming that the groups are different. For example, if a test of significance is reported as p =.05 for a comparison of two groups, then there is less than a 5% chance that the two groups were actually the same. Remember, however, that a statistical significance does not automatically mean there is a clinical significance.

Geographic Information Systems: Mapping Historically, an element of the epidemiological assessment model has been the geographic

102 Chapter 4 Characterizing and Defining the Health Problem

One study of immunization rates (Jia et al., 2006) is illustrative of the challenge inherent in dealing with small numbers. Jia and colleagues color-coded county data to reflect whether the county was more than two standard deviations above or below the state mean, between one and two standard deviations above or below the state mean, or less than one standard deviation above or below the state mean. Their choice of displays on a map of the United States demonstrates the creativity that may be needed to deal with small numbers and small-area data.

Epidemiology Rates TABLE 4-4 is a summary of commonly used epidemiological rates, with their corresponding numerators and denominators. As the table entries suggest, the epidemiological model relies heavily on having accurate counts that can be used in the denominators and numerators. To the extent that data are available, key elements of the epidemiological model are incorporated into most community health assessments.

▸ Stating the Health Problem

Data collected for the community health assess- ment can be organized in a variety of ways, such as a community profile, wellness profile, behavioral profile, or service profile (Paronen & Oje, 1998), or a community diagnosis (Muecke, 1984). Regardless of which format is chosen, the community health assessment ought to lead to a statement of what was found, phrased so that stakeholders, constituents, community members, and multidisciplinary health professionals can understand each health problem identified.

Diagramming the Health Problem In health program planning, understanding what causes a health problem and how those causes

the issue of how to portray rates. The problem is a simple one: If a geographic area has a small population (denominator), then a small variation in the occurrence of a health problem (numerator) will inevitably lead to a large change in the rate or proportion of that health problem. This instability of the rate affects the conclusions that can logi- cally be drawn from the data. The same statistical problem also occurs when the analysis focuses on a small geographic area, such as a parish, county, or legislative district. Another, concern with small numbers is the potential breach of confidentiality, given the possibility of identifying an individual when the number of cases in a row by column cell is small. Guidelines for maintaining confidentiality exist (e.g., VanEenwyk & Macdonald, 2012) and ought to be followed.

Small numbers can also be a problem if the data collected are of a social or qualitative nature, as might be the case in an asset assessment. If the number of respondents to a community survey or the number of participants in a community focus group is small, then the information those individuals provide has a higher likelihood of not being representative of the range of views and opinions in the community.

Once data are collected, those conducting the assessment rarely have an opportunity to go back and gather more data. Given this caveat, careful planning and execution of the data collection must be done to avoid having too few respondents.

Several statistical techniques exist for addressing the small-numbers problem, utiliz- ing counts, rates, or proportions (Dever, 1997) or pooling years of data (Cawley, Schroeder, & Simon, 2006). Additional techniques continue to be developed as well (Yu, Meng, Mendez-Luck, Jhawr, & Wallace, 2007). One set of techniques focuses on comparing the small area (population) with a larger area (population) or a standard. Another set of techniques is based on comparing two small areas. Yet another approach is to use data from multiple time periods, which may cumulatively produce a sufficient sample size to make comparisons either across time periods or with another small area using a similar time period.

Stating the Health Problem 103

model or a root cause analysis. The key distinction is that a logic model focuses on organizing the delivery of the program, whereas the model of the health problem focuses on understanding what causes the health problem. A root cause analysis includes a range of possible causes of a problem, but it does not rely on data or sequence those possible causes in terms of cause and effect. Attention to the causes of the health problem is critical if program planners are to select the best point of intervention and the appropriate intervention for that point. In this way, the model of the health problem contributes not only to

lead to the problem is portrayed in a diagram or conceptual model that organizes key factors along a general sequential time line, as well as a written statement. Articulating the problem both visually and verbally can help in gaining clarity about the causes and possible solutions to the problem. In addition, having both the visual and verbal description allows the planner to communicate better with both those who prefer pictures and those who prefer text.

Each health problem has its own unique set of precursors given the specific context. The model of the health problem is distinct from a logic

TABLE 4-4 Numerators and Denominators for Selected Epidemiological Rates Commonly Used in Community Health Assessments

Rate Numerator Denominator Per

Crude death rate

Total number of deaths in a given period

Total population 1,000

Cause-specific death rate

Number of deaths due to a specific cause in a given time period

Total population 100,000

Disease- prevalence rate

Number of currently existing cases with the disease, during a given time period

Total population during that time period

(Varies depending on the size of the numerator)

Birth rate Number of live births in a given time period

Total population 1,000

Fetal death rate Number of fetal deaths at 28 weeks’ gestation or more that occur in a given period

Number of fetal deaths at 28 weeks’ gestation or more plus number of live births that occur in a given time period

1,000

Neonatal death rate

Number of deaths of infants 28 days old or younger that occur in a given time period

Number of live births that occur in a given time period

1,000

Infant- mortality rate

Number of infants (age birth to 1 year) who died in a year

Number of births in a year

1,000

104 Chapter 4 Characterizing and Defining the Health Problem

causes, mediating factors, and moderating factors. Not included in the causal theory are factors, conditions, variables, and elements that may exist but are not immediately relevant to the health problem or that are so complex that they are not contenders for programmatic interven- tion. In other words, the process of developing the causal theory also is a process of narrowing the problem and of prioritizing. As with all processes, the creation of the causal theory is an iterative, evolving, and ongoing process that cannot be completed in a single short session.

Precursors can be thought of as existing physical or contextual factors as well as the causes of the health condition. Required antecedent factors are those elements that must be present for the health problem to come into existence or are direct precursors of it. For example, required factors may include genetic predisposition, being in the right place at the right time, prior exposure and vulnerability, or legal or policy conditions. From an asset perspective, required factors might also include variables such as the political clout of the local representatives or the existence of economic empowerment zones. The predisposing factors of health services utilization models (Anderson, 1995; Anderson & Aday, 1978; Green & Kreuter, 1991) generally fall into the required antecedent factors category.

Causal factors are those elements that influence whether the health problem will manifest itself, given the presence of the required antecedents. Depending on the health problem, causal factors

the development of a logic model but also to tailoring the programmatic intervention and to designing the evaluation of the program’s effect.

Elements of a Causal Theory The model of the health problem brings together, in a visual display, the key factors that were iden- tified from the community health assessment as being important to the health problem. The combination of these factors ought to explain or hypothesize about what causes the health problem. For this reason, it is called a causal theory. For some health problems, an existing causal theory may be applicable to the current circumstances. If it is not, a new causal theory will need to be created. The decision of what to include in the causal theory and what to exclude intentionally from the model has ramifications through the program planning and evaluation cycle: It guides the intervention choice, establishes the parameters for the evaluation of the program effect, and influences the statistical analyses of the evaluation data.

Throughout this text, the same approach to displaying a causal theory (shown in FIGURE 4-1) is used for the sake of illustration. Each program, reflecting the unique perspective of its set of stakeholders, may develop its own approach to displaying a causal theory visually. The template presented in this chapter has been carefully crafted to include four key elements to be considered in developing any causal theory: existing factors,

FIGURE 4-1 Generic Model of a Theory of Causes

Required antecedent

factors Mediating

mechanisms

Causal Theory

+/– Moderating factors

Main causal factors of the health problem

(determinants)

Health impact

Health problem

Stating the Health Problem 105

reinforcing factors (Anderson, 1995; Anderson & Aday, 1978; Green & Kreuter, 1991) must be reevaluated for their role in the causal theory, as many are likely to function as moderating factors.

Mediating factors come between causes and outcomes. In fact, without the mediating factor, the causes will not result in the health outcome. In other words, without this process or mechanism, the causal factors cannot cause the health outcome. Depending on the health problem, there may not be any mediating factors. For example, if an individual has the genetic mutation that causes cystic fibrosis, the disease will appear—there is no mediating variable. However, if the health outcome is defined as longevity for persons with cystic fibrosis, medi- ating factors would include quality of health care and individual response to treatments. In contrast, if someone has a stroke, both morbidity and mortality depend on the response time and quality of the emergency medical care, which are mediating factors.

Examples TABLE 4-5 shows how data collected during the community assessment can be presented in a tabular format. It contains examples of infor- mation for the five health problems identified in the community assessment of the imaginary Layetteville. Possible required antecedent factors, causal factors, moderating factors, and mediating factors that lead to five different health problems are given. The information that is included in a table such as Table 4-5 and in a causal theory diagram is based on the data collected during the community health assessment, including the scientific literature related to the health problem. These hypothetical details of the five health problems demonstrate how a causal theory diagram might look using the data from this table. These five health problems are carried throughout the text at the subsequent stages of the planning and evaluation cycle.

One example is the health problem of adult immunizations, particularly against influenza. Not surprisingly, we want the rate

might be exposure to the health hazard, suscep- tibility, or the virulence of the hazard. From an asset perspective, causal factors might include health knowledge, the existence of healthy food choices in local grocery stores, the existence of environmental pollutants, the existence of road safety features (intersection lights), or the accessibility and availability of local health and social services agencies. These causal factors could also be called the determinants of the problem because they are directly responsible for outcomes. In recent years, however, the term determinants has been more widely applied to encompass a host of social and ecological factors that lead to health problems. To avoid confusion and to be more specific, the term causal factors is used throughout this text to refer specifically to the factors identified from the community assessment as leading directly to the health problem.

Causal factors include, for example, the water that determines whether a seed will sprout, the hole in a boat that determines whether the boat will sink, and the presence of potassium in the blood that determines whether the heart muscles will contract. Nevertheless, as these concrete examples suggest, the situation involves both required antecedent factors and factors that lead one to say “yes, but. . . .” Those “yes, but” factors can be sorted into two types: moderating factors and mediating factors.

Moderating factors are those elements that have the potential either to exaggerate or to lessen the presence of the health problem. Again, depending on the health problem, these factors might consist of laws and policies or social sup- port. Such factors generally act to affect the causal factors. Complex and interacting relationships exist among required antecedents, causal fac- tors, and moderating factors. In a model of the causal theory, moderating factors are shown as possibly influencing the identified causal factors. By their nature, moderating factors can either increase, potentiate, exaggerate, and stimulate or, alternatively, lessen, diminish, and suppress the presence or strength of the causal factors. Factors that would be classified as enabling and

106 Chapter 4 Characterizing and Defining the Health Problem

TABLE 4-5 Existing Factors, Moderating Factors, Key Causal Factors, Mediating Factors, and Health Outcome and Impact for Five Health Problems in Layetteville and Bowe County

Required Antecedent

Factors or Conditions

Moderating Factors or

Conditions Key Causal

Factors Mediating

Factors Health

Outcome Health Impact

Age, existing health conditions, pathogens in environment

Knowledge about adult immunizations, media attention, quality of medical care

Motivation to be vaccinated, fear of the communicable disease, perceived susceptibility

Vaccine supply and distribution, vaccine cost

Immunity level

Rate of preventable hospitalizations

Age, food availability, type of employment, genetics

Knowledge about folic acid, taking prenatal vitamins, genetic counseling

Inadequate intake of folic acid, quality of prenatal care, genetic counseling

Preconception nutritional status, biological processes

Presence of neural tube defect

Rate of congenital anomalies

Psychological development, physical development stage

Media messages, knowledge, family support, availability of birth control

Sexual activity, sexual self- efficacy, partner and peer pressure

Use of birth control methods

Diagnosis of pregnancy

Child abuse rate

Developmental stage, local history of violence, local lack of jobs, state gun laws

Parental supervision, school antiviolence program, community action

Lack of conflict resolution skills, school dropout rate, local gang activity, gun availability

Individual resilience, inadequate policing, quality of emergency care

Death from gunshot wounds

Adolescent death rate due to gunshot wounds

Genetic predisposition, age, race, safe place to exercise

Knowledge about diabetes prevention, family support for self-care

Specific health behaviors (e.g., exercise), quality of medical supervision

Physiological processes

Diagnosis of type 2 diabetes

Morbidity due to chronic illness

Stating the Health Problem 107

the degree of motivation is based on the person’s age, current health condition, and the presence of the pathogens in the environment.

The factors listed in Table 4-5 related to deaths from adult immunization rates, adolescent death rates, and rates of congenital anomalies are shown as causal theory diagrams in FIGURES 4-2, 4-3, and 4-4, respectively. These examples use hypothetical data, but findings from the literature were used to substantiate those data: Hwang and Jaakkola (2003) found an association between exposure to chlorination and birth defects; Calhoun, Dodge, Journel, and Zahnd (2005) used police

of adult immunizations to increase. The rate of immunizations is based on individuals actually receiving the vaccine. What causes an adult older than age 55 to seek immunization is motivation. However, even a motivated individual cannot be vaccinated if no vaccine is available or if the cost is too high. The level of motivation varies depending on the person’s knowledge about adult immunizations, the amount of media attention given to the importance of adult immunizations, and the quality of medical care in terms of having a provider who recommends getting immunized. Ultimately, the need for adult immunization and

FIGURE 4-2 Diagram of Theory of Causes/Determinants of Receiving Immunizations, as Contributing to Adult Immunization Rates, Using the Layetteville Example

Causal Theory of Adult Immunization

Adult immunization

rate VaccinationVaccine supply

and distribution, vaccine cost

Knowledge about immunizations, media attention,

quality of medical care

Motivation to be vaccinated, fear of the

communicable disease, perceived vulnerability

Age, existing health condition,

participants’ environment

FIGURE 4-3 Diagram of Theory of Causes/Determinants for Deaths from Gunshot Wounds, as Contributing to Adolescent Death Rates, Using the Layetteville Example

Developmental stage, local history of

violence, local lack of jobs,

state gun laws

Individual resilience, inadequate policing, quality of emergency

care

Causal Theory of Adolescent Deaths from Gunshot Wounds

Parental supervision, school antiviolence

program, community action

Lack conflict resolution skills, school dropout,

local gang activity, gun availability

Death from gunshot wound

Adolescent death rate

108 Chapter 4 Characterizing and Defining the Health Problem

point for planning the intervention, writing objectives, and planning the evaluation. It also serves as a reference point if the planning and evaluation process begins to drift away from the original health problem and factors identified in the community health assessment.

The causal theory statement adds two elements to the causal theory diagram. The full statement includes both the “who” in reference to the health problem and the relevant health indicators. The elements of the causative theory are used to develop the causal theory statement. The basic format is as follows.

Basic Template Risk of [health problem] among [population/ community] is indicated by [health outcome indicators] and results from [causative factors].

Note that the population or community is the “who” of the community health assessment.

Using the example gun violence as described in the Layetteville community health assessment, the basic causal theory would read as in the following example.

Example Risk of [death from gunshot wounds] among [adolescents of Layetteville] is indicated in the

records and gun sales records as part of their assessment. The important consideration here is to gain confidence in pulling the diverse com- munity assessment data together in a coherent, systematic, and scientifically defensible manner.

Writing a Causal Theory of the Health Problem Community diagnosis, as a diagnosis-type for- mula, was suggested by Muecke (1984) as one technique for synthesizing needs assessment data into a statement that can be understood by various health disciplines. Since then, the term community diagnosis has been used to encompass the data collection as well as the planning. As a formula, it follows the general diagnosis format that specifies who is at risk for what health problem and to what extent those risks are related (Stanhope & Lancaster, 2010). It has also been the basis for graduate courses (Quinn, 1999) and for preparedness (Matsuda & Okada, 2006; Okada et al., 2006). Because these definitions might more aptly describe the com- munity health assessment process, the focus here is on developing a coherent statement that is the equivalent of the diagram of the causal theory of the health problem. The value in writing the causal theory statement is that it complements and extends the value of the diagram as a reference

FIGURE 4-4 Diagram of Theory of Causes/Determinants for Neural Tube Defects, as Contributing to Rates of Congenital Anomalies, Using the Bowe County Example

Age, food availability,

type of employment

Preconception nutritional status,

biological processes

Causal Theory of Presence of Neural Tube Defect

Knowledge about folic acid, taking prenatal

vitamins, genetic counseling

Inadequate intake of folic acid, prenatal exposure to

chlorine, paternal exposure to organic solvents

Presence of neural

tube defect

Rate of congenital anomalies

Stating the Health Problem 109

violence, lack of job opportunities, and state gun laws] exist prior to the causes.

The community health assessment is likely to identify or uncover numerous health problems or conditions that potentially need to be addressed. For each health problem, a community health statement can be developed. The factors identified through the community assessment may apply differently to the different health problems. A separate community diagnosis statement can be written for each highly prioritized health problem. For example, the birth defects health problem ranked high in priority, so a community diagnosis statement was developed, as shown in Causal Theory Statement B.

Causal Theory Statement B Risk of [birth defects] among [residents of Bowe County], indicated by the [rate of neural tube defects and congenital anomalies], is caused by [low folic acid intake, parental exposure to organic solvents, prenatal exposure to chlorine, and the Zika virus], but is mediated by [preconception nutritional status and biological processes], given that [genetic counseling, use of prenatal vitamins, knowledge about folic acid, and mosquito abate- ment] moderate the causes and that [the mother’s age, type of employment, and availability of food high in folic acid] exist prior to the causes.

A causal theory statement ought to convey information about the health problem so that it stands as a well-articulated base from which to engage in the prioritization and subsequent program designing processes. The statements can be used in a priority setting, as will be seen in the next section of this chapter. Statements about health problems can be compared with regard to the extent to which the required antecedents, causal factors, and moderating and mediating factors are amenable to change as well as the level of seriousness or importance of the health problem. Although a health problem might be considered a high priority initially, data from the community health assessment might in fact potentially lead to a reprioritiza- tion of the problem. In short, prioritization and

[high rate of admissions for gunshot injuries at the local hospitals and police reports] and results from [lack of conflict resolution skills, being a high school dropout, local gang activity, and gun availability].

The basic template can be modified to incorporate findings about existing factors and mediating factors. The resulting template has more of a public health tone. However, the template does not yet include a way to incorpo- rate assets or strengths of individuals, families, or the community. If we consider that assets and resources have the potential to modify the causative factors’ ability to result in the health problem, then including assets identified through the community needs assessment statement is key. Thus, the final full causal theory statement template is as follows.

Causal Theory Statement Template [Health problem] among [population/commu- nity], indicated in [health outcome indicators], is caused by [causative factors], but is mediated by [mediating factors] given that [moderating factors] moderate the causes and that [required antecedent factors] exist prior to the causes.

Continuing with the Layetteville community example, the community diagnosis template statement would read like Causal Theory Statement A.

Causal Theory Statement A Risk of [death from gunshot wounds] among [adolescents of Layetteville], indicated in the [high rate of admissions for gunshot injuries at the local hospitals and police reports], is caused by [lack of conflict resolution skills, being a high school dropout, local gang activity, and gun availability] but is mediated by [individual resilience, adequacy of policing, and quality of emergency medical care], given that [community action, parental supervision, and school antivi- olence programs] moderate the causes and that [adolescent developmental stage, local history of

110 Chapter 4 Characterizing and Defining the Health Problem

can be targeted at either reducing or enhancing key factors.

▸ Prioritizing Health Problems

The final problems to be addressed by a program are those selected from among the many health concerns identified through the needs assessment. A highly rational approach to prioritization is presented here because the decision process generally begins as a rational approach. Indi- viduals with inclinations toward other planning

assessment are often iterative processes rather than a straightforward linear process. The nature of these processes hints at the extent to which community assessors and health planners need to be flexible and act as guides throughout the planning–assessment process.

In summary, the elements contained in the causal theory statement are related to both pro- gram design and program evaluation, as shown in TABLE 4-6. The causal theory diagram and the corresponding causal theory statement become the basis for developing the program theory with the corresponding logic model. One point is critical to understand at this juncture: The program interventions that will be developed

TABLE 4-6 Relationship of Problem Definition to Program Design and Evaluation

Diagnosis Problem ➜ Program ➜ Evaluation

Risk of: Health problem or condition

Program goal Outcome variables

Among: At-risk population or group, target audience

Recipients Intervention group

As demonstrated in: Health indicators Program objectives Outcome and impact variables

Resulting from causal factors:

Specific processes, conditions, and factors

Interventions or treatments for the target population

Outcome evaluation

But is mediated by: Factors that must be present for the health problem to occur

Possible intervention Possible control variables

Given moderation of the causes by:

Factors that increase or decrease the potency of the causative factors

Possible intervention Possible control variables

And required existing factors of:

Sociodemographic characteristics and social ecological factors

Program eligibility criteria

Control variables or comparison groups

Prioritizing Health Problems 111

results will be honored and used as the basis for moving forward in the program planning. It is also important to give the group adequate background information so that group members can make informed decisions when they cast their votes.

Basic Priority Rating System A more systematic approach was developed by Hanlon (1973), whose model now has a history of being used or adapted in efforts to formulate a community health plan (New York Department of Health, 2006; Sogoric et al., 2005). An overview of this process provides some initial insight into the depth and breadth of the data that are needed for making decisions about program directions. Hanlon’s basic priority rating system (BPRS) is now one of several tools used for establishing priorities and is one of the few tools to high- light an objective approach. Over the years, the adaptability of the Hanlon method can be seen in the small variations made that accommodate community participation in the planning process and major local concerns (Neiger, Thackeray, & Fagen, 2011). The extent to which the BPRS has been adapted in practice underscores the useful- ness of knowing the original system (Platonova, Studnicki, Fisher, & Bridger, 2010).

Hanlon’s approach to planning public health programs has been codified into the deceptively simple formula known as the BPRS. This method entails prioritizing health problems based on the magnitude of the problem, the severity or importance of the health problem, and the potential effectiveness of interventions. A key part of the process involves assigning values to each of these three factors. The formula is:

Basic priority rating = (A + 2B) × C

where A is the score for the magnitude of the problem, B is the score for the seriousness of the health problem, and C is the score for the potential effectiveness of the intervention.

Unfortunately, the scores assigned to the problem magnitude, seriousness, and interven- tion effectiveness (TABLE 4-7) can be biased by

approaches may alter this process accordingly. Nonetheless, health professionals involved in a community health assessment need to have the skills to guide the decision process in a way that does not ignore the data and that results in a plan with the highest overall potential to improve the health status of the community and target community.

Nominal Group Technique The nominal group technique is not strictly a health planning or prioritizing method but is more typically used in small-group processes and in research. Because it is widely used and can easily be applied to stakeholders who have little experience, it is included in this section as a prioritization approach. The nominal group technique has been used for a wide variety of prioritization needs, ranging from ranking research priorities (Elliott et al., 2016) to environmental changes or barriers for  physical activity (Lees et  al., 2007; Jahns, McDonald, Wadsworth, Morin, & Liu, 2014), to relative ranking of health interventions under resource constraints (Makundi, Kapiriri, & Norheim, 2007), to develop- ing health programs (Annang, Hannon, Fletcher, Sykes, & Cornish, 2011; Dreer et al., 2013).

This technique involves a round-robin series of voting and narrowing lists based on the results of the voting. In essence, the process begins with the complete, usually long list of health problems identified from the community health assessment. Each member of the planning group is given three votes to be used to select which problems to address. The problems with the most votes are kept, and the problems with the fewest votes are eliminated. Next, the voting is repeated with each member of the planning group having only one vote. The health problem with the most votes becomes the problem to be addressed. The results may or may not be logical, but they often stimulate dialogue and discussion about why the highest priority problem emerged.

For the nominal group process to be successful in selecting and prioritizing health problems, participants must agree before voting that the

112 Chapter 4 Characterizing and Defining the Health Problem

prevalence, and relative risk. One difficulty with using mortality rates as the sole criterion for determining the size of a health problem is that mortality data are medical, making them less helpful in planning that focuses on behavioral or social health problems. In addition, disability, pain, and quality of life are just as important considerations as death, as we have seen with regard to QALYs and DALYs. Thus, the size of a health problem and the factors leading to its manifestation ought to be viewed from various angles and incorporate a diversity of measures or indicators.

Not all health problems are equally serious (B), where seriousness encompasses the degree of urgency for addressing the problem, the degree of severity of the health problem, the degree of economic losses possible from the health problem, and the degree to which others can be motivated to become involved. Each of these four elements of seriousness can be rated on a scale of 1 (at the

the personal preferences of those involved in the planning process. By going through a group process to arrive at a score for each factor, how- ever, members of the planning group are forced to make explicit the assumptions underlying their assignment of values. This understanding in turn helps establish consensus and consistency within the group.

The first factor to determine is the magni- tude of the health problem (A). Magnitude is reflected in expressed need, such as the demand for and utilization of services. The magnitude can also be the burden of disease in terms of the incidence rate (Pooripussarakul, Rewpai- boon, Bishai, Muangchana, & Tanivess, 2016). The normative needs—namely, what health professionals view as being a deviation from a baseline or normally acceptable level—also captures the magnitude. Normative need is reflected in epidemiological measures, such as mortality and morbidity rates, incidence,

TABLE 4-7 Criteria for Rating Problems According to the BPRS

BPRS Factor

A B C

Size Urgency Severity

Economic Conse-

quences

Willingness or Involve-

ment of Others

Intervention Effectiveness

Rating scale

1 (small) to 10 (endemic)

1 (not at all) to 10 (extremely urgent)

1 (low) to 10 (high)

1 (low) to 10 (high)

1 (low) to 10 (high)

1 (low) to 10 (high)

Factors to consider in the rating

Stability of -incidence or prevalence over time

Rate of spread

Extent to which QALYs and DALYs are affected; virulence of health problem

Healthcare costs; extent to which YLL and YPLL are affected

Political support for addressing the problems; popular awareness of the health problem

Recalcitrance to change; entrenchment of contributing factors

Prioritizing Health Problems 113

factors (FIGURE 4-5). The effective and efficient (lower-cost) interventions  can be  viewed as contributing to the extent to which it is possible to change the health problem. Naturally, at the point that the prioritizing process occurs, the planning group may not have complete data on intervention effectiveness. In this situation, some data gathering may be necessary, with subsequent rescoring.

Propriety, Economics, Acceptability, Resources, and Legality (PEARL) Component Once the calculation of the priorities has been done, the high-priority problems can next be discussed in terms of their propriety, economics, acceptability, resources, and legality (PEARL) characteristics (Vilnius & Dandoy, 1990). Propriety refers to whether addressing the health problem is the responsibility of those represented by the planning group. The economic aspect relates to the economic feasibility of addressing the problem. Acceptability is assessed in terms of the culture’s and population’s preference for the potential intervention. Resources refers to the availability of all types of resources. Naturally,

lowest end) to 10 (at the highest end). Again, the specific data derived from the community assessment are used to score each element. The severity of a health problem or condition is also related to its virulence.

Seriousness is best determined by exam- ining information from experts, the scientific literature, and input from key stakeholders on the long-term consequences of the health problem. The degree of economic loss focuses on individual loss due to disability and death, but it also might include the societal costs of providing care and the loss of revenue from disabled individuals. Utility measures that cap- ture individuals’ preferences for different states of health also play a role, implicitly or explicitly, in determining seriousness.

Intervention effectiveness (C) is the third element in the BPRS. Scoring the effectiveness of the interventions that might be used to address a health problem also utilizes a scale of 1 to 10. Interventions for which considerable and favorable evidence exists would be rated high- est, where favorable means having a clinically and practically significant effect on the health problem. The choice of an intervention deserves considerable attention, in terms of whether and how it has the potential to affect causal or other

FIGURE 4-5 Theory of Causes/Determinants with Elements of the BPRS Score: Size, Seriousness, and Interventions

Mediating factors

Main causal factors of the health problem

Health problem A: Size

B: Seriousness

Moderating factors

C: Potential effective pathways of chosen INTERVENTIONS

114 Chapter 4 Characterizing and Defining the Health Problem

through the use of utility measures or through reliance on the Healthy People 2020 objectives, which indicate whether the health condition is sufficiently important to warrant national atten- tion. The changeability of a health problem or condition is the degree to which any intervention has the potential to alter its course.

When multiple health problems or con- ditions are being considered as targets for an intervention or health program, each of the health problems can be rated with regard to its degree of importance and changeability. Using these two dimensions, program planners can use the high and low changeability and high and low importance parameters to form four quadrants of a 2 × 2 matrix into which health problems can be sorted (TABLE 4-8). Health problems classified as having both high changeability and high importance ought to be addressed first. In contrast, health problems in the low changeable, low importance quadrant are either at the bottom of the list or off it entirely.

The above discussion of methods for prioritizing health problems assumes that the planning group and stakeholders will make rational decisions. Against this backdrop of ratio- nal planning, other factors will likely come into play, such as personal experiences with a specific health conditions, political preferences, or time

resources available for addressing a health prob- lem is a concern that affects prioritization. The legality element reflects whether there are legal constraints or mandates in addressing the health problem. Each health problem being considered for program planning is evaluated on the five dimensions and scored as either yes (1) or no (0).

Applying the PEARL component to pri- oritizing among health problems may not be possible until sufficient data about the health problem and characteristics of the community are known. In other words, it may not be possible to apply the PEARL scoring until a substantial amount of information has been collected about the sociopolitical context of the health problem. The need to revisit priorities when new infor- mation is uncovered is likely to be frustrating for everyone involved in the priority setting, but it is a reality of program planning.

Prioritizing Based on Importance and Changeability A simpler approach to understanding the priority of a health problem is to consider only whether an intervention can actually make a change in the health problem and whether the health problem is important or worth addressing. The importance of a health problem can be assessed

TABLE 4-8 Program Prioritization Based on the Importance and Changeability of the Health Problem

Highly Important Health Problem Less Important Health Problem

Highly Changeable Health Problem, More Effective Intervention

High priority for developing a program

Low priority, unless resources are available for developing a program

Less Changeable Health Problem, Less Effective Intervention

High priority, if an innovative program can be developed

No program development is warranted

Across the Pyramid 115

archival data. Accordingly, the information that is incorporated into the health problem statement tends to be personal; episodic; and specific to a small, highly homogenous target audience. At the individual level, the health problem statement resembles a very compre- hensive medical diagnosis that incorporates sociocultural determinants.

At the enabling services level of the pyra- mid, the data used for prioritization need to be applicable to the aggregate of concern. Thus, the assessment data are from community sources, such as local news media and groups advocating for the aggregate. Other sources of data may be gathered by unobtrusive means, such as walking through a park to determine its level of safety. A key archival source of data at the enabling level is the list of existing services, which would be informative with regard to the urgency for creating a program that does not exist. The various data collected can be sorted into the elements of the health problems, perhaps as a first step toward determining which data ought to be woven into the final health problem statement.

At the population-based services level of the public health pyramid, data for the priori- tization process are more typically drawn from epidemiological data sources, and trend analysis may be necessary to determine the urgency of the health problem. As for the elements of the health problem statement, the factors at the pop- ulation level would apply to the entire population and therefore are not specific to individuals or aggregates. In addition, the factors affecting the health problem are likely to be environmental (either social or physical).

Last, at the infrastructure level of the pyramid, sources of data for prioritization come from the organization, the relevant legal jurisdiction, and workforce records. At this level of the pyramid (and at the other pyramid levels), relevant charac- teristics of the infrastructure can be sorted into the elements of the health problem statement. Doing so serves as a nice double check on barriers and facilitators to the proposed program.

needed to complete the prioritization process. Another important factor is the existence and popularity of rankings, particularly of the county health status (http://www.countyhealthrankings .org/). Published rankings, while intuitive in their interpretation, can be misleading for planning purposes. Rankings provide a relative or comparative status, but they do not take into account the magnitude or severity of the item being ranked nor the changeability of the problem, particularly in the local context. Attention to making a thoughtful, transparent, and explicit selection of health problems helps to engage stakeholders in implementing solutions, and minimizes the possibility of idiosyncratic and inappropriate selections.

▸ Across the Pyramid In this chapter, the community health assessment process, problem statement development, and prioritization process have been described as though they constituted a linear set of activi- ties. This is not the reality. These activities are iterative processes that must be adapted to the local situation. Reflecting on whether the data gathered during the assessment process are representative of the levels of the public health pyramid can be helpful in identifying gaps in the assessment and prioritization.

TABLE 4-9 provides a few examples of how prioritization data can be sorted by level of the pyramid. Note that, across all levels of the pyramid, the determination of the effectiveness of proposed interventions ought to come from the scientific literature or existing, rigorously conducted program evaluations. TABLE 4-10 shows the elements of the causal theory and illustrates how health problem statements incorporate data from across the pyramid.

At the direct services level of the public health pyramid, the data for prioritization come mostly from individual patients through surveys or from provider, clinic, and hospital

116 Chapter 4 Characterizing and Defining the Health Problem

TABLE 4-9 Examples of Sources of Data for Prioritizing Health Problems at Each Level of the Public Health Pyramid*

Level

Individual and Direct Services

Community and Enabling

Services Population-

Based Services Infrastructure

Health problem Nonimmunized adult; infant with neural tube defect; diagnosis of pregnancy; diagnosis of diabetes; death from gunshot wound

Insufficient parenting support services; lack of diabetes management classes

Cultural acceptance of obesity as normal; lack of media messages about safe sex for adolescents and diabetes prevention

Availability of guns; inequities in vaccine distribution; lack of school- based clinics; insufficient inspection of workplaces for teratogens

Data sources for size (A)

Vital records data; clinic or service provider data; hospital discharge data; police records; survey

Census data; waitlists; human services sources of data

Epidemiological data; acute care and outpatient discharge data

Personnel records

Data sources for seriousness (B)

Medical literature on course of the health problem; literature on associated DALYs, YLL, and YHL

Advocacy group pressure; local media on health problem

Statistical trends; Medicaid and Medicare cost data

Regulatory requirements; lawsuits; effects on capacity requirements

Data sources for intervention effectiveness (C)

Scientific literature; professional associations’ practice guidelines

Scientific literature; existing health program evaluations

Scientific literature; existing health program evaluations

Scientific literature; existing health program evaluations

* These examples correspond to the five example health problems in Layetteville.

117Discussion Questions and Activities

know the rate of type 2 diabetes in the three census tracts being assessed. How- ever, the only data available are the county statistics. Describe the process by which you would create a synthetic estimate of the rate of type 2 diabetes for whites, blacks, and Hispanics in the three census tracts.

3. Using the hypothetical data in Table 4-5, create a causal theory diagram for either morbidity due to chronic illness or for child abuse rates. Which decisions did you make in developing the diagram?

▸ Discussion Questions and Activities

1. Which statistical tests would be used to determine statistical significance, and which statistical tests would be used to determine the variance from a popula- tion mean? Give a brief description of their key differences.

2. Imagine that you are part of a communi- ty health assessment and planning group. Your group believes it is important to

TABLE 4-10 Examples of Required Existing, Causal, and Moderating Factors Across the Pyramid

Factors

Level

Individual and Direct Services

Community and Enabling

Services

Population- Based

Services Infrastructure

Required existing factors

Genetic predisposition, health beliefs, attitudes, values

Local economic empowerment zones, safe places to exercise and play

Deep cultural practices and beliefs

Legal or policy considerations, organizational mission, workforce capacity

Causal factors

Lifestyle and health practices, individual physiology, quality of medical supervision, exposure to toxins

Road safety features, local gang activity

Exposure to environmental toxins

Resources allocated, capacity

Moderating factors

Lifestyle practices, family, norms, patterns of health services utilization

Culture, accessibility and availability of local health and social services agencies

Income level, educational level

Health workforce competence, needs assessment and planning, information systems

Mediating factors

Quality of emergency medical care

Police presence in community

Vaccine supply and distribution

118 Chapter 4 Characterizing and Defining the Health Problem

THRIVE: Tool for Health and Resilience In Vulnerable Environments Here is an example of creatively using an Internet version of the online priority application. In this example, raters can see their own priorities. Look at the “Collaboraton Multiplier”: https://www .preventioninstitute.org/tools.

Workbook for Prioritization The National Association of County and City Health Officials (NACCHO) has a wealth of re- sources that are available to the public, including these materials related to establishing health priorities: www.naccho.org/topics/infrastructure /accreditation/upload/Prioritization-Summaries -and-Examples.pdf.

References Anderson, R. (1995). Revisiting the behavioral model and

access to medical care: Does it matter? Journal of Health and Social Behavior, 36(1), 1–10.

Anderson, R., & Aday, L. A. (1978). Access to medical care in the U.S.: Realized and potential. Medical Care, 16, 533–546.

Annang, L., Hannon, L., Fletcher, F. E., Sykes, W., & Cornish, D. (2011). Using nominal technique to inform a sexual health program for black youth. American Journal of Health Behavior, 35(6), 664–673.

Baker, S. P., & Haddon, W., Jr. (1974). Reducing injuries and their results: A scientific approach. Milbank Memorial Fund Quarterly, 52, 377–389.

Baker, T. D., & Reinke, W. A. (1988). Epidemiologic base for health planning. In W. A Reinke (Ed.), Health planning for effective management (pp. 117–130). New York, NY: Oxford University Press.

Calhoun, D., Dodge, A. C., Journel, C. S., & Zahnd, E. (2005). The supply and demand for guns to juveniles: Oakland’s gun tracing project. Journal of Urban Health, 82, 552–559.

Cawley, J., Schroeder, M., & Simon, K. I. (2006). How did welfare reform affect the health insurance coverage of women and children? Health Services Research, 41, 486–506.

Cebrecos, A., Diez, J., Gullon, P., Bilal, U., Franco, M., & Escobar, F. (2016). Characterizing physical activity and food urban environments: A GIS-based multicomponent proposal. International Journal of Health Geographies, 15, 35. doi:10.1186/s12942-016-0065-5

4. Create priority scores for the health problems in Table 4-5 using the BPRS. If you had scored these problems as though you were the state health officer, how might the scores be different?

▸ Internet Resources University of North Carolina (UNC) At UNC, students’ community assessment papers are made public at this website: http://hsl.lib.unc .edu/specialcollections/communitydiagnosispa- pers. A perusal of the papers may help generate some ideas of what is involved and what a needs assessment can yield.

County Health Needs Assessments Most counties in the United States, especially the larger ones, have a website devoted to shar- ing their most recent community health need assessments and reports. These informative assessments provide insights into not only how a community’s needs change but also how the approach to the assessment changes over time. Google “county health needs assessment report” and browse the links.

Statistical Tests Two websites list freeware that is available to perform the most commonly used statistical tests. Both websites give brief descriptions and links to the downloads: www.healthcarefreeware .com/calc.htm and http://statpages.org/.

Community Tool Box The Community Tool Box (http://ctb.ku.edu /en) provides resources and guidance on con- ducting community assessments with a focus on building health communities. The “Tools” section includes an entire area dedicated to community assessment.

119

Jung, K., & Valero, J. N. (2016). Assessing the evolutionary structure of homeless network: Social media use, keywords, and influential stakeholders. Technological Forecasting & Social Change, 110, 51–60. http://dx.doi .org/10.1016/j.techfore.2015.07.015

Kaplan, R. M. (1996). Utility assessment for estimating quality-adjusted life years. In F. A. Sloan (Ed.), Valuing health care: Costs, benefits, and effectiveness of pharma- ceuticals and other medical technologies (pp. 32–60). Cambridge, UK: Cambridge University Press.

Lees, E., Taylor, W. C., Hepworth, J. T., Feliz, K., Cassells, A., & Tobin, J. N. (2007). Environmental changes to increase physical activity: Perceptions of older urban ethnic minority women. Journal of Aging and Physical Activity, 15, 425–438.

Makundi, E., Kapiriri, L., & Norheim, O. F. (2007). Combin- ing evidence and values in priority setting: Testing the balance sheet method in a low-income country. BMC Health Services Research, 7, 152.

Matsuda, Y., & Okada, N. (2006). Community diagnosis for sustainable disaster preparedness. Journal of Natural Disaster Science, 28, 25–33.

Muecke, M. (1984). Community health diagnosis in nursing. Public Health Nursing, 1, 23–35.

Murry, C. J. L. (2013). The state of US health, 1990–2010: Burden of diseases, injuries, and risk factors. Journal of the American Medical Association (JAMA), 310, 591–608. doi:10.1001/jama.2013.13805

Neiger, B. L., Thackeray, R., & Fagen, M. C. (2011). Basic Priority Rating Model 2.0: Current applications for priority setting in health promotion practice. Health Promotion Practice, 12(2), 166–171.

New York Department of Health. (2006). Building on community health assessment: Workshop agenda. Retrieved September 9, 2012, from www.health.state .ny.us/statistics/chac/agenda2_bcha.htm

Okada, N., Yokomatsu, M., Suzuki, Y., Hagihara, Y., Tatano, H., & Michinori, H. (2006). Urban diagnosis as meth- odology of integrated disaster risk management. Annals of Disease Prevention Research, 49, 1–7.

Paronen, O., & Oje, P. (1998). How to understand a com- munity: Community assessment for the promotion of health-related physical activity. Patient Education and Counseling, 33(suppl), S25–S28.

Platonova, E. A., Studnicki, J., Fisher, J. W., & Bridger, C. (2010). Local health department priority setting: An exploratory study. Journal of Public Health Management & Practice, 16(2), 140–147.

Pooripussarakul, S., Rewpaiboon, A., Bishai, D., Muangchana, C., & Tanivess, S. (2016). What criteria do decision makers in Thailand use to set priorities for vaccine introduction? BMC Public Health, 16, 684. doi:10.1186 /s12889-016-3382-5

Preedy, V. R., & Watson, R. R. (Eds.) (2009). Quality of life measures. New York, NY: Springer.

Dever, G. E. (1997). Improving outcomes in public health practice: Strategy and methods. Gaithersburg MD: Aspen Systems.

Dever, G. E. (1980). Community health assessment. German- town, MD: Aspen Systems.

Dever, G. E. (2006). Managerial epidemiology: Practice, methods and concepts. Sudbury, MA: Jones and Bartlett Publishers.

Dreer, L. E., Girkin, C. A., Campbell, L., Wood, A., Gao, L., & Owsley, C. (2013). Glaucoma medication adherence among African-Amerians: Program development. Optom Visual Science, 90(8), 883–897.

Elliott, M. J., Staus, S. E., Pannu, N., Ahmed, S. B., Laupacis, A., . . . & Hemmelgarn, B. R. (2016). A randomized controlled trial comparing in-person and Wiki-inspired nominal group techniques for engaging stakeholders in chronic kidney disease research prioritization. BMC Medical Informatincs and Decision Making, 16, 113. doi:10.1186/s12911-016-0351-y

Farivar, S. S., Cunningham, W. E., & Hays, R. D. (2007). Correlated physical and mental health summary scores for the SF-36 and SF-12 health survey, V.1. Health Quality and Life Outcomes, 5, 54–62.

Fos, P. J (2011). Epidemiology foundations: The science of public health. San Francisco, CA: Jossey-Bass.

Friis, R., H. & Sellers, T. A. (2014). Epidemiology for public health practice (5th ed). Burlington, MA: Jones & Bartlett Learning

Green, L., & Kreuter, M. (1991). Health promotion planning (2nd ed.). Mountain View, CA: Mayfield.

Haddon, W., Jr. (1972). A logical framework for categorizing highway safety phenomena and activity. Journal of Trauma, 12, 193–207. Cited in Grossman, D. C. (2000). The history of injury control and the epidemiology of child and adolescent injuries. Future of Children, 10(1), 23–52.

Handler, A., Rosenberg, D., Kennelly, J., & Monahan, C. (1998). Analytic methods in maternal and child health. Vienna, VA: National Maternal and Child Health Clearinghouse.

Hanlon, J. J. (1973). Is there a future for local health depart- ments? Health Services Report, 88, 898–901.

Hwang, B. F., & Jaakkola, J. J. (2003). Water chlorination and birth defects: A systematic review and meta-analysis. Archives of Environmental Health, 58, 83–91.

Jahns, L., McDonald, L. R., Wadsworth, A., Morin, C. & Liu, Y. (2014). Barriers and facilitators to being physically active on a rurual U.S. Northern Plains American Indian reservation. International Journal of Environmental Research and Public Health, 11, 12053–12063. doi:10.3390 /ijerph111112053

Jia, H., Link, M., Holt, J., Mokdad, A. H., Li, L., & Levy, P. S. (2006). Monitoring county-level vaccination coverage during the 2004–2005 influenza season. American Journal of Preventive Medicine, 31, 275–280.

References

120 Chapter 4 Characterizing and Defining the Health Problem

VanEenwyk, J., & Macdonald, S. C. (2012). Guidelines for working with small numbers. Washington State Department of Health. Accessed November 29, 2016, from www .doh.wa.gov/portals/1/documents/5500/smallnumbers.pdf

Vilnius D., & Dandoy S. (1990). A Priority Rating System for Public Health Programs. Public Health Reports, 105(5):463–470.

Webb, E. J., Campbell, D. T., Schwartz, R. D., & Sechrest, L. (2000). Unobtrusive methods (rev. ed.). Thousand Oaks, CA: Sage.

Webb, E. J., Campbell, D. T., Schwartz, R. D., Sechrest, L., & Grove, J. B. (1981). Non-reactive measures in the social sciences (2nd ed.). Boston, MA: Houghton Mifflin.

Yu, H., Meng, Y., Mendez-Luck, C. A., Jhawr, M., & Wallace, S. P. (2007). Small-area estimation of health insurance coverage for California legislative districts. American Journal of Public Health, 97, 731–737.

Quinn, S. T. (1999). Teaching community diagnosis: Inte- grating community experience with meeting graduate standards for health education. Health Education Research, 14, 685–696.

Radloff, L. S. (1977). The CES-D scale: A self-report depres- sion scale for research in the general population. Applied Psychological Measurement, 1, 385–400.

Ramanadhan, S., Mendez, S. R., Roa, M., & Viswanath, K. (2013). Social media use by community-based organiza- tions conducting health promotion: A content analysis. BMC Public Health, 13, 1120. Accessed October 13, 2016, from http://www.biomedcentral.com/1471-2458/13/1129

Sogoric, S., Rukavina, T. V., Brborovic, O., Vlahugic, A., Zganec, N., & Oreskovic, S. (2005). Counties selecting public health priorities: A “bottom up” approach (Croatian experience). Collegium Antropologicum, 29, 111–119.

Stanhope, M., & Lancaster, J. (2010). Foundations of nursing in the community: Community-oriented practice. St. Louis, MO: Mosby/Elsevier.

SECTION III

Health Program Development And Planning

© Lynne Nicholson/Shutterstock

123

© Lynne Nicholson/Shutterstock

Program Theory and Interventions Revealed

After developing statements about health problems that have been ranked as a high priority, the next steps in health program planning involve a more intellectual and creative effort to articulate an explanation of what caused the problem. This is a critical step toward identifying which intervention or group of interventions will be most effective in addressing the health problem. Wild guesses, past experience, and personal preferences might be used as the basis for decision making, but a more rational approach is to identify existing scientific knowledge and theories that can be used to develop a program theory.

A theory is a description of how something works. It is a set of statements or hypotheses about what will happen and therefore contains statements about the relationships among the variables. We use working theories in everyday life, usually in the form of working hypotheses, such as “If I ask the children to clean their rooms, they are not likely to do it.” We also use theories based in science. For example, based on theories of thermodynamics and heat conduction, we can predict how long the turkey needs to roast.

With regard to planning a health program, a primary consideration is to specify what is to be

explained or predicted with a theory. The health problem is what needs to be explained from a programmatic perspective. To explain how to change or affect the health problem, a theory must contain relevant variables, or factors, and must indicate the direction of the interactions among those variables related to the health problem. Identifying the relevant antecedent, contributing, and determinant factors of the health problem gives planners the foundation for developing a working theory of how the programmatic interventions will lead to the desired health outcome. A difficult part of this task is to identify where a health programmatic intervention can have an effect on those factors. As more details and more factors are included in the explanation of the health problem and beliefs about how the programmatic interven- tions will work, the theory becomes increasingly more complex.

The theory development phase of program planning requires thinking rather than doing, so it often receives less attention than is needed to develop an effective health program fully. However, using a systematic approach to develop a program theory and to engage stakeholders in the development of the theory has big and

CHAPTER 5

124 Chapter 5 Program Theory and Interventions Revealed

long-term payoffs that outweigh any delay or costs associated with developing the theory.

▸ Program Theory A sound basis for developing the health program and for guiding the program evaluation is the use of a program theory. Rossi, Freeman, and Lipsey (2004) acknowledged that the need for a program theory has long been recognized by evaluators in the social sciences, and many have advocated for program theory as useful in public health program development (Potvin, Gendron, Bilodeau, & Chabot, 2005). Program theory is a conceptual plan, with some details about what the program is and how it is expected to work. The comprehensive overview of how the program is to work has various names, including logic model, causal model, outcome line, program model, and action theory. These names all refer to a conceptual plan of how the program will work. Whether one is developing a new health program or designing an evaluation for an existing health program, understanding and articulating the program theory is essential.

The use of the words impact and outcome is inconsistent across the literature, in practice, and in government. It is prudent to look beyond the words themselves and ask for definitions. In this text, outcome refers to the immediate effects resulting from an intervention, whereas impact refers to the long-term or cumulative effects attributable in part to the programmatic interventions. The term effect generically refers to changes or consequences of an intervention, regardless of whether the changes are immediate, proximal outcomes, or longer term distal impacts.

There are two main components of program theory, as shown in the top half of FIGURE 5-1. The theory about resources and actions is called the process theory, and the theory about interventions and outcomes is called the effect theory. The con- cept of program theory is used throughout this text rather than the more widely used term logic model. The key difference is that a full program theory, compared to a logic model, contains a far more explicit explanation of the relationship of the factors related to the health problem with the interventions. These relationships are the effect theory. Similarly, the process theory offers a more explicit and detailed description of the resources

FIGURE 5-1 Model of Program Theory Modified from Rossi, P., Freeman, H., & Lipsey, M. (1999). Evaluation: A systematic approach (6th ed.). Thousand Oaks, CA: Sage.

Planning and thinking

foundation

Program theory

Process theory

Organizational plan

(inputs, capacity)

Outputs (products)

Outputs (products)

Initial health effects

(outcomes)

Longer term health effects

(impact)

Service utilization plan (activities, interventions)

Effect theory

Program delivery

and implementation

reality

Program Theory 125

used than is normally found in a logic model. The major similarity is that both a logic model and the program theory provide road maps to creating a successful program. The development of a program theory and its components leads to a stronger program and a more convincing argument for the program’s existence.

Process Theory The process theory includes three components: the organizational plan, the service utilization plan, and specifications of their outputs (Rossi et al., 2004). Process theory includes inputs, which are part of the organizational plan; activities, which are part of the service utilization plan; and out- puts, which are by-products of the organizational and service utilization plans.

The organizational plan, according to Rossi et  al. (2004), encompasses the nature of the resources needed to implement and sustain the program. As such, it includes specifications about personnel; the organization of resources to be used in the program; and elements of capacity, such as infrastructure, information technology, fiscal resources, and personnel. It covers all the behind-the-scenes work needed to provide a program. The organizational plan implicitly contains if–then statements.

For example, if program staff members are adequately supported with regard to supplies and managerial support, then these staff members will deliver the interventions as planned. These if–then statements are useful not only for checking the logic behind requesting specific resources but also for guiding the portion of the evaluation plan that focuses on the processes behind the delivery of the health program.

The service utilization plan, according to Rossi et  al. (2004), specifies how to reach the intended audience and deliver the programmatic interventions and services to that audience. It constitutes the nuts and bolts of providing the program and of implementing the program plan. The service utilization plan includes specifics about social marketing of the program, accessi- bility and availability of the program, screening

procedures, and other logistics of providing the program. Development of the service plan ought to reflect cultural sensitivity and appropriateness of the services and intervention given the target audience.

Within the context of planning a program, the organizational plan needs to be in place before the program can begin. Developing both the organizational plan and the service utilization plan draws on the results of the organizational and community health assessments, particularly with regard to incorporating existing resources into the plans and addressing structural issues that can affect the delivery of the program. The organizational plan is influenced by the service utilization plan to the extent that the planned intervention must be adequately supported by the resources outlined in the organizational plan. As a consequence, the development of the organizational and service utilization plans is an iterative process, with considerable back-and-forth adjustments as each element is more fully ex- plicated. Likewise, the service utilization plan evolves as the effect theory is revised, which then leads to adjustments in the organizational plan. Thus, the process theory elements are continually adjusted throughout this phase of planning for the program. Although the time it takes to make adjustments and revisions may be frustrating, it is much easier to make the adjustments at this stage of planning than it is to do so after the program has begun.

Effect Theory Effect theory consists of the explanations of how the programmatic interventions will affect the causal factors and moderating or mediating factors of the health problem and describes the relationship between the programmatic interventions and the desired immediate and long-term outcomes for program participants. Three sets of relationships, or theories, constitute the effect theory (FIGURE 5-2): the theory of causal/ determinant factors, the theory of intervention mechanisms, and the theory of outcome to impact. Depending on the health problem, it

126 Chapter 5 Program Theory and Interventions Revealed

and education, as well as psychological strategies and policy formulation. Such a broad definition also allows for the inclusion of strategies not typically considered treatments, such as pro- viding transportation (an enabling service) or community development (an infrastructure-level intervention). Clearly identifying and labeling the interventions as such make developing the intervention and outcome theories easier, facilitate developing outcome objectives, and help distinguish outcome and impact objectives from process objectives.

Sometimes interventions are included as or couched in terms of “activities.” Doing so makes the intervention actions indistinguishable from the myriad of other activities done as part of the organizational or service utilization plans; the latter activities are supportive of the interventions but are not actions that will make a difference on the health problem. Interventions are the heart of all health programs. A clear understanding and statement of the role of interventions is made in the intervention theory.

Finding and Identifying Interventions Selecting and then articulating the chosen inter- ventions are cornerstone activities of health

can be useful to develop each of these theories. Often these theories are implicitly stated and understood by health professionals and program staff members. By explicitly stating and discuss- ing these theories, however, program planners can refine programmatic interventions, thereby increasing the likelihood of program success.

This set of three theories and the associated informally stated hypotheses constitute the effect theory portion of the program theory. The term effect theory makes it clear that this part of the program theory deals with both outcomes and impacts. Generating each of the theories that constitute the effect theory may seem complicated. Program experts agree on the complexity of constructing an effect theory as well as its central role in program evaluation (Rossi et al., 2004).

▸ Interventions Interventions are actions that are done inten- tionally to have a direct effect on persons or populations with the health problem. In other words, interventions are the verbs that tell what is being done to make a change in program recipients. Using this definition allows for the inclusion of a broad range of actions, such as medical treatments, pharmacological treatments,

FIGURE 5-2 The Effect Theory Showing the Causal Theory Using Community Diagnosis Elements

Existing factors

Causal Theory Impact Theory

Health outcome

Health impact

E f f e c t

T h e o r yIntervention Theory

Moderating factors

Mediating mechanisms

INTERVENTIONS

Main causal factors of the health problem

Interventions 127

Types of Interventions A simple starting point for thinking about types of interventions is to consider the levels of prevention. In the most common typology in public health, prevention activities are clas- sified into three levels: primary, secondary, and tertiary. Primary prevention includes those activities that are done to prevent a disease or illness from beginning. Adequate exercise, good nutrition, immunizations, and wearing seat belts are examples of primary prevention. Secondary prevention involves screening for undiagnosed problems so that a disease can be treated before it manifests itself. Blood pressure screenings at health fairs, fecal immunochemical tests for colon cancer, testing for anemia, and cholesterol tests are all secondary prevention activities. Tertiary prevention involves activities to limit the extent of an existing disease. For example, it includes taking blood pressure medications, receiving physical rehabilitation after an injury, and taking stress management classes for individuals with cardiac problems. The three levels of prevention provide a starting point, but they are not sufficiently detailed to provide guidance in the development of programmatic interventions.

Another approach to thinking about types of interventions is to consult one of the various classification schemes of interventions that have been developed across the health disciplines. In medicine, the Current Proce- dural Terminology (CPT®) codes (American Medical Association, 2016) enumerate the various procedures that physicians perform. Excluding diagnostic procedures, all other procedures can be thought of as interventions in the sense that they are intended to affect the health of an individual. In nursing, the equally detailed Nursing Intervention Classi- fication (Bulechek, Butcher, Dochterman, & Wagner, 2013) or the Omaha System (Monsen & Newsom, 2011) may be used to categorize interventions. Given the highly clinical na- ture of these intervention classifications, they would be helpful only in the development

program planning. It is important when planning a health program to draw on existing knowledge in multiple disciplines. A literature review, for example, can generate ideas and information with regard to existing theories that have been used to explain what leads to the health problem, as well as explanations of why some interventions have been effective and others have not.

The use of existing theories can expedite the development of the effect theory and lend it credibility. Heaney and van Ryn (1996) provided a nice example of this phenomenon. In their case, the health problem was worksite stress. These authors wanted to develop a health program to reduce worksite stress but were concerned that existing programs had been designed for a target audience of middle-class employees within the cultural majority. Recognizing this fact, Heaney and van Ryn sought to improve the effectiveness of worksite stress-reduction programs for employees of low status or of a cultural minority. Their premise was that the potential exists for different subgroups to vary in both their participation in and benefit derived from a program.

Heaney and van Ryn (1996) began by reviewing the literature on stress and coping. From this literature, they constructed a theoret- ical model of stress and coping, and identified the major variables along with the direction of the interaction among those variables. They also reviewed the literature on the content of worksite stress-reduction programs and the sociological literature on status, class, culture, and stress. From their literature reviews, they were able to identify program interventions that might potentially alter specific variables in the stress and coping model. This informa- tion became part of their effect theory for the worksite stress-reduction program for low-status minority workers.

Unfortunately, for many health problems, widely accepted theories are not available to guide the development of an effect theory or the selection of interventions. However, health program planners and the planning team have options for determining how to proceed.

128 Chapter 5 Program Theory and Interventions Revealed

Specifying Intervention Administration and Dosage Many health program interventions differ from medical interventions because they are thought of in more general terms, such as “hand out infor- mational flyers” or “provide emotional support.” Nonetheless, health program interventions also need to be thought of in terms of dosage, route of administration, and site of administration.

Dosage refers to the amount and strength of the intervention, with a minimum dosage presumed to have an effect. We normally think of dosage in terms of a medication regimen or an exercise program. However, each intervention strategy that is included in a health program needs to be developed and tailored with regard to the dosage of the intervention exposure for the program participants. For example, Morone, Greco, and Weiner (2008) provided a stress-reduction mindfulness program and described the dosage in terms of practicing the meditation three times per week, for a minimum of 50 minutes each time, over an 8-week period. Specifying the dosage is important for achieving the optimal program for the intended audience; it also provides the information needed to develop the process theory adequately and appropriately. Once the dosage is specified, that information is incorporated into the service utilization plan and is used to modify the organizational plan to ensure that adequate resources have been allocated.

Dosage consists of five elements: frequency, duration, strength, route of administration, and administration credibility. The first four of these are fairly straightforward. Frequency is how often the intervention is received, such as hourly, daily, weekly, monthly. Duration specifies over what time period the intervention is delivered, such as one session, 8 weeks of classes, or 6 months of expo- sure. O’Mara-Eves and colleagues (2015) found that the duration of the intervention associated with an effect varied by health outcome studies. This suggests the need to tailor the duration to the specific desired outcome. Strength of the

of health programs at the individual level. One typology of interventions, the Public Health Intervention Wheel, shows a core set of interventions as applied at the individual, community, population, and systems levels (Olson-Keller, Strochschein, Lia-Hoagberg & Schaffer, 2004). Nevertheless, their use with electronic medical or health records is a big advantage in subsequent program monitoring.

A more global intervention typology is needed to identify interventions across the public health pyramid levels. One such typology, devel- oped by Grobe and Hughes (1993), had seven categories of interventions; an eighth category was added by Issel (1997) when studying case management of pregnant women. This typology, by providing an encompassing perspective, can aid in identifying which activities are program- matic interventions. In a meta-analysis of 131 studies of community-based interventions, most frequently used interventions were education (80% of the studies), advice (54%), social support (44%), and skill development training (39%) (O’Mara-Eves et al., 2015).

Each of the eight types of interventions exists at the direct services, enabling services, and population levels of the public health pyramid (TABLE 5-1). The typology also accommodates both secondary and tertiary prevention because these are activities of health professionals undertaken with the intent of having an effect on the health of the program participant. Primary prevention is not included in the typology because providers cannot “do primary prevention” for or to the participant. Rather, primary prevention is a rubric for a variety of interventions: Individuals receive education about primary prevention, are encouraged to engage in primary prevention behaviors, and might be monitored for the extent to which they practice primary prevention behaviors. This is one example of how such a typology of interventions forces program planners to be specific about the actions (as reflected in verbs in the written plan) that are undertaken to affect the health condition or situation of the target audience.

Interventions 129

TABLE 5-1 Examples of Interventions by Type and Level of the Public Health Pyramid

Intervention Type Direct Services Level

Enabling Services Level

Population Level

Infrastructure Level

Treating Medical or dental procedures, medications, physical manipulations, tertiary prevention

Respite care, exercise classes or groups

Water treatment and fluoridation, mass immunizations

Codification of new treatment modalities

Assessing Determination of needs and preferences by asking individuals, secondary prevention

Determination of needs and preferences by needs assessment

Use of epidemiological data to identify trends and rates of illnesses and conditions

Coordinating Care coordination, client advocacy, referral, linking to services

Case coordination, local provider networks and collaborations

Systems integration, records and data sharing, disaster response, planning

Interorganizational and intersectoral collaboration

Monitoring Reassessment, follow-up Local trends and news reports

Trends analysis, epidemiological surveillance

Educating Skills building, information giving

GED programs, job training programs

Media campaigns

Workforce training

Counseling Psychotherapy, emotional support, marital counseling, cognitive behavioral therapy

Group counseling, family counseling, grief counseling for groups

New alerts and advice

Codification of new trauma diagnoses

Coaching and nudging

Role modeling, motivational interviewing, empowerment, encouragement, stress management

Community development

Policy formation Urban redesign, built environment changes

Giving tangibles

Giving vouchers for food or clothing

Medical supplies loan programs

Income supplements, insurance supplements

Establishing food banks, tax incentives

130 Chapter 5 Program Theory and Interventions Revealed

the number of kilocalories expended per kilogram of body weight (strength), and the number of sessions per week (frequency) over 6 months (duration) administered by trained professionals. In another example, Guttentag and colleagues (2014) developed an early parenting intervention at a high and a low intensity. Based on the findings from a comparison of infant and maternal outcomes in the two groups, they hypothesized the need to extend the high- intensity intervention in order to maintain the effects gained. Such findings highlight the need to be very specific in the development of not only the intervention but also the target audience and the format elements of the service utilization plan.

Interventions and Program Components One key challenge in selecting an intervention strategy is deciding whether a single intervention is warranted or whether a package of interventions would be more effective in addressing the health problem. A program component comprises an intervention or set of interventions, with the corresponding organizational plan. Thus, if a health program includes multiple interventions, each addressing one of several causes of the health problem or one of several moderating or mediating factors for the health problem, and these interventions are grouped in some way that makes sense for either effectiveness or efficiency reasons, then the program has multiple program components.

Using program components is appropriate if, to address the health problem, changes must occur across levels, such as at both the family and the community levels. Levels are nested within other levels, and each can be the focus of the program. It is extremely difficult to develop a single intervention that can affect all or most of the causes and moderating or mediating factors for a health problem at multiple levels. Instead, program components are typically needed. For example, if individuals as well as the community as a whole in which those individuals live are

intervention refers to the powerfulness or intensity. For example, a smoking-cessation mass-media campaign is less intense to receive than smoking cessation counseling by an individual’s primary care physician. In an exercise program to improve cardiovascular fitness, stretching has less strength than Zumba even if both are done at the same frequency and duration. A policy intended to prevent gun violence is stronger if there are more and more severe consequences, such as jail time, than if it results in citations and no jail time. Route of administration is the mechanism by which the intervention is delivered or the medium used to deliver the intervention, whether interpersonal communication, public mass media, educational brochures, or injection. Thus, dosage of health program interventions might include the number of hours of education, days of respite, micrograms of fluoride, number of home visits, or number of weeks of counseling, all along with the amount of the intervention provided (e.g., six education topics versus one topic) was provided and at what level of intensity (e.g., leaving a brochure versus providing detailed feedback on performing a task).

Administration credibility refers to the perceived degree to which the person or agency providing the health program is knowledge- able and believable. In other words, it involves whether the intervention is provided by a health professional of a particular discipline, a lay health worker, or a paraprofessional. For some health problems, the cultural values attached to a physician may be a key factor in the effec- tiveness of the intervention, whereas for other health problems and programs, a community member will have more credibility. Thus, among facets of dosage, administration credibility is particularly relevant for health programs.

For many health problems, research reported in the literature or official documents can pro- vide information on which doses are needed to be effective. For example, Martin, Church, Thompson, Earnest, and Blair (2009) found that the amount of exercise per week among postmenopausal women was related to the amount of change in physical and mental quality of life. The dosage of exercise included

Interventions 131

program outcome. These examples highlight the synergistic effects of interventions that can occur when they are provided in a group con- text, as well as the delivery of a psychological intervention that may or may not have been planned. Understanding such interactions and identifying the presence of implicit interventions is critical to later evaluations of what made the difference in health outcomes.

Some interventions are packaged with mne- monics to assist practitioners with remembering the set of interventions. For example, the “five A’s” consist of assess, advise, agree, assist, and arrange. Fisher et al. (2005) suggested that these interventions may be helpful in programs for diabetes self-management, in addition to drawing attention to the resources and support needed for successful self-management. Alternatively, for programs addressing diabetes and other chronic illnesses, standards have been developed by national associations that specify recommended interventions. Use of national standards is encouraged given that national standards tend to be evidence based, updated regularly, and used as the community standard of practice.

Because each program component will have a slightly different effect, acknowledging the individual components is important in subsequent evaluation plans. The intervention and outcome theories will vary slightly for each program component and for each of the different units of intervention of the program.

Characteristics of Good Interventions The final choice of an intervention or a package of interventions can be evaluated against a set of criteria for useful interventions. Having a list of criteria for good interventions is not new (Blum, 1982) but is helpful.

Evidence-Based Choosing an intervention that has been studied for its effectiveness and found to achieve the intended effect enhances the program’s likelihood

targets for the intervention, then interventions tailored to both individuals and communities will be needed. If both individual behavior and actions of the gun industry are targeted to address the problem of gunshot deaths, then different interventions (program components) are needed.

Another reason to include multiple program components is to address micro and macro health problems. As early as 1982, Blum suggested that some health problems or risks require individual behavioral changes, whereas others require group behavioral change. From a public health perspec- tive, an individual behavioral change needed to protect against a health risk is called active protection; in contrast, protection that does not require individuals to make a behavioral change but is instituted through policy, laws, or some other means that does not involve the individual is called passive protection. Passive protection often occurs at a macro level because it encom- passes more than a small group of individuals. However, macro-level changes can also involve active protection, such as the immunization of all infants and vulnerable adults. Immunization involves individual healthcare-seeking behavior but is intended to have a population effect. In contrast, fluoridation of the water supply and reduction of factory pollutant emissions as health programs are both intended to provide passive protection of a population. The distinctions between micro and macro programs, as well as between active and passive protection, may be important in developing the interventions and the effect theory. If the health program is intended to be community-based or community- focused, then it will likely include components at the micro level as well as at the macro level.

Of course, it is important to consider the package of interventions that the recipients actually receive. For example, Harris (2007) used dance and movement therapy as an intervention with African adolescents who were former child soldiers and survivors of torture. The group cohesion that developed during this program was important to the success (i.e., the effectiveness) of the intervention. Similarly, Lipman et al. (2007) identified group cohesion as being critical for

132 Chapter 5 Program Theory and Interventions Revealed

Even widely accepted interventions may need tailoring. For example, Mapuana, Antonio, and Chung-do (2015), in a review of positive youth de- velopment programs for Native Americans, found that culturally grounded programs had higher completion rates, leading to higher dosage and thus more favorable outcomes. At the individual level, Resnicow and colleagues (2009) found that tailoring breast cancer prevention messages to the ethnic identity of African Americans was associated with a greater increase in their fruit and vegetable intake compared to the control group. However, tailoring interventions—and especially public health prevention messages— can be very difficult, as Perchmann and Reibing (2006) discovered when comparing seven different antismoking messages.

Causes Health Gains A third criterion is that health gains must result from the intervention. That is, the problem must be able to be changed with the available knowledge of how to change it. This criterion acknowledges that some interventions may have unintended consequences or side effects. Awareness of potential harmful effects of public health interventions has led to the development of a typology of harms and a call for articulating possible mechanisms of harm when developing interventions (Bonell, Jamal, Melendez-Torres, & Cummins, 2014). Other programs are simply ineffective, such as the Drug Abuse Resistance Education (DARE) program, which has been widely adopted but is ineffective (Des Jarlais et al., 2006; Pan & Bai, 2010).

This criterion also speaks to an advantage of fully articulating the effect theory. A common tendency among health professionals and program planners is to jump to a favorite solution, albeit one that may not necessarily be a good match for addressing the health problem. One technique that helps avoid this tendency is to specify the mechanisms and processes that would result in the health gains. In some scenarios, interventions could be useful and effective with regard to one type of outcome, but they may not lead to the

of success. The increased awareness of the need to have an evidence-based practice has resulted in an increase in the number of meta-analyses and literature syntheses that provide a summary of the effectiveness of interventions for a specific health condition or problem. Some reviews provide information on which interventions are more effective for a specific health problem (Cooke, 2010; Lundgren & Amin, 2014), other reviews provide information on the dosage characteristics of effective programs (Virués-Ortega, 2010; Travagin, Margola & Revenson, 2015), yet other reviews assess the effectiveness of a known intervention on alternative outcomes (De-Regil, Peña-Rosas, Fernández-Gaxiola, & Ryaco-Solon, 2015).

In choosing an intervention based on sci- entific evidence for its effectiveness, program planners sometimes face the question, What constitutes “evidence”? The array of possibilities ranges from meta-analyses of existing studies to a single randomized clinical trial, qualitative reports, or practice guidelines. The other chal- lenge when selecting an evidence-based inter- vention is dealing with equivocal findings. Many meta-analysis studies of interventions conclude few of the interventions had been inadequately described, making comparisons unreliable. Thus, many authors are reluctant to recommend one intervention over others. This ambiguity over the relative effectiveness of interventions may apply to many health problems.

Tailored to the Intended Population A good intervention is tailored to the character- istics of the intended population. Tailoring the intervention encompasses adapting the program for cultural sensitivity, linguistic appropriateness, group similarity, cultural beliefs, and ethnic val- ues. It can occur either through a modification of the intervention to fit the intended audience or through screening members of the intended audience for eligibility based on an important characteristic. Either approach achieves the goal of having an intervention that can be readily accepted by the program recipients.

Interventions 133

complex health promotion programs intended to address social and environmental determi- nants of health.

Another aspect of manipulability is the notion that the intervention ought to be designed to address moderators as well as determinants of the health problem. This means that the intervention needs to have sufficient strength to overcome factors indirectly affecting the health problem. For example, a school-based violence prevention program has an effect not only on reducing violence in general but also notably among youth who had experienced maltreatment (Crooks, Scott, Ellis, & Wolfe, 2011). The program “buffered” the youth from the consequences of maltreatment. Using the psychological theory of buffering guided the inclusion of specific elements in the program. In other words, existing theories can be helpful in manipulating the intervention so that it is sufficiently strong.

Technologically and Logistically Feasible The feasibility of an intervention needs to be considered from the point of view of whether it is technologically realistic and logistically doable within the context in which the intervention will be provided. These aspects of an intervention could be determined through a pilot study in which the intervention is provided on a small scale and on a trial basis. For example, Abroms, Hershcovitz, Boal, & Levine (2015) conducted a feasibility study of a text-messaging program to promote smoking cessation in a new pop- ulation, Israelis. Ensuring involvement of the stakeholders—and particularly those likely to be providing the intervention—in the planning can provide insights into the feasibility of providing the intervention within an everyday context.

Another aspect of feasibility considers the technology to be used as part of the intervention. In some settings or situations, the availability or acceptability of technology is minimal, limiting the nature of interventions. For example, use of mammography for early detection of breast cancer

outcome or impact of interest. For example, health education about family planning methods may be effective in reducing the birth rate in a target audience, but it may not be effective in reducing rates of sexually transmitted infections. Again, having done the work of developing the effect theory helps program planners be certain that the intervention will lead specifically to the desired health gains.

In addition, the program planners need to have the requisite expertise for designing the intervention and activities so that those activities will actually affect the health problem. As was discussed earlier in terms of prioritizing the health problems, the changeability of a health problem is considered to be one aspect of its importance. In terms of interventions, a more technologically feasible intervention ought to result in a more changeable health problem.

Manipulability The fourth criterion is that the intervention must be manipulable (Rossi et al., 2004). Manip- ulability refers to the ability of the program planners and program staff members to adjust the intervention to the specific needs of the participants. A major element of manipulability is dosage, as discussed earlier in this chapter. If the dosage of the intervention can be tailored to the intended audience, then the intervention meets the manipulability criterion. Effective and efficient interventions are customized to some extent to account for the variations among potential participants.

Related to manipulability is the ability to achieve synergy by taking into account other programmatic interventions. For example, Thomas, Sayers, Godon, and Reilly (2009) described a set of community-level interventions to increase physical activity. The program was designed to mutually reinforce the effects of the social marketing and local policy changes. The approach of intentionally developing a program intervention to maximize the effects of program components being delivered to a community is essential as communities become the focus of

134 Chapter 5 Program Theory and Interventions Revealed

during the priority-setting and assessment phases. Lack of the desired healthy state or a high prevalence of the problem may contribute to granting high priority. By contrast, many effective interventions can be used to address trivial problems of low priority.

Health program planners and evaluators might potentially play a role in raising the priority of the issue so that the health problem takes a more prominent place. To some extent, societal priority is set by celebrity spokespersons for specific health problems or by the nightly news covering the current health research. These societal pressures may conflict with the local assessment data. Nevertheless, the intervention must be aligned with the societal priorities assigned to health problems if it is to receive public credibility and backing. Also, the new behavior or health state must be import- ant to the intended audience members or they will not make attempts to change. Although the importance of the health problem to the intended audience may have been included as an element in the community needs assessment, this issue can resurface during program theory development in terms of societal versus public health priorities.

▸ Path to Program Outcomes and Impacts

Careful consideration of which interventions will be used in the health program must go hand-in-hand with specifying which effects are anticipated from the program. Several factors can distract program planners from having a clear vision of the relevant effect. For example, a plethora of possible outcomes from programmatic interventions may exist. There may also be many ways to think about changes resulting from programs (Patton, 1997, p. 160). Yet another distraction is that, with extensive stakeholder involvement, it is quite possible to become sidetracked and end up with an extensive list of what “our program could do.” For these reasons, having the community diagnosis, as

would not be possible in undeveloped nations, but it also might not be feasible in some remote and impoverished regions in the United States.

Reasonable Cost The sixth criterion is that the cost of the interven- tion must be reasonable rather than prohibitive. The cost of the intervention depends on many factors, such as the extent to which the health behavior or problem is resistant to change, the duration of the program, and the number of program components.

Politically Feasible The seventh criterion of a good intervention is that it be politically feasible. Not all interventions are equally acceptable to the intended audience, to funding agencies, or to other stakeholders. During the assessment phase, program planners should have determined the preferences and willingness of various stakeholders to endorse different types of interventions. Interventions need to be culturally appropriate and sensitive as a first step toward being politically feasible. Various strategies, such as conducting focus groups and pretesting an intervention, can be used to design culturally sensitive and compe- tent health program interventions for use with ethnically or racially distinct target populations.

A corollary to the political feasibility criterion is that meeting this criterion helps the program planner, as well as the program, to survive. Proposing interventions that are not politically feasible can result in the planner being used as a scapegoat and blamed for a “bad” intervention. Worse yet, politically sensitive programs run the risk of not being funded, which will reflect poorly on the qualifications of the program planner.

Addresses Societal Priorities The last criterion is that the intervention must address societal priorities; in other words, the problem must be important in the larger picture. Sufficient agreement first needs to exist with regard to the importance of the health problem. This consensus should have been established

Path to Program Outcomes and Impacts 135

and that [existing factors] exist prior to the causes. Outlining and articulating the components of the effect theory help ensure that the program will achieve the desired effect. The effect theory, as the overall explanation of how the program inter- ventions lead to the desired effect, integrates the three narrowly focused subcomponent theories.

Theory of the Causal/ Determinant Forces One component of the effect theory is the causal theory, which is an explanation of the process that currently underlies the health problem. It includes statements or hypotheses that describe which causal factors, including key relevant social ecological determinants, are directly responsible for the health problem. The causal theory ought to include the factors found present through the community needs assessment and draw on the scientific literature to justify the causal theory.

For example, we can use the community diagnosis related to deaths from gunshot wounds to develop a causal theory. The causal theory states that deaths from gunshot  wounds stem from causal factors of local gang activity, lack of conflict resolution skills, dropping out of school, and gun availability. Individual resilience, adequacy of policing, and quality of emergency medical care are mediating factors that determine whether the causal factors actually result in a death. In addition, the adolescent’s develop- mental stage, local history of violence, lack of job opportunities, and state laws, as preexisting forces, influence whether the causal factors exist. Community action, parental supervision, and school antiviolence programs all have the potential to moderate—either decreasing or increasing—the potency of the causal factors.

Similarly, the community diagnosis for birth defects is the basis for a causal theory of birth defects in Bowe County. The causal the- ory states that birth defects among residents of Bowe County are caused by low folic acid intake, parental exposure to organic solvents, prenatal exposure to chlorine, and the Zika virus. However, preconception nutritional status and biological processes (mediating factors) influence whether

written at the conclusion of the community needs assessment, is important because it helps those involved in the planning process stay focused on both the health problem and those health outcomes and impacts that are directly related to the health program.

Further complicating the choice of key health outcomes and impacts is the reality that change is not always the purpose of health pro- grams; some programs are, in fact, intended to stabilize, prevent, or maintain a health state. In addition, health is multidimensional, encom- passing bio-psycho-social dimensions, and social determinants of health might be the focus of health program. Nonetheless, a specific end point is desired and that essentially constitutes the outcome or impact.

Components of the Effect Theory After having considered the type of intervention and the criteria for choosing an intervention, the next step is to articulate more fully the effect theory by enumerating the causal, intervention, and impact theories that constitute the effect theory. This iterative process requires going back and forth between the needs assessment, priorities, and intervention choice.

Both inductive and deductive approaches can be used to generate an effect theory. In other words, theory development can proceed through a deductive process that uses reason and existing knowledge, or it can occur through an induc- tive process that uses experience and intuition. Either approach will lead to an effect theory. In practice, a combination of both inductive and deductive approaches is typically used and yields the optimal results. Generating an effect theory requires both creativity and intellectual rigor.

Elements of the effect theory draw on the community diagnosis developed for each of the high-priority health problems as well as the literature. Recall the template for the commu- nity diagnosis: Risk of [health problem] among [population/community], indicated in [health indicators or measures], is caused by [causative factors], but is mediated by [mediating factors] given that [moderating factors] moderate the causes

136 Chapter 5 Program Theory and Interventions Revealed

education [intervention] changes the behavior of the woman with regard to eating dark green vegetables. Another point at which to intervene on the causal factors might be by encouraging the use of prenatal vitamins [intervention] to remove the causal factor of inadequate folic acid intake. Also, screening for occupational exposures followed by an early ultrasound [intervention] could identify fetuses with abnor- malities. Receiving nutritional education, taking supplements, and making changes in prenatal care can, together, alter the biological processes that result in a neural tube defect.

As this example shows, not all moderating, causal, or mediating factors need to be, or can be, addressed within a single health program. An equally plausible intervention theory might state that education about occupational exposures [intervention] leads to decreased exposures and subsequently fewer infants with neural tube defects. Alternatively, an intervention theory might focus on the environment context related to eliminat- ing the mosquito carrying the Zika virus. The decision regarding which intervention theory to use as the basis of a program is influenced by the preferences of stakeholders, the mission of the organization, and the science regarding which factors are more readily changeable and effective.

Theory of Outcome to Impact The final element of effect theory is the impact theory, which explains how the outcomes lead to impacts. Usually, a health program has a very limited number of health outcomes that it seeks to affect. Impact theory helps substantiate the sometimes seemingly wild and wishful claims of program planners about the effects of their program by specifying the relationship between the immediate outcome of the program and the long-term, ultimate changes to the health problem. It is possible to have multiple impact theories for one long-range impact, especially if multiple intervention theories are used within a single program. Given the complex nature of many health problems and conditions, this is a likely scenario. Continuing with the birth defects example, the impact theory states that fewer

the causal factors actually result in a birth defect. In addition, the mother’s age, type of employment, and availability of food high in folic acid, as con- textual preexisting factors, determine whether the causal factors exist. Genetic counseling, taking prenatal vitamins, knowledge about folic acid, and mosquito abatement all have the potential to moderate the influence of the causal factors by either increasing or decreasing their potency.

Theory of Intervention Mechanisms The intervention theory explains how interventions affect which of the causal/determinant factors, or possibly the moderating or mediating factors. It contains hypotheses about the relationships of the programmatic interventions to the factors in the causal theory that the interventions are intended to affect. More important, it must address how the intervention alters the causal factors or breaks the chain between causal factors and health outcome. The intervention theory includes statements de- scribing the relationships connecting interventions and outcomes. The intervention might also affect some of the moderating or mediating factors. Thus, the intervention theory articulates all the connections between the programmatic intervention and the intended effects on the health problem. Having the intervention theory explicitly stated and understood by the program staff members contributes to the success of the program.

The intervention theory describes how the program “works its magic.” Developing an inter- vention theory is useful to refine the number, types, and quality of interventions that are carried out as part of the health program. Interventions that are not likely to alter or change the key factors in the causal theory can, in turn, be eliminated, which results in a more effective and efficient program.

In the birth defects health problem, planners might identify several possible points at which to intervene to ensure that the causal factors do not lead to neural tube defects. For example, the program might focus on the moderating factor regarding knowledge about the impor- tance of folic acid. Accordingly, one part of the intervention theory would state that nutritional

Path to Program Outcomes and Impacts 137

interventions need to be tailored to reach that specific intended audience, essentially matching the audience level at which the intervention is aimed to the level at which the intended audience exists, and to the level at which the outcome is desired. For example, if the intervention is designed to affect family eating patterns, then the health outcome sought ought to be family nutritional health rather than reducing anemia in children or increasing the daily consumption of milk in a neighborhood. The importance of being clear about the level or unit for the intervention has long been recognized as pivotal for the evaluation phase; the unit of intervention becomes the unit of analysis in the evaluation phase (Jackson, Altman, Howard-Pitney, & Farquhar, 1989). In the family eating pattern example, the family, as a unit, is the level at which the intervention is focused and would be the unit of analysis in the evaluation of effects.

In summary, the effect theory integrates the theories of causal/determinant forces, interven- tion mechanisms, and outcome with impact. The combination of these three components of the effect theory explains the complexity of addressing a health problem. FIGURE 5-3 brings together all the components of the effect theory in the birth defects example.

infants born with neural tube defects leads to a decrease in the rate of birth defects of all types.

Funding agencies commonly specify pro- gram impacts—for example, a decrease in infant mortality or an increase in early detection of preventable disease. These impacts might be stated as program goals that the funded pro- grams are to achieve. In such cases, program planners must essentially work backward to generate the impact theory and the intervention theory. In addition, impact theories show the links and explain the relationships between the program objectives regarding outcomes and program goals specifying desired impacts.

Matching Levels: Audience, Cause, Intervention, and Effects In developing the effect theory, care must be taken to ensure that the level of the pyramid where the intervention functions matches both the intended audience and the level of the public health pyramid at which the outcomes and impacts are expected. Intended audiences may consist of individu- als, families, aggregates, or populations, with effects occurring at each of those levels. Program

FIGURE 5-3 Effect Theory Example: Effect Theory for Reducing the Rate of Congenital Anomalies

Age, food availability,

type of employment

Causal Theory of Presence of Neural Tube Defect Impact Theory

Presence of neural

tube defect

Rate of congenital anomalies E

f f e c t

T h e o r y

Intervention Theory

INTERVENTIONS

Nutrition education

Prenatal screening, encourage taking prenatal vitamins

Knowledge about folic acid,

genetic counseling

Inadequate intake of folic acid, prenatal

exposure to chlorine, paternal exposure to

organic solvents

Preconception nutritional status,

biological processes

138 Chapter 5 Program Theory and Interventions Revealed

not be congruent. The theories-in-use denote how the program is implemented and are the source of the effects on participants. One way to avoid incongruity between the espoused theory and the theory-in-use is to include the theory-in-use explicitly in the effect theory. Being aware of the differences among espoused theories, theories-in-use, and effect theories (TABLE 5-2) can help planners generate an effect theory that incorporates useful elements of both the espoused theories and the theories-in-use. If the program has been in existence for some time, an alternative is to decide either to incorpo- rate the theory-in-use into the program theory or to exclude the theory-in-use explicitly from the program. Modifying the program theory based on the practical experience gained through the theory-in-use may be efficient and prudent if the theory-in-use has had the desired effect on program participants.

Draw Upon the Scientific Literature Program planners should review articles published across the health disciplines for information that can help them generate the theories by providing information on the relationships among the antecedents and causal, moderating, and mediating factors. Abstracts available through online databases are another good source of ideas that can be incorporated into the effect theory. The pub- lished literature is also helpful in developing the process theory, particularly with regard to the service utilization elements.

Existing theories from multiple disciplines can be used to develop the effect theory. For example, if the health program is intended to have a physiological effect or to address a certain pathology, then theories from genomics, bio- chemistry, pharmacology, or physiology might be useful. In contrast, theories from psychology or social work about psychopathology, stress, coping, or family functioning might be used to explain a family problem. For programs

▸ Generating the Effect Theory

Involve Key Stakeholders Generating a program theory is not a solitary task; it requires brain power, diverse ideas, and sustained energy. Involving key stakeholders not only makes good ideas evident but also encourages stakeholders to become invested in the health program and to address the health problem. This type of involvement is a critical step toward having a politically feasible intervention.

Potential program participants and providers typically have their own working explanation, or theory, of how a program will affect participants. One type of theory they may advocate is an espoused theory. Agryis and Schon (1974) were among the first to understand the importance of espoused theories. They found that employees had explanations for why things happen in their organizations; these stated explanations are the espoused theories. People know what they are supposed to do or say, regardless of whether they actually do or say it. The espoused theory consists of this stated and repeated explanation. Agryis and Schon (1974) also found that espoused theories were not always congruent with the behaviors they observed. What people do to achieve their ends is termed their theory-in-use, sometimes called a theory-in-action. For example, Nichols and Badger (2008) found that hospital staff members espoused one practice for infection control but actually had a different practice or theory-in-use. The theory-in-use is crucial in program evaluation because it consists of the interventions that actually make up the health program and affect participants. For example, in a diabetes management program, if the staff members become friends with the patients and provide encouragement in a supportive manner, but they rarely focus on teaching patients, then their theory-in-use is coaching or social support rather than education.

As seen in the diabetes management exam- ple, espoused theories and theories-in-use may

Generating the Effect Theory 139

The theories used by program planners are generally specific to the level of the public health pyramid. In this section, the examples are largely at the individual level. For problems at the other levels of the pyramid, theories can be found in the literature. For example, Gay (2004) relied on the theory of disease transmission as a framework for understanding what is required to develop a program to eliminate measles, an infectious disease. Although measles is an individual illness, the elimination of any infec- tious disease—whether measles, tuberculosis, or HIV/AIDS—requires thinking in terms of populations as well as individual susceptibility. To change population behaviors related to alcohol use problems, Wallin (2007) reported that the successful program was based on the diffusion of innovation theory.

intended for population-level changes, theories from epidemiology, decision sciences market- ing, social networks, or public health may be helpful. Integrating or borrowing theories from across disciplines may be necessary, especially as a means to address multiple determinants of health problems.

Many existing theories can help health program planners develop causal theories for health problems and situations. The examples listed in TABLE 5-3 are grouped by the domain of health outcomes anticipated by the program as a reminder that ultimately the program in- tervention theory must be matched with both the health problem and the desired outcomes of the program. In addition, existing theories can be used in developing the process theory; examples of such theories are shown in TABLE 5-4.

TABLE 5-2 Comparison of Effect Theory, Espoused Theory, and Theory-in-Use

Effect Theory Espoused Theory Theory-in-Use

What it is Explanation of how program interventions affect participants

What staff members say about how the program affects participants

What staff members actually do that affects participants

Where it resides Manuals and procedures; program descriptions

Minds of program staff members; program manuals and descriptions

Actions of program staff members; on- the-job training

How it is identified Review of scientific literature, program materials

Listen to staff members describe the program; read program materials

Watch what staff members do in providing the program and intended interventions

Importance Guides program evaluation; basis for claiming outcomes

Becomes what staff members, clients, and stakeholders believe and expect of the program

Is the actual cause of program outcomes

140 Chapter 5 Program Theory and Interventions Revealed

means of engaging program staff members and getting feedback from other professionals in the field. As the scientific literature is reviewed and assimilated, additional relevant variables and their interrelationships can be incorporated into the map of the causal chain of events. Including every possible variable is neither realistic nor desirable, of course; instead, program planners should include only those variables that relate to the essence of the program and that, according to the community health assessment and available scientific literature, are mostly likely to influence the success of the proposed interventions.

Diagram the Causal Chain of Events Drawing or creating a visual representation of the various theories is important given the complex nature of the causes of health problems and the equally complex systems of services required to address health problems. Diagrams that depict the effect theory, the process theory, and the overall program theory can be created with pencil and paper or by using graphics software. A picture showing how each intervention changes a charac- teristic of the participants provides an expedient

TABLE 5-3 Examples of Types of Theories Relevant to Developing Theory of Causative/ Determinant Factors or Theory of Intervention Mechanisms by Four Health Domains

Physical Health Psychosocial

Health Knowledge and

Abilities Lifestyle Behaviors

Pathophysiology Immunology Endocrinology Pharmacology Wound healing Biochemistry Metabolism

Psychopathology Social cognition Stress and coping Family functioning Addiction Violence Resilience Discrimination

Learning Communication Cognition Attention Memory Diffusion of innovation Acculturation

Peer pressure Decision making Self-efficacy Self-worth Risk taking Social stratification Motivation Social networks Economic nudging Urban design Health services utilization

TABLE 5-4 Examples of Types of Theories Relevant to Developing the Organizational Plan and Services Utilization Plan Components of the Process Theory

Organizational Plan Service Utilization Plan

Social network Communication Leadership Accounting Quality improvement

Social marketing Marketing Cueing Economic incentivizing Pricing

Functions of Program Theory 141

Another assumption relates to parsimony. Improving the health of individuals, families, and communities is a complex task, so most health programs address only one aspect of a complex puzzle of factors affecting health. Including too much in a program theory can lead to confusion, diffuse interventions, and frustration, not to mention exorbitant expenditures. Parsimony is a crucial characteristic of a good theory, includ- ing an effect theory. Relying on the priorities set earlier in the planning process by focusing on the most important factors about the target audience helps achieve parsimony.

▸ Functions of Program Theory

Having an articulated theory of how the health program will lead to improved health, and specifically how the interventions will affect participants, serves several purposes (Bickman, 1987) that range from providing guidance and enabling explanation to forming a basis for communication.

Provide Guidance A program theory that can be stated in one or two sentences provides a description of what is being implemented. To say that a program is helping asthmatic children is less compelling or descriptive than saying that a program teaches children how to be aware of their bodies and thereby avoid situations that may trigger an asthma attack. The latter is a description of how the program works to reduce asthma attacks and provides direct guidance on what to include in the program.

In a world of complex and interactive health problems, identifying the specific health problem and the appropriate intended audience for a program can be difficult. Blum’s (1982) caution against failure to analyze problems adequately is avoided by developing the program theory,

In some instances, a health program is started in response to a mandate or a health policy initiative and therefore may not have an explicit program theory. If a program has been in existence or is ongoing, the development of a program theory is still possible, and its creation instead can contribute to program improvements. In such cases, the espoused theory of program staff members is a good starting point for the development of a program theory. Observations of program staff members would then help identify the theory-in-use. Together with findings from the literature, these elements could be formalized into an effect theory. It is quite possible that new areas for program monitoring and evaluation would emerge from such an exercise with program staff. In addition, program staff members may come to see the value of their work and become more committed to the program and the participants. Involving program staff members in reconciling their espoused theories and theories-in-use can lead to new program approaches and the iden- tification of areas of inefficiencies.

For some health programs, timing is critical such that some intervention components must be accomplished before other intervention components are implemented. If either the intervention or the outcomes must proceed in stages, these increments need to be reflected in the effect theory of the causal chain of events leading to the health outcome.

Check Against Assumptions The program theory—and the effect theory in particular—needs to be checked against alterna- tive assumptions about theories. Patton (1997) referred to these points as validity assumptions. One assumption is that the theory is really about the phenomenon of interest. In other words, pro- gram planners assume that the program theory truly deals with the health problem or condition that is the focus of the health program. Through the multiple interactions and discussions with stakeholders, this assumption can inadvertently be violated.

142 Chapter 5 Program Theory and Interventions Revealed

Enable Explanations The program theory helps identify which interventions are likely to have the greatest effect on program participants and clarify how the interventions cause the desired effect in program participants. In this way, the theory enables planners and evaluators to explain more easily how the program should and does work.

One task of program planners is to antic- ipate the unintended. Careful attention to the development of the program theory can help uncover unintended consequences that may result from the program. The development of an effect theory, in turn, helps generate plausible explanations for those unintended consequences. Engaging in this kind of exercise in speculation helps program planners avoid another source of unsuccessful programs: failure to examine and compare relevant possible interventions (Blum, 1982).

A program theory also enables the evaluators to distinguish between process theory failure and effect theory failure (FIGURE 5-4). If the evaluation results show no effect on program participants, then the evaluator must explain what failed. A  successful program sets into motion the interventions (causal processes) that lead to the desired outcome. However, if a program is not effective, the evaluator needs to identify the roots of that failure. A lack of program success can result from the program not being provided—a process theory failure. A lack of program success also can result from an ineffective intervention—an effect theory failure. This distinction between process and effect theory failures, based on the notions of program and theory failure put forth by Weiss (1972), helps evaluators sort out what went wrong or right with the program and explain the evaluation findings to stakeholders.

Form a Basis for Communication Health programs compete for resources. A program theory helps convince organizational or legislative policy makers that the program is

which specifies the problem and the intended audience. If the program theory is inordinately difficult to develop, it may indicate that the health problem has not been sufficiently narrowed, the intended audience is not specific enough, or too many program elements have been included. Having an intended audience that is too broad can lead to a program theory that is too complex to be of value in designing and implementing the program.

The program theory guides what to measure in both the process and the effect evaluations of the program. In terms of the process evaluation, it specifies what needs to be measured with regard to the delivery of the intervention. In terms of the effect evaluation, the effect theory specifies the desired effects and therefore what needs to be measured. When a health program has several possible outcomes, the effect theory clarifies which outcome is most directly a result of the intervention. This information makes the evaluation of outcomes more efficient and enables program planners and evaluators to design an evaluation that will find those program effects that are arguably the result of the program.

Just as theory is used to guide the develop- ment of the health program, so theory can be used to guide the development of the evaluation. For example, Newes-Adeyi, Helitzer, Caulfield, and Bronner (2000) used ecological theory to guide their formative evaluation of New York State’s Women, Infants, and Children (WIC) nutritional program. Their use of ecological theory strengthened the evaluation in terms of its design and ability to explain how the program worked. Their report also serves as a reminder that the same underlying social or psychological theory that guides the effect theory can be applied to the effect evaluation as well.

When a new health program is first pro- vided, its evaluation helps refine the subsequent delivery of the program. A program theory helps identify needed inputs and determine what needs to be evaluated and where improvements or changes in the delivery of the interventions are appropriate.

Across the Pyramid 143

consensus on the program theory could be critical to the program’s survival.

Make a Scientific Contribution In a sense, every health program is an experiment that tests the program theory. In other words, every evaluation has the potential to contribute to our understanding of human nature and health. Evaluations based on the program theory can be used to modify existing theories relevant to the target population and types of interventions used.

▸ Across the Pyramid At the direct services level of the public health pyramid, the health problems are related to specific individuals, so the relevant theories will focus on individual behavior and intra-individual responses to treatment or pathology. In other words, the focus is on the micro level. As a result, the interventions delivered are one-on-one, with providers directly delivering the interventions to their clients. (Examples of direct services inter- ventions appear in Table 5-1.) If the program will have subcomponents, those components would involve different types of interventions that are delivered directly to individuals.

At the enabling services level, the health problems are related to aggregates of individuals, so the relevant theories will focus on the interactions

worthy and deserving of support. The causal chain of events outlined in the effect theory serves to frame discussions on a more rational basis, leading to a more rational decision-making process about the health program. The effect theory also helps policy makers understand the extent to which the program interventions are ideologically compatible with their stance and are based on science rather than biases and opinions. In other words, the effect theory provides a basis for clear communication of the program intent and content.

Starting and maintaining a program requires that key stakeholders agree on sup- porting the program. Gaining consensus from stakeholders—whether program staff members, administrators, or legislators—is an important step in ensuring the success and acceptance of the health program. If stakeholders understand the program theory, it becomes easier to gain consensus on the usefulness of the program. Having gone through the exercise of developing the causal, intervention, and impact theories, the program planners are in the position of being better able to anticipate questions and provide alternative rationales for the health program. As mentioned earlier, stakeholders can be included in the development of the program theory as a way to gain consensus on the program interventions. For controversial programs, such as those dealing with sexuality education or family planning for adolescents,

FIGURE 5-4 Two Roots of Program Failure Weiss, C. H. (1998). Evaluation, (2nd ed.). Reprinted by permission of Pearson Education, Inc., New York, NY.

Program interventions

set into

motion

which led toCausal process

Desired effect

Successful program

Program interventions

did not set into motion

which would

have led to Causal process

Desired effect

Process theory failure

Program interventions

set into

motion which did not lead to

Causal process

Desired effect

Effect theory failure

144 Chapter 5 Program Theory and Interventions Revealed

of planned behavior as the basis for enhancing genetics literacy among health professionals.

▸ Discussion Questions and Activities

1. Select a health program with which you are familiar. a. Briefly state the hypotheses that

constitute the effect theory of the program.

b. What are the intervention compo- nents and the specific interventions?

c. Develop an effect theory of the pro- gram theory used by the program.

d. Do a brief literature search to deter- mine whether the scientific evidence supports the interventions used.

2. What are the relationships among the possible functions of effect theory and the selection of optimal interventions?

3. Which of the theories that make up the effect theory are likely to be affected by the cultural, ethnic, or racial differenc- es of target populations? In what ways might you make those theories cultur- ally appropriate or sensitive?

4. Identify possible primary, secondary, and tertiary prevention interventions for each level of the public health pyramid.

5. Figure 5-3 shows a possible effect theo- ry, with the interventions, to address the health problem of congenital anomalies. Using this as a template or an example, try developing an effect theory diagram for one specific health problem of inter- est to you.

▸ Internet Resources University of Iowa, College of Nursing This website (www.nursing.uiowa.edu/center-for- nursing-classification-and-clinical-effectiveness)

of individuals with family or community char- acteristics. Because enabling services are still provided to individuals, the focus continues to be at the micro level. Hence, interventions are delivered on a one-on-one basis, as well as to groups with similar characteristics. Different intervention types can be applied at the enabling services level (Table 5-1).

At the population level, the health problems are related to entire populations, so the relevant theories will focus on group responses that lead to the health problem, cultural theories that explain behaviors and beliefs related to the health problem, and social theories about interactions among groups. Liddle and Hogue (2000), for example, described an intervention for high-risk adolescents. One key feature of their intervention model was that the theoretical foundation included risk and protection theory, developmental psychopathology theory, and ecological theory. This blend of theories is consistent with the intent of the program. In terms of the public health pyramid, however, the use of ecological theory reflects the theoretical awareness of the program planners that the population level influences both the enabling level (i.e., the family) and the individual level. At the population level, the interventions are designed and intended to have a universal focus. Such interventions are more likely to be delivered though the mass media or to involve policy formation. Having program components at the population level may create synergies that enhance the intervention, as well as possibly increasing feasibility and decreasing long-term program costs.

At the infrastructure level, the problems are related not to individuals but rather to processes and structures that enable the delivery of health programs, so relevant theories might focus on organizational behavior, management and leader- ship style, personnel motivation, political action, and communication. The interventions can be delivered one-on-one with personnel, as well as with groups of workers or entire organizations. Because workforce capacity building is a key focus at the infrastructure level, it may be appropriate to use individual-level theories. For example, Kirk, Tonkin, and Burke (2008) used the theory

145References

Mapuana C. K., Antonio, M. A., & Chung-Do, J. J. DrPH. (2015). Systematic review of interventions focusing on indigenous adolescent mental health and substance use. American Indian and Alaska Native Mental Health Research (Online), 22(3), 36–56. Retrieved from https:// librarylink.uncc.edu/login?url=http://search.proquest .com/docview/1787152385?accountid=14605

Bickman, L. (1987). The functions of program theory. New Directions for Program Evaluation, 43, 5–18.

Blum, H. L. (1982). Social perspective on risk reduction. In M. M. Farber & A. M. Reinhart (Eds.), Promoting health through risk reduction (pp. 19–36). New York, NY: Macmillan.

Bonell, C., Jamal, F., Melendez-Torres, G. J., & Cummins, S. (2014). ‘Dark logic’: Theorizing the harmful consequences of public health interventions. Journal of Epidemiol- ogy and Community Health, 69, 95–98. doi:10.1136 /jech-2014-2046712014-204671

Bulechek, G., Butcher, H., Dochterman, J., & Wagner, C. (Eds.). (2013). Nursing interventions classification (NIC) (6th ed.). St. Louis, MO: Elsevier.

Cooke, E. V. (2010). The effects of increased dose of exercise-based therapies to enhance motor recovery after stroke: A systematic review and meta-analysis. BMC Medicine, 8(60). doi:10.1186/1741-7015-8-60

Crooks, C. V., Scott, K., Ellis, W., & Wolfe, D. A. (2011). Impact of a universal school-based violence prevention program on violent delinquency: Distinctive benefits for youth with maltreatment histories. Child Abuse and Neglect, 35(6), 393–400.

De-Regil, L. M., Peña-Rosas, J. P., Fernández-Gaxiola, A. C., & Rayco-Solon, P. (2015). Effects and safety of peri- conceptional oral folate supplementation for preventing birth defects. Cochrane Database of Systematic Reviews, 12. Article Number CD007950. doi:10.1002/14651858. CD007950.pub3

Des Jarlais, D., Sloboda, Z., Friedman, S. R., Tempalski, B., McKnight, C., & Braine, N. (2006). Diffusion of the D.A.R.E. and syringe exchange programs. American Journal of Public Health, 96, 1354–1358.

Fisher, E. B., Browston, C. A., O’Toole, M. L., Shetty, G., Anwuri, V. V., & Glasgow, R. E. (2005). Ecological approaches to self-management: The case of diabetes. American Journal of Public Health, 95, 1523–1535.

Gay, N. J. (2004). The theory of measles elimination: Impli- cations for the design of elimination strategies. Journal of Infectious Diseases, 189(1S), S27–S35.

Grobe, S. J., & Hughes, L. C. (1993). The conceptual validity of a taxonomy of nursing interventions. Journal of Advanced Nursing, 18, 1942–1961.

Guttentag, C. L., Landry, S. H., Williams, J. M., Noria, C. W., Borkowsi, J. G., . . . Warren, S. F. (2014). “My Baby & Me”: Effects of an early, comprehensive parenting intervention on at-risk mothers and their children. Developmental Psychology, 50, 1482–1496. doi:10.1037 /a0035682

provides an overview of standard nursing interventions (NIC) and outcomes (NOC). The detailed list can be helpful to show the level of specificity for interventions, which may be needed for some programs.

Wheel of Public Health Interventions Each of 16 different interventions can be explored at this interactive website. The intervention examples are more widely applicable than is the public health nursing focus. http://www.people .vcu.edu/~elmiles/interventions/

Understanding Change and Theories Critical in Developing Program Theory The following websites focus on understanding and generating change. For example, the chapter found in the “Theory at a Glance” published in 2005 (https://cancercontrol.cancer.gov/brp /research/theories_project/index.html) is part of a short, online text and nicely summarizes the theories often used in public health. If you want to broaden your repertoire of change theories, then the information at the Communication Initiative Network (www.comminit.com) would be helpful.

Community Guide to Preventive Services This website lists interventions for various health topics and the degree of scientific evidence for the use of the intervention: www.thecommuni- tyguide.org. These systematic literature reviews focus on interventions for populations.

References Abroms, L., Hershcovitz, R., Boal, A., & Levine, H. (2015).

Feasibility and acceptability of a text messaging pro- gram for smoking cessation in Israel. Journal of Health Communication, 20, 903–990. doi:10.1080/10810730 .2015.1018585

Agryis, C., & Schon, D. A. (1974). Theory in practice: Increasing professional effectiveness. San Francisco, CA: Jossey-Bass.

American Medical Association. (2016). CPT 2016: Professional edition. Chicago, IL: American Medical Association.

146 Chapter 5 Program Theory and Interventions Revealed

Nichols, A., & Badger, B. (2008). An investigation of the division between espoused and actual practices in infection control and of the knowledge sources that may underpin this division. British Journal of Infection Control, 9(4), 11–15.

Olson-Keller, L., Strochschein, S., Lia-Hoagberg, B., & Schaffer, M. A. (2004). Population-based public health interventions: Practice-based and evidence-supported, Part I. Public Health Nursing, 21, 453–468.

O’Mara-Eves, A., Brunton, G., Oliver, S., Kavanagh, J., Jamal, F., & Thomas, J. (2015). The effectives of community-engagement in public health interventions for disadvantaged groups: A meta-analysis. BMC Public Health, 15, 129. doi:10.1186/s12889-015-1352-y

Pan, W., & Bai, H. (2010). A multivariate approach to a meta-analytic review of the effectiveness of the D.A.R.E. Program. International Journal of Environmental Re- search and Public Health, 6(1), 267–277.

Patton, M. Q. (1997). Utilization focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.

Perchmann, C., & Reibing, E. T. (2006). Antismoking ad- vertisements for youths: An independent evaluation of health, counter-industry, and industry approaches. American Journal of Public Health, 96, 906–913.

Potvin, L., Gendron, S., Bilodeau, A., & Chabot, P. (2005). Integrating social theory into public health practice. American Journal of Public Health, 95, 591–595.

Resnicow, K., Davis, R., Zhang, N., Strecher, V., Tolsma, D., Calvi, J., William, C. (2009). Tailoring a fruit and vegetable intervention on ethnic identity: Results of a randomized study. Health Psychology, 28(4), 394–403.

Rossi, P., Freeman, H., & Lipsey, M. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.

Thomas, I. M., Sayers, S. P., Godon, J. L., & Reilly, S. R. (2009). Bike, walk, and wheel: A way of life in Columbia, Missouri. American Journal of Preventative Medicine, 37(6), 322–328.

Travagin, G., Margola, D., & Revenson, T. A. (2015). How effective are expressive writing interventions for adoles- cents? A meta-analytic review. Clinical Psychology Review, 36, 42–55. Retrieved from http://dx.doi.org/10.1016/j .cpr.2015.01.003

Virués-Ortega, J. (2010). Applied behavior analytic interven- tion for autism in early childhood: Meta-analysis, meta- regression and dose–response meta-analysis of multiple outcomes. Clinical Psychology Review, 30(4), 387–399.

Wallin, E. (2007). Dissemination of prevention: Community action targeting alcohol use-related problems at licensed premises. Substance Use and Misuse, 42, 2085–2097.

Weiss, C. (1972). Evaluation. San Francisco, CA: Jossey-Bass.

Harris, D. A. (2007). Dance/movement therapy approaches to fostering resilience and recovery among African adolescent torture survivors. Torture, 17, 134–155.

Heaney, C. A., & van Ryn, M. (1996). The implications of status, class, and cultural diversity for health education practice: The case of worksite stress reduction programs. Health Education Research, 11(1), 57–70.

Issel, L. M. (1997). Measuring comprehensive case man- agement interventions: Development of a tool. Nursing Case Management, 2, 3–12.

Jackson, C., Altman, D. G., Howard-Pitney, B., & Farquhar, J. W. (1989). Evaluating community-level health pro- motion and disease prevention interventions. In M. T. Braverman (Ed.), Evaluating health promotion programs (pp. 19–32). San Francisco, CA: Jossey-Bass.

Kirk, M., Tonkin, E., & Burke, S. (2008). Engaging nurses in genetics: The strategic approach in the NHS national genetics education and development center. Journal of Genetic Counseling, 17, 180–188.

Liddle, H. A., & Hogue, A. (2000). A family-based, developmental–ecological preventive intervention for high-risk adolescents. Journal of Marital and Family Therapy, 26, 265–279.

Lipman, E. L., Waymouth, M., Gammon, T., Carter, P., Secord, M., Leung, O., Hicks, F. (2007). Influence of group cohesion on maternal wellbeing among participants in a support/education group for single mothers. American Journal of Orthopsychiatry, 77, 543–549.

Lundgren, R., & Amin, A. (2014). Addressing intimate partner violence and sexual violence among adoles- cents: Emerging evidence of effectiveness. Journal of Adolescent Health, 56, S42–S50. doi:10.1016/j .jadohealth.2014.08.012

Martin, C. K., Church, T. S., Thompson, A. M., Earnest, C. P, & Blair, S. N. (2009). Exercise dose and quality of life: A randomized controlled trial. Archives of Internal Medicine, 169(3), 269–278.

Monsen, K. A., & Newsom, E. T. (2011). Feasibility of using the Omaha System to represent public health nurses manager interventions. Public Health Nursing, 28(5), 421–428. doi:10.1111/j.1525-1446.2010.00926

Morone, N. E., Greco, C. M., & Weiner, D. K. (2008). Mindfulness meditation for the treatment of chronic low back pain in older adults: A randomized controlled pilot study. Pain, 134, 310–319.

Newes-Adeyi, G., Helitzer, D. L., Caulfield, L. E., & Bronner, Y. (2000). Theory and practice: Applying the ecological model of formative research for a WIC training program in New York State. Health Education Research, 15(3), 283–291.

147

Program Objectives and Setting Targets

In this chapter, the focus is on setting the parameters by which the program is judged as successful—in other words, developing goals and objectives for the program. Setting goals and objectives gives clarity and specificity to the implementation plan and guides subsequent evaluation effects. After a logic model has been developed, setting goals and objectives are the next step in program planning.

▸ Program Goals and Objectives

Goals and objectives are terms that are widely used in program planning and evaluation. Goals, in a strict sense, are broad, encompassing statements about the impact to be achieved, whereas objectives are specific statements about outcomes to be achieved and are stated in measurable terms. Funding bodies do not use the terms objectives and goals consistently, so program planners and evaluators must understand the difference. The distinction between objectives and goals forms the basis for the conceptual distinctions between short-term outcomes and long-term impacts of the program.

Goals and their corresponding objectives reflect the logic model and the program theory. Involving the stakeholders and program staff members in the development of the program goals and objectives can be useful in gaining their support, stimulating good ideas, and reaching a consensus on what will constitute the program. However, the process of reaching a consensus, particularly on objectives, can be a bit of a struggle if stakeholders have vested interests in achieving particular health outcomes for their constituents. In addition, program planners often must cope with tight schedules for preparing a program proposal, making timely involvement of stakeholders a challenge. The efforts devoted to arriving at a set of clearly articulated goals and objectives do pay dividends, however; they lay the foundation from which to develop the evaluation and establish standards against which to assess the success of the program.

Goals Goals are always statements about the health impact or status of the intended audience, and generally apply to a longer time horizon, such as 5 years. Typically, goals do not incorporate a quantifiable

CHAPTER 6

© Lynne Nicholson/Shutterstock

148 Chapter 6 Program Objectives and Setting Targets

measure but instead refer in broad terms to the most important anticipated effect of the program. A program will have at least one goal, and a well-focused program with several components may have more than one. In general, however, the number of goals is quite low. The use of creative activities, stories, and clear communication can make writing goals a positive experience.

Good goals are congruent with and contrib- ute to the strategic plan, whether it consists of a national (e.g., Healthy People 2020), state, or local health plan. For health programs being developed by local health agencies or local community-based organizations, this larger context of health programs can be crucial in achieving synergies within the program as well as between existing programs with complementary foci. Having program goals and objectives that also are compatible with the strategic or long-term plan of the healthcare organization can affect the priority given to the program and hence the fiscal support provided for, and organizational approval of, the program.

On a cautionary note, Friedman, Rothman, and Withers (2006) provocatively remind eval- uators that using goals for evaluation creates a paradox. Goals implicitly or explicitly embody values. The paradox arises when goals are used to guide an evaluation because the evaluation then also embodies those values. This is not nec- essarily a bad thing, but it serves as a reminder that who is involved in developing and approving the goals is important.

Foci of Objectives Objectives are precise statements about the work to be done or an immediate effect on program participants, and have a short time frame. Develop- ment of objectives begins with having conceptual clarity regarding whether the objective is related to the program’s process theory or effect theory. Objectives focus specifically on the process theory or the effect and thus are called either process or effect objectives. Objectives address specific program processes or effects, so they provide more direct guidance for the program implementation. For this reason, program planners and evaluators

need to understand the particulars of the format and development of objectives.

An easy format that helps in remembering the parts of a good objective is this: “by when, who will achieve what, by how much.” For example, one objective may be: “By 2012, the Layetteville Innovation—Adolescent Preventing Pregnancy (i-APP) Program will reduce the pregnancy rate among program participants by 20% compared to girls not participating.” The “by how much” portion of the objective, or the target value, is the quantifiable measure that distinguishes an objective from a goal. The target value is the essence of the objective; without it, no objective exists.

The statement, “The percentage of pregnant adolescent girls among girls enrolled in the Layetteville i-APP Program during 2010 will be reduced” is a goal, not an objective. A goal may have several objectives that delineate more precisely what achieving the goal entails. Thus, adding a target value would yield the following for reaching that goal: “The percentage of adolescent girls who become pregnant among girls enrolled in the Layetteville i-APP Program during 2010 will be 8%.” The “8%” quantifies the reduction and is measurable.

The time frame used in the objectives needs to be short term and well within the life span of the program. For direct services and enabling services, objectives generally are set with a 1- or 2-year time horizon. This time line contrasts with that for objectives for population services, which are more likely to have 5- or 10-year horizons that are aligned with the national Healthy People targets set for a particular decade.

Process Objectives: TAAPS The process theory component of the program theory—and specifically the organizational plan and the service utilization plan—provides a framework for stating process objectives (FIGURE 6-1). The process objectives encapsulate the essentials of the process theory that describes how the program is delivered, focusing on the activities of the program staff members or the program participant. Process objectives ought to have the following elements: Time frame,

Program Goals and Objectives 149

Amount of what Activities done by which Participants/program Staff (TAAPS). TAAPS objectives can be written in the general format for writing objectives; process objectives would then state, “by when, which staff members will do what, to what extent” (TABLE 6-1).

TAAPS objectives focus on actions of par- ticipants or on the activities of the program staff members that don’t directly cause the effect. The organizational plan and service utilization plan provide insights into what ought to be included in each process objective, particularly for the “do what” portion. The “to what extent” portion will be determined based on past experience with the capabilities of the staff and on the amount of work to be done within the time frame. Objectives can be identified for the capacity of the infrastructure, as is commonly done in terms of personnel qual- ifications. Capacity objectives are best considered as objectives about the organizational plan.

Thus, a process objective might be, “By month 6, 100% of program staff will have par- ticipated in 90% of the training sessions on how to use health education modules being used in Layetteville’s i-APP Program.”

Effect Objectives: TREW Effect objectives focus on benefits that program participants will experience as a result of receiving the program interventions. Following the formula for writing objectives, effect objectives would state, “by when, how many of which program participants will experience what type of health benefit or state and to what extent.”

Effect objectives have the following elements: in what Time frame, what portion of Recipients experience what Extent of Which type of change (TREW). TREW objectives can be written in the general format for writing objectives; effect objectives would then state, “After how much intervention, how many recipients will experi- ence what extent of which type of change.” The extent refers to how much or what degree of change, which is anticipated by having received a sufficient dosage of the intervention.

The effect theory—most notably, the theo- ries of cause/determinants, of the intervention, and of outcomes—provides the basis for stating intervention, outcome, and impact objectives, remembering that impacts are more appropriately

FIGURE 6-1 Using Elements of Program Theory as the Basis for Writing Program Objectives

Planning and Thinking

Foundation

Program Theory

Process Theory

Organizational Plan

(Inputs, Capacity)

Outputs (Products)

Outputs (Products)

Initial Health Effects

(Outcomes)

Longer term health Effects

(Impacts)

Service Utilization Plan (Activities, Interventions)

Effect Theory

Program Delivery

and Implementation

Reality

Process Objectives (TAAPS) Outcome Objectives (TREW) Goals

150 Chapter 6 Program Objectives and Setting Targets

the Layetteville i-APP Program participants will have a 20% lower pregnancy rate compared to girls not participating,” uses nonparticipants as the comparison—that is, the benchmark. Another approach to writing an outcome ob- jective is to have the “extent” parameter reflect a preferred level of achievement or a target value that the program seeks to achieve. An objective written in this way might state, “After attending 90% of the sessions, 100% of Layetteville i-APP Program participants will have a 3% pregnancy rate.” Regardless of how the “extent” parameter is stated, the objectives need to reference a quantity of dosage that can be translated into a time frame, the program participants to be affected, a health outcome related to the pro- gram interventions, and a quantifiable target value for that health outcome. Clearly stated objectives that include these components serve

called goals (FIGURE 6-2). In most program litera- ture, all three types of objectives are referred to as outcome objectives. The purpose of distinguishing among the three types is to ensure that, during the planning process, connections between the planned interventions and health changes are made explicit. Being explicit at this phase of the planning facilitates subsequent development of the evaluation, particularly with regard to which changes, benefits, or health outcomes should be measured. Because funding agencies generally require objectives dealing with effects, interven- tion objectives can be included with outcome and impact objectives.

The format for writing good objectives can be used to write objectives in terms of increasing or reducing the level of a certain outcome com- pared to some benchmark level. For example, “After attending 90% of the sessions, 100% of

TABLE 6-1 Aspects of Process Objectives as Related to Components of the Process Theory, Showing the TAAPS Elements

Organizational Plan Service Utilization

Plan Process Theory

Outputs

Objective By when (T ), how much (A) of obtaining or organizing which types of resource in what ways (A) will be done by which program staff members (PS).

By when (T ), how many (A) of what types of interactions will be done (A) by which participants/program staff members (PS).

By when (T ), how many (A) of what types of outputs or products will be created or finalized (A) by whom (PS).

Objective examples

By [date] (T ), full funding (A) for three new computers with electronic clinical record software will be secured (A) by the program manager (PS).

By [date] (T ), 4 hours of training about community gang violence (A) will be provided (A) by the medical director (PS).

By [date] (T ), three evidence-based interventions for improving self- management of diabetes (A) will be identified (A) by the health educator (PS).

By [date] (T ), 100 brochures (A) will be distributed to women receiving genetic counseling at the clinic (A) by outreach liaisons employed by the program (PS).

T for Time; A for Amount; A for Activities; PS for Participant/Program Staff.

Program Goals and Objectives 151

specific than the parental report, they can detect a smaller true change.

By choosing—or at least considering— indicators when developing objectives, planners and evaluators can set reasonable target numbers for the extent of or how much change is expected, given the indicator chosen. For full-coverage programs at the population level, it may be more appropriate to think of indicators in terms of benchmarks. For example, the national standard for a healthy birthweight could be used as an indicator, as in the objectives for infants born to women in the Women, Infants, and Children (WIC) nutrition program.

Most health programs address one or more domains of health or well-being. Typically, these domains encompass physical, mental, cognitive, behavioral, knowledge, social, and financial issues. For each of these domains of health, specific variables (indicators) are used to measure the program effect on that domain. TABLE 6-2 lists some commonly used variables for each health or well-being domain. The developers of a health program would need to select those indicators that reflect the specific health domain targeted by that program. Reliance on the effect theory and identification of the antecedent, causal, moderating, or mediating factors of the

as clear guideposts in designing the evaluation of program effect.

Objectives and Indicators One aspect of developing objectives is to consider indicators. Like so many other terms in program planning and evaluation, indicator has many uses and interpretations. It can refer to the “what” portion of the objective, to the variables used to measure that “what,” or to performance benchmarks used to determine the failure or success of a program. There is no easy way to distinguish among these uses or to state prescriptively that one is better than the others. It is important to be aware, however, that the results of the evaluation regarding “extent” can be influenced not only by the program’s true effect but also by the sensitivity of the measure (indicator) selected. For example, if an outcome objective concerns improvement in cognitive functioning of children with special healthcare needs, indicators of cognitive functioning might consist of a score on a standardized scale such as the Bayley Scales of Infant Development or the Denver Developmental Screening Test, or a parental report of cognitive functioning. Because the standardized scales are more sensitive and

FIGURE 6-2 Diagram Showing Relationship of Effect Theory Elements to Process and Outcome Objectives

Organizational plan process objectives

Causal Theory Impact Theory E f f e c t

T h e o r yIntervention

Theory

Moderating factors

Mediating mechanisms

INTERVENTIONS: Service plan

process objectives

Main causal factors of the health problem

Health outcome

objectives

Health impact goals

Eligibilty criteria:

Service plan process

objectives

152 Chapter 6 Program Objectives and Setting Targets

criterion to consider is any indicators that are required or mandated by the funding agency. For example, the Maternal and Child Health Bureau (MCHB) of the Health Resources and

health problem being targeted may also suggest the optimal indicators of program effect.

A variety of criteria can be applied when selecting indicators. The first and foremost

TABLE 6-2 Domains of Individual or Family Health Outcomes with Examples of Corresponding Indicators and Standardized Measures

Outcome Domain Examples of Indicators (Variables) to Measure Objectives

Physical health General measures: cardiovascular fitness, weight, dental diagnosis, medical diagnosis of acute illness, medical diagnosis of chronic illness.

Standardized measures: normal range on laboratory tests, ICD-10, DSM-5.

Mental health General measures: motivation, values, attitudes, emotional bonding, medical diagnosis of mental illness, medical diagnosis of addiction, stress.

Standardized measures: CES-D, Edinburgh Postnatal Depression Scale, Daily Hassles Scale.

Cognitive processes General measures: decision making, judgments, problem-solving ability, cognitive development, cognitive impairment.

Standardized measures: Bayley Scales of Infant Development, IQ tests, Short Mental Status Exam.

Behavior General measures: smoking, exercise, acts of aggression, seat-belt use, food purchasing behavior, specific parenting behavior, risk-taking or risk-seeking behaviors.

Standardized measures: Self-Care Inventory, Child Behavior Checklist, Adolescent Sexual Activity Index, items from the Behavioral Risk Factors Surveillance Survey.

Knowledge General measures: skill, ability, performance, education of others, recall of facts, synthesis of facts.

Standardized measures: Smith Alcohol Knowledge Test, Self-Rated Abilities for Health Practices.

Social health General measures: marital status, social network, recreation activities, volunteerism.

Standardized measures: Norbeck Social Support Questionnaire, Dyadic Adjustment Scale, Life Events Scale.

Resources General measures: income, insurance source, housing situation, employment status, education level.

Standardized measures: Hollingshead Index, dissimilarity index.

Program Goals and Objectives 153

collect. Indicators such as variables also ought to be scientifically defendable—hence the use of standardized or existing questionnaires and tools. Indicators of any type also ought to be relevant to users, such as the program managers and program stakeholders. Indicators (measures) need to be relatively easy to analyze. It is point- less to rely on an indicator (measure) that is so difficult to analyze that it is not used in program management or improvement.

Indicators may also be selected by returning to the community health diagnosis statements developed about health problems. In those state- ments, the health status indicator can be directly applied to the outcome objectives. For example, for each the five health problems identified as being a high priority in Bowe County, indicators or variables are used in the objectives (TABLE 6-3). TABLE 6-4 provides examples of intervention, outcome, and impact objectives for the goal of

Services Administration (HRSA) requires that all grantees of Title V funds use its set of 18 performance measures, 6 outcome measures, and 8 capacity-related objectives. MCHB Title V indicators include measures such as the rate at which children are hospitalized for asthma, the percentage of women with a live birth who have had an adequate number of prenatal visits according to the Adequacy of Prenatal Care Utilization index (Kotelchuck, 1997), and the percentage of live births where the newborns weigh less than 2,500 grams (Maternal and Child Health Bureau of the Health Resources and Services Administration, n.d.). Although some of the MCHB indicators could be consid- ered related to process, grantees must use these indicators as outcomes.

Another criterion for selecting indicators includes the requirement that the data for the indicator, if it is a variable, must be feasible to

TABLE 6-3 Bowe County Health Problems with Indicators, Health Outcomes, and Health Goals

Health Problem Indicator of Health

Problem Health Outcome Health Goal or

Impact

Vaccination Rates of underimmunization, by age group

Vaccine-preventable illness

Decrease vaccine- preventable hospitalizations

Presence of neural tube defect

Rates of neural tube defects and congenital anomalies

Absence of neural tube defects

Reduce current rate of congenital anomalies

Diagnosis of pregnancy during adolescence

Pregnancy rate, by age group

Diagnosis of pregnancy

Reduce child abuse related to unwanted pregnancy

Hospital admissions for gunshot wounds

Rate of admissions for gunshot injuries at local hospitals, number of police reports

Adolescent death rate due to gunshot wounds

Reduce adolescent death rate due to gunshot wounds

Diagnosis of type 2 diabetes

Incidence rate of diabetes, prevalence rate of diabetes

Rates of amputation and vision loss due to diabetes

Reduce morbidity due to chronic illness

154 Chapter 6 Program Objectives and Setting Targets

TABLE 6-4 Effect Objectives Related to the Theory of Causal/Determinant Factors, Theory of the Intervention Mechanisms, and Theory of Outcome to Impact, Using Congenital Anomalies as an Example, Showing the TREW Elements

Effect Objectives Related to the . . .

Theory of Causal/ Determinant Factors

Theory of the Intervention

Theory of Outcome to Impact

Format of objective

After how much intervention (T ), what portion of recipients (R) will experience what extent of change (E) on which causes of the health problem (W )

After how much intervention (T ), what portion of recipients (R) will experience what extent of change (E) on which immediate or mediating factors that lead to the health problem (W )

After how much intervention (T ), what portion of potential recipients (R) will experience what extent of change (E) on which long-term or global health status indicators (W )

Example After attending the 3 sessions (T) on environmental hazards that are teratogenic, 100% of the women (R) will have avoided (E) exposure to environmental hazards (W) during the year before becoming pregnant

After taking prenatal vitamins the year before pregnancy (T ), 100% of women in the program (R) will have normal (E) blood levels of iron and folic acid (W)

After three complete cycles of the program (T), newborns in Bowe County (R) will have no (E) preventable congenital anomalies (W)

T for Timeframe; R for Recipients; E for Extent of change; W for Which type of change.

reducing birth defects, and TABLE 6-5 provides examples of intervention, outcome, and impact objectives for adolescent pregnancy. In each example, the objectives are derived from—and thus correspond to—the intervention, causal, and impact theories.

Good Goals and Objectives Obviously, good goals and objectives are both meaningful and useful (Patton, 1997). The objectives need to be distinctly related to either process or effect. Likewise, both process and effect objectives need to be tailored to the specific health program being planned. Thus, program planners are encouraged to adapt—rather than plagiarize—objectives from similar programs.

Each objective ought to convey only one idea so that each statement can be related to only one measure. This precludes using the word and in a goal or objective; each part ought to be separate goal or objective. Ideally, the objectives will be understandable to any stakeholder who might read them. Most important, objectives facilitate implementation by keeping the processes and activities in mind and facilitate direct evaluation by specifying the desired effect to be found.

Goals and objectives are often written using language that indicates a direction, such as improve or reduce. This use of a direction in the objective can be confusing and misleading, and therefore ought to be avoided. A goal of “improving birth outcomes” encompasses both reducing congenital anomalies and increasing

Program Goals and Objectives 155

birthweight, which may not accurately reflect the breadth of the program. Objectives and goals ought to be written to reflect the final rate or state of health, not the change needed to get there. A way to double-check that the program is directly responsible for the elements addressed in the objective is to ask the question, Without the program, would whatever is stated in the objective occur? Similarly, the program goals and objectives need to be reviewed for alignment with the needs, problems, and assets identified through the community health assessment.

Although this chapter presents the develop- ment of the goals and objectives as being derived from the logic model and the program theory, in actuality, discussions that develop about objec- tives may prompt the program planners and the

program stakeholders to revise the logic model or the program theory. Similarly, the process of selecting indicators for objectives may cause the objectives to be revised. These iterations ought to be viewed as a positive sign that ways to strengthen and streamline the health program are being identified and attempts to do so are occurring during the planning phase rather than after the program has been implemented.

Sooner or later someone will mention the acronym SMART in reference to objectives. SMART stands for five qualities of a good objective: specific, measurable, achievable, real- istic, and time. Specific refers to the “what” that is to be achieved. Measurable refers to the metric or measure being used to determine whether the objective was met. Achievable is a reality check

TABLE 6-5 Effect Objectives Related to the Theory of Causal/Determinant Factors, Theory of the Intervention Mechanisms, and Theory of Outcome to Impact, Using Adolescent Pregnancy as an Example, Showing the TREW Elements

Effect Objectives Related to the . . .

Theory of Causal/ Determinant Factors

Theory of the Intervention

Theory of Outcome to Impact

Format of objective

After how much intervention (T), what portion of recipients (R) will experience what extent of change on which causes (E) of the health problem (W)

After how much intervention (T ), what portion of recipients (R) will experience what extent of change (E) on which immediate or mediating factors that lead to the health problem (W )

After how much intervention (T ), what portion of potential recipients (R) will experience what extent of change (E) on which long-term or global health status indicators (W )

Example After attending two i-APP sessions (T ), 100% of adolescents (R) will be able to describe three ways (E) to avoid pregnancy (W)

After 18 months of the i-APP (T ), among sexually active adolescents (R) in Bowe County, less than 1% (E) will give birth (W )

After 2 years of full program implementation (T ), among residents of Bowe County (R) the incidence of infant abuse (W ) will be 8.0 per 1,000 (E)

T for Timeframe; R for Recipients; E for Extent of change; W for Which type of change.

156 Chapter 6 Program Objectives and Setting Targets

reasonable target values directly influences the extent to which a program is perceived as successful, particularly with regard to outcomes and impacts. Consequently, the measurement of success must be scientifically credible. During the development of the objectives and their corresponding targets, planners and evaluators will want to agree on a strategy that accounts for both the program theory and any extraneous factors, and they will also want stakeholders to become involved during this crucial initial stage.

Developing a rational target-setting strategy is preferred over using mere “guesstimation.” Using a rational approach to setting targets leads to targets that are meaningful from a program- matic perspective, that are achievable to the extent that they represent an outgrowth of the program theory, and that are based on empirical data. The target-setting approaches outlined here begin with the process of establishing guidelines based on a decision framework and then move to choosing one or more relatively simple statistical procedures to yield target values consistent with the decisions. These approaches are best suited for setting tar- get values for effect objectives. The framework described in the next section was developed by Rosenberg (2012) as an outgrowth of an effort by the MCHB to provide states with enhanced skills for program planning and evaluation.

Decisional Framework for Setting Target Values The first and most basic element in developing a target-setting strategy is deciding how a program will be deemed successful. Success can be defined as meeting or exceeding a target, or as making meaningful progress toward the target but not necessarily meeting it. If success is defined as meeting a target, then targets will probably be chosen more cautiously than if success is defined less strictly. If program planners and staff mem- bers wish to claim success even when a target value is not achieved, then “making meaningful progress” must be quantified, in addition to setting the target value itself. For example, the efforts to reduce infant mortality could be called “successful” based on the steady decrease in the

ensuring that the target being set can actually be achieved or attained. Realistic asks whether, given the resources and conditions, it is plausible that the objective will be achieved. Time refers to the time frame for achieving the objective. In developing objectives, it can be helpful to reflect on the SMART qualities to ensure that the objective is good.

▸ Using Data to Set Target Values

All types of objectives, whether related to pro- cess theory or effect theory, have the “by how much” portion for each “what.” A critical step in developing a meaningful objective is choosing a numeric value as the target for the “by how much” portion of the objective. Despite the importance of quantifying the amount, surprisingly little literature exists about providing explanations of how to calculate the amount or about exam- ples of calculating the amount. That gap in the literature reflects either the assumption that the knowledge is intuitive or that the calculation is totally idiosyncratic. As the following demon- strates, neither assumption holds.

For process objectives, the procedure for establishing target values for the “by how much” portion generally means using data from the organizational and marketing assessments. National standards or objectives are not always available for use as a guide. Nevertheless, profes- sional standards can often be used, particularly for organizational plan objectives. For example, legal and professional standards have been established for minimum qualifications for personnel. These standards can be used as a starting point for setting targets—say, for the percentage of program staff members with a given certification.

For effect objectives, the target-setting process relies on the effect theory. The assumption is that as long as the objectives are consistent with the program theory and the level of programmatic effort, targets can be achieved. What is achiev- able, of course, depends on many factors both internal and external to the program. Having

Using Data to Set Target Values 157

to a long-term objective. A different target value might be selected depending on which cell is relevant to the outcome of interest. For example, if a program is being implemented for a target population that has been experiencing worsen- ing conditions over time and that has a current indicator value far from a long-term objective, the target value might be set more cautiously than if the program is being implemented for a target population that has been experiencing gradual improvement and that has a current indicator already fairly close to the long-term target value.

Another component of making decisions about target setting is choosing which types and sources of data will be used. A wide variety of data sources are often available and appropriate for assessing health programs and measuring objectives. Ideally, multiple data sources will be used in setting target values because each source contributes slightly different information to the target-setting process. For example, one source of data might consist of police reports on the use of guns, and another source might be medical diagnoses of gunshot wounds in emer- gency departments. The statistics reported by each of these sources might be similar, but they might also be different. Having access to both data sets would be useful in setting a reasonable target for the rate of intentional gun violence in Layetteville.

Both the number of data sources available and the consistency of the data across these sources influence the target-setting process. TABLE 6-7 shows the intersection of these two dimensions. For example, if many data sources are available and their data are in reasonable agreement, then arriving at a target value is relatively straightfor- ward because it will reflect the consistent values. In such a case, similar target values would be reached no matter which data sources are used. If many data sources are available, however, but the information is inconsistent or conflicting, then decisions must be made regarding which data source should be given precedence or which combination of data sources will be used. These decisions should be based on the strengths and weaknesses of each data source, including

rate of infant mortality over the past decade, despite the fact the infant mortality rate did not reach the target specified in Healthy People 2020. Either definition of program success is acceptable, but the definition to be used in the later program evaluation needs to be agreed upon and, more important, made explicit to all relevant parties during the planning stage.

The way in which program success is defined influences whether targets are chosen primarily according to past or baseline indicator values or whether more emphasis is placed on the values of longer term objectives for the program. This difference in perspective can have a dramatic effect on a final target value. Referencing targets to past or baseline values is typically a more cautious approach because the target values tend to be set to a level that represents a very modest improvement in the health outcome being measured—in other words, a minimum expectation for program effectiveness. In contrast, referencing targets to longer term objectives is a bolder approach, often resulting in target values that are somewhat more difficult to reach but that challenge program managers to examine the program implementation continually and to advocate for changes if necessary. Both approaches are appropriate, and a decision needs to be made about which focus is more important to the particular program under consideration.

Once program success is defined and a con- sensus has been reached regarding the relative importance of past, present, or future indicator values, program planners can begin developing a specific methodology for incorporating indicator values into the target. Sometimes only current data values are used in setting a target; sometimes a combination of current values and trend data is considered; and sometimes current values, trend data, and a local or national standard are all incorporated into the target-setting process. For example, if data have already been collected over time and a well-established objective or national standard specifying a long-term outcome exists, it may be important to set a target based both on the trend in the data and on the distance between the existing standard and the desired outcome.

TABLE 6-6 shows a matrix depicting combi- nations of patterns over time and relationships

158 Chapter 6 Program Objectives and Setting Targets

TABLE 6-6 Matrix of Decision Options Based on Current Indicator Value, Population Trend of the Health Indicator, and Value of Long-Term Objective or Standard

Population Trend of

Indicator

Current Value of Health Indicator in Target Audience

Better Than Long-Term Objective or Standard

Meets Long-Term Objective or

Standard

Worse Than Objective or

Standard

Improving Set target to maintain current level; better than the long- term objective and limits to further improvement

Set target to surpass the long- term objective; continuing the improving trend

Set target to a better level; accelerate improving trend to approach the long- term objective

No change Set target to slightly better level; better than the long- term objective, but want to see improving trend

Set target to surpass the long-term objective; begin improvement in trend

Set target to moderately better level; begin improvement in trend

Deteriorating Set target to maintain current level; stop the worsening trend

Set target to maintain current value; stop the worsening trend

Set target to maintain current level or adjust it slightly downward; stop or slow the worsening trend

TABLE 6-7 Framework for Target Setting: Interaction of Data Source Availability and Consistency of Information

One or Only a Few Sources Many Sources

Consistent Information Across Sources

Need to consider whether the available data are of high quality. Need to consider whether it is relevant to the program and objective target being considered.

Can use any of the data sources.

Inconsistent Information Across Sources

If the one data source is markedly different from the literature, need to either change the objective or verify the data.

Need to decide which data source to use, given the strengths and weaknesses of each data source. Need to consider which data source is most relevant to the program and the objective being considered.

Using Data to Set Target Values 159

different outcomes for different groups within the population. For example, the unstratified rate of pregnancy among 15- to 17-year-olds in a county might appear relatively close to the Healthy People 2020 objective of 36.2 per 1,000. When pregnancy is stratified by ZIP code, however, it may become clear that the rate of adolescent pregnancy in one area is far from the national target value and much different from the rate in another area. Target values may or may not be chosen based on stratified data, but program evaluators should incorporate stratified values into their interpretation of why targets are or are not met. With respect to the program theory, the moderating factors might also be stratified. Continuing with the adolescent preg- nancy example, cognitive development or family income may be variables that can be stratified as a means of understanding which subgroup of adolescents has a disproportionately higher or lower pregnancy rate.

It is not always possible to stratify data in the way that program planners or evaluators may want. Data sources related to the indicator of interest may not include data for the variables to be stratified. For example, data from emergency department records are not likely to include infor- mation on the educational level of the patient, and data from police reports are not likely to include information on the severity of the injury.

Stratification can result in having only a few individuals being grouped within one stratum; this situation poses statistical and interpretation problems. Various approaches to addressing this problem of small numbers exist; one approach is to reduce the number of strata by combining data across strata, whether the strata are years, geographic areas, or sociodemographic character- istics. For example, while it might be desirable to stratify a population or intended audience based on age, broad (rather than narrow) age strata might be defined to ensure adequate numbers in each group. This problem of small numbers is particularly challenging for programs in rural areas or for intended audiences with rare health needs or problems.

its sample size, data completeness, and other aspects of data quality. The goal is to integrate the data and the information in a way that permits arriving at one target value.

The choice of data source also needs to be congruent with or to correspond to the target population or audience. For example, if a target value is being developed for effect objectives for a full-coverage, population-level program with a goal of improving birth outcomes, an appropriate data source would be vital records data. In con- trast, for the Layetteville i-APP program, which is a partial-coverage program geared toward a smaller intended audience with the goal of reducing adolescent pregnancy, appropriate data sources might include medical records and surveys of the women who are program recipients, as well as the county vital records data.

The data sources for setting the target may or may not be the same as the sources of the evaluation data or the data for the community needs assessment. The choice of data source for each of these program planning and eval- uation activities must always correspond to the purpose for which the data will be used. Also, if different sources of data are used, then the program planners and evaluators need to agree that the different sources yield the same information.

Stratification and Object Target Values Another factor to consider in the decision-making process is the extent to which disparities exist across or within the intended populations. During the community health assessment, some sense of the disparities in health may be evident. If the disparities exist by different income levels, race/ ethnicity, or geographic location, the data may need to be stratified by those factors. Stratification is the process by which a specific characteristic of a group is partitioned into levels, called strata, in order to see variations in the subgroups across each strata. When data are not stratified, the values are simply averages that may mask very

160 Chapter 6 Program Objectives and Setting Targets

community health assessment, along with input from stakeholders. Developing the if–then state- ments becomes part of the program planning process and helps create realistic objectives.

Although some target-setting decisions can be applied to all the program objectives, other decisions may vary depending on the objective. For each different health outcome, the objective may rely on differing pools of data sources, and have differing trends over time and differing patterns of disparities. In addition, objectives for population-based and full-coverage programs will require a target-setting strategy that differs from that developed for a direct services-level program that addresses a specific health domain within individuals. Program planners must also recognize that target setting is an iterative process, taking place over the life of a program. Rarely are target values set beyond 1 year, making it necessary to revisit the targets on an annual basis for health programs that are institutionalized or that are planned to last for a longer period of time.

Options for Calculating Target Values Many options for calculating target values are available, each of which may be appropriate in some circumstances but not in others. The fact that target values for any one health problem can be calculated in so many ways underscores the importance of establishing a consensus on the underlying logic reflected in the if–then statements that lead to a particular value. Ten options for calculating target values are described here. The calculations can easily be done using a calculator or a spreadsheet.

FIGURES 6-3, 6-4, and 6-5 show the calculations for a program whose goal is to reduce adolescent pregnancy in Bowe County. For this program, outcome objectives are needed regarding the extent to which that goal is being met.

In planning some health programs, very limited information may be available on which to base the calculations for some target values. This can be the case, for example, for innovative programs,

Use of Logic Statements to Develop Targets One technique for explicitly organizing and documenting the process of setting targets is to use logic statements. These statements can be written as the decision-making process unfolds, which is a way to keep the decisions explicit and the discussions focused. Logic statements are written in an “if, then;” or “otherwise, if, then” format. For example, for integrating different data sources, a logic statement for gun violence might be as follows:

If the emergency department data and the police department data do not agree,

then the [one or other of the data sources] will be given precedence in setting the target value.

To integrate different types of data about gun violence, a logic statement might be something like the following:

If trend data for gun violence show steady improvement, but the current value is still far from a long-term objective,

then the target value will be set to reflect an increase in the rate of improvement;

otherwise, if the trend in gun violence shows steady improvement and the current rate is already close to a long-term objective,

then the target value will be set to reflect a continuation of the existing rate of improvement;

otherwise, if no trend data are available,

then the target value will be set to reflect an improvement in the current value of X percent.

Sets of such statements can be drafted for each indicator about which decisions are being made, such as data sources, data consistency, data types, existing or perceived disparities, and resource availability. These statements should incorporate information obtained during the

Using Data to Set Target Values 161

FI GU

R E

6- 3

C al

cu la

ti o

n s

o f O

p ti

o n

s 1

Th ro

u g

h 4

U si

n g

a S

p re

ad sh

ee t

A 1

L o n g T

e rm

C u rr

e n t

V a lu

e

3 7.

6

3 7.

6

3 7.

6

3 7.

6

– 0 .0

2

– 0 .0

4 3 0

5 16

,5 5

6

16 ,5

5 6

16 ,5

5 6

1 D

e fa

u lt

(n o

n e

) 3

7. 6

3 3

.5

3 6

.8 0

.8 – 2 .0

%

4 .1

– 10

.9 %

0 0 .0

%

U se

s ta

tis tic

a lly

si g n ifi

ca n t

ch a n g e

(a p p ro

x. )

U se

c u rr

e n t

tr e n d

a s

ch a n g e d

e si

re d

U se

m e e tin

g lo

n g -

te rm

o b je

ct iv

e

th e

n , Y

r 1

th e

n , Y

r 2

th e

n , Y

r 3

th e

n , Y

r 4

th e

n , Y

r 5

(B 9

* A

9 )

+ A

9

(B 9

* J

11 )

+ J

11

(B 9

* J

12 )

+ J

12

(B 9

* J

13 )

+ J

13

(B 9

* J

14 )

+ J

14

3 6

.1

3 4

.6

3 3

.2

3 1.

9

3 0

.6

1. 5

1. 5

1. 4

1. 3

1. 3

– 4 .0

%

– 4 .0

%

– 4 .0

%

– 4 .0

%

– 4 .0

%

7. 0

18 .6

% To

ta l i

m p ro

ve m

e n t

(B 7

* A

7 )

+ A

7

A 5 –

(S Q

R T

(2 *(

A 5 *(

10 0 0 – A

3 ))

/E 5 )*

2 )

2 3 4

16 ,5

6 6

% C

h a n g e

Ta rg

e t

V a lu

e N

u m

b e r

o f Y

e a rs

P o p u la

tio n

a t R

is k

O p

ti o

n D

e sc

ri p

tio n

F o rm

u la

Ta rg

e t

V a

lu e

A b

so lu

te C

h a

n g

e P

e rc

e n t

C h a n g e

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

B C

D E

F G

H I

J K

L

162 Chapter 6 Program Objectives and Setting Targets

FI GU

R E

6- 4

C al

cu la

ti o

n s

o f O

p ti

o n

s 5

Th ro

u g

h 8

U si

n g

a S

p re

ad sh

ee t

A 1

C o

u n ty

O P Q R S T

S u

b to

ta l/ R

a te

fo r

A p

p ro

x.

5 0 %

o f

P o

p u

la ti

o n

8 9 74

12 ,6

4 8

16 ,5

5 6

S u

b to

ta l/ R

a te

fo r

A p

p ro

x.

7 5 %

o f

P o

p u

la ti

o n

To ta

l/ O

ve ra

ll R

a teV W X

9 9 4

7 9 3

2 7 8 5

8 5 9

2 2 0 5

13 3 8

2 6 6 4

3 0 2

3 9 0 8

2 3 3

3 7 8

6 2 2

3 29

6 6

2 2

6 4

4 0

10 6 15

2 4 4

2 6 .0

2 9 .9

3 7.

6

3 2 .2

11 .3

2 3 .7

2 5 .6

5 6 7 8

2 9 .0

2 9 .9

U 7 0 8

2 4

3 3 .9

3 9 .8

4 9 .7

6 2 .4

N o

t c

o n

si d

e r-

in

g s

am p

le s

iz e:

C o

n si

d e

ri n

g sa

m p

le s

iz e

:

3 7.

6

3 1.

0

3 3

.8

6 .6

– 17

.4 %

– 10

.2 %

3 .8

A ve

ra g

e (

D 2

:D 11

) M

id p

o in

t b

e tw

e e

n C

o u

n ty

S

a n

d T

:D 6

+ ((

D 7-

D 6

)/ 2

5 0

% =

B 14

x 0

.5 0

= 8

,2 7

8 .

S u

m s

iz e

s u

n til

r e

a ch

8

2 7

8 , O

,P ,Q

,R ,S

,T

(C 8

/B 8

)* 10

0 0

(C 12

/B 12

)* 10

0 0

7 5 %

= B

14 x

0 .7

5 =

1 2 ,4

17 .

S u

m s

iz e

s u

n til

r e

a ch

1 2 ,

4 17

, P,

Q ,R

,S ,T

,U ,V

,W

2 6

.0

2 9

.9

11 .6

7. 7

– 3 0 .9

%

– 2 0 .5

%

3 7.

6

3 7.

6

3 7.

6

3 7.

6

M e a n r

a te

M e

d ia

n o

f th

e r

a te

s

R a te

f o

r "b

e st

" 5

0 %

o f

p o

p u

la tio

n

R a te

f o

r 7

5 %

o f p

o p

u la

tio n

(C o

u n

tie s

P t o

S iz

e o

f P

o p u la

tio n

a t

R is

k

N u m

b e r

o f T

e e n

B ir

th s

Te e n B

ir th

R a te

p e r

10 0 0

O p

ti o

n D

e sc

ri p

tio n

F o rm

u la

C u

rr e

n t

V a

lu e

Ta rg

e t

V a

lu e

A b

so lu

te C

h a

n g e

P e rc

e n t

C h a n g e

2 3 4 5 6 7 8 9 10 11 12 13 14

B C

D E

F G

H I

J K

L

Using Data to Set Target Values 163

FI GU

R E

6- 5

C al

cu la

ti o

n s

o f O

p ti

o n

s 9

an d

1 0

U si

n g

a S

p re

ad sh

ee t

A S tr

a ta

P o ve

rt y

Y e

s

N o

To ta

l

2 5

3 3

14 ,0

2 3

16 ,5

5 6

14 8

4 74

6 2

2

5 8

.4

3 3

.8

9 10

3 7.

6 3

3 .8

– 10

.1 %

3 .8

5 8

.4 5

2 .6

– 10

.0 %

5 .8

S te

p 1

. U s

e O

p ti

o n

2 fo

rm u

la p

e r

s tr

a ta

S te

p 2

. C a

lc u

la te

f in

a l

ta rg

e t,

w e

ig h

ti n

g b

y %

o f

p o

p u

la ti

o n

i n

e a c h

g ro

u p

10 %

d e

cr e

a se

f o

r P

o ve

rt y

= (–

0 .1

* 1

6 )

+ 1

6

3 3

.8 3

3 .1

– 2

.0 %

0 .7

3 7.

6 3

6 .1

– 4

.0 %

1. 5

2 %

d e

cr e

a se

f o

r N

o P

o ve

rt y

= (

– 0

.0 2

* 1

7 )

+ 1

7

(J 6

*( B

3 /B

5 ))

+ (J

7 *(

B 4

/B 5

))

R a te

f o

r b

e st

st ra

ta (n

o n

e )

O ve

ra ll

ra te

b a

se d

o n

s tr

a ta

sp e

ci fic

r a

te s

S iz

e o

f P

o p

u la

tio n

a t

R is

k

N u

m b

e r

o f T

e e

n B

ir th

s

Te e

n B

ir th

R a te

p e r

10 0

0 O

p ti

o n

D e

sc ri

p tio

n F

o rm

u la

C u

rr e

n t

V a

lu e

Ta rg

e t

V a

lu e

A b

so lu

te C

h a

n g

e P

e rc

e n

t C

h a

n g

e 1 2 3 4 5 6 7 8 9 10

B C

D E

F G

H I

J K

L

164 Chapter 6 Program Objectives and Setting Targets

programs addressing rare health problems, and programs that are highly tailored to the location in which they are delivered. In such cases, only one piece of information from the community health assessment may be relevant—namely, a numeric value for the current level of the health problem. Four options for calculating a target value can be used in developing the outcome objective under these conditions. TABLE 6-8 summarizes the conditions under which each of the 10 options would be best and outlines the advantages and disadvantages of each option.

Option 1 assumes that no change in the health problem will occur because of the program. This is equivalent to accepting the current level or value. As a default position, it provides a starting point, particularly if the health program is in its first year and no data are available on how much change can occur. This option may also be appropriate for health programs that are mature and are seeking to maintain the current value because it is already at an acceptable, healthy level. The formula is as follows:

Target value = current value

As an example, the target value for the birth rate per 1,000 female adolescents aged 15 through 17 is stated in the objective as 37.6 per 1,000. This target value is used in the program objective as the “how much” value.

Option 2 identifies a value that, when com- pared to the current value, results in a statistically significant improvement. This option would be appropriate if the data source is credible, the program has a rigorous intervention, or policy makers need to be convinced that the program is a worthwhile investment. Because change may happen by chance and not just because of the health program, planners must be able to argue that the amount of change is greater than would occur by chance alone and hence is attributable to the program. An approximate Z-test can be used to derive the amount of change needed to be statistically significant. Typically, the significance level is set at p < .05, meaning that the probability of reaching that target by chance alone is less than 5 in 100, or 5%. The .05 significance level translates

into a Z score of 1.96, which is used in the formula to estimate the target value. The formula is quite complex but has been simplified somewhat here so that it can be used with a spreadsheet:

Target value = Current value −

2   current value    multiplier   current value× × −( )) ×

 

 population at risk

   1 

This formula assumes that the current value is an integer—that is, it is a percentage or a number per 1,000, 10,000, or whatever the usual units are for reporting the indicator. The multiplier, then, is that unit value. In the adolescent birth rate example, the current value is 37.6 and the multiplier is 1,000. In addition, the formula is written so that the target value will be less than the current value. If improvement in an indicator translates into a target value that is larger than the current value, then the minus (−) sign in the formula will change to a plus (+) sign.

Any test for statistical significance is very sen- sitive to sample size. For a full-coverage program using population data, the number of recipients is typically large, so even a modest improvement in the current value can lead to a statistically significant result. If the number of participants or recipients is small, as is likely in a partial-coverage program using data for only the program recipients, it is likely that an unrealistically large target would be needed to achieve statistical significance.

In the adolescent pregnancy example, the statistical test is based on more than 16,000 adolescents, so a reasonable target value of 33.5 would result in a significant result. In contrast, suppose that this method were to be used in a program serving only 500 adolescents. In this case, a target value of 14.0 would be required to result in statistically significant improvement— clearly an impossible target to meet. Statistical testing, then, should really be used as an aid to understanding what a reasonable target value might be rather than for determining the target value per se.

Option 3 is to select a desired percentage change in the health problem and convert the percentage into a definitive target value. This

Using Data to Set Target Values 165

TABLE 6-8 Summary of When to Use Each Option

Option Description

of Option

Type of Program for

Which the Option Is Ideal

Advantages of Option

Disadvantages of Option

1 Default, no change

Mature, stable program

Does not require historical data

Does not require improvement

2 Change based on results of statistical test

Population- based or program with large numbers of recipients

Supports argument that improvement was more than by chance

Sensitive to sample size; may result in unreasonable target; requires some statistical knowledge

3 Percentage change in health problem based on current trend, literature, or hopeful guess

Stable program; stable target population

Very straightforward and easy to understand; can easily take into account trend data if available

Must know the trend

4 Use existing benchmark or standard to project target values for several years

Program must show improvement

Comparable programs can be compared

Requires existence of long-term objective or standard; requires long-term program

5 Mean rate across geographic areas

Population- based

Easily understood Requires having data for each area

6 Median rates across geographic areas

Population- based

Easily understood Requires having data for each area

7 Overall rate for best 50% across geographic areas

Population- based or multisite

Takes into consideration the best and worst values in the target population; moves entire target population to an achievable value

Requires having data for each area; may be more difficult to understand; overlooks sample size

(contiunues)

166 Chapter 6 Program Objectives and Setting Targets

Option Description

of Option

Type of Program for

Which the Option Is Ideal

Advantages of Option

Disadvantages of Option

8 Overall rate for best 75% across geographic areas

Population- based or multisite

Takes into consideration the best and worst values in the target population; moves entire target population to an achievable value

Requires having data for each area; may be more difficult to understand

9 Rate for best stratum using sociodemographic groupings

Population- based or diverse target audience with evidence of disparities

Takes into consideration the best and worst values in the target population; moves entire target population to an achievable value

Requires having data for each group; may be more difficult to understand

10 Overall rate based on differential targets for each stratum

Population- based or diverse target audience with evidence of disparities

Program must show improvement; more intense program intervention aimed at group with the most need for improvement

Requires having data for each group; may be more difficult to understand

TABLE 6-8 Summary of When to Use Each Option (continued )

option is the most straightforward approach and can be understood intuitively by stakeholders. It can be used with health programs that are situated at any level in the public health pyra- mid and in any health domain. The percentage change used in the formula can be chosen based on information gained from published literature, or it may merely be a hopeful “guesstimate.” The formula is as follows:

Target value = (% change desired × current value) ± current value

If trend data exist for the health outcome, then the percentage decrease (or increase) can be refined based on past and recent experience. The percentage change can be chosen to reflect

either a continuation of the observed trend or a change in the trend (i.e., either an acceleration of improvement or a slowing of deterioration, depending on the health outcome of interest).

In the example of the adolescent birth rate, trend data indicate an average 2% annual decrease. Using this percentage in the formula for option 3, the target value for the birth rate per 1,000 female adolescents is 36.8 per 1,000 (see Figure 6-3). If program planners decide that a 4% decrease is more appropriate—that is, if they assume that the program can accelerate improvement—the target value for the birth rate per 1,000 female adolescents would be 36.1. The target value chosen for this calculation is then used in the program objective as the “how much” value.

Using Data to Set Target Values 167

As shown in Figure 6-3, an annual 4% decrease results in an adolescent birth rate of 30.6 per 1,000 at the end of 5 years, for an overall decrease of 7 births per 1,000. This decline represents an 18.6% decrease in the birth rate among adolescents. In this example, a rate of 30 births per 1,000 was the long-term objective; thus, using the method described here, the final target value was not exactly met because the rate of improvement was maintained at 4% each year. To reach the long- term objective, the rate of improvement would have to increase slightly each year. The question for discussion among the planning team, however, is whether a 4% decrease every year for 5 years is possible for the program, and whether the 18.6% decrease over 5 years will be acceptable to funding agencies and other stakeholders. Program planners must also consider whether the change can be identified using the methods currently chosen for use in the effect evaluation.

Options 5 through 10 are relevant for population-based and multisite programs when the data can be stratified, either by geographic area or by some characteristic such as age, race/ ethnicity, or income. The adolescent pregnancy example is a population-based program, using data from all high schools in Bowe County as well as data on whether the adolescents’ family income is below or above the poverty level. When stratification is used for target setting, the planners often assume that some sites may already have reached a very desirable level and therefore would not be expected to improve dramatically when the program is implemented. A corollary is that some sites will likely be drastically far from any target that might be set, which means they must make radical improvements to reach any reasonable target. The extent to which a site may already be at an ideal level warrants attention from the planning team and ought to be reflected in the logic statements and the subsequent decisions about selecting target values.

Option 5 sets the target value as the mean of the rates across the sites, and option 6 sets the target value as the median of the rates across the sites. Options 5 and 6 are likely to give very similar target values, especially if the rates of

This exercise reveals that, although a 4% decrease in adolescent births may require con- siderable programmatic resources to achieve, the reduction in the rate may be barely noticeable. It may be useful to repeat the calculation with slightly different percentage changes and consider which elements in the organizational plan and service utilization plan would need to be mod- ified to achieve those other percentage changes.

Option 4 is used when programs are on- going, are multiyear projects, or are expected to have long-term effects. For such programs, the time-frame portion of the objective may be several years into the future. In this case, it becomes necessary to have annual target values that cumulatively reach the desired long-term target value. Essentially, the total amount of change is dispersed across the time period for the program. For this reason, the target values for each year will be affected by the anticipated length of the program and the starting or cur- rent value. Option 4 can be used for programs at any level of the public health pyramid, but it is appropriate only for objectives related to a long-term goal. To use option 4, health program planners’ first decision is to select an existing benchmark or standard, such as a Healthy People objective that identifies the desired target value for the health problem that is to be achieved over the long term.

Calculating annual target values requires first estimating the amount of annual change needed to get close to the long-term target. This annual percentage change is then used in calculations like those in option 3 to find the target value for each subsequent year. The following set of formulas is used in sequence to carry out option 4:

Annual % change = [(Long-term objective − current value)/current value]/number of years

Next-year target value = (Annual % desired change × current value) ± current value

Subsequent-year target value = (Annual % desired change × past year value) ± past year value

168 Chapter 6 Program Objectives and Setting Targets

option 8 is based on 75% of the target population. The pared means method can actually be used with any proportion of the target population. The higher the proportion, the easier it will be to reach the target; the lower the proportion, the more dif- ficult achieving that the target will be. Choosing 50% means that half of the target population has already achieved the target but the other half will have to improve; choosing 75% is a more conser- vative approach because improvement will have to occur in only 25% of the population.

Continuing with the example, to calculate the target value according to option 7, program planners would take as many schools as necessary to incorporate 50% of all the adolescents in the 10 schools that have the lowest (best) birth rates. They would then calculate the overall birth rate for this subset of schools. In this example, the calculation must include 6 of the 10 schools and thus include 50% of female adolescents. The calculation is as follows:

Target value =

Number with the health outcome in the top 50%

Number in target population in the top 50%

Using these data gives a target value of 26.0 births per 1,000, which is a 30.9% decrease from the overall, current rate of 37.6. In comparison, if the counties that have 75% of the adolescents are used in the calculation (option 8), then the target value is 29.9, or a 20.5% change.

Options 9 and 10 are examples of approaches to using stratified data (see Figure 6-5). If data are available on the health status or rates of groups within the target population, then it is possible to use those rates to calculate target values for those groups. Option 9 is simply an extension of the pared means method used in options 7 and 8; it uses the “best” rate of the two groups as the overall target. In contrast, option 10 starts with two separate targets, based on the two strata, by choosing different percentage decreases or increases for each. The different percentages chosen may reflect a more intense program- matic effort aimed at the group with the most urgent need for improvement. Thus, distinct,

the health outcome across the sites are normally distributed. Conversely, if there is not a normal distribution, they may not yield similar values. In the adolescent (females aged 15–17 years) birth rate example (see Figure 6-4), the county birth rates range from 11.3 per 1,000 adolescent females to 62.4 per 1,000 adolescent females, with the mean of all high school rates being 33.8 and the median being 31.0. A disadvantage of these two options is that they do not take into account the differing sizes of the target population in each group, such as in each county or in each clinic. If the sites have very different adolescent birth rates and target population sizes, and if those data are not normally distributed, then options 5 and 6 may not be the optimal approaches.

Options 7 and 8 take into account the population sizes in the area targeted by the pro- gram. The overall rate of 37.6 births per 1,000 adolescent females for all schools combined (the value used in options 1 through 4 as the current value) is the mean for the whole population. Nevertheless, because it combines the data for all schools, it obscures the school-by-school information. Options 7 and 8, by contrast, also calculate overall current values; rather than using all schools, however, each option uses only a portion of the population with the “best” outcomes. Options 7 and 8 are based on the idea that the rate achieved by a certain portion of the target audience should be reachable by the whole target audience, so the target value ought to be set based on that existing rate.

Options 7 and 8 use what is called a “pared means method” (Kiefe et al., 1998). This approach reinforces the idea that the target value for a program should aim to move the entire target population to a value already achieved by a portion of the target population. In other words, for options 7 and 8, the target value for the objective would be for the adolescent birth rate in the schools to improve to match the birth rate already achieved by the schools encompassing 50% or 75% of adolescents.

The difference between options 7 and 8 lies in the proportion of the target population that is used to calculate the target value: option 7 is based on 50% of the target population, whereas

Using Data to Set Target Values 169

on information in published evaluations of the intervention. The studies are reviewed during the planning stage and include statistics that indicate how much difference the intervention made for the experimental group. By utilizing the size of the effect found in those evaluations of the intervention, the amount of possible im- provement can be estimated. Similarly, the effect size statistic, which is reported in meta-analyses, can be used to estimate a target value, albeit with the help of a statistician.

In summary, a variety of techniques can be used to calculate the target value to be used in the effect objectives. Each calculation technique results in a different value (TABLE 6-9). In the example cited here, the potential target value for the rate of births to adolescents ranges from a low of 26.0 births per 1,000 using option 7 to a

stratum-specific targets are calculated, but these can then be combined into a single target value for the whole population by calculating an average weighted by the size of the pop- ulation in each group. The formulae for this calculation follow:

Target valueGroup 1 = (% change desired × current value) ± current value

Target valueGroup 2 = (% change desired × current value) ± current value

Overall target = (% of population in group 1 × target valueGroup 1) + (% of population in group 2 × target valueGroup 2)

An evidence-based approach to establish- ing objective targets can be determined based

TABLE 6-9 Range of Target Values Derived from Options 1 Through 10, Based on the Data from Figures 6-3 Through 6-5

Option Description Resulting

Target Value

1 Default, no change, overall rate 37.6

2 Result of statistical testing 33.5

3 Percentage change in health problem: based on trend data 36.8

4 Use existing benchmark or standard to project target values for several years: first-year target

36.1

5 Mean of rates across geography/sites 33.8

6 Median of rates across geography/sites 31.0

7 Overall rate for best 50% 26.0

8 Overall rate for best 75% 29.9

9 Rate for “best” stratum (i.e., adolescents not living in poverty) 33.8

10 Overall rate based on stratum-specific rates 36.0

170 Chapter 6 Program Objectives and Setting Targets

entrenched expectation, particularly in federal and state governmental agencies. The purpose of MBO is to motivate individuals to work toward a common organizational goal, thereby keeping the organization focused and internally well coordinated.

Since its introduction, relatively little research has focused on the effectiveness of MBO. Poister and Streib (1995), in a survey of governmental agencies, found that MBO is widely used but inconsistently implemented. Dinesh and Palmer (1998) commented that MBO is effective but only when it has been implemented as intended. Spe- cifically, they contended that MBO is unsuccessful in practice because it is partially implemented, and the human relations aspect of MBO has been forgotten over the years. As a consequence, MBO is seen as an older fad (Gibson & Tesone, 2001), albeit with some potential relevance for the present. The use of objectives as the basis for decision making, whether for a program or an organization, must be complemented and supplemented with attention to gaining staff acceptance of and support for the objectives, deploying human resources in a manner that supports the activities necessary to achieve the objectives, and using a reward system that makes achieving the objectives meaningful.

Second, being guided by goals and objectives is a logical, linear systems approach to planning a health program. By contrast, the growing body of scientific literature on complexity theory suggests that, in many workgroup settings, greater flexibility and spontaneity lead to more productive work teams. If a health program is being designed and provided in response to community needs and a controlled evaluation of the program is deemed unnecessary, then allowing for self-organizing teams could lead to a “better” program. This approach certainly will not be feasible for health programs that are, say, federally funded. Nonetheless, health program planners and managers may find situations in which less control—in other words less reliance on objectives—leads to a better program.

The list of objectives, once created, is too often forgotten and becomes disconnected from

high of 37.6, which is the current value, using option 1. This range of possible and reasonable target values underscores the importance of having a decisional framework for target setting, including developing explicit logic statements, to define realistically what constitutes the success or effectiveness of a health program.

Which target value is ultimately chosen depends on the program theory, the availability of resources, and the strength of the intervention. Although options 2 and 5 through 10 are best suited to population-based programs, they can be adapted to very large programs at the direct services and enabling services levels of the public health pyramid. To use these options, sufficient data must be available for each site, and enough sites must have reasonable numbers in the groups. Options 3 and 4 are straightforward and can be used for any program.

▸ Caveats to the Goal-Oriented Approach

All aspects of planning, implementing, and evaluating a program are open for critique and reflection. Although goals and objectives may seem noncontroversial, at least three caveats highlight the need to be self-critical during these processes: the tenuous effectiveness of using objectives to guide work, the need for sponta- neity, and the messy interface of objectives and performance measures.

First, the high degree of emphasis placed on having objectives—both process and effect— warrants some reflection on its history and a cursory review of the evidence of the usefulness of objec- tives. The concept of management by objectives (MBO) was developed and popularized in the 1950s by Peter Drucker, a management scholar, based on a theory of goal setting as being motivational for workers. MBO was subsequently adopted by the Nixon administration (Dahlsten, Styhre, & Willander, 2005), and it has since become an

171Discussion Questions and Activities

the coordination of efforts necessary to imple- ment the health program, and on the garnering of adequate and appropriate resources to provide a population-based health program. The effect objectives can have either an outcome or impact focus, and the “who” portion will consist of the community or a specific population. Processes for setting targets for population-based services— particularly those provided to state and metropolitan populations—will draw heavily on national data.

At the infrastructure level, process objectives will dominate. The infrastructure, by virtue of its nature, emphasizes developing and sustaining an organization and obtaining and managing the resources needed to implement a health program. Nonetheless, effect objectives can be written in relation to the infrastructure, most probably about the effectiveness and efficiency of services. For example, Allison, Kiefe, and Weissman (1999) proposed using a pared means benchmark method to arrive at a target value for the best-performing physicians in terms of patient outcomes. Effect objectives may also apply more directly to the infrastructure itself. For example, effect objectives might address impacts from and educational training for staff, or outcomes from employee screening programs.

▸ Discussion Questions and Activities

1. The organizational plan and the service utilization plan may include many ele- ments and processes. What would you use as criteria for developing a set of ob- jectives about the process theory? Would you set targets for process objectives?

2. For effect objectives at each level of the public health pyramid, which sources of data might be commonly used for estab- lishing targets?

3. Imagine that you have been asked to explain to your colleagues in 10 minutes how to set targets for program objectives. Develop an outline of the steps involved.

the ongoing program oversight. This is more likely to be the case for programs that emerge in response to community demands or are provided inconsistently. For the most part, in health care, health outcome objectives are linked to performance measures, which are used for quality improvement. The relationships among objectives, performance measures, and program monitoring are discussed in detail elsewhere in the text.

▸ Across the Pyramid At the direct services level of the public health pyramid, process objectives are likely to focus on how providers interact with program participants and how the program supports those providers in their involvement with the program. Effect objectives for programs at the direct services level will focus on individual client behavior or health status change. Setting targets for direct services programs may involve translating national objectives into local program objectives. Although national targets may or may not be appropriate for local programs, the national targets need to be considered at least as an accepted benchmark or goal.

At the enabling services level, in addition to the foci at the direct services level, process objectives are likely to include a focus on the involvement of community resources in the program, as well as emphasis on interagency collaboration and cooperation. Effect objectives for the enabling services level are likely to address changes in the behavior or health status of families and other aggregates, such as students in a school or resi- dents of a public housing project. Setting targets for enabling services can be more challenging because national or state data regarding the problem being addressed likely will not exist. For enabling services, past experience, experiences of similar programs, and data from the community needs assessment may be the only data available for health program planners to use with a rational approach to setting the target numbers.

At the population services level, process objectives will need to include an emphasis on

172 Chapter 6 Program Objectives and Setting Targets

References Allison, J., Kiefe, C. I., & Weissman, N. W. (1999). Can

data-driven benchmarks be used to set the goals of Healthy People 2010? American Journal of Public Health, 89(1), 61–65.

Dahlsten, F., Styhre, A., & Willander, M. (2005). The unin- tended consequences of management by objectives: The volume growth target at Volvo Cars. Leadership and Organization Development Journal, 26, 529–541.

Dinesh, D., & Palmer, E. (1998). Management by objectives and the balanced scorecard: Will Rome fall again? Management Decision, 36, 363–369.

Friedman, V. J., Rothman, J., & Withers, B. (2006). The power of why: Engaging the goal paradox in program evaluation. American Journal of Evaluation, 27, 201–218.

Gibson, J. W., & Tesone, D. V. (2001). Management fads: Emergence, evolution, and implications for managers. Academy of Management Executive, 15, 122–133.

Kiefe, C. I., Weissman, N. W., Allison, J., Farmer, R., Weaver, M., & Williams, O. D. (1998). Identifying achievable benchmarks of care: Concepts and methodology. International Journal for Quality in Health Care, 10(5), 443–447.

Kotelchuck, M. (1997). Adequacy of prenatal care utilization. Epidemiology, 8, 602–604.

Maternal and Child Health Bureau of the Health Resources and Services Administration. (n.d.). National Outcome Measures. Retrieved from, https://mchb.tvisdata.hrsa .gov/PrioritiesAndMeasures/NationalOutcomeMeasures

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.

Poister, T. H., & Streib, G. (1995). MBO in municipal gov- ernment: Variations on a traditional management tool. Public Administrative Review, 55, 48–57.

Rosenberg, D. (2012). Performance and outcome measure- ment: Strategies for target setting. Retrieved from www .amchp.org/Calendar/Conferences/amchp-conference /Handouts2012/Documents/B01%20-%20Performance%20 and%20Outcome%20Measurement%20-%20Strategies%20 for%20Target%20Setting.pdf

4. Which of the 10 options for setting targets would be best suited to developing tar- gets for the other four health problems in Layetteville and Bowe County? Provide some rationale for your choice. Try using your choice of target setting based on the following data: the adult immunization rate is 30%, with 25 hospitalizations per 1,000 adults for influenza-related pneu- monia; for children younger than 1 year old, the incidence of maltreatment is 9.4 per 1,000 children (University of Cali- fornia at Berkeley Center for Social Ser- vices Research, http://cssr.berkeley.edu /ucb_childwelfare/RefRates.aspx); 1.2 ado- lescents are discharged with a diagnosis of gunshot wounds for every 1,000 hos- pital admissions; the neural tube defects rate is 18 per 100,000 (National Center for Health Statistics); and new diagnosis of diabetes is 7.5 per 1,000 (Centers for Disease Control and Prevention).

▸ Internet Resources Taxonomy of Educational Objectives Bloom’s Taxonomy (Bloom, B., et  al., [1956]. Taxonomy of educational objectives: Handbook I: Cognitive domain. New York, NY: Longman) is a classic and worth knowing if the intervention relates to knowledge: www.personal.psu.edu /bxb11/Objectives/ActionVerbsforObjectives.pdf

Rapid BI This business-oriented page on writing SMART objectives has some useful tips: http://rapidbi .com/created/WriteSMARTobjectives/

SECTION IV

Implementing and Monitoring the Health Program

© Lynne Nicholson/Shutterstock

175

Process Theory for Program Implementation

Implementation of a health program is done ideally in a manner consistent with the pro-gram theory and the purpose of the health program. Such implementation requires both acquiring adequate resources and then ensuring that those resources are used appropriately. The amount of effort needed at each stage of the planning and evaluation cycle varies throughout that cycle. Implementation of the health program requires the most and longest sustained effort of all the phases of a health program (FIGURE 7-1).

This chapter introduces the logistics associated with managing a health program, with special attention paid to general managerial issues. These logistics fall within the organizational plan and the services utilization plan portions of the process theory (FIGURE 7-2). The operational logistics of implementing a program tend to be straightforward but do require attention to ensure proper implementation of the health program.

Health programs are projects that can be viewed as miniature organizations. In the management literature, the organizational plan and the services utilization plan would be considered elements of the tactical plan. A common frame of reference for thinking about organizations and health programs is to consider inputs, throughputs, outputs, and

outcomes (Turnock, 2009). In the process theory, various inputs into both the organizational plan and the services utilization plan are specified. In addition, specific outputs of both the organizational plan and the services utilization plan are expected. Distinguishing between the inputs and outputs of these plans aids in acquiring the appropriate resources and in being able to communicate both programmatic needs and successes. The chapter concludes by relating the various process theory elements to the logic model format.

▸ Organizational Plan Inputs

The organizational plan encompasses the program inputs and resources, as well as the way in which those resources are organized. The type and amount of resources required for a health program vary with the interventions to be used. Nonetheless, the expertise of personnel, the characteristics of the intended audience, and the degree of attention to acquiring and managing resources all affect the potential for having a successful program. The organizational plan objectives serve as a

CHAPTER 7

© Lynne Nicholson/Shutterstock

176 Chapter 7 Process Theory for Program Implementation

FIGURE 7-2 Diagram of the Process Theory Elements Showing the Components of the Organizational Plan and Services Utilization Plan

Services utilization plan

inputs

Information system

Budget

Time line

Operations manual

Organizational chart

Coverage

Units of service

Service completion

Materials produced

Work flow

Participants

Queuing

Invervention delivery

Social marketing

Human resources

Informational resources

Monetary resources

Physical resources

Transportation

Managerial resources

Time resources

Organizational plan inputs

Outputs Outputs

FIGURE 7-1 Amount of Effort Across the Life of a Health Program

Amount

of

Effort

Needs Assessment

Program Development

Program Implementation

Process Evaluation

Outcome Evaluation

Termination; Impact

Evaluation

Organizational Plan Inputs 177

guide for the most critical organizational plan activities in implementing a health program. Many aspects of the organizational plan will not be in the organizational plan objectives, yet they still need to be addressed. This section presents an overview of the key inputs and outlines the rationale for considering them as key.

Human Resources Human resources encompass the quantity and quality of personnel needed to carry out the program, in terms of their expertise, ex- perience, and capabilities. Human resources come at a cost, of course, and personnel costs are almost always the largest portion of any

program budget. The dollar cost of personnel includes not only wages (amount paid hourly) or salaries (amount paid monthly), but also fringe benefits as a percentage of the wage or salary. Estimating the dollar cost of personnel is a rather straightforward arithmetic problem, as explained in Chapter 8 in the discussion about budgeting.

Licensure and Statutory Regulation of Health Professionals TABLE 7-1 indicates whether individuals from each of the health disciplines are subject to legal and

TABLE 7-1 List of Health Professionals with a Summary of Typical Legal and Regulatory Considerations

Health Discipline State Licensure

Required

State Regulation of Scope of

Practice Professional

Certification Exists

Dentistry Yes Yes Yes

Community health No No Certificate programs

Diet and nutrition Yes, as registered dietician Yes Yes

Health administration

No (except for long-term care administration)

No Yes

Health education No No Yes

Industrial hygiene Yes Yes Yes

Medicine Yes Yes Yes, for sub-specialties

Nursing Yes Yes Yes, for sub-specialities

Physical therapy Yes Yes Yes

Social work Varies Yes, for those with licensure

Yes

178 Chapter 7 Process Theory for Program Implementation

complex intervention to an intended and recipient audience that is resistant or difficult to change.

A separate consideration with regard to human resources is workforce cultural diversity. As health programs are increasingly intended for audiences with specific cultural and racial charac- teristics, it becomes important that program staff members are not only culturally competent with regard to the intended audience but—ideally—are members of the intended audience. While having program personnel who are culturally diverse may help in the delivery of the intervention, it can sometimes lead to other problematic issues among staff members. Program managers must be attentive to signs of interracial or cross-cultural tensions among program personnel and address these issues as soon as they surface.

Volunteers as Human Resources Community involvement in the development and implementation of health programs can occur through the establishment of advisory committees and boards, councils, or consortia. Through such groups, community members have a formal venue through which to participate in the program delivery and are best considered as being volunteers and a human resources input.

Volunteers are widely used to deliver health program interventions. For example, volunteers have delivered a physical activity program for women in Iran (Pazoki, Nabipour, Seyednezami, & Imami, 2007), served as community health workers in Nepal (Glenton et  al., 2010), and provided cardiac health peer education in On- tario (Karwalajtys et al., 2009).

The use of volunteers in health programs makes them a key human resource requiring particular attention, especially if they create a synergy between themselves and the program recipients (Hintgen, Radichel, Clark, Sather, & Johnson, 2000). In other words, volunteers gain from the experience even as participants benefit from their service in terms of personal attention (Chapin et al., 2013). Evidence suggests that volunteers may experience improved health outcomes themselves (Hong & Morrow-Howell,

state statutory regulation with regard to their practice. The legal parameters of practice for each of the licensed health disciplines ought to influence who is hired into which position. Some familiarity with the state regulations of scope of practice is important for matching the program needs with the qualifications of program per- sonnel. In addition, for some health disciplines, professional associations oversee certification of individuals. For some health programs, it may be wise or important to ensure that the program services are provided by individuals with specialty certification, which serves as an indicator of advanced or specialized knowledge and skills.

Training for Program Delivery Training staff members and volunteers for their roles in the program implementation is a key aspect of human resources management. Training costs include both staff time for this activity and the trainer’s time. In addition, costs are incurred for the materials used during train- ing, such as handouts, equipment, or audiovi- sual materials. These costs may mount quickly, but such training is nevertheless necessary for ensuring the standardized delivery of program interventions. Without this standardization of the intervention through training, the actual intervention delivered can deviate from what was intended by the program, depending on the personal preferences of the individuals provid- ing the intervention. In other words, training helps align the theory-in-use and the espoused theory with the effect theory. This has serious implications for achieving the desired outcomes and subsequently for ensuring the long-term sustainability of the program.

Training also helps maintain the morale and self-efficacy of the staff members. Receiving training contributes to program staff members feeling valued, trusted, and capable of providing the interventions. These feelings influence the sense of having a higher ability to carry out the intervention as designed—in other words, they promote greater task self-efficacy. This is an es- pecially important ingredient in the delivery of a

Organizational Plan Inputs 179

for volunteers than for employees (Ferrari, Luhrs, & Lyman, 2007). And volunteers for programs involving disasters and violence may experience emotional distress and exhaustion (Clukey, 2010; Thornton & Novak, 2010).

One issue germane to community health programs and the use of volunteers was identified in the study by Trettin and Musham (2000), which found a difference between program developers and community volunteers in their perceptions of the role of volunteers. This finding hints at an undercurrent in the relationship between pro- gram planners and the community—namely, the perception that individuals are being exploited under the guise of being volunteers (Maes, 2012). This undercurrent will be more evident and influential in determining the success of a health program in some communities.

Physical Resources Physical resources include material resources, facilities, supplies, and equipment (Kettner, Moroney, & Martin, 1999).

Material resources are those tangible items that are needed to provide the intervention and to provide program support. Intervention equipment might include, for example, blood pressure cuffs, syringes, or imaging machines. Space—usually called facilities—is another ma- terial resource needed for both the intervention and the program support. The costs associated with having or renting a classroom, auditorium, community meeting room, waiting room, exam- ination room, and offices need to be taken into account. Supplies constitute another material cost. They encompass miscellaneous office supplies (e.g., paper, stationery, pens, clips, computer supplies), specific resources for program staff members (e.g., journal subscriptions, resource or reference manuals), and any items related to the intervention (e.g., tote bags, clipboards, stopwatches, Band-Aids, flip charts). Physical resources may be needed for the evaluation as well—generally office supplies and storage space for data collected. Maintaining adequate supplies, without hoarding, is the function of

2010). This could be used as an incentive to recruit volunteers.

Volunteers are motivated by factors that are different from those that motivate paid employees. Altruism is a major motivation to be a volunteer (Bowen, Kuniyuki, Shattuck, Nixon, & Sponzo, 2000). Other motivators include personal gains in terms of feeling a sense of belonging; assistance and allowances from the project, and pathways to paid work (Topp et al., 2015); and translating health knowledge into action (Singh, Cumming, Mohajer, & Negin, 2016). Older adults are likely to perceive the opportunity to engage in a new endeavour as the strongest reason to volunteer, although a sense of life satisfaction, a need to be productive, and social interaction are also important reasons to volunteer (Morrow-Howell, Kinnevy, & Mann, 1999).

Recruitment of volunteers can be accom- plished through a variety of techniques. In a study of volunteers for one cancer prevention program, Bowen and associates (2000) found that half of the volunteers were recruited through media, such as television, radio, or newspapers, and from work-related sources. Although these methods are similar to those used to recruit em- ployees, the messages for recruiting volunteers will be different. Specifically, these messages will focus on motivating factors such as those mentioned previously.

Volunteers bring variable relevant back- grounds in trainings and experiences with the focal population. This makes training volunteers particularly critical to the success of the health program, including clear written guidance and ongoing support tailored to the volunteer’s needs (Allen et  al., 2016). Volunteers may need not only the skills to implement the intervention effectively but also emotional support in dealing with challenging clients and knowledge about the clients and the health condition. Volunteers in programs for high-risk clients may be particularly vulnerable themselves, including being subject to police harassment (Bluthenthal, Heinzerling, Anderson, Flynn, & Kral, 2008). Another human resources issue arises from the fact that the source of satisfaction and work-related stress is different

180 Chapter 7 Process Theory for Program Implementation

Navas-López, & Cruz-González, 2013; Rossi, Cricelli, Grimaldi, & Greco, 2016).

Information as a resource can be present in the form of professional networks (Gulati, 1999), street smarts that affect program implementation, and professional knowledge and experience. Staff members bring their professional knowledge to the program, and they gain additional knowledge through the training sessions. In terms of health programs, this means the knowledge held by employees involved in the program is valuable and deserves management attention.

As staff members become more qualified for other positions based on their knowledge and experience, it can become difficult to retain those personnel. Consequently, retention of such employees is a key issue for health programs and represents another reason to make efforts to keep the program personnel satisfied with their jobs and the organization. Loss of staff is not just an issue of replacing personnel; rather, it is more appropriately thought of as replacing knowledge and expertise—hence the importance of maximizing staff retention and minimizing unplanned staff turnover.

Time Time generally translates into personnel costs, but it must also be thought of as a separate re- source. Time is relevant to the overall time line for program design and implementation. The sequencing of events, especially in the start-up phase of a program, is very time dependent. Any delay in accomplishing one step could easily affect the overall time line. If meeting a start date is es- sential, then additional personnel resources may be required during the start-up phase, which in turn will affect the budget. Time also affects the budget in less obvious ways through depreciation, inflation, and interest. These points are discussed later in this chapter as part of planning via budgets.

Managerial Resources The qualities and characteristics of the mana- gerial personnel are also key resources. When

inventory control; having too many unused supplies drains monetary resources that may be needed for other expenses.

A large category of physical resources is office equipment: computers, printers, cables, fax machines, and photocopiers. If the costs of these items exceed an amount set by the organization, the purchase of the item will be considered a capital expense and will require special approval and possibly special ordering procedures. Also, some funding agencies look unfavorably on the purchase of standard office equipment exceed- ing a certain amount. For these reasons, many program managers minimize purchases that are considered capital expenditures.

Transportation Transportation is a separate type of resource and expense. Transportation must be thought of from the perspective of program staff members and program participants. For staff, the issue is one of reimbursement for any travel related to providing the program. For example, if the health program includes an outreach component, then staff members need to document carefully their mileage or keep track of expenditures for public transportation. From the perspective of program participants, the issue is accessibility of the program site given the usual mode of transportation used by potential program participants. Thus, in a rural program, the transportation issue might be one of travel time required to get to the program site, whereas in an urban program, the issue might be proximity to a major mass-transit stop.

Informational Resources Computer hardware and software costs are always included in the budget, usually as physical re- sources. By contrast, information is a key intangible resource. The knowledge and expertise of the staff members must be viewed as a resource to be managed. Information possessed by individuals is considered an asset for the organization, and managerial attention is devoted to managing knowledge as resource (Castro, Delgado-Verde,

Organizational Plan Inputs 181

alternative to the negotiated agreement, analysis of all the interests of each party, and the relative importance of each party’s interests. Armed with this information, the negotiator enters into a process in which strategies are used to arrive at an agreement by breaking out of the existing definitions of differences. These strategies include many actions commonly encountered in good communication—specifically, building good relationships and starting with defined goals (Halpert, Stuhlmacher, Crenshaw, Litcher, & Bortel, 2010). Program managers will negotiate not only with funding agencies and with program staff members and community stakeholders. Thus, a rudimentary understanding of negoti- ation techniques is important.

Team building is another essential managerial skill, especially for participatory, community-based, or multidisciplinary health programs. Team building includes mobilization of a group, fostering group cohesion, and facil- itation of teamwork. Information about team building and teamwork can be found in both the lay managerial literature available at most bookstores and in professional management journals. Managers must consciously attend to the quality of interactions within teams and the quantity of teamwork. Problems can arise quickly, of course, and they must be addressed expeditiously. Otherwise, the team can become dysfunctional and drain resources rather than being a resource itself.

An indicator that team building has been successful is the use of stories. As program staff members begin to develop as a team, stories will be told about clients, managers, and events. These stories, which are ubiquitous in most organizations, serve the function of building team norms and establishing communication among the team members.

Leadership is the ability to inspire and motivate others into action that is purposeful and organized. Program managers are rarely thought of as leaders, yet they must motivate staff members, serve as role models of produc- tivity, and generate enthusiasm for the health program. As a consequence, program managers

selecting the project manager, Posner (1995) suggests considering the skills required of a person in the position—namely, organizational abilities, communication skills, team-building skills, leadership qualities, coping skills related to managing complex and ambiguous tasks and environments, and technical skills related to the health program intervention.

Organizational abilities include the logistics of juggling multiple tasks and persons as well as the ability to keep track of important dates and information. The ideal program manager will be able to manage the project in terms of structuring the relationships among project personnel as well, as is usually reflected in organizational charts that identify the flow of communication, delegation, and responsibilities. A somewhat different but equally crucial aspect of organizational abilities is the understanding of how the health program fits within the par- ent organization, and the knowledge of how to position the program for success within the parent organization.

Communication skills encompass not only the ability to communicate verbally with peers but also the ability to guide discussions; communicate nonverbal messages that are con- sistent with verbal messages; and, perhaps most important, listen and appreciate various points of view. In addition, written communication skills are essential for a health program manager to function in the larger context of grant writing, report generation, and e-mail communication. Another facet of communication skills is the ability to understand who needs what and when as prerequisites to the creation and manipulation of information to have an effective and efficient health program.

Negotiation is a major component of com- munication skills required of program managers. During this process, two or more parties reach a shared decision despite having different preferences with regard to the options. Much research about negotiation has revealed what we know about how people think and respond to information. Three pieces of information are needed in any negotiation: each party’s best

182 Chapter 7 Process Theory for Program Implementation

▸ Organizational Plan Outputs

Not all inputs into the organizational plan will be directly linked to specific organizational plan outputs; some will be linked to services utiliza- tion plan outputs. The outputs discussed in the following subsections of the chapter are examples that could be measured or at least documented as effective use of the organizational plan inputs.

Time Line A time line is a way to represent graphically the dates, time span, and sequence of events involved in planning, initiating, sustaining, and evaluating the health program. Time lines vary from sim- ple to highly intricate. At a minimum, the time line ought to reflect the necessary activities, be related to the expenses detailed in the budget, and be easily understood by those who will use it (EXHIBIT 7-1). A time line can be created by using the table function in a word processing program, a spreadsheet, or specific project management software. This communication tool conveys deadlines, helps keep activities coordinated and sequenced, communicates accountability for assigned tasks, and helps program managers estimate personnel and material costs.

Operations Manual The operations manual contains the policies, procedures, guidelines, and protocols related to the health program. It also includes job descrip- tions and workplace policies and procedures.

One element of many health program opera- tions manuals is a section on safety. For both staff members and participants, safety is an issue—safety not only in their cars as they are coming to or from the program site but also on the street surrounding the program site. In particular, if program staff members are outreach workers, their safety must be considered. In such a case, safety strategies, including monitoring, become an important management responsibility (Reese, 2015).

need to be knowledgeable about the motivational process in regard to both staff members and program participants. Motivation of program staff members results in all members doing what they were hired to do in a timely, efficient, and high-quality manner.

The technical skills required of program managers consist of skills related to the health program intervention. In most health programs, the manager must have basic scientific or practice knowledge about the health problem and the type of interventions that are provided. In some situations, program staff members, program participants, or funding agencies may not view the health program favorably if the program manager does not have some professional credibility. Technical skills are also necessary to supervise the program staff adequately and ensure the integrity of the program intervention.

Fiscal Resources Fiscal resources encompass money, investments, and income. Existing fiscal resources are gen- erally listed on the income side of a balance sheet or budget because they comprise funding and income generated through fees and billing. Monetary donations are another resource that need to tracked and included in accounting records. However, if participants are given any cash for participation, then monetary resources are placed on the expense side of the budget.

Budgets are mechanisms for planning and tools for communicating and refining priorities. As projections of dollar amounts, budgets enable the program planner to assess the fiscal feasibil- ity of doing a project. Developing a budget for a program highlights which programmatic changes may be needed for the health program to be fiscally responsible and efficient. For these reasons, budgets are an input to the health program organizational plan. Each organization will have its own particular format for developing budgets and may use special software for this task. The financial officer in the organization will set forth the rules and accounting specifics used across programs and departments within the organization.

183Organizational Plan Outputs

EXHIBIT 7-1 Example of an Abbreviated Time Line for a Short-Term Health Program

Activity

Month

1 2 3 4 5 6 7 8 9 10 11 12

Convene program planning group

Conduct community needs and asset assessment

Translate assessment information into health problem statement and objectives

Formulate program theory; articulate process theory

Initiate institutional process regarding human subjects’ protection

Advertise for program personnel

Hire and train program personnel

(contiunues)

184 Chapter 7 Process Theory for Program Implementation

council that serves as an advisory committee to the health program. Inclusion of such groups is important because it reflects, in a legitimate and visible fashion, the involvement of the community in the health program.

Also, the health program ought to be identified specifically within the appropriate organizational chart. For health programs that are population-based, such as state child health insurance, the program ought to appear some- where in the organizational chart of the state agency. The program also ought to be included

Organizational Chart Most organizations construct graphic represen- tations to depict the relationships among work units, departments, and individuals. As a result of having personnel designated for the health program and having program accountability, health programs have their own organizational chart. One aspect of the health program orga- nizational chart that may be different from the organizational charts of other work units is the inclusion of a community-based consortium or

Activity

Month

1 2 3 4 5 6 7 8 9 10 11 12

Advertise the health program (social marketing)

Deliver the health program

Conduct process evaluation

Conduct outcome evaluation

Analyze evaluation data

Produce reports

Disseminate findings

EXHIBIT 7-1 Example of an Abbreviated Time Line for a Short-Term Health Program (continued )

Inputs to Service Utilization Plan 185

considerations focus not only on the charge for the service but also on the secondary costs, such as transportation or loss of peer group status. Place refers to where the product is available, whether it is in a clinic, on a billboard, or in a convenient location within a store. Product accessibility, convenience, and visibility are qualities of place that merit attention. Promotion is the more visible, publicity type of activities, including paid media, public service media, and word-of-mouth sharing by opinion leaders. Social marketing goes beyond these classic four P’s to include other P’s, such as partnership and policy.

Social marketing can be combined with health communications, which is sometimes called health marketing (Centers for Disease Control and Prevention [CDC], 2011). Principles from social marketing have been used successfully in a variety of health educational programs to deliver the health program to the intended audience (French, 2010). In this way, the development and implementation of the social marketing strategy serve as an input to the service utilization plan because that strategy enables the health program to reach the intended audience.

Eligibility Screening One of the first decisions facing program planners is to define for whom the program is designed. Although general agreement may exist about who is the focus of the program based on the needs assessment and the logic model, further specificity is required. The intended population is the entire population in need of the program, whereas the intended audience is the segment of the population for whom the program is specifically envisioned and planned. The term recipient refers to those individuals who actually receive or participate in the program.

For programs at the population level of the public health pyramid, the intended population is also the intended audience and—ideally—the recipients. At the direct services and enabling services levels of the pyramid, however, no pro- gram can hope to accommodate all those persons in need; thus, the program may be designed for

in the organizational charts of the community organizations involved in promoting or providing the program.

Information System The information system has outputs that are part of the organizational plan. The programs and templates created for the program are outputs, as well as the analyzed data. New report capac- ities constitute information system outputs. To the extent possible, and without losing sight of program objectives, evaluators should try to use existing data. Whatever the data source, it is wise to analyse a small batch of pilot data as early as possible, ideally before the program begins. This can enable program staff members to improve data quality or enable evaluators to adjust evaluation plans based on what is actually available.

▸ Inputs to Service Utilization Plan

Social Marketing Kotler and Zaltman (1971) were the first au- thors to advocate social marketing as a method to reach a wide audience with health or social messages. Subsequently, Walsh, Rudd, Moeykens, and Moloney (1993) defined social marketing as the design, implementation, and control of the program calculated to influence the acceptability of social ideas.

Social marketing adapts the four P’s of classical marketing: product, price, place, and promotion (Cheng, Kotler, & Lee, 2009). Product refers to the services, tangibles, or ideas that are delivered with the intent of being beneficial to the intended audience. Social marketing focuses on the beneficial aspect and understanding from the perspective of the recipient how those benefits are valued and perceived. Price in social marketing is the cost, of any type, that poses a barrier to accessing or using the product. Price

186 Chapter 7 Process Theory for Program Implementation

In terms of program expenditures, overin- clusion can result in a shortage of funds, whereas underinclusion can result in unspent funds that may need to be returned to the funding agency. Underinclusion and overinclusion are also un- desirable from the perspective of the program evaluation. Providing services to individuals who do not need the program (overinclusion) can decrease the measurable effect of the program on participant outcomes. That is, the extent of change experienced by those individuals who do not need the program is likely to be less than the extent of change experienced by those individuals who do need the program. This diffusion of the program’s effects translates into a decrease in the average amount or degree of change found when all participants in the pro- gram are considered. In addition, overinclusion may artificially inflate the normative need for the program. If current enrollment or requests for participation in the program are used for future planning of the program, overinclusion would falsely increase the apparent number in the intended audience. Overinclusion also can lead to a decreased availability of funds to in- clude true intended individuals in the program. This result is particularly likely if members of the true intended audience are more likely not to be the first persons to enroll in the program.

Underinclusion can also affect evaluation results, particularly for programs designed for and delivered at the population level. Having too few members of the intended audience in the program: (1) could make it difficult to find significant effects due to small numbers, (2) could increase the amount of program services received by individual participants and thereby falsely inflate the program effects, and (3) definitely increases the cost per participant. At any level of the public health pyramid, underinclusion can lead to biased evaluation results if members of the intended audience who do and do not participate in the program differ from one another in ways that are related to the program’s effectiveness.

Several steps can be taken to help minimize overinclusion or underinclusion. The first step comes in developing the process theory in terms

a subpopulation. For such programs, intended audience is a better term.

The distinction between the intended population or audience and the recipients is critical in terms of both budgetary issues and program implementation and evaluation issues. The program can have an impact only on the recipients, so the evaluation will focus primarily on this group. Nevertheless, planners also need to quantify the broader intended audience to estimate underinclusion and overinclusion in the program and describe how these variations may influence the evaluation.

Reach: Underinclusion and Overinclusion Ideally, only members of the intended audience would receive the program. In reality, this ideal can be difficult to achieve, resulting in overin- clusion or underinclusion of individuals in the program.

Overinclusion occurs when some participants in the program are not part of the intended audience. It can be minimized by developing procedures to exclude correctly individuals who are not members of the intended audience. For example, in a dental sealant program, children who are younger than 2 years of age, as well as children who are between the ages of 5 and 14 and who already had sealant treatment, might be excluded to avoid overinclusion.

Underinclusion occurs when some members of the intended audience do not receive the program. It can be minimized by developing procedures to include correctly members of the intended audience for whom the program is designed. Underinclusion in the dental sealant program would be having fewer children of the appropriate age receive the dental sealant than the number of children of that age who need the sealant and who are within the catchment area of the dental clinic. Underinclusion can occur if the program is not well publicized, if some characteristic of the program is unappealing to the intended audience, or if a barrier prevents members of the intended audience from accessing the program.

Inputs to Service Utilization Plan 187

Scope: Full and Partial Coverage The distinction between intended population, intended audience, and recipient is also critical in determining whether the program has partial or full coverage (Rossi & Freeman, 1993). This distinction between partial- and full-coverage programs has implications for public health and health policy.

Partial-coverage programs are designed to serve some portion of the intended population, and participation in the program is based on a set of criteria that focuses recruitment strate- gies and takes into account limited resources. During the planning stage, the decision to have a program that provides only partial coverage generally stems from having limited capacity to serve all those in the target population. Partial-coverage programs are likely to occur at the direct care or enabling services levels of the public health pyramid. Examples of such programs include early childhood intervention programs for children at developmental risk, and hospice care for those dying who choose that service.

Full-coverage programs are delivered, or are intended to be delivered, to the entire target population. By definition, these programs are more likely to occur at the population services level of the public health pyramid. Examples of full-coverage programs include seat-belt laws and water fluoridation.

of specifying how those in need of the program get into the program; this is part of the services utilization plan. Another step is to have a solid, thoughtful marketing plan, which is another element in the services utilization plan.

Once the intended population or audience has been clearly specified, screening tests that are both highly sensitive and specific can be used to minimize both overinclusion and underinclusion (TABLE 7-2). Test sensitivity refers to the proba- bility that the screening test is positive when an illness, need, or existing risk factor is actually present. Using a highly sensitive screening test to identify individuals who are eligible for the program increases the likelihood that more individuals who actually need the program will be in it, thereby reducing underinclusion. Test specificity refers to the probability that the test is negative when there is no illness, need, or risk factor. Using a highly specific screening test to identify individuals who are not eligible for the program results in fewer individuals in the program who do not need it, which reduces overinclusion.

In practice, it is never possible to have a screening test that is both 100% sensitive and 100% specific. Typically, a trade-off must be made between sensitivity and specificity, with the screening test being either more sensitive or more specific. Nonetheless, a screening mechanism is often the best way to minimize overinclusion and underinclusion.

TABLE 7-2 Relationship of Test Sensitivity and Specificity to Overinclusion and Underinclusion

Sensitivity

Specificity

High Low

High Ideal inclusion and coverage; minimal overinclusion and underinclusion

Overinclusion

Low Underinclusion Overinclusion and underinclusion

188 Chapter 7 Process Theory for Program Implementation

regarding whether a program ought to provide full or partial coverage. Frontline staff members may hesitate to decline referrals from colleagues with whom they have ongoing interactions, regardless of budgetary or logistical issues. The expectations of these partners must be taken into account and reconciled with issues of feasibility if the program is to be successful in gaining their support or en- dorsement. Also, if the health program is to provide partial coverage, program planners must establish eligibility criteria and procedures for prioritizing and enrolling potential recipients. This issue, of course, can lead to considerable debates over the particulars of the cutoff criteria chosen for program eligibility. The other reason to consider whether the health program will provide full or partial coverage is that the scope of the program affects the design of the evaluation and potentially the cost of conducting an effect evaluation of the program.

In addition, the scope of the program needs to be considered with respect to underinclusion

Some programs are less readily identified as providing full or partial coverage because, although the program is designed with the population level in mind, the intended population is restricted to those meeting the criteria for participation, such as income level for state child health insurance programs; federal Women, Infants, and Children (WIC) nutrition programs; or Medicare. Because these programs are intended to serve the entire intended population, they could be considered full-coverage programs, although the eligibility criteria are such that they are partial-coverage programs. Across the public health pyramid, both partial- and full-coverage programs reflect whether the program is designed primarily to make changes at the individual, aggregate, or population level (TABLE 7-3).

Making a distinction between full- and partial-coverage programs during the planning phase may not be so easy. Stakeholder issues may arise, and advocacy positions may be taken

TABLE 7-3 Examples of Partial- and Full-Coverage Programs by Level of the Public Health Pyramid

Pyramid Level Partial Coverage (for Segment of

Intended Population) Full Coverage (for Entire

Intended Population)

Individual—direct services

Dialysis for portion of those with kidney failure, early-childhood- intervention programs for children at developmental risk, hospice care for those dying who choose the service

Ambulance and emergency medical care for all individuals, immunization clinics available to all individuals

Aggregate— enabling services level

Needle exchange programs for some substance abusers, Medicare services for the disabled and homebound, Head Start for low-income children

Medicaid coverage for dialysis for those with kidney failure

Population-based level

WIC program for low-income families, state child health insurance plans (SCHIP) for low-income families

Seat-belt laws, Medicare coverage for all individuals older than age 65, fluoridation of the water supply for all residents

Infrastructure level Portable electronic devices for nurses making home visits

Licensure for all physicians, nurses, and dentists; national cancer registry

Inputs to Service Utilization Plan 189

are all aspects of being put in a queue. Waiting lines and wait times reflect the degree of match between the capacity to provide the service and the demand for the service. The services utiliza- tion plan ought to include a plan for handling wait-lists and such.

For example, if an immunization clinic is being held, the services utilization plan ought to balance the anticipated number of individ- uals seeking immunizations against the rate at which individuals can be processed for the immunization, the length of time needed to give the immunization, and the number of program staff members available to implement the im- munization clinic. An imbalance will result in either people waiting for long periods or staff members not having work.

Because the particulars of studying queues can be complex, large health programs—and particularly those that are ongoing—may find it valuable to hire an operations specialist to study the issue of queuing. Consultation with such experts can ensure that the health program is provided in the most timely and efficient manner possible.

Intervention Delivery The fact that the discussion of intervention delivery appears halfway through this text is no accident. The actual delivery of the intervention, though it takes the most effort (see Exhibit 7-1), is relatively easy if the planning has been well done and the process theory has been thought out thoroughly.

Delivery of the intervention ought to follow the protocols and procedures developed specifi- cally for the health program. Such adherence to the plan ensures that the intervention is deliv- ered in a standardized and consistent manner. The level of detail included in the protocols and procedures will vary across programs, of course. For example, if the health program intervention involves secondary prevention of breast cancer with mammography screening, then the intervention protocol should include considerable details about the procedure for

or overinclusion. Overinclusion is much more difficult to detect in a full-coverage program than in a partial-coverage program because the presumption in the former type of program is that the intended program recipients are the members of an entire intended population. Given that a full-coverage program, by its nature, is likely to have a large number of recipients, it will be diffi- cult to identify those few recipients who are not members of the intended population. By contrast, in a partial-coverage program, overinclusion is more likely to occur than underinclusion.

The interaction of the scope of a program with the appropriateness of inclusion may have implications that need to be considered, espe- cially when developing the program marketing plan and the program budget. It will also be a consideration in establishing the eligibility cri- teria and screening procedures. In addition, an awareness of the potential for this interaction may help explain later findings during evaluations of either the process or program effect.

Screening Program eligibility screening is another services utilization plan input. Such a procedure is nec- essary to ensure that the program is provided to members of the intended audience, thereby minimizing underinclusion or overinclusion. Despite the inclination to want to provide the program to anyone interested, screening en- hances the efficiency and the effectiveness of the program. Efficiency is enhanced by providing the program only to those participants to whom the intervention is tailored, thereby making it less likely that the program intervention will need to be individually tailored. Effectiveness is enhanced because only those who need the program, and thus are more likely to experience the benefits of the program, are included in the outcome and impact evaluations.

Queuing Waiting to be seen for services, being on hold, and having to wait until services become available

190 Chapter 7 Process Theory for Program Implementation

pretested, with the program developers paying attention to the various program components. Pretesting can take many different forms, such as having focus groups review materials and make comments, providing the program for free to a small group, or having experts com- ment on the program design and materials. In particular, marketing materials and media messages ought to be pretested. Characteristics to be assessed include the materials’ attrac- tiveness, comprehension, acceptability, and persuasion. Any materials that will be read by program participants should also be tested for readability. The rule of thumb is the lower the level, the better. No one wants to struggle with technical language, complex sentences, or large words. Generally, an eighth-grade reading level is recommended for materials. EXHIBIT 7-2 shows this paragraph as rewritten at an eighth-grade level.

Pretesting of the interventions ought to include pretesting of the evaluation instruments that will be used during the program and after the program has been completed. As more health programs are required by funding agencies to document their success, the evaluation is be- coming more integral to the actual intervention and overall program delivery.

doing the actual mammogram, the taking of a history before the mammogram is per- formed, and notification and referral of those screened. In contrast, if the health program is a metropolitan-wide mass-media awareness cam- paign about the value of adult immunizations, then the intervention protocol should allow for flexibility in accessing and communicating with media contacts while providing guidance in who to approach, which topics to address and to avoid, and which products (e.g., public service announcements, video clips, flyers) to distribute to which media sources.

Implementing evidence-based interventions requires managers’ commitment to ensure the successful uptake of these practices in diverse settings. For example, researchers studying five evidence-based practices in community mental health settings found more actively engaged leadership, attention to workflow dynamics such as policy and documentation, and reinforcement such as monitoring and feedback to be associated with better implementation fidelity (Torrey, Bond, McHugo, & Swain, 2012).

One aspect of the intervention that makes common sense, but is often neglected, is pilot-testing the program. Prior to full pro- gram implementation, the program may be

EXHIBIT 7-2 Chapter Text Paragraph Rewritten at an Eighth-Grade Reading Level

It makes common sense to try out a program before it begins, but this is often not done. Before starting a program, try out the handouts and the program parts. There are many ways to see if the people in the program will understand the handouts and program. One way is to have a focus group look over the materials and make comments. Another way is give the program to a small group of people for free and see if there are any problems. Also, experts can help by making comments on the program and the handouts. Advertisements and media messages need to be tried out before they are used, too. Look at how attractive they look, how easy it is for people to understand them, whether people will accept the message, and how good the messages are at convincing people to participate in the program. Also, handouts need to be checked for how easy they are to read. Today, word processing software can check them and show the grade level they are written at. The rule of thumb is that lower grade levels are better because no one wants to work at reading technical words, long sentences, or large words. For example, this paragraph is written a little below the eighth-grade level.

Services Utilization Plan Outputs 191

Ensuring that a mechanism is in place to track the service plan outputs is a critical managerial re- sponsibility. TABLE 7-4 is a template for tracking the interventions and the units of service outputs. Fuller et al. (2007) used this format to show the amount of reach across different intended populations.

One other services utilization output exists as well: the materials developed and produced as part of the effort to provide the health program, such as public service announcements (PSAs), educational videos, annual reports, or curricula. Another key output is the workflow—that is, the extent to which program staff members have work over a given time period or that work is done in a coordinated manner.

Keeping a record of the various outputs as the health program is implemented makes it possible later to conduct a process evaluation of activities focused on program implementation. These evaluation activities are discussed in detail elsewhere in the text.

▸ Services Utilization Plan Outputs

Outputs of the services utilization plan include the number of units of service provided and the quantity of service completions. Units of service is a term used to refer the agency- or program-specific quantification of what was provided, such as hours per client, number of inpatient visits, number of educational sessions, or number of hours of client contact. Because what constitutes a unit of service can vary widely, each health program must specify what it considers a unit of service. Another service utilization output is the number of services that have been completed. For health programs, this might be the number of immunization clinics held, the number of completed referrals for medical follow-up, or the number of health educational courses provided.

TABLE 7-4 Template for Tracking Services Utilization Outputs Using Example Interventions and Hypothetical Activities

Intervention Component

Intended Audience A:

Persons with a Health Problem

Intended Audience B:

Clinics

Intended Audience C:

City

Individual education 100 persons at risk 76 providers visited (Not applicable)

Individual screening 600 persons 30 providers 1,000 persons screened at health fairs

Group education 15 groups of persons at risk

8 groups at 3 clinics 15 groups at library; 6 groups at schools

Population education 2,000 flyers; 600 stickers

50 posters 25 public service announcements (PSAs); 10 health fairs

Individual support for behavior

125 persons at risk 32 providers (Not applicable)

192 Chapter 7 Process Theory for Program Implementation

Having a diagram such as Figure 7-4 allows planners to determine what might be missing as part of the effort to address the health problem and to deliver the intervention, as well as what has been added that might not be essential to achieving the desired program effect.

▸ Alternative Plan Formats

In summary, the program theory shows the inputs and outputs of the organizational plan and the service utilization plan. Other formats for summarizing this information exist. Each format has advantages and disadvantages, and will resonate with different audiences. Regard- less of which format is used, the intention is to anticipate, to the fullest extent, the needs and

Summary: Elements of Organizational and Services Utilization Plans This chapter has explained the elements of the organizational plan and services utilization plan, which collectively make up the process theory. In designing and planning for a health program, attention to the process theory becomes a reality check that the needed resources and processes are in place to initiate the program.

For example, continuing with the Layette- ville example, FIGURE 7-3 gives a possible pro- cess theory for a neural tube defect prevention program. However, the process theory needs to correspond to and support the intervention theory. When the process theory is combined with the effect theory, a comprehensive view of the program becomes possible (FIGURE 7-4).

FIGURE 7-3 Process Theory for Neural Tube Defects and Congenital Anomalies Health Problem

Services Utilization Plan

Inputs

Monthly reports

Revenue– expense totals

Dates for hire, training, start

Organizational chart

Instructors’ manual,

intervention protocol

% coverage

# lab tests, # visits,

# counseling sessions

# women who got folic acid

education

# handouts created,

evaluation forms created

Eligibility criteria,

enroll women

Waitlist, triage

Advertising, PSA,

announce program

Deliver interventions per protocol

RNs, MDs, Geneticists

Software for electronic

record

Grants plus fees

Office, classroom, educational

charts

Transportation to geneticist

Program manager

2-year cycle

Organizational Plan Inputs

Outputs Outputs

Alternative Plan Formats 193

to outputs, immediate outcomes, and long-term outcomes. Since their introduction in the late 1980s, logic models have become ubiquitous and are frequently required by funding agencies. The CDC has used logic models for its programs, and the Public Health Foundation includes them as one of the quality improvement tools. A logic model is one-page tabular summary of the program theory. The logic model comes in

activities that will make the health program as successful as possible. The two most commonly used formats are reviewed here.

Logic Models As a tool for program planning and evaluation, a logic model is a diagram in a table format that shows the relationship of inputs and activities

FIGURE 7-4 Effect and Process Theory for Neural Tube Defect Prevention Program

Age, food availability,

type of employment

Causal Theory of Presence of Neural Tube Defect Outcome Theory

Presence of neural

tube defect

Rate of congenital anomalies E

f f e c t

T h e o r y

Intervention Theory

INTERVENTIONS

Nutrition education

Prenatal screening, encourage taking prenatal vitamins

Knowledge about folic acid and workplace

hazards, genetic counseling

Inadequate intake of folic acid, prenatal

exposure to chlorine, paternal exposure to

organic solvents

Preconception nutritional status,

biological processes

P r o c e s s

T h e o r y

RNs, MDs, Geneticists

Program manager

Funding for 2-year

program

Organizational Plan Inputs

Tracking information

system

Eligible women

Program recruitment

Services Utilization

Plan Inputs

194 Chapter 7 Process Theory for Program Implementation

the program encompasses that can be used to guide process and effect evaluations.

Kaplan and Garrett (2005) drew on ex- perience with three comprehensive, multisite programs when they reviewed the correspond- ing logic models to identify the value of logic models. They found that development of a logic model not only fostered collaboration for already strong coalitions but also proved challenging for diverse and underresourced organizations or coalitions. The logic models they reviewed did reveal assumptions that led to changes in staffing for the program, but often the assumptions were unstated or unrealistic. These authors also found that the lack of attention to the rationale for the program—in other words, the intervention theory—was related to the heavy emphasis that the logic model put on activities and outcomes. A key benefit of the logic model was that it facilitated communication with internal and external stakeholders, including

many variations. TABLE 7-5 is an example of a logic model for the neural tube defect health problem and the interventions chosen to address the problem.

Comparing the process and effect theories for a congenital anomalies health problem (Figure 7-4) with the logic model for the same problem (Table 7-5), the differences and areas of overlap become clear. The logic model lists activities but does not distinguish between the actual intervention activities and the various activities that are part of the organizational plan inputs. In contrast, the process theory clearly distinguishes the actual intervention from other activities associated with the organizational plan and service utilization plan inputs. The logic model can be thought of as a different repre- sentation of the overall program theory. Both logic models and the program theory models function as tools to help anticipate needs and gaps, to engage in thinking about the entire program, and ultimately to have a plan of what

TABLE 7-5 Hypothetical Logic Model of a Program for Reducing Congenital Anomalies

Assumptions Inputs Activities Outputs Immediate

Outcome Long-Term Outcome

Health problem is important to community of providers and residents. Residents have access to health care and food sources.

Program staff members. Location to hold the screening. Donation for printing.

Create a time line and tracking system. Conduct staff training. Create program protocol and manual. Create PSAs and recruitment materials.

Number of educational materials; program manual. Number of staff trained. Number of women screened for type of employment and serum folic acid. Percentage of women who start folic acid preconception.

Improved knowledge of folic acid. Percentage of births with neural tube defects (anencephaly, spina bifida), cleft lip or palate.

Rate of congenital anomalies.

Across the Pyramid 195

and the least specific information. More detail, particularly about the financial characteristics of the proposed program, is often needed to communicate effectively and convincingly the need for the proposed health program.

▸ Across the Pyramid At the direct services level of the public health pyramid, implementation of a health program is very similar to the implementation of other health services. However, particular attention should focus on tailoring the human resources to the programmatic intervention. The social marketing plan is also targeted to individuals and their behaviors.

At the enabling services level, a match between providers and the health program intervention is necessary. If the program in- tervention is an enabling service, the providers (human resources) are more likely to have a background or expertise in social services. Enabling service programs are also more likely to use volunteers, a decision that has impli- cations for the managerial resources needed. The social marketing plan is also needs to be tailored to the aggregate targeted by the pro- gram. Because many enabling services either require a referral or are accessed via a referral, the social marketing of the program may focus as much on the providers making the referrals as on the intended audience of clients.

At the population services level, as at the other levels of the public health pyramid, a match must exist between the abilities and skills of the providers and the health program intervention. Social marketing to a population should have a broad base of appeal and should almost certainly use mass media.

This chapter has dealt mostly with the infrastructure level of the public health pyra- mid. Having a highly specified, comprehensive organizational plan and services utilization plan provides a strong foundation for implementing an effective health program. Also, having the necessary and appropriate resources for the

funders. Overall, logic models are complex and time consuming to develop well, but they can pay off through improved alignment among intended population health needs, programs, and subsequent process evaluation (Gervais, Montigny, Lacharité, & Dubeau, 2015).

Business Plans A business plan is a document that summa- rizes the analyses behind the development of a product, service, or program; substantiates the development of a program based on a wide range of crucial considerations; and provides details for implementation of the program. Business plans, in contrast to the diagrammatic layout of the process  theory and logic models, are written documents. Nevertheless, as with the process theory diagram and logic models, the overriding intent is to communicate the details of the program and to provide evidence that the details have been thoroughly thought out and actions are well planned and coordinated.

Research has shown that business plans facilitate decision making and accelerate devel- opment of the product or service (Burke, Fraser, & Greene, 2010). The use of business plans is advocated in medicine (Cohn & Schwartz, 2002), nursing (Hamric, Hanson, Tracy, & O’Grady, 2014), and public health (Orton, Umble, Zelt, Porter, & Johnson, 2007). Although development of health programs is rarely thought of as an en- trepreneurial activity, the development of health services is very appropriately viewed from this perspective. As Ellis, Embree, and Ellis (2015) note, business cases are especially relevant for more costly initiatives. They advocate clarifying implementation plans and outcomes measures in order to demonstrate the thoughtful devel- opment of new initiatives.

TABLE 7-6 lists the basic elements of a business plan, along with the corresponding elements of the organizational plan, services utilization plan, and logic model. As the table makes clear, the business plan, because it is a written document, has greater detail in many areas. By comparison, logic models provide the least amount of information

196 Chapter 7 Process Theory for Program Implementation

TABLE 7-6 Generic Elements of a Business Plan, with Their Purpose and Corresponding Element of the Process Theory and Logic Model

Business Plan Format Purpose

Process Theory Element

Logic Model Element

Title/cover page

Gives a first impression

Executive summary

Gives a first impression

Business concept

Describes the program design, with goals and objectives

Entire program theory Entire logic model

Market analysis Analyzes the demand, need, competition, and effect on existing services and health status

Community needs assessment

Financial analysis

Projects revenues and expenses; states the fiscal assumptions used in analyses

Organizational plan inputs: monetary resources, budget

Risk and competitive analysis

Discloses the sources and types of possible failures, with alternatives to avoid those failures; balances failure risks with merits of the program

Operational plan

Shows how personnel, management, space, and equipment come together for the program; delineates resource requirements

Organizational plan inputs: human, informational, physical, managerial, time resources, transportation

Inputs, activities

Marketing plan Describes strategy to reach the intended audience, branding, distribution, price, promotion

Service utilization plan: social marketing, participants

Activities

Milestones Gives time frame for accomplishment of key tasks and outcomes

Process and outcome objectives

Immediate and long-term outcomes

health program is an indicator of the quality of the infrastructure. Even the most creative ideas and the most scientifically sound programs will fail if they lack an adequate infrastructure. Too

often attention is focused on the health program interventions and clients, without the prerequisite attention and effort being devoted to developing and maintaining the programmatic infrastructure.

197Internet Resources

Clear and Effective” (https://www.cms.gov /outreach-and-education/outreach/written- materialstoolkit/downloads/toolkitpart07.pdf ) contains a subsection on using reading formulas.

Kellogg Foundation Evaluation Manual This text, which is available as a PDF file, is a comprehensive publication on evaluation. Chap- ter 5 (https://www.wkkf.org/resource-directory /resource/2010/w-k-kellogg-foundation-eval- uation-handbook) relates to the development of program theory. You will be asked to type in an e-mail address in order to download a PDF copy.

ReCAPP BDI Logic Model Course For an audio experience, cruise through the highly recommended online walkthrough on logic models found at http://recapp.etr.org /recapp/documents/logicmodelcourse/index .htm. The material is presented in PowerPoint.

Centers for Disease Control and Prevention (CDC) The CDC has a metasite for logic models with some good links. Find it at www.cdc.gov/eval /resources/index.htm. The CDC also provides scholarly papers on marketing at http://www .cdc.gov/healthcommunication/

Tools of Change The Canadian website at http://www.toolsofchange .com/en/home/ has links to social marketing as well as numerous examples of programs using various techniques for motivation.

Social Marketing Institute Working papers and published articles at the Social Marketing Institute website (http:// socialmarketing.org/) are scholarly resources.

Business Plans Numerous resources on developing business plans can be found on the Internet, including some sites that provide how-to guides and

For this reason, the organizational plan must identify resources for the infrastructure as well as resources for the services utilization plan activities. In addition, if the health program is intended to increase the capacity of the infrastructure—for example, to improve workforce capacity—an organizational plan and services utilization plan for such a program is still warranted as a step to ensure the success of the program.

▸ Discussion Questions and Activities

For all of the questions below, please pick one of the following health issues identified as sa- lient in Bowe County’s most recent community health needs assessment: vaccination, adolescent pregnancy, gunshot wounds, or type 2 diabetes.

1. Give examples of how the outputs of the organizational plan and the services utilization plan relate to the process theory objectives.

2. What type of social marketing would you use?

3. How might you minimize underinclu- sion or overinclusion in such a health program?

4. Draft a logic model for this health pro- gram.

▸ Internet Resources There is a wealth of resources on developing various types of logic models and program theory models. The following list represents diverse perspectives and approaches.

Reading Level of Written Materials A Google search using “reading statistics” will yield instructions for assessing the reading grade level of text highlighted in Word. Also, “CMS’s Toolkit for Making Written Material

198 Chapter 7 Process Theory for Program Implementation

Cheng, H., Kotler, P., & Lee, N. R. (2009). Social marketing for public health: An introduction. Sudbury, MA: Jones and Bartlett.

Clukey, L. (2010). Transformative experiences for Hurricanes Katrina and Rita disaster volunteers. Disasters, 34(3), 644–656. doi:10.1111/j.1467-7717.2010.01162.x

Cohn, K. H., & Schwartz, R. W. (2002). Business plan writing for physicians. American Journal of Surgery, 184, 114–120. http://dx.doi.org/10.1016/S0002-9610(02)00921-2

Ellis, R. J. B., Embree, J. L., & Ellis, K. G. (2015). A business case framework for planning clinical nurse specialist–led interventions. Clinical Nurse Specialist, 29(6), 338–347. doi:10.1097/NUR.0000000000000162

Ferrari, J. R., Luhrs, T., & Lyman, V. (2007). Eldercare volunteers and employees: Predicting caregiver expe- riences from service motives and sense of community. Journal of Primary Prevention, 28, 467–479. doi:10.1007 /s10935-007-0108-6

French, J. (2010). Social marketing and public health: Theory and practice. Oxford, UK: Oxford University Press.

Fuller, C. M., Galea, S., Caceres, W., Blaney, S., Sisco, S., & Vlahov, D. (2007). Multilevel community-based inter- vention to increase access to sterile syringes among injection drug users through pharmacy sales in New York City. American Journal of Public Health, 97, 117–124. doi:10.2105/AJPH.2005.069591

Gervais, C., Montigny, F., Lacharité, C., & Dubeau, D. (2015). The Father Friendly Initiative within Families: Using a logic model to develop program theory for a father support program. Evaluation and Program Planning, 52, 133–141. http://dx.doi.org/10.1016/j .evalprogplan.2015.04.006

Glenton, C., Scheel, I. B., Pradhan, S., Lewin, S., Hodgins, S., & Shrestha, V. (2010). The female community health volunteer program in Nepal: Decision makers’ percep- tions of volunteerism, payment, and other incentives. Social Science & Medicine, 70, 1920–1927. http://dx.doi .org/10.1016/j.socscimed.2010.02.034

Gulati, R. (1999). Network location and learning: The influ- ence of network resources and firm capabilities on alliance formation. Strategic Management Journal, 20, 397–420. doi:10.1002/(SICI)1097-0266(199905)20:53.0.CO;2-K

Halpert, J. A., Stuhlmacher, A. F., Crenshaw, J. L., Litcher, C. D., & Bortel, R. (2010). Paths to negotiation success. Negotiation and Conflict Management Research, 3(2), 91–116.

Hamric, A. B., Hanson, C. M., Tracy, M. F., & O’Grady, E. T. (2014). Advanced practice nursing: An integrative approach. 5th edition. ElsevierSaunders. St. Louis, MO.

Hong, S.I., & Morrow-Howell, N. (2010). Health outcomes of Experience Corps: A high-commitment volunteer program. Social Science & Medicine, 71, 414–420.

Hintgen, T. L., Radichel, T. J., Clark, M. B., Sather, T. W., & Johnson, K. L. (2000). Volunteers, communi- cation, and relationships: Synergistic possibilities.

templates. Entrepreneur.com has an entire sec- tion dedicated to developing business plans at www.entrepreneur.com/businessplan/. The U.S. Small Business Administration (SBA), created by Congress to help, counsel, and protect small businesses, provides an online guide to writing a business plan at https://www.sba.gov/starting -business/write-your-business-plan?interi- orpage2015. The nonprofit My Own Business offers an entire online course about developing a business; the section on business plans (www .myownbusiness.org/s2/) includes basics, for- matting, examples, and templates.

References Allen, R. S., Azuero, C. B., Csikai, E. L., Parmelee, P. A.,

Shin, H. J., Kvale, E., . . . Burgio, L. D. (2016). “It was very rewarding for me . . .”: Senior volunteers’ experi- ences with implementing a reminiscence and creative activity intervention. The Gerontologist, 56(2), 357–367. doi:10.1093/geront/gnu167

Bluthenthal, R. N., Heinzerling, K. G., Anderson, R., Flynn, N. M., & Kral, A. H. (2008). Approval for syringe exchange programs in California: Results from a local approach to HIV prevention. American Journal of Public Health, 98(2), 278–283. doi:10.2105/AJPH.2005.080770

Bowen, D. J., Kuniyuki, A., Shattuck, A., Nixon, D. W., & Sponzo, R. W. (2000). Results of a volunteer program to conduct dietary intervention research for women. Annals of Behavioral Medicine, 22, 94–100. doi:10.1007 /BF02895173

Burke, A., Fraser, S., & Greene, F. J. (2010). The multiple effects of business planning on new venture perfor- mance. Journal of Management Studies, 47(3), 391–415. doi:10.1111/j.1467-6486.2009.00857.x

Castro, G., Delgado-Verde, M., Navas-López, J., & Cruz-González, J. (2013). The moderating role of innovation culture in the relationship between knowl- edge assets and product innovation. Technological Forecasting & Social Change, 80, 351–363. http://dx.doi .org/10.1016/j.techfore.2012.08.012

Centers for Disease Control and Prevention. (2011). Health marketing. Retrieved from www.cdc.gov/healthcom munication/HealthBasics/WhatIsHC.html

Chapin, R. K., Sergeant, J. F., Landry, S., Leedahl, S. N., Rachlin, R., Koenig, T., & Graham, A. (2013). Re- claiming joy: Pilot evaluation of a mental health peer support program for older adults who receive Medicaid. Practice Concepts, 53(2), 345–352.doi:10.1093/geront /gns120

199References

Reese, C. D. (2015). Occupational health and safety manage- ment: A practical approach (3rd ed., pp. 35–46). Boca Raton, FL: CRC Press.

Rossi, C., Cricelli, L., Grimaldi, M., & Greco, M. (2016). The strategic assessment of intellectual capital assets: An application within Terradue Srl. Journal of Business Research, 69(5), 1598–1603.

Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Thousand Oaks, CA: Sage.

Singh, D., Cumming, R., Mohajer, N., & Negin, J. (2016). Motivation of community health volunteers in rural Uganda: The interconnectedness of knowledge, rela- tionship and action. Public Health, 136, 166–171. http:// dx.doi.org/10.1016/j.puhe.2016.01.010

Thornton, L. A. & Novak, D. R. (2010). Storying the temporal nature of emotion work among volunteers: Bearing witness to the lived traumas of others. Health Communications, 25, 437–448. doi:10.1080/10410236 .2010.483340

Topp, S. M., Price, J. E., Nanyangwe-Moyo, T., Mulenga, D. M., Dennis, M. L, & Ngunga, M. M. (2015). Motivations for entering and remaining in volunteer service: findings from a mixed-method survey among HIV caregivers in Zambia. Human Resources for Health, 13(72), 1-14. doi:10.1186/s12960-015-0062-y

Torrey, W. C., Bond, G. R., McHugo, G. J., & Swain, K. (2012). Evidence-based practice implementation in community mental health settings: The relative importance of key domains of implementation activity. Administration and Policy in Mental Health and Mental Health Services Research, 39(5), 353–364.

Trettin, L., & Musham, C. (2000). Using focus groups to design a community health program: What roles should volunteers play? Journal of Health Care for the Poor and Underserved, 11, 444–455.

Turnock, B. J. (2009). Public health: What it is and how it works (4th ed.). Sudbury, MA: Jones and Bartlett.

Walsh, D. C., Rudd, R. E., Moeykens, B. A., & Moloney, T. W. (1993, Summer). Social marketing for public health. Health Affairs, 104–119. doi:10.1377/hlthaff.12.2.104

Topics in Stroke Rehabilitation, 7(2), 1–9. http://dx.doi .org/10.1310/4R0L-ULJM-MEP1-XBGK

Kaplan, S. A., & Garrett, K. E. (2005). The use of logic models by community-based initiatives. Evaluation and Program Planning, 28, 167–172. http://dx.doi .org/10.1016/j.evalprogplan.2004.09.002

Karwalajtys, T., McDonough, B., Hall, H., Guirguis-Younger, M., Chambers, L. W., Kaczorowski, J., . . . Hutchison, B. (2009). Development of the volunteer peer educator role in a community cardiovascular health awareness program (CHAP): A process evaluation in two com- munities. Journal of Community Health, 34, 336–345. doi:10.1007/s10900-009-9149-5

Kettner, P. M., Moroney, R. M., & Martin, L. L. (1999). Designing and managing programs: An effectiveness based approach (2nd ed.). Thousand Oaks, CA: Sage.

Kotler, P., & Zaltman, G. (1971). Social marketing: An approach to planned social change. Journal of Marketing, 35, 3–12. doi:10.2307/1249783

Maes, K. (2012). Volunteerism or labor exploitation? Harnessing the volunteer spirit to sustain AIDS treatment programs in urban Ethiopia. Human Organization, 71, 54–64. http:// dx.doi.org/10.17730/humo.71.1.axm39467485m22w4

Morrow-Howell, N., Kinnevy, S., & Mann, M. (1999). The perceived benefits of participating in volunteer and educational activities. Journal of Gerontological Social Work, 32(2), 65–80. doi:http://dx.doi.org/10.1300 /J083v32n02_06

Orton, S., Umble, K., Zelt, S., Porter, J., & Johnson, J. (2007). Management Academy for Public Health: Creating entrepreneurial managers. American Journal of Public Health, 97, 601–605.doi:10.2105/AJPH.2005.082263

Pazoki, R., Nabipour, I., Seyednezami, N., & Imami, S. R. (2007). Effects of a community-based healthy heart program on women’s physical activity: A randomized controlled trial guided by community-based participa- tory research (CBPR). BMC Public Health, 7(147), 216. doi:10.1186/1471-2458-7-216

Posner, B. Z. (1995). What it takes to be a good manager. In J. R. Meredith & S. J. Mantel, (Eds.) Project management: A managerial approach (3rd ed., pp. 146–149). New York, NY: John Wiley & Sons.

201

© Lynne Nicholson/Shutterstock

Monitoring Implementation Through Budgets and Information Systems

Two key elements exist behind the scenes that contribute to the success of a program and to the ability to evaluate a program. One element is the ever-important budget and the accompanying skills to develop and monitor the program’s budget. The language and skills needed to create and monitor the operating budget, the budget that includes all the costs and revenues related to delivery of the program, are introduced. The other element, equally important, is the information system used by the agency for tracking and recording program participation and outcomes of participants. The informatics landscape continues to evolve rapidly, so only the basics as they relate to health program de- velopment and monitoring are presented.

▸ Budgets and Budgeting Program managers need a mechanism for mon- itoring the fiscal status of the program during implementation. This requires understanding the

basic concepts related to budgeting and having basic costing and budgeting skills, allowing for the development of a sound fiscal plan for the health program. General budget principles that apply across budget formats and specific organizational requirements are presented in this chapter. The intent is to introduce basic program budget concepts, particularly as they relate to the operating budget, so that health program planners and managers can communi- cate with financial personnel, funding agencies, and administrators to present the program in a positive light.

Budgets have the appearance of being factual and neutral. The reality is otherwise. Budgets are inherently political, are contextually influenced, and reflect values and preferences of the program developers. One only needs to consider the an- nual debate over the federal budget to appreciate the politicized nature of budgets. The  conse- quence of defunding health programs can have direct health effects, even for noncontroversial health programs. For example, preventive care

CHAPTER 8

202 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

decreased and emergent care increased after an orthopedic trauma service for indigent patients was defunded (Forsh , Amanantullah, Coleman, & Wolinsky, 2014). Underfunding of projects can result in not meeting project goals (Hazzard, Moreno, Beall, & Zidenberg-Cherr, 2012). Po- litical interference in program administration, despite wide public support, has implications for achieving population health goals (Petticrew & Krishnaratne , 2014). Such issues will likely weigh on the minds of health program planners and man- agers as they prepare and monitor their budgets.

Budgeting Terminology The broadest categories within budgets are expenditures and revenues. Expenditures are classified as either fixed or variable and as either direct or indirect.

Fixed costs do not vary with the number of clients served. Examples of fixed costs include rent, salaries of administrative personnel, and insurance costs. In contrast, variable costs do vary with the number of clients served. Exam- ples of variable costs include copying program handouts, program advertising, and refresh- ments for participants. Depending on how the health program is designed and implemented and how the organization does its accounting, certain costs may be counted as fixed or vari- able. For example, if program staff members are paid regardless of how many clients show up, then personnel wages and salaries represent a fixed cost. Conversely, if the program uses staff members on a part-time, as-needed basis, then personnel costs are variable. Budgets prepared based on the distinction between variable and fixed costs are more likely to be useful for pro- gram management via fiscal monitoring and for later conducting of economic and accounting analyses of the program.

Another way to think about costs is as di- rect or indirect. In the purest sense, direct costs reflect those resources that are used directly in the delivery of the program. Generally, the wages and salaries of staff members providing the intervention are a direct cost, as are materials

or supplies used with clients. Similarly, in the purest sense, indirect costs are those costs not associated with the delivery of the program but, more generally, with supporting the pro- gram. Utility bills, telephone charges, and legal and human resources staff are all examples of indirect costs. Indirect costs associated with overhead expenses (e.g., rent, utilities, facilities management, shared clerical support staff, office equipment) are typically estimated based on a standard rate that is set by the program funding agency or the organization’s financial officer. Although the federal government can negotiate indirect costs as a percentage of direct costs to be over 50%, funders can also set much lower caps on this crucial source of support for general expenses such as facilities and administration. Given this wide range, it is important to obtain the correct rate at which indirect costs should be applied to the expenditure side of the budget.

In developing a budget for planning a program, Foster, Johnson-Shelton, and Taylor (2007) remind us that costs associated with time are very important. They estimated the costs associated with participant recruitment time, staff training time, intervention time of staff members and participants, and time of volunteers. The need for this level of detail can be anticipated and the actual expenditures tracked as the program is delivered.

A critical distinction exists between the cost of a service and the charge for that service. The cost of the service is the simple sum of all re- sources required to provide the service, including indirect costs. However, clients may be asked to pay more than the cost; charges typically include the cost plus a profit margin. When budgeting, the program planners must consider both the cost of the service and the charges, given that ultimately the charges influence participation and acceptance of the program.

Identifying revenues is a bit simpler. Funds for health programs come from grants from federal funding agencies or foundations, fees collected from program participants, reimbursements from third-party payers, or charitable fund raising. Revenues might also be matched from state or

Budgets and Budgeting 203

federal agencies for local dollars allocated to the health program.

One source of revenue is often invisible: in-kind donations. Those services are provided to the program free of charge, but the program would have to pay for them if they were not donated. A common example is printing costs given as an in-kind donation; volunteer time is another in-kind donation for staff time. In some nonprofit agencies, the in-kind donations can be substantial. One study found in-kind donations to be a key resource during the program start-up phases (Tangka et al., 2008). It is important to track in-kind donations for two reasons. Funding agencies look at in-kind donations favorably as an indication of community support for the pro- gram. Also, if adequate in-kind donations are not received, a contingency plan for paying for those services must be developed and implemented.

Opportunity costs refer to purchases that cannot be made because of having spent the money on something else. In our daily lives, we instinctively calculate opportunity costs; for example, if I buy a ticket to fly home for the holiday, then I cannot buy a new smartphone. The opportunity costs accrued by program participants depend on the type of program. Two examples of opportunity costs used in cost analyses include loss of regular class time for a school-based health program delivered during the school day (Kilmer, Burgdorf, D’Amico, Miles, & Tucker, 2011), and parents’ work loss due to a child’s medical visits (Tai & Bame, 2011). Other examples of opportunity costs include losses or gains in work productivity, the family burden related to the health problem or program, out-of-pocket expenses related to participation, and cost of participating in the program (Drum- mond, Sculpher, Claxton, Stoddart, & Torrance, 2015). In terms of participants, because they are paying, in some form, for being in the program, they no longer have that money to spend on something else. For example, if a mother takes her child for immunizations, the cost of getting to the clinic and the copayment may cost her the opportunity (the money) to purchase new clothes for that child. Opportunity costs also

exist for the agency; for example, providing program A may mean that program B cannot be delivered. In summary, opportunity costs to the participants and the organization must be considered during the program planning stage.

Two other budget concepts deserve to be mentioned. Discounting is the process of converting future dollars, and future health benefits, into the present dollar value. It involves decreasing the current value by a rate, usually between 1% and 3%, on an annual basis. In a sense, discounting is the reverse of interest. Through this approach, expenses and health benefits that might be expected in 2 or 20 years can be valued on a par with one another. This facilitates interpretation of the dollar values and makes it simpler to see the costs so that decision makers can make choices based on data that are, so to speak, standardized. In practice, discounting is rarely used for operating budgets but is critical in budgets related to capital improvement or long-term financing.

One other concept related to costs is exter- nalities. Secondary effects, whether anticipated or not, may occur, and the effects can spill over to individuals not participating in the program; these are called externalities. Externalities may be indirect or even unanticipated conse- quences of the program, whether beneficial or harmful. Including them can be important in comprehensive economic analyses that attempt to monetize a broad range of program-related effects, such as from the societal perspective. Externalities are identified by conducting a so-called thought experiment in which one imagines possible programmatic effects on the intended audience as a whole, regardless of participation. Identifying externalities is important because they then become effects (if beneficial) or costs (if harmful or in some way costly). Some externalities for participants may not be easy to monetize. For example, participation in a substance abuse program may cost the participant older friendships based on substance use activities. Being able to list and discuss the externalities can be important in justifying a program to various stakeholders.

204 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

value to the inputs listed in the process theory. Costing out the elements of the process theory can feel odd or uncomfortable, but it must be done if the budget is to capture accurately and fully the costs in monetary terms.

Most program participants want to know what it will cost them to participate. The charge for the program or service is the most obvious cost. Other costs for program participants may exist, such as new or special equipment (e.g., handrails or cooking utensils), educational ma- terials (e.g., books or magazine subscriptions), transportation, or child care while attending the program. Being able to provide this infor- mation to potential participants can be helpful in recruiting them into the program.

▸ Budgeting as Part of Planning

Monetize and Compute Program Costs Most costs are related to the program imple- mentation and include the resources utilized by the program staff members and participants. Detailed program expenditures are used as the basis for computing the indirect and direct costs associated with providing the program. The process theory continues to be a guide for identifying which program costs to include in the budget (FIGURE 8-1) and for attaching a dollar

FIGURE 8-1 Relevance of Process Theory to Economic Evaluations

Organizational plan costs and expenses

Service utilization plan expenses

Other societal expenses and costs

Human resources expenses

Facilities and capital expenses

Transportation and travel expenses

Managerial expenses (e.g., consulting)

Monetary expenses (e.g., interest)

Informational expenses (e.g., computers,

telephone

Social marketing expenses

Participants’ out-of- pocket expenses

Intervention delivery expenses

Opportunity cost for other programs

Secondary effects costs

Total program costs

Budgeting as Part of Planning 205

Break-Even Analysis After the program budget is complete and nearly final, it is possible to do a break-even analysis. A break-even analysis is the math- ematical determination of the point at which the expenses related to providing the program are equal to or less than the revenues generated for or from the program. This type of analysis uses the price of the service (the charge), the variable costs of program, and the fixed costs of the program. The rather straightforward formula for a break-even analysis (Finkler, Ward, & Baker, 2007) follows:

Quantity of services = fixed cost/(charge per client − variable cost per client)

When the total fixed costs associated with the program are divided by the difference between the amount charged per participant and the variable cost per participant, the result gives the number of services—in other words, per paying participant—needed to be provided to break even financially. When the break-even analysis is done for classes rather than for one-on-one services, the number of participants per class needs to be taken into account in calculating the variable costs. This can be a little tricky, but it is not complicated. The following exhibits show the step-by-step process. EXHIBIT 8-1 gives a narrative scenario in which a break-even analysis is needed. The data from the narrative are then shown in a budget format in EXHIBIT 8-2. The formulas in TABLE 8-1 are applied to the budget information, using a spreadsheet program, Microsoft Excel. Sometimes it is not immediately clear where the break-even point might be. EXHIBIT 8-3 shows, in a table format, a range of the costs and revenues, with the break-even point noted.

Even programs provided by nonprofit agencies would be wise to conduct a break-even analysis. This rudimentary process provides useful insights into the amount of funding needed. Un- fortunately, public agencies and public programs neglect their fiscal accountability and efficiency accountability all too often by not conducting

Budget for Start-Up and Evaluation Costs For large-scale health programs that are being implemented for the first time, some costs will be related specifically to the start-up phase and scaling up. These costs can include the one-time purchase of large or small equipment, the extra time needed to find locations or negotiate contracts, or other one-time purchases needed to launch a program. In one study, the cost of scaling up a large adolescent sexual health program was approximately 20% of the total program costs (Terris-Prestholt et al., 2006). Cost effectiveness analyses of community-based rapid HIV testing included varying start-up costs across communities (Shrestha et  al., 2008). Another example of start-up costs for large-scale programs relates to the 2014–2015 Ebola virus outbreak in West Africa. The outbreak resulted in the Centers for Disease Control and Prevention (CDC) recommending preparing 55 hospitals for Ebola care, at an average cost of over $1 million per hospital, to become an Ebola treatment center (Herstein et al., 2016).

The budget must also include anticipated expenses related to the program evaluation. Whether the program staff members will be involved in the evaluation activities or whether a consultant is hired, funds must be allocated to support the evaluation before the program begins. Retrospectively acquiring funds to con- duct an evaluation can be difficult. In general, program grant proposals and budgets that do not include evaluation funds receive lower-priority scores. Evaluation expenses generally fall into the same categories as program expenses, al- though material expenses are typically limited to supplies and copying costs. Incentives given to individuals to participate in the evaluation can account for a substantial portion of the evaluation budget. Personnel will usually be the largest expense. At a minimum, a meaningful evaluation cannot be done for less than 10% of the direct program costs.

206 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

EXHIBIT 8-1 Example of a Scenario Needing a Break-Even Analysis

Layetteville’s Lighthouse Agency has decided to implement an updated evidence-based program, Bright Light, to address self-care for elderly diabetes patients, based on the needs identified during the community health assessment. Operational fixed expenses for Bright Light are budgeted at $1,500 per year.

Because Bright Light is an intensive educational program and Lighthouse tries to have a consistent ratio of teachers to clients, the number of teachers varies with the number of clients. For every 10 clients, Lighthouse employs 1 teacher, at a salary of $300, but Lighthouse will not hold a class with fewer than 5 clients. Lighthouse can provide up to 1 class per month.

Based on this information, how many clients must be served, billed, and pay for Bright Light classes if the program is to break even? What recommendation would you make to Lighthouse?

The team created a budget (Exhibit 8-2) and applied the formulas (Table 8-1) using Excel. Conclusion: Assuming that the classes are full with 10 clients, Lighthouse loses money if it chooses Option A, but it makes a little money with Option B.

EXHIBIT 8-2 Example of a Budget Used for a Break-Even Analysis for Bright Light on an Excel Spreadsheet

A B C D

1 Fixed costs (annual)—TOTAL $1,500

2 Rent $1,000

3 Clerical support $100

4 Cleaning service $100

5 Telephone (base charge) $300

6

7 Revenue

8 Charge per class $100

9

10 Variable costs per class—TOTAL $350

11 Teacher per class of 10 $300

12 Materials cost per student $50

Budgeting as Part of Planning 207

TABLE 8-1 Formulas Applied for Options A and B

Option A Option B Using data from Exhibit 8-2

7 classes 8 classes

Fixed costs $1,500 $1,500 = D1

Variable costs $5,600 $6,400 = (number of classes × D10)

Expenses $7,100 $7,900 = Fixed costs + variable costs

Revenue $7,000 $8,000 = (number of classes × 10 students × C8)

Balance ($100) $100 = Revenue - expenses

a break-even analysis. If clients are not paying for services, as is often the case in public health programs or mass-media campaigns, the price an individual might be willing to pay for the service or other such information can be used in place of the charge or price. For example, if residents of a community are willing to pay only $0.10 for information about how to prevent sexually transmitted diseases, then a safe-sex mass-media campaign will need to reach a very high number of persons theoretically to break even.

Thinking in terms of break-even analysis may seem unethical or contrary to the public health ethic. A break-even analysis would not be appropriate for health services that are required to ensure the health and safety of a population, such as a program to ensure that persons infected with tuberculosis take their medications or a national infant immunization program. In programs such as these, program participants are not expected to pay and thus the revenue portion of the break-even equation is zero. However, a break-even analysis could be conducted to understand the fiscal implications of these programs. In reality, use of a break-even analysis is a fiscally responsible way to make

decisions among programs or programmatic options. It also provides a quantifiable ratio- nale for proceeding or for modifying a health program. A break-even analysis may reveal that additional funding is required to provide the program as intended; it is far better to identify this potential problem before the program is initiated.

Budget Justification Budget justification is a requirement for almost all grant proposals, although the degree of detail expected varies by funding agency. A safe rule of thumb is to provide a very detailed budget justification; more detailed budget justifications demonstrate a more thorough program imple- mentation and evaluation plan. Most budget justifications involve some narrative explanation of why the dollar amounts are requested, but they must also include fairly detailed arithmetic formulas that show the derivation of specific costs. For example, budget narratives typically show the cents per mile paid to staff members for travel, the estimated number of miles staff members are expected to travel, and the number of staff members traveling those miles. Even if a

208 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

EX H

IB IT

8 -3

B re

ak -E

ve n

Ta bl

e Sh

ow s N

um be

r o f P

ay in

g St

ud en

ts N

ee de

d to

B re

ak Ev

en *

E F

G H

I J

K L

N um

be r o

f Cl

as se

s To

ta l N

um be

r of

S tu

de nt

s Fi

xe d

Co st

s Va

ri ab

le C

os ts

To ta

l C os

ts To

ta l R

ev en

ue Ba

la nc

e

= D

1 =

(C 11

× F

17 ) +

(C 12

× G

17 )

= H

17 +

I1 7

= G

17 ×

C 8

= K

17 -

J 17

1 7

1 10

$1 ,5

00

$8

00

$2 ,3

00

$1 ,0

00

($ 1,

30 0)

1 8

4 40

$1 ,5

00

$3 ,2

00

$4 ,7

00

$4 ,0

00

($ 70

0)

1 9

4 45

$1 ,5

00

$3 ,4

50

$4 ,9

50

$4 ,5

00

($ 45

0)

2 0

5 50

$1 ,5

00

$4 ,0

00

$5 ,5

00

$5 ,0

00

($ 50

0)

2 1

5 55

$1 ,5

00

$4 ,2

50

$5 ,7

50

$5 ,5

00

($ 25

0)

2 2

6 60

$1 ,5

00

$4 ,8

00

$6 ,3

00

$6 ,0

00

($ 30

0)

2 3

6 65

$1 ,5

00

$5 ,0

50

$6 ,5

50

$6 ,5

00

($ 50

)

2 4

7 70

$1 ,5

00

$5 ,6

00

$7 ,1

00

$7 ,0

00

($ 10

0)

2 5

7 75

$1 ,5

00

$5 ,8

50

$7 ,3

50

$7 ,5

00

$1 50

2 6

8 80

$1 ,5

00

$6 ,4

00

$7 ,9

00

$8 ,0

00

$1 00

2 7

8 85

$1 ,5

00

$6 ,6

50

$8 ,1

50

$8 ,5

00

$3 50

2 8

9 90

$1 ,5

00

$6 ,9

00

$8 ,4

00

$9 ,0

00

$6 00

* Us

in g

da ta

fr om

E xh

ib it

8- 2.

Budget as a Monitoring Tool 209

spreadsheet set up so that makes it easy to include columns for variance (EXHIBIT 8-4). Because of delays associated with billing, timing of orga- nizational fiscal reports, and the various dates selected by the organization at the beginning of a fiscal year, the actual expenses and income will always be based only on what is available at any given time. As bills are posted and income recorded, the expenditures and income must be updated.

Severe negative variances can occur because of overexpenditures or inadequate income, whereas severe positive variances can arise due to underexpenditures or greatly increased income. Either way, the presence of a significant variance alerts program managers that the program may not be delivered as planned and that some aspect of the program may need further scrutiny and modification.

Monthly updating of the budget variance usually suffices. As the end of a grant-funded program nears, at which time the balance needs to be near zero, more frequent updating would be wise. During the program implementation, the overall variance ought to be no more than 10% to 20% of the projected budget for an expense category. Some funding agencies specify the degree of budget variance that is acceptable and indicate at what point the budget needs to be renegotiated with the funding agency. Program managers, by monitoring the budget variance on at least a monthly basis, can make needed adjustments to spending throughout the fiscal year so that the year-end variance is within an acceptable range.

Types of Cost Analyses Several different types of analyses related to program costs are possible. The four main types of economic analyses are briefly reviewed as an introduction to the various approaches. The types can be classified along two dimensions (Drummond et al., 2015). One dimension is whether one or more programs are under consideration, and the other is whether costs only or costs and effects are included in the analysis (EXHIBIT 8-5).

health program is being sponsored by the parent organization, a budget justification is typically presented to departmental administrators or advisory boards when requesting their support.

Most grant proposal budgets focus on the major categories of direct costs. However, some federal funding agencies have begun to ask for budgets that are linked more directly to the program objectives, while others ask for budgets broken out by levels in the public health pyramid. Such budgets enable the funding agencies and program managers to determine the merit of the budget in terms of what is planned and which outcomes are anticipated. While creating such a budget can be challenging and requires some degree of speculation, assigning costs per pro- gram objective can be a powerful motivational and managerial tool.

▸ Budget as a Monitoring Tool

A budget developed during the planning stage becomes a tool to monitor implementation after the program has started. In addition, the careful attention to tracking expenses and income can be helpful in doing different types of cost analyses as part of a monitoring and subsequent planning cycle. The main approach to using a budget as a monitoring tool is tracking the budget variance.

Budget Variance On an ongoing and regular basis (usually monthly), the program manager should determine the extent to which current expenditures exceed (or not) the projected program expenditures. The difference between the budgeted and actual expenditures or income is called the budget variance.

The variance is calculated for each category or line item as a simple subtraction of expendi- tures from the budgeted amount, which can be easily calculated with the use of spreadsheets. When the program budget is developed during the planning stage, it would be wise to have the

210 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

EX H

IB IT

8 -4

E xa

m pl

e of

a B

ud ge

t S ho

w in

g Ye

ar -t

o- Da

te Va

ria nc

e

A B

C D

E F

G H

Ca te

go ry

To ta

l Bu

dg et

Cu m

ul at

iv e

Ex pe

ns es

Ex pe

ct ed

Do lla

r Va

ri an

ce

Am ou

nt

Le ft

to

Sp en

d Pe

rc en

ta ge

Un

sp en

t Pe

rc en

ta ge

Va

ri an

ce

b ×

(n u

m b

er o

f m

o n

th s

in to

F Y/

12 )

= d

− c

= b

− c

= f/

b =

(c /d

)/ 10

0

1 Sa

la rie

s (p

ro fe

ss io

n al

s) $1

01 ,9

65

$7 7,

17 4

$7 6,

47 4

$( 70

0)

$2

4, 79

1 24

% 1.

0%

2 A

ss is

ta n

t sa

la ri

es $9

,2 82

$6 ,8

78 $6

,9 62

$(

84 )

$ 2,

40 5

2 6%

1. 0%

3 Fr

in g

e b

en ef

it s

$3 2,

47 7

$2 4,

45 4

$2 4,

35 8

$9 6

$8 ,0

23

2 5%

1. 0%

4 Fi

xe d

S u

b to

ta l

$1 43

,7 24

$

1 0

8 ,5

0 6

$

1 0

7 ,7

9 4

$

(6 8

8 )

$ 3

5 ,2

1 9

2

5 %

5 6 M

at er

ia ls

/s u

p p

lie s

$1 ,6

00 $1

,1 64

$1

,2 00

$

(3 6)

$4 36

27

% 1.

0%

7 Tr

av el

$3 ,0

00 $2

,3 57

$2

,2 50

$1

07

$6 43

2

1% 1.

0%

8 Se

rv ic

es $8

00 $

57 8

$ 60

0 $

(2 2)

$2 22

28

% 1.

0%

9 Pr

in ti

n g

$1 ,2

07 $

83 8

$ 90

5 $

(6 7)

$3 69

3

1% 0.

9%

10 Su

b co

n tr

ac ts

$4 ,4

19 $1

,6 89

$3

,3 14

$(

1, 62

5) $

2, 73

0 62

% 0.

5%

Budget as a Monitoring Tool 211

11 Eq

u ip

m en

t $1

,5 00

$2 ,6

37

$ 1,

12 5

$ 1,

51 2

$(

1, 13

7) −

76 %

2. 3%

12 Va

ria b

le S

u b

to ta

l $

1 2

,5 2

6 $

9 2

6 3

$

9 3

9 4

$

(1 3

1 )

$ 3

2 6

3

2 6

% 0

.0 %

13 14 In

d ire

ct c

o st

s (1

2% )

$1 8,

75 0

$1 3,

99 6

$1 4,

06 3

$( 67

) $4

,7 55

25

%

15 16 E

X P

E N

D IT

U R

E

T O

TA L

$1 75

,0 00

$

1 3

1 ,7

6 4

$

1 3

1 ,2

5 0

$

(8 8

6 )

$ 4

3 ,2

3 6

2

5 %

0 .0

%

17 18 C

a te

g o

ry To

ta l

B u

d g

e t

C u

m u

la ti

ve

R e

ve n

u e

E x

p e

ct e

d D

o ll

a r

V a

ri a

n ce

A m

o u

n t

Le ft

t o

G

e n

e ra

te P

e rc

e n

ta g

e

U n

fu n

d e

d P

e rc

e n

ta g

e

V a

ri a

n ce

19 Fe

es $7

8, 50

0 $5

9, 58

3 $

58 ,8

75

$ 70

8 $1

8, 91

7 24

% 1

.0 %

20 D

o n

at io

n s

$6 ,5

00

$ 3,

92 8

$4 ,8

75

$( 94

7) $

2, 57

2 40

% 0

.8 %

21 G

ra n

t fu

n d

s $9

0, 00

0. 0

$ 75

,0 00

.0

$ 67

,5 00

$

7, 50

0 $

15 ,0

00

17 %

1 .1

%

22 R

E V

EN U

E TO

TA L

$1 75

,0 00

$

1 3

8 ,5

1 1

$

1 3

1 ,2

5 0

$

7 ,2

6 1

$

3 6

,4 8

9

23 24

B A

L A

N C

E

$ 6

,7 4

7

212 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

EXHIBIT 8-5 Types of Cost Analyses

One Program Two or More Programs

Costs Only Cost description Cost analysis Cost minimization

Cost comparison

Costs and Effects Cost analysis Cost effectiveness Cost–benefit Cost utility

Classifying economic analyses of programs along these two dimensions helps to discern the types and the subsequent requirements for conducting each type of analysis.

Cost-Only Analyses When only one program is considered and effects are not included, the type of analysis is termed a cost description. Cost description is the simplest form of economic analysis in that it is a straight- forward presentation of expenses related to the delivery of the health program. Most program managers routinely prepare cost descriptions, particularly for any annual reports that require an accounting of expenses by category or line item. A cost description is best thought of as part of process monitoring, particularly with regard to accountability and the budget aspect of the organizational plan. However, more sophisti- cated cost analyses might focus on a specific stage in the life cycle of a program, such as the start-up stage. Tangka et al. (2008) did just that by estimating the costs associated with starting a colorectal cancer program.

When the cost description includes a break- down of total expenses by an analytic factor, such as time periods, staff activities, or funding source, it becomes a cost analysis. A cost analysis is also useful as a process monitoring tool. It essentially analyzes costs by other elements of either the

organizational plan or the service utilization plan. In this way the findings of a cost analysis begin to provide information about the efficiency of a program, using, for example, dollars spent per program participant. These findings can be compared to published reports, benchmarks, or the original program plan to interpret the extent to which the program is more or less efficient than similar programs. Cost analyses can also focus on the costs or savings that result from the health program. For example, Lairson, Huo, Ball Ricks, Savas, and Fernández (2013) used call logs and additional expenditure data to quantity the staffing and material costs associated with the addition of conducting cancer screenings to existing 211 help center services.

Cost minimization is an analysis to deter- mine the best ways to provide the program at the lowest cost. This type of analysis ought to be part of the basic budgeting process because it allows program managers to select the lowest reasonable expense for a category or to comparison-shop for expenses such as supplies and capital equipment.

Published cost descriptions and cost analyses can be used during the planning stage to estimate program expenses for different intervention options. Clearly, cost descriptions and cost analyses are important for process monitoring and for understanding the basic cost parameters of a program. The basic information generated through these analytical approaches is easy for

Information Systems 213

Dave, 2017). Simulation can yield insights into direct and indirect positive and negative potential effects of alternatives before starting a new pro- gram. This can help decision makers avoid costly mistakes in program selection and implementa- tion. However, developing such models requires specialized expertise and labor-intensive reviews of prior research, potential new data analyses, consultation with informed stakeholders, and iterative programming and stakeholder review. Hence, the practical application of these methods for most health program evaluators may be using previous simulations conducted in reasonably comparable contexts.

Like a cost-effectiveness analysis, a cost–benefit analysis compares two programs. In this type of analysis, however, the programs need not address the same health problem. The program effects are compared based on larger societal benefits in addition to the outcomes and impacts of the program. The two dissimilar programs are compared on the basis of cost per dollar value of benefits achieved. Often the program under consideration is compared to a do-nothing option. Again, a review of existing cost–benefit analyses can help in choosing a program during the planning stage.

A cost–utility analysis measures the outcome of health programs in terms of the potential participants’ preference for the health outcome. This makes it the most complex and theoretical economic analysis. Programs are compared on the basis of their cost per unit of preference, called utility. For all practical purposes, only researchers perform such calculations and cost–utility analyses.

▸ Information Systems Since 2009, several federal acts and initiatives have been passed that are dramatically chang- ing the health information landscape and will require the ongoing attention of public health program managers. The strategic goals of these pieces of legislation, essentially, are to have: (1) a nationally adopted standard terminology

program staff members and stakeholders to understand, so it facilitates decision making about the program’s future.

Costs and Effects Analyses Cost comparison compares the costs of two or more programs without looking at outcomes or impacts. These comparisons may focus on the costs per participant for the programs being considered or the revenues generated by various programs. Cost comparisons might be done by a single agency with multiple health programs as the basis for deciding which program to continue. For example, Jerrell and Hu (1996) reported on a comprehensive analysis of costs associated with clients in three types of mental health and substance abuse prevention programs administered by one agency: a 12-step program, a behavioral skills program, and case management. An agency might undertake a cost comparison if it provided different intervention programs for the same health problem and needed to eliminate one of the programs. Cost comparisons can in- clude standard practice in comparison to a new program. Cost comparisons can also be done via simulations (e.g., Yonas et al., 2013), which do not require actual program implementation or actual cost data.

Cost-effectiveness analysis always compares the costs of two programs against one type of impact that is measured the same way in both programs. Using this approach, the programs are then compared on the basis of cost per unit of outcome. During the program planning phase, a review of cost-effectiveness reports could help program planners decide which interventions to implement. After the program has been im- plemented, program personnel can perform a cost-effectiveness analysis, with some help from a program evaluator. As with cost comparisons, simulations can be conducted to compare both the costs and various potential outcomes of al- ternative programs and implementation choices. Among this set of approaches, system dynamics modeling examines feedback loops that can lead to unintended effects (Lich, Urban, Frerichs, &

214 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

Within the healthcare industry, health infor- mation technology (HIT) broadly encompasses all of the technology in use within healthcare organizations, from computerized imaging to tweets of health messages. Until the passage of the American Recovery and Reconstruction Act (ARRA) in 2009, focus centered on medical technology linked to imaging or devices or on the documents related to the medical record. The ARRA provided financial incentives for physicians and healthcare organizations to speed up the adoption and use of HIT. Thus, emphasis has shifted from the electronic med- ical record (EMR), the more traditional paper medical record maintained by a physician, to the electronic health record (EHR), which has been defined as the longitudinal electronic record of a patient’s health information generated by one or more encounters in any care delivery setting (Healthcare Information and Management Systems Society [HIMSS], 2016).

If the health program is connected to a hospital or a clinic and the participants are patients of those healthcare organizations, then some form of an EHR exists. A concern from the perspective of a health program is the potential need to interface with the existing EHR. If the health program is based in an agency or orga- nization that does not provide direct medical services, however, such as a community parks district or a church, then linkages to an EHR may not be of concern.

Information Systems Considerations Throughout the program planning and evaluation cycle, the information system plays an important role as the means to retrieve existing or collect new data, store data and other program data elements, and then utilized the data elements in the form of reports. FIGURE 8-2, a simplified representation of an information system for a health program, shows varying arrow sizes to indicate the major direction of flow of data. At the center of the information system is a

for health data elements, (2) secure mechanisms for electronically sharing and transmitting health information, and (3) a data repository that can be used to monitor in real-time health events and trends. Each organization has its own informa- tion system, but for many health programs, that system needs to interface with the information systems of the funder and regulatory bodies. The following introduction to health informatics and public health informatics forms a base for understanding the issues and processes relevant to health program managers and developers.

Health Informatics Terminology Knowing how to use a computer and the Internet is not the same as understanding health infor- matics. In the broadest sense, an information system encompasses both the hardware and the software used to collect, store, and exchange digital information for a work unit, whether a it is a healthcare organization or a utilities company. Information technology refers the products used within the information system, such as Microsoft Office and Ethernet Internet connections. Most organizations have a management information system (MIS), with a subset of the information system designed to address managerial, fiscal, and human resources processes and data. Appointment scheduling software might be a component of the MIS for a healthcare organization, as would be any accounting software.

Geographic information systems (GISs) are used increasingly to collect, analyze, and display public health data using maps. John Snow, who mapped new cases of cholera in London in 1854, was essentially the first to use GIS in public health. With the advance of software specifically designed to map data points, GIS can be used at each phase of the program planning cycle. For example, GIS has been applied to tracing envi- ronmental health risk exposure, disease spread, and primary care access (Craglia & Maheswaran, 2016). ZIP codes are another convenient unit for mapping both participation and need and for assessing the correlation between the two (Hsu et al., 2010).

Information Systems 215

FIGURE 8-2 Information System Processes Throughout the Program Planning Cycle

Health Program Planning

Effects Evaluation Data Elements

Process Evaluation Data Elements

Data for analysis Reports on trends over time

Timeline Budget Goals and TAAPS and TREW Objectives

Participation counts Expenditures and Revenues Staff hours and effort Program dosage tracking

R eports based

on queries

Retri eval

of

docu ment

s

Health status measures Knowledge measures Behavior measures Social health measures Health care utilization

Reports oneffects andprocesses

Re po

rts b

as ed

on q

ue rie

s

P rim

a ry d

a ta

co lle

cte d fro

m

co m

m u n ity

Program Information System and Data Warehouse;

Links to Existing Databases

Program Planning Data Elements

Community Assessment Data

Elements

computer server or data warehouse, implying that the data submitted and retrieved are com- patible with the various types of hardware and software used throughout the organization. In larger healthcare organizations, the issues of ensuring compatibility fall within the domain of the information officer. In smaller organizations providing health programs, the program manager may be the one facing compatibility problems. Nonetheless, informatics systems can be used to generate data for making program decisions (Paneth-Pollak et al., 2009).

Developing and using a data dictionary, a list of the names of data elements to be collected, with definitions of those date elements, helps ensure a standardized definition and consistency in use (American Health Information Management

Association [AHIMA] e-HIM Work Group on EHR Data Content, 2006). Once the program objectives have been established, then a data dictionary can be created for the data elements needed to conduct the process and the effect evaluations. Thus, data elements in the data dictionary for the program range from specific expenditure items to participant demographic characteristics. For example, if participant race/ ethnicity is a data element, the dictionary defines the categories of race/ethnicity. The task of creating the data dictionary can feel tedious and mundane, but having it facilitates communication with the information officer or technology staff members, and minimizes having unreliable or missing data.

As the availability and flexibility of handheld electronic devices increases, so will their usefulness

216 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

be selected from a drop-down menu. Accuracy also depends on everyone entering data having the same understanding and definitions of the data being entered—for example, not having two people interpret “mostly followed instructions” in different ways.

Backing up data must not be ignored. Fol- lowing organizational procedures for saving and backing up data ensures that the data will not be lost. For small programs, the program manager needs to establish that policy for a frequent routine and then monitor whether the policy has been strictly followed by staff members.

▸ Across the Pyramid At the direct services level of the public health pyramid, participants experience the cost of the program, which becomes program revenue. Thus, good budgeting and creative generation of revenue from other sources may influence the level of participation because it reduces the out-of-pocket cost to participate in the health program. Program  participants should not be distracted by the presence of an information system or the data entry of their information into such a system. Programs may be intentionally focused on increasing the individual’s use of his or her own EHR as a health awareness and health promotion tool.

At the enabling services level, as with the direct services level, budgeting and information systems should be transparent and not noticeable. Budgeting for programs at the enabling services level includes the same key budgeting categories as with a direct services health program. Infor- mation systems at this level of the pyramid are more difficult to conceptualize and implement, and thus rely heavily on data from individuals and families receiving the enabling services.

At the population services level, budgets tend to be those of the state or federal govern- ments and earmarked for programs that serve the population. Large healthcare systems, such as Kaiser Permanente or the U.S. Department

in monitoring the implementation of a health program. The options and potential applications seem endless, from using cell phones for texting reminders about health behaviors to providing links to existing web-based health informa- tion. Mobile health (mHealth) and wearable health information sensing devices allow for continuous monitoring at the individual and population levels (Kumar et al., 2013). These can be particularly helpful to programs that are not connected to EHRs. In addition, social media websites can be used to recruit participants and to disseminate key health information across a population.

Health organizations continue to expand the use of a web-based system for entering data in real time via a form at a secure website. Online surveys have nearly replaced pen-and-paper questionnaires. Web-based systems have greater flexibility for generating multiple outputs. For example, a web-based system for tracking the amount of time staff members spent on given tasks can be used to determine the true costs of staff activities (James et  al., 2011). Mobile healthcare applications are becoming increas- ingly popular for monitoring and delivering healthcare interventions as well as collecting data. Advanced applications include monitor- ing vital signs and blood glucose levels (Baig, GholamHosseini, & Connolly, 2015). Program managers need to decide whether the size of the health program warrants the development of a web-based data collection system or whether use of paper forms is more appropriate given the intended program audience. The decision must fit the anticipated number of participants, the planned duration of the intervention program, and available resources.

Data entry issues are substantial. Accuracy at the point of data entry is paramount. The adage of “garbage in, garbage out” holds so true for health information. The responsibility for ensuring accurate data entry falls to the pro- gram manager. Accuracy depends on entering the data as intended and not having the cursor move, which would cause an incorrect answer to

217Internet Resources

of Veterans Affairs, may also have budgets for programs that address their enrollees. Nation- ally, various efforts are underway to utilize the EHR along with other data sources to conduct public health surveillance for sentinel events foreshadowing possible communicable disease outbreaks.

This chapter has dealt mostly with the infra- structure level of the public health pyramid. Being fiscally responsible in planning and monitoring a program has ramifications for receiving subse- quent program funding and for being viewed as credible by the program stakeholders.

▸ Discussion Questions and Activities

1. Speculate how fixed and variable costs might change for a health program at the direct services level, the enabling services level, or the population services level of the public health pyramid. Dis- cuss the implications of these changes on the results of a break-even analysis.

2. Imagine that Layetteville’s Lighthouse Agency has learned that the class lead- er will be available for only 6 months. It now needs to adjust its budget so that it will break even after the sixth class. It remains committed to having no more than 10 students per class. How would you go about generating a recommen- dation for the agency?

3. Create a data dictionary with at least seven data elements related to program implementation and five data elements related to the participants of an adult immunization health program being provided through community-based clinics. Which data elements were most difficult to define and why?

4. Imagine that your healthcare organiza- tion is starting a health program for dia- betes prevention. You have been charged

with leading a team to address the health information technology issues related to that program. What steps would you take? What would be your top three concerns?

▸ Internet Resources There is a wealth of online resources on develop- ing budgets and data systems. The following list represents diverse perspectives and approaches.

Break-Even Analysis Online and software calculators make break-even calculations much simpler. Dinkytown.net offers free trials for its financial calculators with options to purchase. Microsoft Excel comes with a built-in financial calculator (http://www.tvmcalcs.com /index.php/calculators/excel_tvm_functions /excel_tvm_functions_page1).

Public Health and Health Informatics For those with an interest in gaining an overview of the complex world of public health informatics, the CDC offers an introductory course on this subject (http://www.cdc.gov/publichealth101 /informatics.html). In addition, the Health Information and Management Systems Society (HIMSS) provides a wealth of information, including a resource library and professional development opportunities (www.himss.org).

Budget Justification Generating the budget justification for a grant is simplified by using the checklist and examples developed by the University of Colorado Boulder (http://www.colorado.edu/ocg/sites/default /files/attached-files/Budget%20 Justification%20 Checklist%2011%202014.pdf ). An example of a national budget justification is presented by the US Department of Health and Human Services (HHS) for the 2017 fiscal year on the public health and social services emergency fund (http://www.hhs.gov/sites/default/files /fy2017-budget-justification-phssef.pdf ).

218 Chapter 8 Monitoring Implementation Through Budgets and Information Systems

Kilmer, B., Burgdorf, J. R., D’Amico, E. J., Miles, J., & Tucker J. (2011). Multisite cost analysis of a school- based voluntary alcohol and drug prevention program. Journal of Studies on Alcohol and Drugs, 72(5), 823–832.

Kumar, S., Nilsen, W. J., Abernethy, A., Atienza, A., Patrick., K., Pavel, M., . . . Swendeman, D. (2013). Mobile health technology evaluation: The mHealth evidence workshop. American Journal of Preventive Medicine, 45(2), 228–236. doi:http://dx.doi.org/10.1016/j.amepre.2013.03.017

Lairson, D. R., Huo, J., Ball Ricks, K. A., Savas, L., & Fernández, M. E. (2013). The cost of implementing a 211 call center–based cancer control navigator program. Evaluation and Program Planning, 39, 51–56. doi:http:// dx.doi.org/10.1016/j.evalprogplan.2013.04.001

Lich, K. H., Urban, J. B., Frerichs, L., & Dave, G. (2017). Extending systems thinking in planning and evaluation using group concept mapping and system dynamics to tackle complex problems. Evaluation and Program Planning, 60, 254–264. http://dx.doi.org/10.1016/j. evalprogplan.2016.10.008

Paneth-Pollak, R., Schillinger, J. A., Borrelli, J. A., Handel, S., Pathela, P., & Blank, S. (2009). Using STD electronic medical record data to drive public health program decisions in New York City. American Journal of Public Health, 100(4), 586–590. doi:10.2105/AJPH.2009.175349

Petticrew, M., & Krishnaratne, S. (2014). The Fag Lady, revisited: Margaret Thatcher’s efforts on behalf of the tobacco industry. Public Health, 128(10), 904-910.

Shrestha, R. K., Clark, H. A., Sansom, S. L., Song, B., Buckendahl, H., Calhoun, C. B., . . . Heffelfinger, J. D. (2008). Cost-effectiveness of finding new HIV diag- noses using rapid HIV testing in community-based organizations. Public Health Reports, 123 (suppl 3), 94–100.

Tai, T., & Bame, S. I. (2011). Cost-benefit analysis of child- hood asthma management through school-based clinic programs. Journal of Community Health, 36(2), 253–260. doi:10.1007/s10900-010-9305-y

Tangka, F. K., Subramanian, S., Bapat, B., Seeff, L. C., DeGroff, A., Gardner, J., . . . Royalty, J (2008). Cost of starting colorectal cancer screening programs: Results from five federally funded demonstration programs. Prevention and Chronic Disease, 5(2), A47.

Terris-Prestholt, F., Kumaranayake, L., Obasi, A. I., Cleophas-Mazige, B., Makokha, M., Todd, J., . . . Hayes, R. J. (2006). From trial intervention to scale-up: Costs of an adolescent sexual health program in Mwanza, Tanzania. Sexually Transmitted Diseases, 33(10), S133–S139. doi:10.1097/01.olq.0000200606.98181.42

Yonas, M. A., Burke, J. G., Brown, S. T., Borrechach, J. D., Garland, R., Burke, D. S., & Grefenstette, J. J. (2013). Dynamic simulation of crime perpetration and reporting to examine community interventions. Health Education and Behavior, 40(10), 87S–97S. doi:10.1177/1090198113493090

References American Health Information Management Association

(AHIMA) e-HIM Work Group on EHR Data Content. (2006). Guidelines for developing a data dictionary. Journal of AHIMA, 77(2), 64A–64D.

Baig, M. M., GholamHosseini, H., & Connolly, M. J. (2015). Mobile healthcare applications: system design review, critical issues and challenges. Australasian Physical & Engineering Sciences in Medicine, 38(1), 23–38. doi:10.1007 /s13246-014-0315-4

Craglia, M., & Maheswaran, R. (2016). GIS in public health practice. Boca Raton, Florida: CRC Press.

Drummond, M. F., Sculpher, M. J., Claxton, K., Stoddart, G. L., & Torrance, G. W. (2015). Methods for the eco- nomic evaluation of health care programmes. Oxford, UK: Oxford University Press.

Finkler, S. A., Ward, D. M., & Baker, J. J. (2007). Essentials of cost accounting for health care organizations (3rd ed.). Sudbury, MA: Jones and Bartlett.

Forsh, D. A., Amanantullah, D. F., Coleman, S., & Wolinsky, P. R. (2014). The financial impact of the loss of county indigent patient funding on a single orthopedic trauma surgery service. Journal of Trauma and Acute Care Surgery, 76(2), 529-533.

Foster, E. M., Johnson-Shelton, D., & Taylor, T. K. (2007). Measuring time costs in interventions designed to reduce behavior problems among children and youth. American Journal of Community Psychology, 40, 64–81. doi:10.1007/s10464-007-9125-0

Hazzard, E. L., Moreno, E., Beall, D. L., & Zidenberg-Cherr, S. (2012). An evaluation of the California Instructional School Garden Program. Public Health Nutrition, 15(2), 85–90. doi:10.1017/S1368980011001248

Healthcare Information and Management Systems Society (HIMSS). (2016). Electronic Health Record (EHR). Retrieved from www.himss.org/ASP/topics_ehr.asp

Herstein, J. J., Biddinger, P. D., Kraft, C. S., Saiman, L., Gibbs, S. G., Smith, P. W., . . . Lowe, J. J. (2016). Initial costs of Ebola treatment centers in the United States. Emerging Infectious Diseases, 22(2), 350. doi:10.3201 /eid2202.151431

Hsu, C. E., Watson, K., Boekeloo, B., Shang, N., Metzger, C., & Downer, G. (2010). The use of emerging informatics techniques to evaluate the delivery of NMAETC pro- grams to address HIV and AIDS disparities. Journal of the National Medical Association, 102(12), 1116–1122.

James, P., Bebee, P., Beekman, L., Browning, D., Innes, M., Kain, J., . . . Waldinger M. (2011). Creating an effort tracking tool to improve therapeutic cancer clinical trials workload management and budgeting. Journal of the National Comprehensive Cancer Network, 9(11), 1228–1233.

Jerrell, J. M., & Hu, T. W. (1996). Estimating the cost impact of three dual diagnosis treatment programs. Evaluation Review, 20(2), 160–180.

219

© Lynne Nicholson/Shutterstock

Implementation Evaluation: Measuring Inputs and Outputs

Once the health program has started, stakeholders, funding agencies, and program staff members want to know if the program was implemented successfully and as planned. Answering this question becomes possible by devoting attention to its imple- mentation. This chapter covers techniques and issues related to documenting, monitoring, and evaluating the implementation of the program. These topics are addressed in relationship to the components of the program theory, including the process of creating data that can be used to assess the achievement of the process objectives established during the planning phase. Docu- menting and assessing the extent to which the process objectives have been achieved, and at what cost, are important aspects of providing a program. Maintaining a focus on the quality and quantity of implementation helps identify gaps between program accomplishments and process objective targets. As with the development of the program interventions and plan, obtaining input from recipients and stakeholders while gathering

and interpreting data enriches the understanding about the health program implementation.

▸ Assessing the Implementation

Questions that focus on elements of the organiza- tional and service utilization plans are essentially questions about implementation, rather than about effects of the program. These questions tend to fall into one of three levels of sophistication in regard to program implementation. Although slightly different in focus and emphasis, all three categories of questions are concerned with assess- ing the elements of the process theory portion of the program theory (FIGURE 9-1). Thus, the measures and data collection for each are likely to be quite similar—in fact, depending on the programmatic circumstances, they may even be the same. Therefore, this chapter covers these three types of questions in an integrated manner.

CHAPTER 9

220 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

FIGURE 9-1 Elements of the Process Theory Included in a Process Evaluation

Services utilization plan

inputs

Immediate health effects (outcomes)

Information system

Budget

Time line

Operations manual

Organizational chart

Coverage

Units of service

Service completion

Materials produced

Work flow

Participants

Queuing

Invervention delivery

Social marketing

Human resources

Informational resources

Monetary resources

Physical resources

Transportation

Managerial resources

Time resources

Organizational plan inputs

Outputs Outputs

Process Evaluation

Unfortunately, terms describing implementa- tion-focused questions are not used consistently in the literature. The following description of the three levels of questions provides a framework for understanding the various terms, such as process monitoring, program monitoring, process evaluation, and implementation monitoring.

Implementation Documentation Implementation documentation refers to the simple tallying of activities and processes carried out as implementation activities of the program. Questions of this sort represent the simplest level of sophistication. As simple documentation, this line of questioning does not lead to subsequent interpretations of what was documented and therefore does not prompt program managers to take actions to change or improve the program.

Implementation documentation certainly involves collecting the data specified in the process objectives. Accurate, complete, and timely implementation documentation is the foundation for the next levels of questions about implementation. In other words, although implementation documentation is necessary, it is not sufficient.

Implementation documentation is carried out in part to meet the requirements of fund- ing agencies—specifically, to demonstrate the extent of program implementation. Required reporting often entails a predetermined set of data that will need to be collected and used in the report to the funding agency. Thus, not all aspects of implementation documentation are under the control of the program administrators or evaluators. The data collection requested by the funding agency will ideally be consistent with and built into the process objectives.

Assessing the Implementation 221

Implementation Assessment The next level of sophistication focuses on implementation assessment. Implementation assessment is the ongoing, nearly real-time activity of collecting data about the implementation of the program for the purpose of making timely corrections or modifications to the implementa- tion through changes to elements of the process theory. Implementation assessment is called either program monitoring or process monitoring.

Process monitoring, the term used in this text to refer to this level of implementation assess- ment, is the ongoing, real-time assessment of the implementation of the program. It is integrated into the implementation of the program as a managerial or oversight tool to ensure that the intervention is being delivered within the parame- ters established during the planning stage. Ideally, process monitoring could lead to a favorable subsequent implementation evaluation.

Program monitoring is concerned with the elements described in this chapter. In addi- tion, some of its aspects are similar to quality improvement techniques. Each program is associated with a set of process objectives, and the implementation assessment activities are focused on achieving those objectives. This linkage between the implementation assessment activities and the process objectives keeps the monitoring activities reasonable in scope and focused on key or critical processes for having a successful program.

One purpose of implementation assessment is to provide managerial guidance and oversight of the implementation of the program. Program monitoring, like the quality improvement meth- odologies, can provide data on which to base changes or corrections in the delivery of the program. The interactive and iterative nature of planning and implementing a health program inevitably requires some degree of flexibility, particularly for programs that are continuous, ongoing, or repeated. This state of flux may be why community coalition program monitor- ing systems sometimes need to be modified (Chalmers et al., 2003).

The process monitoring information can inform decision making regarding which aspects of the organizational plan or the service utilization plan are ineffective in accomplishing the process objectives. The actualization of this flexibility and the willingness to make changes amount to corrective managerial practice. Through such actions, program managers address issues of accountability and quality. In addition, process monitoring can function as a warning system for problems in delivery of the program and can provide a basis for modifying the process theory for subsequent revisions of the program.

Implementation Evaluation The most sophisticated level of questioning comprises implementation evaluation. Im- plementation evaluation is a comprehensive, retrospective determination of the extent to which the program was delivered as designed and whether the variations might have had significant or important implications for the program effects. Implementation evaluation is generally called process evaluation. Process evaluation entails systematic research to assess the extent to which the program was delivered as intended. Thus, it is the systematic examination of programmatic coverage and delivery. Patton (2008, p. 324) defined process evaluation as focusing on “the internal dynamics and actual operations of a program in an attempt to under- stand its strengths and weaknesses.” Although the term is not used consistently in the literature, process evaluation is used in this text to refer to implementation evaluation.

Process evaluation has at least two purposes. One purpose is to gather data about the delivery of the interventions so that the results of an effect evaluation can be interpreted within the context of the program delivery. That is, if the intervention does not have effects on the health problem, it will be important to verify that the program was in fact delivered and to quantify the degree of intensity and faithfulness to the interventions design. Process evaluations are intended to demonstrate that program

222 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

Of the three terms, efficacy is probably the one most likely to be misused, and it is usually used instead of the term effectiveness. Efficacy is the maximum potential effect under ideal conditions. Because ideal conditions are diffi- cult to create, the efficacy of an intervention is determined through rigorous studies—usually clinical trials, especially randomized clinical trials. By controlling many of the potential influences, randomized clinical trials provide a context in which the greatest possible effect of a treatment or intervention can be directly attributed to that treatment or intervention. Because of the costs and ethical considerations involved in clinical trials, efficacy evaluations of health programs are seldom done unless they are performed as part of evaluation research. In efficacy studies, the role of a process evaluation is to establish that the health program intervention is being carried out precisely according to the interven- tion protocol.

Effectiveness is the realistic potential for achieving the desired outcome when a treatment or intervention is used in real life. The degree of effectiveness of an intervention reflects what can be expected given the normally messy situations that occur when health programs are delivered to real-world clients. Data from outcome assessments and outcome evaluations, as well as from evaluation research, provide

specifications are met and, as such, are useful for ensuring that the work being done by program staff members is consistent with the program’s process objectives and plan. Process evaluations help identify whether the implementation of the program contributed to  the program’s failure, as distinct from whether the effect theory was incorrect (FIGURE 9-2).

The second purpose of process evaluation relates to the dissemination or replication of the health program. Put simply, the process evaluation provides comprehensive operational information to the new sites so that the program can be replicated successfully.

▸ Efficacy, Effectiveness, and Efficiency

In evaluating programs, three key concerns are often whether the program was efficacious, effective, and efficient. These three terms are often used as synonyms, with minimal attention being paid to the important differences among the concepts. In fact, these differences have implications for the role of the process evalu- ation, and they reveal the underlying reasons for the different evaluation activities needed to quantify each.

FIGURE 9-2 Roots of Program Failure Weiss, C. H. (1998). Evaluation, (2nd ed.). Reprinted by permission of Pearson Education, Inc., Upper Saddle River, NJ.

Program interventions

set into

motion

which led toCausal process

Desired effect

Successful program

Program interventions

did not set into motion

which would

have led to Causal process

Desired effect

Process theory failure

Program interventions

set into

motion which did not lead to

Causal process

Desired effect

Effect theory failure

Quantifying Inputs to the Organizational Plan 223

seven categories of methods of data collection are appropriate for process evaluations: activity logs, organizational records, client records, ob- servations, questionnaires, interviews, and case studies (TABLE 9-1). This list of data collection methods does not preclude disciplined creativity in developing and using data collection methods or tools that are uniquely tailored to the pro- gram, including checklists (Simbar, Dibazari, Saeidi, & Majd, 2005). The use of activity logs, whether paper and pencil or web-based (Turner, Yorkston, Hart, Drew, & McClure, 2006), and organizational records tends to be more specific to process evaluation.

The choice of a data collection method needs to be congruent with the indicators in the process objectives and the best method for arriving at a conclusion about whether the process objective target was reached. Each method has its own advantages and disadvantages, all of which need to be considered carefully when choosing a data collection method.

For all data collection methods, high stan- dards must be met in terms of the quality of the data collected, the reliability of the tools, and the accuracy of data entry. The need for reliability extends to assessing and establishing an accept- able degree of interrater reliability in the use of checklists and activity logs. Having a reliable means of assessing delivery of the intervention is especially important when the consistency of intervention delivery is problematic or when the program is a pilot that is being evaluated as an experiment.

▸ Quantifying Inputs to the Organizational Plan

During program development, specific organi- zational resources are identified as being keys to implementing the health program. Both inputs and outputs of the organizational plan are included in an implementation evaluation (FIGURE 9-3), which determines the extent to

practical experience with the anticipated level of intervention effectiveness. The degree of ef- fectiveness may be reflected in several different statistics, depending on the evaluation design and methods. Any statistic that denotes the degree of difference related to having received the programmatic intervention—whether a difference score, a correlation coefficient, or an odds ratio—provides information on the degree of effectiveness. In this way, existing studies of the effectiveness of interventions provide a benchmark against which to gauge the success of subsequent or similar programs.

As a consequence, such studies have value beyond their specific findings. The role of a process evaluation in effectiveness studies is not only to ensure that the program is being provided according to protocol but also to document situations, events, and circumstances that influenced the delivery of the intervention.

Efficiency is, generically, the relationship between the amount of output and the amount of input, with higher outputs that are achieved with fewer inputs being deemed more efficient. With regard to health programs, efficiency is generally thought of in terms of the amount of effect from the program intervention—that is, the ultimate output—compared to the amount of inputs that went into providing the intervention. Data are collected as part of implementation documentation on the expenditures for all types of inputs into the program as well as on the outputs from the organizational and service utilization plans. Efficiency is then calculated as the cost per unit of output, where the unit of output is selected from the various outputs of the program. Using this basic notion of cost per output, efficiency could also be calculated per process objective.

▸ Data Collection Methods

Along with the question of what to measure is the question of how to collect data. At least

224 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

TABLE 9-1 Methods of Collecting Process Evaluation Data

Method When to Use Examples of

Measures Pros Cons

Activity log Have a list of actions that are discrete, and a common understanding for what those are, need quantitative data

Number of sessions, number of participants, time received inquiry phone calls, date of press release

Can tailor activity log to the program activities, easy to use, easy to analyze data, applicable across pyramid levels, can develop for use on handheld devices

May become too long, may not be completed on a regular basis, easy to falsify

Checklist Have a list of actions or behaviors that can be observed

Set up room for session, gave supportive comments, distributed program materials

Simple to use, can be developed to include a time frame, data entry is straightforward, applicable across pyramid levels, can develop for use on handheld devices

Difficult to narrow list items, challenging to write items for consistent interpretation, reliability needs to be established

Organizational records

Have existing records that capture information needed and can legally access those records, need quantitative data

Length of time on waiting list, number of computers bought or upgraded, number of hours worked

Accessibility to the information, applicable across pyramid levels, can be electronic databases

Need a data abstraction form, records may not include what is needed, may require complex data linking and data analysis

Client records Have existing records that capture information needed and can legally access those records, need quantitative data

Program attendance, client compliance with program elements

Accessibility to the information, likely to be electronic record

Need a data abstraction form if paper records are used, records may not include what is needed, may require complex data linking and data analysis

225Quantifying Inputs to the Organizational Plan

Method When to Use Examples of

Measures Pros Cons

Observation Need to have data on interpersonal interactions or sequences of events

Number of staff–participant interactions

Data may reveal unexpected results, naturalistic, can quantify observations, can be recorded on handheld devices

Time intensive, need observation checklist, complex data analysis

Questionnaire Need to collect data quickly from reliable respondents and have a reliable and valid questionnaire, need quantitative data

Degree of satisfaction with program, degree of compliance with program interventions

Can collect pencil-and-paper or web-based versions for many programs, applicable across pyramid levels

Respondent must have good reading skills and motivation to complete the questionnaire, will gather useless data if not well written, can be expensive for population-level programs

Interview Have time and need qualitative data or have respondents for whom questionnaire is not appropriate

Commitment of staff members to program and intervention

Able to get detailed descriptions during one-on- one interview, possibly new insights

Time intensive, need private place for the interview, need interview question and format, more complex data analysis

Case study Need to understand the full set of interactions around the program and the context in which it is functioning

Degree to which managerial personnel make changes to ensure fidelity of intervention

Gives very thorough picture of program and provides new insights

Extremely complex because it uses multiple methods over period of time, time intensive, very complex data analysis

226 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

A variety of measures can be used to measure each organizational plan input and output (TABLE 9-2), although the specific measures depend on the program and its objectives. At a minimum, the human resources and physical resources devoted to and utilized by the program must be monitored. These two organizational plan inputs are used as examples of considerations that would be involved in monitoring inputs. These discussions are not intended to be defin- itive but rather illustrative of how to approach the development of a process evaluation plan.

FIGURE 9-3 Examples of Organizational Plan Inputs and Outputs That Can Be Measured

Organizational Plan Inputs

Outputs

Informational Resources: Computers and network

capabilities, interpersonal networks, Internet accessibility

Personnel: Number of hires per position type, types of positions, licensure of staff, volunteer use

Monetary Resources: Revenue amount, stipulations on expenditures, budget

Physical Resources: Quantity and quality of specific items

needed, projected use

Transportation: Types available, projected frequency of use

Managerial Resources: Quality of leadership, delegation of

responsibility and accountability

Time Resources: Projected time line, funding constraints

Personnel: Levels of productivity, turnover rates, amount of training,

job satisfaction

Budget: Variance from projected, within category variance

Information System: Use ease and frequency, committee structures,

generation and distribution of reports

Physical Resources: Types used, frequency of use, changes made

Time Line: Variance from projected, changes to time line

Managerial: Operations manual, style, organizational structure, coalition formation

which those inputs were available and used and the quantity of outputs. Although every input could be monitored, tracking all of them would be neither prudent nor feasible. Each program has a set of concerns and interests with regard to the organizational plan inputs, as reflected in organizational plan process objectives. Thus, the choice of which organiza- tional plan inputs are evaluated is influenced by the earlier work of the program planners and the current concerns of program staff members.

Quantifying Inputs to the Organizational Plan 227

TABLE 9-2 Example of Measures of Inputs and Outputs of the Organizational Plan

Input Measures of Input Measures of Output

Human resources Number of full-time equivalents (FTEs), number of new hires, number of volunteers, percentage of licensed personnel, percentage of personnel with certification, educational level of staff members, hours of training and orientation

Number of hours worked, staff–recipient ratio, hours of client contact per staff member, degree of job satisfaction of staff members and volunteers, degree of commitment of staff members to program and intervention

Informational resources

Number of computers bought or upgraded, number of program recruitment efforts, availability of communication hardware and software, ease of process data entry and retrieval, ease of impact data entry and retrieval

Degree to which computer and telecommunication systems facilitate delivery of the intervention; availability and accessibility of personnel, budget, operating, meeting, or other reports

Monetary resources Amount of grant monies and donations, amount of indirect costs deducted from the program, number of proposals submitted for program funding

Dollar or percentage variance from budgeted per line item, number of grants awarded, profit or loss

Physical resources Number and type of capital equipment, number and type of office or clinical equipment, square footage of office space

Extent to which changes are made to physical resources needed for intervention delivery, replacement of aged equipment

Transportation Parking fees, total mileage per month, number of bus passes used, program vehicle expenses

Mileage per staff member, number of clients receiving transportation assistance, transportation cost per staff member or per program participant

Managerial resources Place in organizational chart, years of experience, educational level, degree of ability to communicate clearly and persuasively

Extent to which managers are viewed by staff members as controlling or delegating, degree to which managerial personnel make changes to ensure fidelity of intervention

Time resources Time line developed, presence of deadline dates

Number of days delayed, percentage of deadlines met, number of repeated requests

228 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

Human Resources Both the quantity and the quality of the human resources used by the health program ought to be assessed, given that human resources often constitute the largest cost component of a pro- gram budget. The importance of tracking the staffing levels of a program is reflected in the results of a study carried out by Rosenheck and Seibyl (2005). Using 5 years of data on staffing levels, these authors were able to find an asso- ciation between higher staffing levels and better outcomes for program participants. Similarly, nurse staffing levels have been associated with various indicators of inpatient quality, such as the number of falls and mortality rates (Minnesota Department of Health, 2015). This section does not present a definitive list of measures of human resource inputs but rather is meant to provide some guidance and ideas for monitoring related to human resources.

Not only is the quantity of personnel rele- vant, but those individuals’ level of commitment to the program, their competencies in terms of knowledge and skills, and their attitudes are also critical considerations. The degree of commitment to the program is important to assess, given that program staff members who lose commitment are less likely to implement the program fully as designed. A change in commitment levels may reflect unanticipated challenges to the delivery of the program. For example, the program staff members may develop lower levels of commitment because of obvious failures of the intervention or because of undesirable side effects of the program. Thus, monitoring the program staff members’ commitment to the program and to the inter- ventions is one avenue for gaining insights into program implementation.

The extent to which program personnel are competent to deliver the program also provides information on the extent of program imple- mentation. If licensed health professionals are required to deliver the intervention, but the pro- gram managers have insufficient fiscal resources to hire such individuals, then the program is not likely to be implemented as planned. Thus,

the extent to which the credentials of program staff members match the competencies needed for program implementation provides more evidence of the extent to which the designed interventions are delivered. This statement is not meant to imply that unlicensed personnel are not qualified. Indeed, many types of inter- ventions and health programs rely heavily and appropriately on a wide range of qualifications for program staff members. Rather, the issue is the extent to which the program staff members’ qualifications actually match what is required to implement the program fully and thus gain the maximum effect for program participants.

As the health program evolves over iterations of implementation, the qualifications needed by program staff members might change as well. These changing needs in relation to staff qualifications ought to be uncovered and noted through an implementation assessment or evaluation. The observed change may lead to a revision of the process theory for the program.

The participation of stakeholders continues into the process evaluation. The inclusion of volunteers and stakeholders as human resources in the organizational plan is intended to serve as a continual reminder that these individuals and groups need to be actively engaged in the various stages of the health program. Ideas from program staff members about ways to improve the program can be part of the process evaluation. Their ideas may help increase the effectiveness and efficiency of the program without compromising the integrity of the intervention as conceptualized. Their ideas may also provide insights into which aspects of the intervention were not implemented as intended. For example, staff members are very likely to pick up on and articulate the difference between what they were asked to do and what they are actually able to do with the given resources.

Another human resources input into many public health programs is the membership of a community consortium, coalition, or advisory board. The rationale for using community coalitions or consortia is based on the belief that the inclusion of such groups of individuals from the target population not only fosters the

Quantifying Inputs to the Organizational Plan 229

document, monitor, and evaluate human re- sources. The number of staff members per job category, the number of full-time equivalent (FTE) workers per job category, and the number of staff members with licensure and certifications are the absolute minimum data to be collected.

Physical Resources Documenting the extent to which facilities are adequate in terms of the types of rooms needed to provide the health program or the accessibility of equipment can be particularly relevant for some health programs, especially if specific physical resources are essential for the success of the program. Lack of or failure to use the necessary physical resources may indicate that the program was not fully implemented. For example, if a health promotion program includes both an educational component requiring only a classroom and a cooking demonstration component requiring a kitchen, then the process monitoring needs to include documentation of the use of both facilities. If no kitchen facil- ities were used for the program, then only the educational component of the program was implemented. Knowing that facility limitations were an issue because only one component of the program was implemented provides an explanation for a weak program effect on the stated health outcomes.

The simplest measure of physical facilities would be a dichotomous variable (i.e., yes/no) as to whether the facilities or equipment specified in the process objective were used. However, for physical resources such as a classroom, it may also be useful for process monitoring purposes to collect data on whether the room was heated, well lit, and so forth. Similarly, if physical re- sources include items such as blood pressure cuffs or supplies for vaccination, then it may be informative to document or monitor their placement, adequacy, expiration dates, disposal safety, and such. These seemingly mundane aspects of the physical resources can influence whether program staff members utilize those resources as intended.

development of culturally appropriate health programs but also influences the context of the program in ways that enhance the service utilization plan. Determining the extent to which this human resource was utilized may be mandated by various federal funding agencies. For example, as part of a Centers for Disease Control and Prevention (CDC) initiative to reduce racial and ethnic disparities across five health conditions, each grantee was required to report on the status of its community coalition. As part of a Maternal and Child Health Bureau program to reduce infant mortality, grantees had to have a community consortium and report on the functioning of that consortium.

Attendance records for community consortia or advisory board meetings are indicators of the amount of input of that type of human resource. The positions and organizations represented by the membership of the consortia or advisory board are other indicators of the human resources capability and diversity. Process monitoring data related to coalitions can reveal difficulties in mobilizing the community as a whole and elucidate the preferential interests of various groups. Process evaluation data help assess whether coalition activities have contributed any program effect. For example, members of a chronic disease prevention network reported needing to work with representatives of other sectors to address poverty-related issues effec- tively (Geneau, Legowski, & Stachenko, 2009). Such process information can be important for future coalition development.

Whereas commitment to and belief in the program and passion for the health problem are viewed as human resources inputs, job satisfac- tion is viewed as an output of the program. This distinction is important as a way to differentiate whether the correct inputs were obtained— namely, motivated individuals—from whether the human resources inputs were converted to satisfied staff members. The effort to screen and hire volunteers and staff members might also be important in this regard.

The preceding discussion hints that a wide variety of measures might be appropriate to

230 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

include the use of inter- and intraorganizational communication. Measures of interpersonal in- formation systems include the number of active committees; the number of community meetings attended by staff members; the perceived accuracy and timeliness of information received; and the number and types of modes of intraorganizational communication used, such as e-mail, memos, electronic mailing lists, and meetings.

Monetary Resources The monetary resources generated to support the health program are the most obvious out- put of the budget plan. Essentially, the output of budget-focused efforts is the  total dollars generated. Thus, the number and size of grants from philanthropic, federal, or state agencies, as highly valued measures, are counted. In addition, some health programs may be funded through contracts, which are also counted as outputs of the budgeting efforts.

▸ Quantifying Inputs to the Services Utilization Plan

Process evaluation typically is equated with ele- ments that are included in an evaluation of the services utilization plan—namely, data on the participants and program delivery (FIGURE 9-4). As mentioned earlier in regard to evaluating the organizational plan, the process objectives serve as a guide for deciding which possible inputs and outputs are crucial to evaluate. TABLE 9-3 provides some examples of measures that are useful for service utilization plan monitoring.

Participants and Recipients The most basic data about participants are a simple count of how many individuals inquired about the program or were served. However, other types of data about program participants are reasonable to collect.

▸ Quantifying Outputs of the Organizational Plan

Just as only key organizational plan inputs are evaluated, so only key organizational outputs can realistically be evaluated. Those outputs associated with process objectives are the minimum to be included in the evaluation. For some programs, it may be important to understand the organizational structure, such as where in the organizational hierarchy the health program director is located. The program director’s position in the organiza- tional chart can indicate the relative importance of the health program and hence the ability of the program manager to garner resources for the program. Here, two organizational plan outputs—information systems and budgets—are used as examples of different approaches to the measurement of organizational plan outputs.

Information Systems The ability to process data and generate needed reports is an output of the information systems. Given the central role played by information systems in acquiring program funds and doc- umenting program implementation and effects, the information system must be able to generate the needed reports. The quality and timeliness with which reports can be generated can be considered an output of the information system. Tracking the number of problems, requests, and complaints would be another approach to measuring the information system outputs.

Part of the process evaluation may include an assessment of the system’s capacity to handle data from the effect evaluation.

Measures of the information system output includes items such as staff members’ perception of ease of use, frequency of using critical databases, amount of time clients are on hold, number of calls waiting to be answered, number and frequency of reports generated, and perceived usefulness of the reports generated. Information systems also

Quantifying Inputs to the Services Utilization Plan 231

Some basic information about the health status of participants can be collected and can prove important for later analyses of program effect. Measures of health status information can range from the simple, such as a single question rating of overall health, to a checklist of health conditions, to findings from a detailed medical history and physical examination. The complexity of the health measure corresponds to the level of the public health pyramid at which the program is aimed and the target audience characteristics.

Intervention Delivery and Fidelity Intervention fidelity indicates whether the intervention was provided as designed and

FIGURE 9-4 Examples of Services Utilization Inputs and Outputs That Can Be Measured

Services Utilization Plan Inputs

Outputs

Social Marketing: Type, frequency, PSA use

Participants: Numbers, health status, characteristics

Queuing: Number and time on waiting list

Intervention Delivery: Quantity, fidelity to plan

Coverage: Under, over, efficiency index, program efficiency index

Social Marketing: Medical events and coverage

Intervention: Dosage received, satisfaction

Units of Service: Quantity, type

Service Completion: Quantity, type

Queuing and program logistics: Work flow, staff workload,

materials produced

A simple, straightforward documentation of demographic characteristics—whether through interviews, self-report questionnaires, or clinic records—is one means to collect this type of data. Demographic data are necessary for determining the extent to which the target audience has been reached and whether the so- cial marketing approaches were effective. These data can be readily collected during program sign-in or intake procedures, especially for di- rect and enabling services. Demographic data on recipients of population-focused programs may be more difficult to obtain directly given that population-based health programs do not require recipients to be physically present at a given location. For example, it would be very difficult to determine the number of recipients of health messages delivered via billboards.

232 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

self-reports, video observations, and audio re- cordings (Breitenstein et al., 2010). Failures in implementation fidelity can discredit potentially effective interventions (Dobson & Singer, 2005). This reinforces the notion that adequate and ongoing process monitoring can help prevent the program failure due to inadequate program implementation by detecting the source of the failure to have full intervention fidelity.

Intervention fidelity may be compromised in three ways. First, an intervention can fail because of the lack of a program (nonprogram, no treatment), meaning that the program was not provided. If the intervention requires a sophisti- cated delivery system, it may not be delivered. For example, a program that involves coordinating services among a variety of health professionals

TABLE 9-3 Examples of Measures of Inputs and Outputs of the Services Utilization Plan

Element Measures of Input Measures of Output

Program reach Number of requests for program Percentage undercoverage, percentage overcoverage

Participants Number of recipients or participants, number of persons denied or unqualified for the program

Efficiency index, program efficiency index, degree of satisfaction with the program

Queuing and program logistics

Number on waiting list, presence of system to move individuals from waiting list to program or alternative programs

Length of time on waiting list, evenness of work among staff members and across time (workflow), number and types of materials produced

Mazlan program based on social marketing efforts

Type of social marketing, quality of marketing, extent of social marketing analysis

Number of advertising events, number of requests for program based on social marketing efforts

Intervention Number of meetings to standardize program, extent of revisions based on previous cycle of intervention delivery, extent of revisions based on new research evidence

Fidelity to intervention plan, number of sessions, hours of program delivery, number of participants completing intervention (service completion), number of requests for additional program delivery, use of materials produced

planned; that is, it comprises the alignment of the intervention activities with the elements of the intervention theory. The concept of intervention fidelity applies to single, one-time programs as well as to standardized interventions adopted by numerous organizations. Unless high inter- vention fidelity is achieved, the program may fail to reach the desired outcomes. Achieving high implementation fidelity can be difficult for community-based organizations to achieve (Mihalic, Fagan, & Argamaso, 2008). Historically, relatively few evaluators of community-based health programs reported how they monitor implementation fidelity. Attention has been devoted more recently to the measurement of implementation fidelity in community-based prevention programs, including the use of

Quantifying Inputs to the Services Utilization Plan 233

Inconsistency in intervention delivery can also have a more insidious and hidden cause. Both program participants and program staff members have their own working explanations or theories of how a program affects participants. The espoused theory is the stated or espoused explanation for how things happen (Argyris, 1992). In a properly functioning program, the effect theory becomes known and understood by program staff members; therefore, it also should be the espoused theory. Of course, staff members inevitably know what they are supposed to say about the program interventions, regardless of whether they believe it or whether their actions match their espoused theory.

The actions that staff members take to achieve the ends constitute their theory-in-use—that is, the working theory that staff members actually use. One key aspect of process monitoring is to observe the theory-in-use because it provides insights into which intervention was provided to recipients, and then compare it to the intervention theory developed during the planning stage. It is especially important to observe the theory-in-use during the process monitoring and to compare it to the program intervention theory.

Naturally, some programs will be more prone to inconsistency than other programs. For example, the espoused theory and the theory-in-use among staff members in immunization clinics are likely to be quite congruent. In contrast, staff members working in gun violence prevention programs are more likely to demonstrate incongruence between the two theories because such a program entails complex interpersonal interactions and addresses socially sensitive problems.

Without standardized interventions, it is difficult to connect the health effects of the program to the program interventions. For this reason, the process evaluation of the intervention needs to take into account the possible sources of inconsistency in how the program intervention is delivered. Findings from the program evaluation may be confusing or misleading if the espoused theory and the theory-in-use are incongruent. Implementation of the program may be incon- sistent, with some staff members providing the

employed by different agencies can easily result in no program. An alternative way for no treatment to occur is that the delivery of the intervention in some way negates the intervention. Having program personnel who are not supportive, encouraging, or empathetic would negate in- tended program interventions that require such qualities in staff members. Similarly, physical resources—for example, a building with limited and difficult access—might negate interventions for persons with disabilities.

The second way that an intervention can fail is if an intervention other than the one designed and planned is provided, or if the dosage is drastically reduced. If an educational program is designed to cover five topics but only one topic is covered, despite holding the classes for the designated number of hours, the program will be dramatically diluted, possibly to the point of having no effect on participants. The process objectives that specify the program content and quantity can be used to assess whether the strength (dosage) of the intervention was delivered as intended. It is also possible that the intervention theory was flawed; if so, the inade- quacy of the intervention as conceptualized will lead to intervention failure, although the actual program may be delivered as planned.

The third way that an intervention can fail is if the intervention is provided in an inconsistent manner, resulting in a nonstandardized treatment over time or among recipients. A standardized intervention is necessary to ensure that the intervention is responsible for the outcomes. If program personnel use their own discretion to alter the intervention, there is no assurance that the intervention as planned is responsible for the health effects found in an outcome evalua- tion. One approach to minimize this potential source of program failure is to incorporate the program theory in the training of program personnel, thereby ensuring that they appreciate the need to follow the guidelines for delivery of the program. Of course, the organizational plan outputs of policies, procedures, and standardized materials need to be in place and used as tools to help standardize the program.

234 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

The service utilization plan outputs dis- cussed here are coverage, units of service, service completion, workflow, and materials produced. These outputs are assessed with regard to the extent that corresponding process objectives have been achieved.

Coverage as Program Reach The extent to which the program reached its intended audience is called coverage, but it may also be referred to as reach. Coverage, in the context of state or federal health policy and subsequent health programs, has become a topic of political debate, which increases the importance of understanding coverage measures and interpretations. Studies of programs have identified attention to coverage as important to understanding why the programs did not have the expected effects (Gottfredson et al., 2006) and for determining which program effects were associated with coverage (Macinko, de Souza, Guanais, & Simoes, 2007). Means used to track program coverage have included documentation, monitoring, and evaluation. If the coverage rates are less than expected or predicted, then it is important to identify the barriers to accessing the program, which may entail a qualitative ap- proach (Sobo, Seid, & Gelhard, 2006). There are several ways to quantify and interpret coverage.

Measures of Coverage Monitoring the degree of participation in a health program is a basic aspect of process evaluation. All funding agencies and program managers want assurances that the program had participants. The mechanism for track- ing the number of individuals served by the program must be in place before the process evaluation begins. Measures of coverage require having accurate data on the number of pro- gram participants within a given time period. Collecting data on a frequent or periodic basis allows for ongoing monitoring and still makes possible the aggregation of the numbers to get totals for a given time period. For example, if

program according to the espoused theory and other personnel delivering the program based on their theory-in-use.

The inconsistencies among the three theo- ries (intervention theory, espoused theory, and theory-in-use) can be a source of decision drift. Although a decision may be made as a milestone in the planning process, over time the decision can evolve in unexpected ways among planners and program staff members. Decision drift is a natural process in long-term health programs, but it can also occur within a short span of time—even within a single meeting. Decision drift is detrimental only if it results in a program that lacks coherence or that no longer addresses the health problem. Through process evaluation, the extent to which decision drift has occurred can be assessed, and the specific areas in which it occurred can be pinpointed. Recognizing that decision drift has occurred provides a basis for either revising the decisions or modifying the program elements and objectives to bring them more in line with the current decisions. Which action to take depends on whether the decision drift has resulted in a more or less efficient and effective health program.

▸ Quantifying Outputs of the Services Utilization Plan

During a process evaluation of each of the services utilization plan outputs, one of the first decisions to be made is which time frame will be used. Because many health programs are vulnerable to seasonal fluctuations, the time frame used is often highly significant. For some programs, annual measures are more reasonable; for shorter programs, the end of the program is sufficient. Ideally, the evaluation of the services utilization plan outputs should occur as close to real time as possible rather than being retrospective to ensure that programmatic changes are made in a timely fashion.

Quantifying Outputs of the Services Utilization Plan 235

for coverage measures. A narrative example of coverage measures is shown in EXHIBIT 9-2, with the corresponding data in an Excel spreadsheet shown in EXHIBIT 9-3.

Undercoverage is measured as the number of individuals who are in need of the service and actually received the service divided by the number of individuals who are in need. It occurs when the program is not delivered to a large portion of the target audience. On the other side of the coin, overcoverage occurs when the program is being used by individuals not in the target audience. It is calculated as the number of individuals who are not in need of the service but who receive the service divided by the number of those who receive the service. Overcoverage can put a strain on existing human and physical

immunization clinics are offered three times per month, managers may count the number of participants per clinic and then add the totals for all three clinics to obtain a total number of persons served per month.

Coverage is assessed with regard to un- dercoverage and overcoverage. Data from the needs assessment are required regarding the numbers of individuals in need and not in need of the program. These data, along with the actual number of individuals served by the program, form a matrix of undercoverage, ideal coverage, and overcoverage (TABLE 9-4). Calculating the different measure of coverage is simple, yet it can yield a great deal of information about which program component needs attention due to undercoverage. EXHIBIT 9-1 lists the formulas

TABLE 9-4 Matrix of Undercoverage, Ideal Coverage, and Overcoverage

Persons Not Served by the Program

Persons Served by the Program

Persons Not in Need of Program

Ideal coverage Overcoverage

Persons in Need of Program Undercoverage Ideal coverage

EXHIBIT 9-1 Formulas for Measures of Coverage

Percentage of overcoverage number not in need

5   but served

number served

Percentage of undercoverage number in need an

5 dd served

number in need

Coverage e�ciency 5 number in intended populat

(number served − number served but not in need) iion

236 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

EXHIBIT 9-2 Example of Narrative Background about Coverage and Dosage Measures

Last year, Bowe County funded programs that address the top five health problems in Layetteville. The community needs assessment data revealed the number of individuals or families in need of each program. Of the 1,000 persons on whom assessment data were collected, 300 were at risk for diabetes, 200 were in need of adult immunizations, 600 adolescents were at risk for violence, 250 women were at risk for congenital anomalies, and 250 adolescent girls were sexually active. The county requires that each program provide annual reports on its coverage efficiency and weighted program dosage average. The grant contract also stipulates that the programs must meet at least 75% of their coverage efficiency target, and the program intervention dosage must average at least 80%.

Each program set a target value for its coverage efficiency objective. The Diabetes Prevention Program chose 90%. The Adult Immunization Program set its target at 98%, and the Adolescent Violence Prevention Program estimated it could achieve 70% coverage. The Adult Immunization Program considered full dosage to mean that an adult received three vaccines; the program had 200 persons receive the first vaccine, 190 persons receive the second vaccine, and 160 persons receive all three vaccines. The Congenital Anomalies Prevention Program set its coverage efficiency at 85%; that program served 200 women, of whom 100 were at risk and needed the program; an additional 100 women who were at risk did not receive the program.

Using the formulas in Exhibit 9-1, three of the programs provided the coverage information as shown in Exhibit 9-3, and one program provided the intervention dosage information shown in Exhibit 9-4 [this exhibit is shown later in this chapter]. The county is reviewing the data provided by these programs, and it intends to use the data to make recommendations to neighboring counties that have expressed interest in funding similar programs.

EXHIBIT 9-3 Examples of Coverage Measures Using an Excel Spreadsheet A B C D E F G H I J

1

2

3 Not in

Program In

Program Total 90

4

5

Already diagnosed 650 50 700 Over-coverage 20% C4/C6

At high risk 100 200 300 Under-coverage

Over-coverage

33% B5/D5

6 Total 750 250 1000 Coverage Efficiency 67% (C6-C4)/D5

7 74% (C5/D5)/H3*100 8 9

10 No Vaccine Got

Vaccine 98

11 Not need vaccine 790

Total

5% C11/C13

12 Need vaccine 10 190 200 5% B12/D12

13 Total 800 200 1000 Coverage Efficiency 95% (C13-C11)/D12

14 97% (C12/D12)/H10*100 15 16

17 Non-

participant In

Program Total

70

18 Low risk, no need 150 250 400 50% C18/C20

19 High risk, has need 350 250 600 58% B19/D19

20 Total 500 500 1000 Coverage Efficiency 42% (C20-C18)/D19

21 60% (C19/D19)/H17*100 22

Diabetes Prevention Program

Adult Immunization Program

Adolescent Violence Prevention Program

Formulas for Calculating Coverage

Coverage Efficiency Objective, Target% :

Coverage Efficiency Objective, Target% :

Coverage Efficiency Objective, Target% :

% Coverage Efficiency Target Achieved

% Coverage Efficiency Target Achieved

% Coverage Efficiency Target Achieved

Over-coverage

Under-coverage

Under-coverage

10 800

Quantifying Outputs of the Services Utilization Plan 237

think in terms of intended populations and recipients. The number of individuals in need of the program—specifically, the size of the intended population—should have been deter- mined through the community assessment. If the community needs assessment data do not include some estimation of the size of the in- tended audience, it becomes almost impossible to determine whether the program is dealing with undercoverage or overcoverage. At best, the level of coverage can be estimated only from alternative data sources.

Units of Service For some programs, it will be important to distinguish between the number of program participants and the number of contacts. Outreach programs may be quite successful in making a high number of contacts with potential program participants, but the number of individuals who actually show up for the program may be a small fraction of that number. In this case, a decision must be made about whether the number of individuals contacted is equivalent to the number of outreach recipients. This is a gray area, and each program will have a slightly different answer to this question. The key is to make clear the item being counted and the definition on which it is based. In particular, the process objective should specify what is being counted.

The unit of service (UOS) must be clearly defined and articulated before the count can begin. A UOS is a predetermined unit, such as number of contact hours, number of individual clients seen, number of educational sessions offered, or size of caseload. This measure is used primarily for programs at the direct or enabling services levels of the public health pyramid. For some programs, the UOS is specified by the funding agency, in which case, data for that UOS become a minimum of what is included in the process evaluation of services delivered.

resources (Osuch et al., 2015). Assessments of under- and overcoverage can be used to deter- mine how to improve a program (Countdown Coverage Writing Group, 2008) and ultimately to justify its continuance.

A more telling indicator is coverage effi- ciency. Coverage efficiency is calculated as the number served minus the number served but not in need divided by the number in target population. If there is no under- or overcoverage, then the coverage efficiency is 100%. As shown in Exhibit 9-3, it is difficult to reach that level of coverage efficiency.

The coverage efficiency index can be applied to the program to create the program efficiency index. The program coverage efficiency index is calculated by summing the efficiency indices for each program component and then dividing this sum by the number of program components.

Although these measures of coverage provide information about the extent to which the program is reaching the intended audience, they do not provide information on the extent to which the program is meeting its objectives. To make this determination, another calculation is needed. That is, the percentage of coverage efficiency achieved can be compared to the percentage coverage efficiency specified as the objective target value (see Exhibit 9-3). This calculation uses the numbers from the under- and overcoverage matrix and the target value set in the objective. It represents another way for program managers and staff members to determine whether the program is reaching its objectives.

High coverage results indicate that the program has achieved good marketing and program recruitment. Taken as a set, coverage measures indicate the extent to which efforts to enroll individuals in the program are effective and the target audience is being reached. They also indicate areas warranting managerial action and further tailoring of the program.

When thinking about the sources of data needed to measure participation, one must

238 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

of participation might be a simple yes or no on attendance at, say, an immunization clinic, or it might comprise a weighted calculation of the per- centage of time during a set of sessions in which the participant was actively engaged with the group. The measure chosen depends on the nature of the program and the target value and indicator specified in the process objective. A low level of participation can result if program personnel are not skilled in engaging participants (a managerial issue) or if the intervention is not appealing to the participants (a process theory issue).

The dosage received by a program participant can be calculated if accurate and complete data have been collected on each participant and the quantity of intervention received. Continuing with the congenital anomalies prevention program in Layetteville, EXHIBIT 9-4 shows the dosage for three women. Suppose that the program has four components—screening, prenatal counseling, use of prenatal vitamins, and a cooking class—and each component has a different planned amount specified in the intervention theory and the corresponding process objectives. If the amount of each component received by each woman is recorded, then the program dosage can be calculated for each woman. Using the program average dosage for each component, a weighted average dosage can then be calculated. The advantage of calculating the weighted average dosage is that it takes into account any variation in planned dosage for components.

The main challenge in calculating the dosage is the collection of data in a manner that enables its calculation. If they know the program average per program component, the program manager and planners can make more informed decisions about program modifications and changes, such as which components need revision or additional support.

Satisfaction Measurement Participant satisfaction with the program is an element of process evaluation and not a health outcome of the program. This perspective stands in contrast to the general vernacular of classifying

Participant-Related Issues Dosage Measurement Of the five dosage elements (frequency, duration, strength, route of administration, and adminis- tration credibility), frequency and duration have the greatest relevance for ongoing monitoring. Frequency of the intervention—whether hourly, daily, weekly, or monthly—and duration of the intervention—whether one session, 8 weeks of classes, or 6 months of exposure—are the elements of dosage that may vary from the objectives laid out in the plan because of either program or participant factors. Thus, to ensure that the planned dosage was received by the participants, it is important to determine the degree to which program participants completed the health program—in other words, service completion. The inverse of service completion is the dropout rate of the program.

Singular event programs, such as screening clinics, are likely to have different completion rates than programs with longer-term involve- ment with the participants, such as substance abuse counseling, Meals on Wheels, or exercise classes. For some enabling services, service com- pletion is the achievement of the service plan or care plan. Counting the number of participants who have completed their service plan provides different information on the extent to which the intervention was implemented and the dosage received by the average participant. Service plan completion can be estimated only if staff mem- bers keep good records of enrollment and record participant attendance on an individual basis.

Drastic changes in completion rates may signal problems with program staff members or with the design of the program. The process objectives should include a threshold for what is an acceptable program completion rate.

Level of participation is a corollary of service completion. If the intervention or impact theories are predicated on a certain level of participation, then data on the level of participation need to be collected as a means of determining whether participants received the appropriate “dose” of the intervention. For some health programs, level

Quantifying Outputs of the Services Utilization Plan 239

Martinez, Cosentino, & Pronk, 2010). What is received, what is expected, and how one is treated are all elements of the service utilization plan rather than parts of the effect theory of how the intervention leads to health changes.

Client satisfaction may not predict program outcomes for individuals. For example, the associations between satisfaction with provid- ers and use of preventive services, such as flu vaccinations and colorectal cancer screening, was actually explained by the patient’s charac- teristics (Jerant, 2014). Nonetheless, satisfaction with health services provides some insight as a

satisfaction as an outcome. The rationale for thinking of satisfaction as a process output stems from the definition of satisfaction: the degree to which participants receive what they expect to receive and the extent to which their expectations are met with regard to how they are treated (Para- suraman, Zeithaml, & Berry, 1985). Similarly, the Expectancy Confirmation Model describes patient satisfaction as the difference between patient experience and patient expectation (Lo, 2014; Oliver, 2014). Thus, program satisfaction can be affected by the degree which the partici- pants experience beneficial effects (VanWormer,

A B C D E F 1 2 3 Participant A Screening Session 2 2 100% 4 Prenatal counseling Session 4 4 100% 5 Prenatal vitamins Weeks 52 43 83% 6 Cooking class Hours 8 7 88% 7 8 Participant B Screening Session 2 2 100% 9 Prenatal counseling Session 4 3 75% 10 Prenatal vitamins W eeks 52 32 62% 11 Cooking class Hours 8 4 50% 12 13 Participant C Screening Session 2 1 50% 14 Prenatal counseling Session 4 2 50% 15 Prenatal vitamins W eeks 52 20 38% 16 Cooking class Hours 8 1 13% 17 18 Average Amount Avg Percent 19 Screening Session 2 1.7 83% 20 Prenatal counseling Session 4 3.0 75% 21 Prenatal vitamins W eeks 52 31.7 61% 22 Cooking class Hours 8 4.0 50% 23 W eighted average dosage 84.2% 24

25 Screening Session 2 (e3+e8+e13)/3 (f3+f8+f13)/3

26 Prenatal counseling Session 4 (e4+e9+e14)/3 (f4+f9+f14)/3

27 Prenatal vitamins Weeks 52 (e5+e10+e15)/3 (f5+f10+f15)/3

28 Cooking class Hours 8 (e6+e11+e16)/3 (f6+f11+f16)/3

29 30

Program Average

Program Component

(((F19*D19) + (F20*D20) + (F21*D21) + (F22*D22)) / D19 + D20 + D21 + D22) /100

Pla nned Amount

Amount Receive d

Percentage Receive d

Unit

W eighted average dosage (assuming all participants have the same planned amount)

Formulas for Ca lculating Program Ave rage, Based on 3 Participa nts

EXHIBIT 9-4 Examples of Calculating Dosage for the Congenital Anomalies Prevention Program Using Excel

240 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

The last difficulty in measuring satisfac- tion concerns the scope of what is important in terms of satisfaction. For example, inpatient satisfaction questionnaires generally include items about parking and food service alongside items about the courtesy of staff members (Gesell, 2001; Mostyn, Race, Seibert, & Johnson, 2000). If such items are included, their relevance to the organizational and service utilization plans should be explicit.

Manary, Boulding, Staelin, & Glickman (2013) suggest taking the following into consideration when developing measures of satisfaction: whether and which specific visit/ program participation measures correlate with which outcome measures, the nature of the interactions between recipient and provider, promptness of data collection after receiving the service or program, statistically adjusting for potential moderators and mediators, and the difficulty in a defining “patient satisfaction” (Manary et al., 2013).

Program Logistics Workflow Interaction inevitably occurs between the pro- cedures used for managing waiting participants and the amount of work done by program staff members. Measures of workflow are one indica- tor of the amount of work done by the program staff members and the queuing of participants. Examples of workflow measures include minutes that participants wait to be seen, number of days between signing up for the program and beginning the program, number of days between being referred to the program and being accepted into the health program, and amount of time required for program staff members to complete a specific task. Of course, the amount of work done by program staff members is influenced by the volume of program participants and the rate at which they participate in the program. For direct services health programs, the volume and queuing greatly affects the workflow of program staff. For population-based programs, the level of

process indicator of the experience of program participants.

Most funding agencies are interested in knowing the satisfaction level of participants in the programs they are funding. Most participants, clients, and patients report being somewhere in the range “satisfied to very satisfied” in regard to the services they receive, perhaps because it is difficult to measure satisfaction in a way that does not lead to a ceiling effect. A ceiling effect occurs when the measurement tool is constructed so that respondents do not have an opportunity to distinguish among levels at the high end of the scale. It becomes apparent when the item has a high mean value and very low standard devia- tion. In other words, the distribution becomes highly skewed to one side. For example, if the satisfaction scale is a 5-point Likert-type scale, the difference between 4 (“somewhat satisfied”) and 5 (“very satisfied”) will lead to 5 being chosen more often than 4. One remedy for avoiding the ceiling effect is to use a scale of 1 to 10, which then allows respondents to distinguish among 7, 8, 9, and 10 as levels of satisfaction.

Other difficulties exist in developing a measure of satisfaction. One challenge is to create a measure that is culturally sensitive and appropriate to subgroups of participants. Fongwa, Hays, Gutierrez, and Stewart (2006) specifically created a measure for African Americans but caution that the measure needs testing with other ethnic groups.

Another challenge is that self-report responses on a questionnaire may not be the same as re- sponses in an interview. Marcinowicz, Chlabicz, and Grebowski (2007) found a discrepancy in this regard, with more negative responses being offered by patients when interviewed compared to their questionnaire responses.

Yet another difficulty stems from the con- ceptual definition of satisfaction, which entails a match between expectations and experience. Developing a measure that captures both expec- tations and actual experience will be longer, more complex to complete, and more challenging to analyze. For these reasons, the use of an existing satisfaction measure is highly recommended.

Across the Pyramid 241

be considered in the process evaluation. Having data about the materials provides insights into the work done by staff members and the extent to which the intervention was delivered in the manner planned. One managerial insight that has emerged from studies tracking the produc- tion of materials is that resources are sometimes directed more toward materials than toward implementation of the intervention. Data about production of materials can be difficult to obtain and will be very program specific.

▸ Across the Pyramid Across the public health pyramid, process monitoring and evaluation focus on the inputs and outputs of the organizational and service utilization plans—albeit tailored to the specific program, of course (TABLE 9-5). For programs at

cooperation from others, such as media represen- tatives, may affect the workflow of program staff, such as in delivery of a mass media campaign.

Data related to both volume of participants and workflow come from a variety of sources. Observations of program staff members, participant records, appointment logs, class sign-in sheets, and billing statements are common sources of these data. For some programs, planners may decide to develop specific data collection forms. If this step is taken, the program staff members should be involved in the process because using an empowerment approach to planning increases the likelihood that optimal measures will be developed and used by the staff members.

Materials Produced Both the quantity and the quality of the mate- rials produced for the health program need to

TABLE 9-5 Examples of Process Evaluation Measures Across the Public Health Pyramid

Direct Services Enabling Services

Population Services Infrastructure

Organizational Plan Input

Provider credentials, location

Provider credentials, physical resources (e.g., cars)

Provider credentials, managerial resources

Personnel qualifications, managerial resources, fiscal resources

Organizational Plan Output

Protocols and procedures for service delivery, data about individual participants

Protocols and procedures for service delivery, data about participants

Protocols and procedures for service delivery

Budget variance, fiscal accountability, data, management information systems

Service Utilization Plan Input

Wait times, characteristics of participants

Wait times, characteristics of participants

Characteristics of the population

Characteristics of the workforce

Service Utilization Plan Output

Measures of coverage

Measures of coverage

Measures of coverage

Materials produced, number of participants

242 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

the direct services level, data measure units of service, such as number of individuals served and number of contact hours with individuals. Such measures are consistent with the nature of health programs designed for the direct services level of the pyramid. Many measures of patient satisfaction relate to the direct services. For in- stance, the Visit-Specific Questionnaire assesses perceptions of the convenience and technical competence of a specific encounter (Ware & Hays, 1988), whereas the Client Satisfaction Questionnaire measures overall satisfaction with a specific healthcare provider (Atkinson & Zibin, 1996). The Consumer Assessment of Healthcare Providers and Systems® includes sets of items relating to perceived access to care and communication among healthcare providers.

At the enabling services level, process monitoring and evaluation indicators are likely to be similar to those used at the direct services level, but they are modified to reflect the specific program and the use of different sources of data.

At the population-based services level of the public health pyramid, as at the direct services and enabling services levels, program process evaluations and monitoring efforts ought to address inputs and outputs of the organizational and service utilization plans. This is true if the program is implemented at the community level. For example, Glick, Prelip, Myerson, and Eilers (2008) focused on documenting the extent to which a fetal alco- hol syndrome prevention campaign reached the intended audience. The units of service measured at this level of the pyramid could include number of individuals served, number of agencies involved, or number of households reached.

At the infrastructure level, process mon- itoring and process evaluation focus on the program infrastructure. If the health program is designed for one of the other levels of the pyramid, then the infrastructure becomes the source of the inputs and outputs of the organizational and service utilization plans. Of course, a program may be designed and intended to change the infrastructure itself. For example, Pearson and colleagues (2005) conducted a process evaluation of the imple- mentation of the chronic care model across

42 organizations, focusing on the changes made at the sites to implement the program. Infrastructure changes may not always lead to the intended outcomes. An evaluation of the Substance Abuse and Mental Health Services Administration’s incentive grant program to nine states for infrastructure changes concluded that infrastructure changes did not result in better service changes and consumer outcomes (Leff, Chchocki, Chow, & Lupton, 2014). Infrastructure units of service could comprise the number of employees involved, and outputs might consist of the number of policy or procedure updates and job satisfaction, especially if the employees are considered inputs into the program.

▸ Discussion Questions and Activities

1. Involvement of community coalitions and consortia in the implementation of health programs has become widespread. What would be possible and appropriate measures or indicators of having imple- mented community coalitions or consor- tia as part of the program delivery?

2. What would you suggest as methods and techniques to avoid the failure of interventions? Justify your ideas in terms of the various ways that interven- tions can fail.

3. To obtain accurate measures of cover- age, which information systems and data collection methods need to be in place? Which steps can ensure that these ele- ments are put in place in a timely manner?

4. Process monitoring and process evalua- tion data are useful only when they are interpreted correctly and subsequently used to make program changes. Outline a plan, with actions and stakeholders, for increasing the likelihood that the process data will contribute to accurate and responsible program delivery.

5. Using the information in Exhibit 9-2, calculate the measures of coverage for

243References

References Argyris, C. (1992). On organizational learning. Cambridge,

MA: Blackwell Publishers. Atkinson, M. J., & Zibin, S. (1996). Evaluative review of

quality of life instruments, Chapter 5 in Quality of life measurement among persons with chronic mental illness: A critique of measures and methods. Retrieved from http://www.phac-aspc.gc.ca/mh-sm/mentalhealth /pubs/quality_of_life/

Breitenstein, S. M., Gross, D., Garvey, C., Hill, C., Fogg, L., & Resnick, B. (2010). Implementation fidelity in community-based interventions. Research in Nursing & Health, 33(2), 164–173. doi:10.1002/nur.20373.

Chalmers, M. L., Housemann, R. A., Wiggs, I., Newcomb-Hagood, L., Malone, B., & Brownson, R. C. (2003). Process evaluation of a monitoring log system for community coalition activities: five-year results and lessons learned. American Journal of Health Promotion, 17(3), 190–196.

Countdown Coverage Writing Group. (2008; on behalf of the Countdown to Core). Countdown to 2015 for maternal, newborn, and child survival: The 2008 report on tracking coverage of interventions. The Lancet, 371(9620), 1247–1258. doi:http://dx.doi.org/10.1016 /S0140-6736(08)60559-0

Dobson, K. S., & Singer, A. R. (2005). Definitional and practical issues in the assessment of treatment integrity. Clinical Psychology: Science and Practice, 12(4), 384–387. doi:10.1093/clipsy.bpi046

Fongwa, M. N., Hays, R. D., Gutierrez, P. R., & Stewart, A. L. (2006). Psychometric characteristics of a patient satisfaction instrument tailored to the concerns of African Americans. Ethnicity & Disease, 16, 948–955.

Geneau, R., Legowski, B., & Stachenko, S. (2009). An inter- sectoral network for chronic disease prevention: The case of the Alberta Healthy Living Network. Chronic Diseases in Canada, 29(4), 153–161.

Gesell, S. B. (2001). A measure of satisfaction for the assisted-living industry. Journal of Healthcare Quality: Promoting Excellence in Healthcare, 23, 16–25.

Glick, D., Prelip, M., Myerson, A., & Eilers, K. (2008). Fetal alcohol syndrome prevention using community-based narrowcasting campaigns. Health Promotion Practice, 9, 93–103. doi:10.1177/1524839907309044

Gottfredson, D., Kumpfer, K., Polizzi-Fox, D., Wilson, D., Puryear, V., Beatty, P., Vilmenay M. (2006). The Strength- ening Washington D.C. Families Project: A randomized effectiveness trial of family-based prevention. Prevention Science, 7(1), 57–74. doi:10.1007/s11121-005-0017-y

Jerant, A. (2014). Satisfaction with health care providers and preventive care adherence: A national study. Medical Care, 52(1), 78.

Leff, S., Cichocki, B., Chow, C. M., & Lupton, C. (2014). Infrastructure change is not enough: An evaluation of SAMHSA’s mental health transformation state incentive grants. Psychiatric Services, 65(7), 947–950. http://dx.doi .org/10.1176/appi.ps.201300324

the Congenital Anomalies Prevention Program, specifically the percentage overcoverage, percentage undercover- age, and coverage efficiency. Of the var- ious programs for which coverage data are available, which programs seem to be most efficient?

6. Using the information in Exhibit 9-2, calculate the dosage for the Adult Im- munization Program for 150 people who needed and received the full vaccination.

▸ Internet Resources Workbook for Designing a Program Evaluation The CDC has developed a workbook for program managers for a range of evaluation topics (http:// www.cdc.gov/obesity/downloads/cdc-evalua tion-workbook-508.pdf ) and on writing pro- cess evaluation questions (https://www.cdc.gov /healthyyouth/evaluation/pdf/brief4.pdf )

United Nations Children’s Fund (UNICEF) UNICEF has created a one-page tool that includes key questions to ask as part of a process evaluation: www .unicef.org/lifeskills/index_10489.html#Process %20indicators%20for%20the%20programme.

Children, Youth, and Families at Risk (CYFAR) CYFAR, through a cooperative agreement with the University of Minnesota, offers instructional videos on program evaluations as well as process and outcome evaluations at https://cyfar.org /resource-core-competencies/program-evaluation.

World Health Organization (WHO) Process Evaluation Handbook This handbook presents a lot of content in a simple and straightforward manner, with lots of examples and links to tools: http://whqlibdoc .who.int/hq/2000/WHO_MSD_MSB_00.2e.pdf.

244 Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs

for future research. Journal of Marketing, 49, 41–50. doi:10.2307/1251430

Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

Pearson, M. L., Wu, S., Schaefer, J., Bonomi, A. E., Shortell, S. M., & Mendel, P. (2005). Assessing the implementa- tion of the chronic care model in quality improvement collaboratives. Health Services Research, 40, 978–996. doi:10.1111/j.1475-6773.2005.00397.x

Rosenheck, R. A., & Seibyl, C. L. (2005). A longitudinal perspective on monitoring outcomes of an innovative program. Psychiatric Services, 56, 301–307. http://dx.doi .org/10.1176/appi.ps.56.3.301

Simbar, M., Dibazari, Z. A., Saeidi, J. A., & Majd, H. A. (2005). Assessment of quality of care in postpartum wards of Shaheed Beheshti Medical Science Univer- sity Hospitals, 2004. International Journal of Health Care Quality Assurance, 18, 333–342. http://dx.doi .org/10.1108/09526860510612180

Sobo, E. J., Seid, M., & Gelhard, L. R. (2006). Parent-identified barriers to pediatric health care: A  process-oriented model. Health Services Research, 4, 148–171. doi:10.1111/j .1475-6773.2005.00455.x

Turner, C., Yorkston, E., Hart, K., Drew, L., & McClure, R. (2006). Simplifying data collection for process evaluation of community coalition activities: An electronic web-based application. Health Promotion Journal of Australia, 17(1), 48–53. http://dx.doi.org/10.1071/HE06048

VanWormer, J. J., Martinez, A. M., Cosentino, D., & Pronk, N. P. (2010). Satisfaction with a weight loss program: What matters? American Journal of Health Promotion, 24(4), 238–245. doi:10.4278/ajhp.080613-QUAN-92

Ware, J. E., & Hays, R. D. (1988). Methods for measuring patient satisfaction with specific medical encounters. Medical Care, 26(4), 393–402.

Lo, J. Y. (2014). Measuring patient satisfaction in cancer care. Doctoral dissertation, The University of Wisconsin– Madison. Madison, WI.

Macinko, J., de Souza, M., Guanais, F. C., & Simoes, C. (2007). Going to scale with community-based primary care: An analysis of the family health program and infant mortality in Brazil, 1999–2004. Social Science and Medicine, 65, 2070–2080. doi:10.1016/j.socscimed.2007.06.028

Manary, M. P., Boulding, W., Staelin, R., & Glickman, S. W. (2013). The patient experience and health outcomes. New England Journal of Medicine, 368(3), 201-203.

Marcinowicz, L., Chlabicz, S., & Grebowski, R. (2007). Open-ended questions in surveys of patients’ satisfaction with family doctors. Journal of Health Services Research and Policy, 12, 86–89. doi:10.1258/135581907780279639

Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Im- plementing the Life Skills Training drug prevention program: Factors related to implementation fidelity. Implementation Science, 3, 5. doi:10.1186/1748-5908-3-5

Minnesota Department of Health. (2015). Hospital Nurse Staffing and Patient Outcomes: A Report to the Minnesota Legislature. St. Paul, Minnesota, MN: Division of Health Policy/Health Economics Program.

Mostyn, M. M., Race, K. E., Seibert, J. H., & Johnson, M. (2000). Quality assessment in nursing home facilities: Measur- ing customer satisfaction. American Journal of Medical Quality, 15, 54–61. doi:10.1177/106286060001500203

Oliver, R. L. (2014). Satisfaction: A behavioral perspective on the consumer (2nd ed.). New York, New York: Routledge.

Osuch, E. A., Vingilis, E., Summerhurst, C., Forster, C. I., Ross, E. E., & Wrath, A. J. (2015). Process evaluation of an early-intervention program for mood and anxiety disorders among older adolescents and young adults. Psychiatric Services, 66, 1113–1117. doi:10.1176/appi .ps.201400256

Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1985). A conceptual model of service quality and its implications

245

© Lynne Nicholson/Shutterstock

Program Quality and Fidelity: Managerial and Contextual Considerations

This chapter reviews the current trends and approaches to managing healthcare organizations that collectively form the context within which a health program must function. The trends and approaches described here are important for the influences they exert on healthcare organizations. The assumption is that health programs are based in a wide range of types of healthcare organizations, such as federal agencies, state and local health departments, non- profit and for-profit health systems, community- based nonprofit organizations, church-affiliated organizations, and international relief and assis- tance organizations. Commonalities exist across these organizations, with many being subject to the same legal constraints and obligations. Most healthcare organizations experience similar pressures to ensure and document the quality of health care provided, and all face a growing need for electronic information systems. The content of this chapter is intended to illuminate that broader

healthcare environment as a contextual influence that managers and planners of health programs must navigate to achieve high program quality and intervention fidelity.

Of primary importance is accountability, and thus the chapter begins with a discussion of this topic. It is followed by a review and synthesis of the currently pervasive approaches used by healthcare organizations to achieve and maintain healthcare quality. Most health programs need to fit within the broader quality system used by the healthcare organization. Processes dealing with ensuring a desired level of quality can lead to change efforts intended to remedy a quality problem or enhance the existing quality of the program. Whether on a small or large scale, managers and directors of health programs need to draw on their knowledge of and skills at facili- tating group change. For this reason, the chapter includes a review of key group process concepts that are related to making changes happen.

CHAPTER 10

246 Chapter 10 Program Quality and Fidelity: Managerial and Contextual Considerations

▸ The Accountability Context

Accountability and responsibility are cornerstones for program implementation. Accountability means being answerable for actions taken and the subsequent success or failure of the pro- gram. Responsibility means being charged with ensuring that things are done, and done within the specified parameters. Program managers are generally both accountable for the program and responsible for seeing that the program is carried out. Accountability with regard to program imple- mentation fits within the twin realms of program accountability and professional accountability.

Program Accountability Program managers are accountable for the pro- gram in six areas, as summarized in TABLE 10-1. Each area of accountability requires some thought, planning, and oversight. In other words, through careful attention to the organizational plan and the services utilization plan, each type of accountability can be achieved.

Two types of accountability relate to the organizational plan: efficiency and legal. Effi- ciency accountability means that the program is delivered with efficient use of the resources. Legal accountability means that staff member act in accordance with local, state, and federal laws and within their professional licensure limits.

Two types of accountability relate to the services utilization plan: coverage and service delivery. Coverage accountability relates to the program reaching the intended recipients; it is documented with the calculation procedures described elsewhere in the text. Service delivery accountability comprises the extent to which the intervention is provided as planned. It is indicated by not only the number of units of service provided but also by the number of times that the program intervention protocol was not followed or by the number of changes made to the intervention.

One type of accountability relates to the effect theory: impact accountability. Impact

accountability is concerned with the program having intended outcomes on the intended audi- ence and recipients. The indicators for impact accountability are highly tailored to reflect the effect theory of the program.

Professional Accountability Professional accountability refers to an individual from a health profession being bound by the corresponding professional norms and codes, including the moral and ethical codes related to serving the public interest. Members of the mul- tidisciplinary team may be licensed and required by law to practice within those boundaries. These professional ethics and norms generally provide a broader, more encompassing, and not as well- codified set of rules to guide professional behavior. Professional accountability becomes important to programs in a variety of ways that can influence the program planning or implementation, with its concerns typically falling into the realm of either personal professional accountability or generic public health professional accountability.

Personal professional accountability relates to the extent to which an individual is performing according to professional standards and norms. This mandate creates the need for managerial support and possible supervision to ensure that professionals are held accountable for meeting their professional standards. Failure to perform according to professional standards should be rare among the program staff members. If it does occur, however, a program manager should seek help from the human resources department to resolve the issue.

The more complex concept of generic public health professional accountability is concerned with the extent to which the program addresses social justice and disparity problems. Given that program budgets are always tight, with corre- spondingly specific program eligibility criteria usually established, some individuals in need of the program and who would participate may potentially not be accepted into the program. This situation has the potential to create an ethical dilemma that puts professional accountability

Performance and Quality: Navigating the Interface 247

at odds with the program limitations. In other words, public health professionals who have social justice as an element of their professional accountability may view the program as being at odds with their professional accountability. The takeaway message is that wise program planners and managers never lose sight of the relevance of professional accountability in their program and take that issue into consideration during the planning stages.

▸ Performance and Quality: Navigating the Interface

Health programs exist within the complex and constantly evolving healthcare system, which is affected by fads and managerial fashions as much as by new scientific evidence. One fad

TABLE 10-1 Types of Program Accountability, with Definitions and Examples of Process Evaluation Indicators (Source: Adapted from Rossi, Lipsey, and Freeman 2004.)

Accountability Type

Definition: The Extent to Which Examples of Indicators

Organizational Plan Related

Efficiency Resources are utilized without waste or redundancy

Dollars spent on the program, cost per client served, cost per unit of outcome

Fiscal Resources are managed according to the budget

Existence of receipts and bills paid, number of errors found during annual audit, percentage variance from budget

Legal Legal, regulatory, and ethical standards are met

Number of malpractice suits, number of investigations, number of personnel with current licensure

Service Utilization Plan Related

Coverage The target population is reached

Coverage efficiency, coverage bias (differing participation across subgroups)

Service delivery The intervention is provided as planned

Number of units of service provided, number of breaches of intervention protocol, number of modifications to intervention

Effect Theory Related

Impact Participants change or are changed because of the intervention

Very program-specific health and behavior indicators

248 Chapter 10 Program Quality and Fidelity: Managerial and Contextual Considerations

Quality Improvement Approaches Continuous quality improvement (CQI) (Juran, 1989) was adopted by healthcare organizations as a tool to reduce costs while improving the quality of services. By the 1990s, CQI had be- come a popular way to enhance organizational effectiveness and was commonplace in healthcare organizations (Shortell et al., 2000). CQI is based on the premise that problems are best addressed through attention to the system as a whole and that employees are the best source of possible solutions. For the most part, these approaches focus on examining organizational processes using statistical and other scientific tools. The explosion of attention to quality in health care has led to the development of other packaged approaches, such as Six Sigma and Lean meth- odology. These proprietary systems are similar because they focus on making organization-wide changes to improve both the quality of services and the overall performance of the organization. Health program staff members may become in- volved in or may become affected by the actions of those in the organization who are applying these approaches.

In addition to systems theory, CQI has ori- gins in engineering and operations management. Over time, CQI has evolved into more generic ongoing processes for assessing the inputs into key organizational processes that influence the use of resources and influence patient outcomes. As ongoing organizational processes, quality improvement efforts are conducted by standing quality improvement committees that include employees who are directly involved in the processes being addressed.

Quality Imp