Need help with this assignment
saramaidaa
Evidence_Based_Decision_Making_in_Public_Health.pdf
86
J Public Health Management Practice, 1999, 5(5), 86–97 © 1999 Aspen Publishers, Inc.
HE MANY accomplishments of modern public health can be lauded, including the 30-year gain in life expectancy in the United States during this century. Much of
Evidence-Based Decision Making in Public Health
Ross C. Brownson, James G. Gurney, and Garland H. Land
Ross C. Brownson, PhD, is Professor of Epidemiology and Chair of the Department of Community Health at the Saint Louis University School of Public Health and directs the Prevention Research Center at Saint Louis University, in St. Louis, Missouri.
James G. Gurney, PhD, is an Assistant Professor in Pediatrics and an Adjunct Assistant Professor in Epidemiology at the University of Minnesota, Minneapolis, Minnesota.
Garland H. Land, MPH, is the Director of the Center for Health Information Management and Epidemiology of the Missouri Department of Health, Jefferson City, Missouri.
The authors are grateful for the contributions from the other mem- bers of the Work Team on Evidence-Based Decision Making in Public Health: H. Denny Donnell, Michael Kramer, Wayne Schramm, Eduardo Simoes, Joseph Stockbauer, and Mark Van Tuinen. This project was supported in part by the Centers for Dis- ease Control and Prevention (CDC) cooperative agreement #U82/ CCU712891, awarded to the Missouri Department of Health, and CDC cooperative agreement #U48/CCU710806, awarded to the Saint Louis University School of Public Health.
As noted in this article, a course entitled Evidence-Based Decision Making in Public Health is taught for public health practitioners in Missouri. Copies of the syllabus and readings for the course are available from Dr. Brownson ([email protected]). Information on other information system strategies that are being implemented within the Missouri Department of Health is available from Mr. Land.
A stronger focus on evidence-based
decision making in day-to-day public
health practice is needed. This article
describes the rationale for this need,
including (1) the inter-relationships
between evidence-based medicine and
evidence-based public health (EBPH);
(2) commonly used analytic tools and
processes; (3) keys to when public
health action is warranted; (4) a
strategic, six-step approach to more
analytic decision making; and (5)
summary barriers and opportunities
for widespread implementation of
EBPH. The approach outlined is being
tested through a series of courses for
mid-level managers in the Missouri
Department of Health—initial results
from a pilot test are encouraging. It is
hoped that the greater use of an
evidence-based framework in public
health will lead to more effective
programs.
Key words: data, decision making, epidemiology, public health practice, surveillance
T this increase can be attributed to the provision of safe water and food, sewage treatment and disposal, to- bacco use prevention, injury prevention, control of infectious diseases through immunization, and other population-based, public health interventions.1 De- spite these successes, many additional public health challenges remain. To meet and exceed expectations for continuing improvement, a drive for more wide-
spread use of evidence-based strategies for effec- tively addressing current challenges in public health is needed.
Ideally, public health practitioners always incor- porate scientific evidence in making management decisions, developing policies, and implementing programs. However, in reality, these decisions often are based on short-term demands rather than long- term study, and policies and programs are developed frequently around anecdotal evidence. These con- cerns were noted a decade ago when the Institute of Medicine (IOM) determined that decision making in public health often is driven by “…crises, hot issues, and concerns of organized interest groups.”2(p.4) In addressing these issues, many factors may lead to a more evidence-based approach to decision making, including enhanced individual skills, wider use of data and analytic tools, and a more favorable organi- zational climate.
In this article, the authors describe: (1) the inter- relationships between evidence-based medicine (EBM) and evidence-based public health (EBPH), in- cluding examples from contemporary public health practice; (2) commonly used analytic tools and pro- cesses; (3) keys to when public health action is war- ranted; (4) a strategic approach to decision making that currently is being tested in Missouri; and (5) summary barriers and opportunities for widespread implementation of EBPH. A major goal of this discus- sion is to move the process of decision making to- ward a pro-active approach that incorporates effec- tive use of scientific evidence and data.
Inter-Relationships between EBM and EBPH
Evidence-based medicine
The concept of EBM has grown in prominence in recent years.3,4 EBM involves the delivery of optimal individual patient care through the integration of current best evidence on pathophysiological knowl- edge, cost effectiveness, and patient preferences. Necessary EBM skills include the ability to track down, critically appraise, and rapidly incorporate scientific evidence into a clinician’s practice. Key steps in the EBM process5 include the abilities to:
1. convert information needs into answerable questions
2. track down, with maximum efficiency, the best evidence with which to answer these questions (from the clinical examination, the diagnostic laboratory, the published literature, or other sources)
3. critically appraise that evidence performance for its validity (closeness to the truth) and use- fulness (clinical applicability)
4. apply the results of this appraisal in clinical practice
5. evaluate performance
Uses of evidence in public health decision making
The authors define EBPH as the development, implementation, and evaluation of effective pro- grams and policies in public health through applica- tion of principles of scientific reasoning including systematic uses of data and information systems and appropriate use of program planning models. In EBPH, the most viable approach to a public health problem is chosen from among a set of rational alter- natives. This process relies on several related disci- plines including epidemiology, biostatistics, behav- ioral sciences, health economics, and health care management. Any process or method that is estab- lished should recognize that public health practitio- ners often have substantial administrative duties; therefore, EBPH must be time efficient. Two ex- amples from contemporary public health practice help to illustrate decisions based on varying degrees of evidence.
In the field of tobacco control, decades of research and thousands of epidemiologic studies have estab- lished cigarette smoking as the “leading cause of pre- ventable premature death”6(p.10) Economic studies have shown that increased tobacco taxes are an im- portant tool for decreasing tobacco consumption.7 To address the issue, California voters passed an ear- marked tobacco excise tax in 1988.8 California raised the excise tax on cigarettes by 25 cents per pack and placed an initial tax of 42 cents on other tobacco products, with the rate on other tobacco products adjusted annually by the State Board of Equalization. This effort launched one of the most intensive and aggressive public health interventions ever under- taken.9 This excise tax and media campaign were ef- fective in sharply accelerating the drop in both sales of cigarettes and in smoking (for 1988–1993, which
Evidence-Based Decision Making in Public Health 87
88 JOURNAL OF PUBLIC HEALTH MANAGEMENT AND PRACTICE/SEPTEMBER 1999
was double the rate expected based on the 1974– 1987 trend).
A second example involves the implementation and evaluation of a Missouri program designed to decrease motor vehicle injuries and deaths among children. In addressing this issue, the Missouri legis- lature enacted a child restraint law in 1984. The law required children younger than age 4 to travel in an approved safety seat in the front seat of a vehicle or in either a safety seat or seatbelt in the back seat. Even eight years after enactment, compliance was estimated at only 50 percent.10 In response to con- cerns from health care providers and safety organiza- tions, the Missouri Department of Health began the Take a Seat, Please! (TASP) Program in 1992, mod- eled after a similar program in Virginia.10 For the TASP program, volunteers were provided with busi- ness-reply postcards on which they could report the license plate number of a vehicle in which a child was not restrained properly. The owner of the ve- hicle then was sent a letter from the health depart- ment stating that the recipient had been observed breaking the law, where and when the observation had taken place, information on child passenger safety, and program information through a toll-free number. Two years after implementation, the TASP program was evaluated through a telephone survey of participants and observational studies in child care centers.10 The findings showed little evidence of program effectiveness; therefore, it was discontinued in September 1995. Similar approaches have been adopted in at least 15 other states with little support- ing evidence of effectiveness.
Contrasting EBM and EBPH
There are important distinctions between EBM and EBPH. First, the quality and volume of evidence differ. Medical studies of pharmaceuticals and pro- cedures often rely on randomized controlled trials, the most scientifically rigorous of epidemiologic studies. In contrast, public health interventions fre- quently rely on cross-sectional and quasi-experimen- tal designs lacking any “true” comparison group, which may limit the quality of the evidence. During the past 50 years, there have been approximately 1 million randomized controlled trials of medical treatments.3 There are fewer studies of the effective- ness of public health interventions. Second, public
health studies often encounter a longer time period between intervention and outcome. An intervention to reduce smoking may have the ultimate outcome of reducing lung cancer deaths, yet it would take de- cades to evaluate this long-term endpoint. Third, the formal training of persons working in public health is much more variable than that in medicine. Unlike medicine, public health relies on a variety of disci- plines and there is not a single (or even small number of) academic credential(s) that “certifies” a public health practitioner.
In light of these issues, two key questions are: (1) What are some useful tools and processes for evaluat- ing public health evidence? and (2) When is evi- dence sufficient for public health action?
Analytic Tools and Processes for Evaluating Evidence
Several important tools and processes are avail- able to practitioners to assist in determining when public health action is warranted. This section pro- vides a very brief overview of five of these.
Meta-analysis
Meta-analysis is a quantitative approach that pro- vides a systematic, organized, and structured way of integrating the findings of individual research stud- ies.11,12 In a meta-analysis, the study results become the unit of analysis, with the goal of identifying con- sistent patterns and sources of disagreement among results.13 Meta-analysis has been used increasingly during the past two decades to synthesize the find- ings of multiple independent studies and has been called “…possibly the most important policy-related research method that has developed in the past two decades.”14(p.229) Detailed descriptions of how to con- duct meta-analysis are described in detail else- where.11,14
Risk assessment
Quantitative risk assessment is a widely-used term for a systematic approach to characterizing the risks posed to individuals and populations by environ- mental pollutants and other potentially adverse ex- posures.15,16 Risk assessment has been described as a “bridge” between science and policy making17 and it has become an established process through which expert scientific input is provided to agencies that
Evidence-Based Decision Making in Public Health 89
regulate environmental or occupational exposures. In the United States, its use is required either explic- itly or implicitly by a number of federal statutes and its application worldwide is increasing. There has been considerable debate over the U.S. risk assess- ment policies and the most widely recognized diffi- culties in risk assessment are due to extrapolation- related uncertainties, specifically, extrapolating low-dose health effects from higher exposure levels.
Economic evaluation
Economic evaluation, commonly through cost-ef- fectiveness studies, should be an important compo- nent of evidence-based decision making.18 These methods provide information to help assess the rela- tive appropriateness of expenditures on public health programs and policies. Cost-effectiveness compares the net monetary costs of an intervention with some measure of health impact or outcome (for example, years of life saved).19,20
An example of a cost-effectiveness analysis in rela- tion to a public health intervention was shown by Hatziandreu et al,21 who assessed the benefits of regular exercise among a cohort of 1,000 35-year-old men. They estimated that regular exercise would re- sult in 78 fewer coronary heart disease deaths and 1,138 quality-adjusted life years (QALY) gained in this cohort. The cost per QALY was favorable com- pared with other preventive or therapeutic interven- tions. In public health practice, a continuing chal- lenge is the difficulty in measuring cost effectiveness for community-based interventions because cost data often are not reported and indirect costs (for ex- ample, lost work productivity) are difficult to mea- sure.
Public health surveillance
Public health surveillance involves the ongoing, systematic collection, analysis, and interpretation of outcome-specific health data, which are closely inte- grated with the timely dissemination of these data to those responsible for preventing and controlling dis- ease or injury.22 A viable surveillance system can provide a wealth of valuable information for deci- sion making in public health. Public health surveil- lance systems should have the capacity to collect and analyze data, disseminate data to public health pro- grams, and regularly evaluate the effectiveness of the use of the disseminated data.23 For example, docu-
mentation of the prevalence of elevated levels of lead (a known toxicant) in blood in the U.S. population has been used as the justification for eliminating lead from gasoline and for documenting the effects of this intervention.24
Expert panels and consensus conferences
Most government agencies, in both executive and legislative branches, and voluntary health organiza- tions utilize expert panels when examining scientific studies based on explicit criteria and determining their relevance to health policies and interventions.25
Ideally, the goal of expert panels is to provide scien- tific peer review of the quality of the science and sci- entific interpretations that underlie public health recommendations, regulations, and policy decisions. When conducted well, peer review can provide an important set of checks and balances for the regula- tory process. One of the successful outcomes of ex- pert panels has been the production of guidelines for preventive medicine.26 In related work, the Council on Linkages between Academia and Public Health Practice has concluded that “the potential benefits of public health practice guidelines are immediate and far reaching.”27,28(p.5) These recommendations have helped to stimulate a current effort to develop a Guide to Community Preventive Services.29 This guide will document the effectiveness of a variety of population-based interventions in public health through systematic review and evaluation of the sci- entific evidence.
Consensus conferences are a related mechanism commonly used to review epidemiologic evidence. Expert panels can take time (sometimes years) to de- velop their recommendations, while the consensus panel commonly must make decisions within a con- ference held over two and one half days. Thus, the “consensus” of many consensus panels occurs in the middle of the night to meet a deadline imposed by the conference.30
A viable surveillance system can provide a wealth of valuable information for decision making in public health.
90 JOURNAL OF PUBLIC HEALTH MANAGEMENT AND PRACTICE/SEPTEMBER 1999
Determining When Evidence Is Sufficient for Action
One of the key and difficult issues in public health is determining when evidence is sufficient for public health action (in the form of a specific program or policy) and what the specific action should entail. It may be useful to consider two levels of evidence. The first involves analytic data that shows the impor- tance of a particular health condition and its link with some preventable risk factor. For example, a large body of research shows that the burden of childhood illness and death can be reduced signifi- cantly through widespread immunization.
Factors important to consider include the condi- tion’s magnitude (number, incidence, or prevalence), severity (morbidity, mortality, or disability), and pre- ventability (what can be done to prevent the health condition). This type of evidence, level one evi- dence, may lead one to the conclusion that “some- thing should be done.” The second level of evidence focuses on the relative effectiveness of specific inter- ventions to address a particular health condition. For example, in relation to immunizations, which is a more effective and cost-effective strategy: provider feedback and reminders, parental reminders and education, or school and daycare immunization laws? This type of evidence, level two evidence, points the practitioner toward the conclusion that “specifically, this should be done.”
Whether considering level one or level two evi- dence, the impetus for action is strengthened by con- sistent findings from a series of well-conducted stud- ies. When considering such a series, the practitioner should examine each individual study for its strength of design and execution (internal validity) as well as its ability to generalize findings to other populations (external validity). In the end, public health deci- sions must integrate the full array of considerations regarding risks and benefits of different courses of action. These considerations should not be limited to the scientific evidence but should also include social, cultural, economic, and political factors.
A Sequential Framework for Practitioners
To foster an enhanced approach to EBPH, the Mis- souri Department of Health convened a work group
in 1996 with the main goal of developing a sequen- tial framework and training course for busy public health practitioners that would lead to greater use of evidence in day-to-day decision making. This sec- tion briefly describes the six-stage EBPH process de- veloped by the workgroup that the authors believe will be useful in addressing a variety of public health program and policy decisions. It is important to note that this process is seldom a strictly prescriptive or linear one, but should include numerous feedback “loops” and may benefit from strategies outlined in other program planning models.31,32 Through this it- erative process, the authors have noted that an addi- tional benefit of EBPH is the potential to build multidisciplinary teams that engage practitioners in problem solving. As with any analytical process, practitioners should maintain a “healthy skepti- cism” in approaching an issue or problem.
Stage one: develop an initial, concise, operational statement of the issue
The practitioner should begin by developing a concise statement of the issue or problem being con- sidered. To build support for any issue (with an orga- nization, policy makers, or a funding agency), the is- sue must be articulated clearly. In many senses, this problem definition is similar to the beginning steps in a strategic planning process, which often involves describing the mission, internal strengths and weak- nesses, external opportunities and threats, and the vision for the future.33,34 The key components of an issue statement include the health condition or risk factor being considered, the population(s) affected, the size and scope of the problem, prevention oppor- tunities, and potential stakeholders.
The following example is provided for an issue commonly encountered in public health practice— infant mortality.
Background/Public Health Issue. Based on epide- miologic data, the rate of infant mortality in state X has decreased by 5 percent during the past five years. Despite this trend, the rate remains among the high- est in developed countries.
Programmatic Issue. The state health department has been charged by the governor with developing a plan for reducing the rate of infant mortality. This plan must be developed within six months and implemented within 12 months.
Evidence-Based Decision Making in Public Health 91
Solutions Being Considered. Program staff, policy makers, and advisory groups have proposed numer- ous solutions, including: (1) increased funding for family planning services, (2) a mass media campaign to encourage better prenatal care, and (3) global poli- cies that are aimed at increasing health care access for pregnant women.
Stage two: determine what is known through the scientific literature
After the issue to be considered has been defined clearly, the practitioner needs to become knowledge- able about previous or ongoing efforts to address the issue. This should include a systematic approach to identify, retrieve, and evaluate relevant reports on scientific studies, panels, and conferences related to the defined topic of interest. The most common method for initiating this investigation is a formal lit- erature review. There are many databases available to facilitate such a review. Most common among da- tabases for epidemiology and public health purposes are MEDLARS, MEDLINE, PubMed, Current Con- tents, HealthSTAR, and CancerLit. These databases can be subscribed to by an organization, can be found selectively on the Internet, or sometimes can be ac- cessed by the public through institutions (such as the National Library of Medicine, the Combined Health Information Database, universities, and public librar- ies). There also are many organizations that maintain Internet sites that can be useful for identifying rel- evant information including many state health de- partments, the Centers for Disease Control and Pre- vention (CDC), and the National Institutes of Health (NIH). The methods for conducting a formal litera- ture search can be found elsewhere.11,35,36
After relevant articles and reports have been iden- tified and retrieved, an evaluation of the information should be conducted. Depending on the specific pur- pose of the review, this may take the form of a sys- tematic analysis and synthesis11,35,36 Such a synthesis can be primarily qualitative, in which the analytic results are reviewed carefully with respect to the va- lidity of the studies, the generalizability of the results beyond the study populations, and the applicability of the findings in the context of the specific problem definition that originated the review. As described earlier, the synthesis also can be quantitative, in the form of a meta-analysis.
Stage three: quantify the issue
After developing a working understanding of the current state of knowledge regarding the public health problem of interest through the literature re- view, a second type of search usually is warranted, one in which sources of existing data are identified. Such descriptive data may be available from ongoing vital statistics data (birth or death records), surveil- lance systems, special surveys, or from national studies.
Descriptive studies can take several forms. In pub- lic health, the most common type of descriptive study involves a survey of a scientifically valid sample (a representative cross section) of the popula- tion of interest. These “cross-sectional” studies are not intended to change health status (as an interven- tion would), but rather, they serve to quantify the prevalence of behaviors, characteristics, exposures, and diseases at some point (or period) of time in a defined population. This information can be valu- able for understanding the scope of the public health problem at hand. Descriptive studies commonly pro- vide information on patterns of occurrence accord- ing to such attributes as person (for example, age, gender, ethnicity), place (such as county of resi- dence), and time (for example, seasonal variation in disease patterns). Additionally, under certain cir- cumstances, cross-sectional data can provide data for use in the design of analytic studies and can be used as baseline data to compare the effectiveness of pub- lic health interventions.
Analytic studies are designed to evaluate specific scientific hypotheses, although they too may serve descriptive goals. If the objective of a given study is to quantify the degree to which a suspected risk fac- tor contributes to the disease burden in a defined population, then case-control and cohort study de- signs should be considered. Results from these types of studies can be used to generate information needed to judge whether a suspected risk factor is related causally to a disease (or other outcome), thus justifying public health intervention. If such causal- ity is established, analytic epidemiologic studies also can be used to quantitatively describe the pre- ventive effect that could be expected if the risk factor in question was reduced or eliminated in the target population (population-attributable risk). On the other hand, more commonly in public health prac-
92 JOURNAL OF PUBLIC HEALTH MANAGEMENT AND PRACTICE/SEPTEMBER 1999
tice, the objective of the study is to change some be- havior or practice that is understood to be deleteri- ous to public health. Therefore, intervention studies are designed and implemented. These may involve either experimental or quasi-experimental studies and may involve small numbers of individuals or larger population bases such as schools or entire communities.37
Stage four: develop program or policy options
In the next step, a variety of health program or policy options are examined. The list of options can be developed from a variety of sources. The initial review of the scientific literature can sometimes highlight various options. More often, expert panels provide program or policy recommendations on a variety of issues. There are several assumptions or contexts underlying any development of options. These considerations focus on five main areas: po- litical/regulatory, economic, social values, demo- graphic, and technological.34
In particular, it is important to assess and monitor the political process when developing policy op- tions. To do so, “stakeholder” input may be useful. The stakeholder for a policy might be the health policy maker, whereas the stakeholder for a coali- tion-based community intervention might be a com- munity member. In the case of health policies, sup- portive policy makers frequently can provide advice regarding timing of policy initiatives, methods for framing the issue, strategies for identifying sponsors, and ways to develop support among the general pub- lic. In the case of a community intervention, ad- ditional planning data may include key informant interviews, focus groups, or coalition member sur- veys.38 Several of these planning issues also are a part of the science of policy analysis.39
In developing options, it is useful to remember that many public health interventions are founded on the notion that action at the level of a social unit can im- prove health outcomes at the individual level. This notion is embodied in a causal model—one that leads from program inputs (programs and resources) to health outputs (changes in health behaviors or health status) if the program works as intended and one that guides program planners in designing inter- ventions. It is important for evaluation purposes that what has been termed this “small theory” of the in- tervention be made explicit early in the planning
process.40 The causal framework should lead toward explicit determination of mutable and immutable factors, assisting in option development.
Stage five: develop an action plan for the program or policy
This aspect of the process again deals largely with strategic planning issues. Key issues are covered here briefly, with more extensive discussions by oth- ers.33,41,42 When an option has been selected, a set of goals and objectives should be developed. A goal is a long-term, desired change in the status of a priority health need and an objective is a short-term, measur- able, specific activity that leads toward achievement of a goal.41,42 The course of action describes how the goals and objectives will be achieved, what resources are required, and how responsibility of achieving ob- jectives will be assigned. Excellent examples exist that show how to construct strategic goals and objec- tives with a “fill in the blank” format.43 It is impor- tant that objectives are:42
1. performance, behavior, or action oriented 2. precise in their language (do not use general or
vague verbs) 3. measurable 4. results oriented with stated outcomes 5. clear in their description of content and perfor-
mance 6. tied to specific time tables for completion.
Stage six: evaluate the program or policy
In simple terms, evaluation is the determination of the degree to which program goals and objectives are met. Most public health programs and policies are evaluated through “quasi-experimental” designs, for example, those lacking random assignment to inter- vention and comparison groups. More complete de- scriptions of research designs can be found else- where.44,45 In general, the strongest evaluation designs acknowledge the roles of both quantitative and qualitative evaluation. Furthermore, evaluation designs need to be flexible and sensitive enough to assess intermediate changes, even those that fall short of changes in behavior. Genuine change takes place incrementally over time, in ways that often are not visible to those too close to the intervention. Sev- eral important considerations for evaluating commu- nity-based interventions are shown in Table 1.46
Evidence-Based Decision Making in Public Health 93
Measuring the impacts of a program or policy should rely commonly on three inter-related levels of evaluation: process, impact, and outcome. The au- thors will discuss briefly potential contributions of each type of evaluation, with a more comprehensive discussion available.32,47 Initially, one should seek to determine which (if any) changes have occurred as the result of a particular intervention. This often in- volves process evaluation—the analysis of inputs and implementation experiences to track changes as a result of a program or policy.32 Process evaluation occurs at the earliest stages of a public health inter- vention and often is helpful in determining “mid- course corrections.” Impact evaluation can be con- sidered a subset of outcome evaluation that assesses whether intermediate objectives have been achieved. Indicators may include changes in knowledge, atti- tudes, or risk factor prevalence.32 The long-term mea- sures of effects rely on outcome evaluation such as changes in morbidity, mortality, and quality of life. As discussed earlier, economic evaluation often is an
important component of an overall evaluation plan. The usual sources of measurement error should be
considered when developing and implementing an evaluation plan, for example, validity and reliability. The most useful community-based interventions show high internal validity (for example, whether the observed results be attributed to the program or intervention). Further, external validity relates to whether the observed results can be generalized to other settings and populations. Reliability (or repro- ducibility) refers to the extent to which the same measurement is obtained on the same occasion by the same observer, on multiple occasions by the same observer, or by different observers on the same occasion. An additional source of error in commu- nity-based interventions may result from inadequate implementation (a so-called type III error).48
Testing the Sequential Approach in a Public Health Agency
Using the approach outlined above, the authors re- cently pilot tested a new course in evidence-based public health with 12 mid-level managers in the Mis- souri Department of Health. Participants took part in a four-day course that included didactic presenta- tions, case study exercises, and hands-on computer lab instruction. Following the course, a written evaluation was conducted. For the overall course
Table 1
A summary of evaluation principles and tools
Principle Tool
Community programs should include an assessment of program theory Logic models Instruments that are used to measure community programs must be Questionnaires and surveys Social indicators
contoured to each individual community Approaches should be guided by the questions asked and often require Experimental and quasi-experimental designs
both a quantitative and qualitative orientation Qualitative designs Evaluation should be informed by social ecology and social system Ecology and systems designs
concepts Community evaluation should involve local stakeholders in meaningful Participatory planning
ways
Source: Adapted from Goodman, R.M. “Principles and Tools for Evaluating Community-Based Prevention and Health Promotion Programs.” Journal of Public Health Management and Practice 4, no. 2 (1998): 39.
Measuring the impact of a program or policy should rely commonly on three inter-related levels of evaluation: process, impact, and outcome.
94 JOURNAL OF PUBLIC HEALTH MANAGEMENT AND PRACTICE/SEPTEMBER 1999
evaluation, the following statements each received mean scores between 8.5 and 9.3 (on a 10-point scale where 1 and 10): (1) “the course had clearly stated objectives;” (2) “the course objectives and course content were related appropriately;” (3) “course ob- jectives were achieved;” (4) “I learned new informa- tion;” and (5) “I can apply what I learned to my job. In addition, 10 of 12 participants stated they would use the course information in their day-to-day work (two were unsure). Based on this positive initial evaluation, plans are underway to revise the course content and to offer it among a wider audience within the state health department.
Barriers and Enablers
Through this effort, the authors have identified several potential barriers that may impede the ability of an organization to implement EBPH (Table 2). Pos- sible approaches for overcoming these barriers have been discussed by others.2,49,50 Leadership is needed from public health practitioners on the need and im- portance of EBPH. Such leadership is evident in
training programs such as the regional leadership network for public health practitioners51 and the ef- forts underway to develop evidence-based guide- lines for interventions.29 To address the gap between curricula in academic institutions and “real world” needs, the Public Health Faculty/Agency Forum de- veloped a useful set of universal and discipline-spe- cific competencies and recommendations.52
Summary
Resources in public health always are limited and development of new programs often is a zero-sum game. In addition, bodies of evidence for numerous public health interventions are growing continually, making choices increasingly complex. To rationally choose among alternatives and make the most pru- dent use of resources, stronger skills in EBPH are needed—these are a blend of art and science. Al- though work remains and testing of the proposed framework is ongoing, the authors hope the issues raised will stimulate debate on the strategies that will make EBPH a reality.
Table 2
Potential barriers and solutions for use of evidence-based decision making in public health
Barrier Solution
Lack of leadership in setting a clear and focused agenda for EBPH Commitment from all levels of public health leaders to increase the use of effective public health interventions
Lack of a view of the long-term “horizon” for program implementation Adoption and adherence to causal and evaluation frameworks and formative evaluation plans
External pressures drive the process away from an evidence-based Systematic communication and approach dissemination strategies
Inadequate training in key public health disciplines Wider dissemination of new and established training programs, including use of distance learning technologies
Lack of time to gather information, analyze data, and review the Enhanced skills for efficient analysis and literature for evidence review of the literature
Lack of comprehensive, up-to-date information on the effectiveness of Increased dissemination of guidelines in programs and policies clinical and population-based strategies
Lack of data on the effectiveness of certain public health interventions Increased funding for applied public health or for special populationsa research
aSpecial populations are defined as groups that have not been widely studied for a particular health condition or intervention—e.g., certain racial/ethnic populations or women.
Evidence-Based Decision Making in Public Health 95
REFERENCES
1. Centers for Disease Control and Prevention. “Public Health in the New American Health System. Discussion Paper.” At- lanta, GA: Centers for Disease Control and Prevention, 1993.
2. Institute of Medicine, Committee for the Study of the Future of Public Health. The Future of Public Health. Washington, D.C.: National Academy Press, 1988.
3. Taubes, G. “Looking for the Evidence in Medicine,” Science 272 (1996): 22–24.
4. Sackett, D.L., et al. “Evidence-Based Medicine: What It Is and What It Isn’t.” British Medical Journal 312 (1996): 71–72.
5. Sackett, D.L., and Rosenberg, W.M.C. “The Need for Evi- dence-Based Medicine.” Journal of the Royal Society of Medi- cine 88 (1995): 620–624.
6. U.S. Dept of Health and Human Services. Reducing the
Health Consequences of Smoking—25 Years of Progress: A Report of the Surgeon General. Rockville, MD: U.S. Dept of Health and Human Services, Public Health Service, Centers for Disease Control, Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, 1989.
7. Sweanor, D., et al. “Report of the Tobacco Policy Research Study Group on Tobacco Pricing and Taxation in the United States.” Tobacco Control 1(Suppl) (1992): 531–536.
8. Bal, D., et al. “Reducing Tobacco Consumption in California: Development of a statewide anti-tobacco use campaign.” JAMA 264 (1990): 1570–1574.
9. Elder, J.P., et al. “Independent Evaluation of the California Tobacco Education Program.” Public Health Reports 111 (1996): 353–358.
10. Land, G., et al. “Missouri’s Take a Seat, Please! and Program Evaluation.” Journal of Public Health Management and Prac- tice 3 (1997): 51–58.
Figure 1. A sequential framework for enhancing evidence based public health
96 JOURNAL OF PUBLIC HEALTH MANAGEMENT AND PRACTICE/SEPTEMBER 1999
11. Petitti, D.B. Meta-Analysis, Decision Analysis, and Cost-Ef- fectiveness Analysis: Methods for Quantitative Synthesis in Medicine. New York: Oxford University Press, 1994.
12. Glass, G.V. “Primary, Secondary and Meta-Analysis of Re- search.” Education Research 5 (1976): 3–8.
13. Greenland, S. “Meta-analysis.” In Rothman, K.J. and Greenland, S., eds. Modern Epidemiology. Second Edition. Philadelphia, PA: Lippincot-Raven Publishers, 1998.
14. Goodman, S.N. “Meta-Analysis in Health Services Research.” In Armenian, H., and Shapiro, S., eds. Epidemiology and Health Services. New York: Oxford University Press, 1998.
15. Samet, J.M., and Burke, T.A. “Epidemiology and Risk Assess- ment.” In Brownson, R.C., and Petitti, D.B., eds. Applied Epi- demiology: Theory to Practice. New York: Oxford University Press, 1998.
16. World Health Organization. Assessment and Management of Environmental Health Hazards. Geneva: World Health Orga- nization, 1989.
17. Hertz-Picciotto, I. “Epidemiology and Quantitative Risk As- sessment: A Bridge from Science to Policy.” American Journal of Public Health 85 (1995): 484–491.
18. Gold, M.R., et al. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press, 1996.
19. Petitti, D.B. “Economic Evaluation.” In Brownson, R.C., and Petitti, D.B., eds. Applied Epidemiology: Theory to Practice. New York: Oxford University Press, 1998.
20. Weinstein, M.C., and Stason, W.B. “Foundations of Cost-Ef- fectiveness Analysis for Health and Medical Practices.” New England Journal of Medicine 296 (1977): 716–721.
21. Hatziandreu, E.I., et al. “A Cost-Effectiveness Analysis of Exer- cise as a Health Promotion Activity.” American Journal of Public Health 78 (1988): 1417–1421.
22. Thacker, S.B., and Berkelman, R.L. “Public Health Surveil- lance in the United States.” Epidemiologic Reviews 10 (1988): 164–190.
23. Thacker, S.B., and Stroup, D.F. “Public Health Surveillance.” In Brownson, R.C., and Petitti, D.B., eds. Applied Epidemiol- ogy: Theory to Practice. New York: Oxford University Press, 1998.
24. Annest, J.L., et al. “Chronological Trend in Blood Lead Levels between 1976 and 1980.” New England Journal of Medicine 308 (1983): 1373–1377.
25. Brownson, R.C. “Epidemiology and Health Policy.” In Brownson, R.C., and Petitti, D.B., eds. Applied Epidemiology: Theory to Practice. New York: Oxford University Press, 1998.
26. U.S. Preventive Services Task Force. Guide to Clinical Pre- ventive Services. Baltimore, MD: Williams and Wilkins, 1996.
27. Novick, L.F. “Public Health Practice Guidelines: A Case Study.” Journal of Public Health Management and Practice 3 (1997): 59–64.
28. Council on Linkages between Academia and Public Health Practice. Practice Guidelines for Public Health: Assessment of Scientific Evidence, Feasibility and Benefits: A Report of the Guideline Development Project for Public Health Practice.
Albany, NY: University of Albany School of Public Health, 1995.
29. Pappaioanou, M., and Evans, C. “Developing a Guide to Community Preventive Services: A U. S. Public Health Ser- vice Initiative.” Journal of Public Health Management and Practice 4 (1998): 48–54.
30. Savitz, D.A. et al. “Methods in Chronic Disease Epidemiol- ogy.” In Brownson, R.C., Remington, P.L., and Davis, J.R., eds. Chronic Disease Epidemiology and Control. Washing- ton, D.C.: American Public Health Association, 1998.
31. Journal of Health Education 23 (1992). 32. Green, L.W., and Kreuter, M.W. Health Promotion Planning:
An Educational and Environmental Approach. Mountain View, CA: Mayfield, 1991.
33. Bryson, J.M. Strategic Planning for Public and Nonprofit Or- ganizations. A Guide to Strengthening and Sustaining Organi- zational Achievement. San Francisco: Jossey-Bass Publishers, 1995.
34. Ginter, P.M., Duncan, W.J., and Capper, S.A. “Keeping Stra- tegic Thinking in Strategic Planning: Macro-Environmental Analysis in a State Health Department of Public Health.” Pub- lic Health 106 (1992): 253–269.
35. Goldschmidt, P.G. “Information Synthesis: A Practical Guide.” Health Services Research 21 (1986): 215–237.
36. Rundall, T.G. “Conducting and Writing Research Reviews.” Medical Care Research and Review 53 (supplement) (1996): S132–S145.
37. Koepsell, T.D. “Epidemiologic Issues in the Design of Com- munity Intervention Trials.” In Brownson, R.C., and Pettiti, D.B., eds. Applied Epidemiology: Theory to Practice. New York: Oxford University Press, 1998.
38. Florin, P., and Stevenson, J. “Identifying Training and Techni- cal Assistance Needs in Community Coalitions: A Develop- mental Approach.” Health Education Research 8 (1993): 417–432.
39. Ham, C., and Hill, M. The Policy Process in the Modern Capi- talist State. New York: St. Martin’s Press, 1984.
40. Lipsey, M.W. “Theory as Method: Small Theories of Treat- ment.” In Sechrest, L., Perrin, E., and Bunker, J., eds. Research Methodology: Strengthening Causal Interpretations of Nonexperimental Data. Washington, D.C.: Government Printing Office, 1990.
41. Turnock, B.J. Public Health: What It Is and How It Works. Gaithersburg, MD: Aspen Publishers, Inc., 1997.
42. Timmreck, T.C. Planning, Program Development, and Evalu- ation. A Handbook for Health Promotion, Aging and Health Services. Boston: Jones and Bartlett Publishers, 1995.
43. American Public Health Association. Healthy Communities 2000: Model Standards. Guidelines for Community Attain- ment of the Year 2000 Health Objectives. Washington, D.C.: American Public Health Association, 1991.
44. Campbell, D.T., and Stanley, J.C. Experimental and Quasi- Experimental Designs for Research. Boston: Houghton Mifflin, 1963.
45. Cook, T.D., and Campbell, D.T. Quasi-Experimentation: De-
Evidence-Based Decision Making in Public Health 97
sign and Analysis Issues for Field Settings. Boston: Houghton Mifflin, 1979.
46. Goodman, R.M. “Principles and Tools for Evaluating Com- munity-Based Prevention and Health Promotion Programs.” Journal of Public Health Management and Practice 4 (1998): 37–47.
47. Rossi, P.H., and Freeman, H.E. Evaluation. A Systematic Ap- proach. Newbury Park, NJ: Sage Publications, 1993.
48. Goodman, R.M., and Wandersman, A. “FORECAST: A For- mative Approach to Evaluating Community Coalitions and Community-Based Interventions.” Journal of Community Psy- chology (1994): 6–25.
49. Institute of Medicine, Committee on Public Health. Healthy
Communities: New Partnerships for the Future of Public Health. Washington, D.C.: National Academy Press, 1996.
50. Brownson, R.C., and Kreuter, M.W. “Future Trends Affecting Public Health: Challenges and Opportunities.” Journal of Public Health Management and Practice 3 (1997): 49–60.
51. Gordon, R.L., et al. “Prevention and the Reforming U. S. Health Care System: Changing Roles and Responsibilities for Public Health.” Annual Review of Public Health 17 (1996): 489–509.
52. Sorensen, A.A., and Bialek, R.G., eds. The Public Health Fac- ulty/Agency Forum: Linking Graduate Education and Prac- tice: Final Report. Gainesville, FL: University Press of Florida, 1991.
