Characteristics, Challenges, and Opportunities of Evidence-Based Design
In recent years, there has been increased pressure from funding agencies and federal, state and local governments for greater effectiveness and accountability of prevention and intervention programs. This rising demand for program quality, and evidence of that quality, has fueled a growing interest in evidence‐ based programs (EBPs). However, there remains some confusion about what constitutes an EBP, whether some EBPs are better than others, and the advantages and disadvantages of implementing EBPs. In this Research to Practice brief, we provide an overview of what it means for a program to be evidence‐based, discuss the advantages and disadvantages of implementing EBPs, and point readers in the direction of resources to help locate these programs and learn more about them.
What are evidence‐based programs? A growing body of research in the social and behavioral sciences has demonstrated that certain approaches and strategies for working with youth and their families can positively impact important social problems such as delinquency, teen pregnancy, substance abuse and family violence. Many of these effective approaches and strategies have been packaged into programs targeting outcomes specific to individuals, schools, families, and communities. Those programs that have been found to be effective based on the results of rigorous evaluations are often called “evidence‐based.”
WHAT WORKS, WISCONSIN – RESEARCH TO PRACTICE SERIES
Evidence‐based programs: An overview
ISSUE #6, OCTOBER 2007 BY SIOBHAN M. COONEY, MARY HUSER, STEPHEN SMALL, AND CAILIN O’CONNOR
University of Wisconsin–Madison and University of Wisconsin–Extension
Evidence‐based programs: An overview 2 What Works, Wisconsin – Research to Practice Series, #6
The importance of rigorous evaluation A rigorous evaluation typically involves either an experimental design (like that used in randomized controlled trials) or a quasi‐experimental design. In an experimental design, people are randomly assigned to either the treatment group, which participates in the program, or the control group, which does not. After the program is completed, the outcomes of these two groups are compared. This type of research design helps ensure that any observed differences in outcomes between the two groups are the result of the program and not other factors. Given that randomization is not always possible, a quasi‐ experimental design is sometimes used. In evaluations using this design, the program participants are compared to a group of people similar in many ways to the program participants. However, because a quasi‐experimental design does not randomly assign participants to program and non‐program groups, it is not as strong a design as the experimental approach. Because there may be unobserved differences between the two groups of people who are being compared, this design does not allow program evaluators to conclude with the same certainty that the program itself was responsible for the impacts observed. Most programs have evaluation evidence from less rigorous studies. Evaluations that do not include any type of comparison group, for example, do not allow for any conclusions to be made about whether the changes seen in program participants are related to or caused by the program. These studies sometimes show the promise of positive results, but they do not allow the program to be classified as evidence‐based. Programs with evidence from less rigorous studies are often referred to as “promising” programs.
An important element of EBPs is that they have been evaluated rigorously in experimental or quasi‐experimental studies (see box on this page). Not only are the results of these evaluations important, but it is also essential that the evaluations themselves have been subjected to
critical peer review. That is, experts in the field – not just the people who developed and evaluated the program – have examined the evaluation’s methods and agreed with its conclusions about the program’s effects. Thus, EBPs often have evaluation findings published in peer‐reviewed scientific journals.
When a program has sufficient peer‐ reviewed, empirical evidence for its effectiveness, its developer will typi‐ cally submit it to certain federal agencies and respected research organizations for consideration. These organizations “certify” or “endorse” programs by including them in their official lists of effective programs. This lets others in the field know the program meets certain standards of effectiveness. (See Appendix A for examples of these organizations.) Simply put, a program is judged to be evidence‐based if (a) evaluation re‐ search shows that the program pro‐ duces the expected positive results; (b) the results can be attributed to the program itself, rather than to other extraneous factors or events; (c) the evaluation is peer‐reviewed by experts in the field; and (d) the program is “endorsed” by a federal agency or respected research organization and included in their list of effective programs. Given this definition of an EBP, it is important to distinguish the term “evidence‐based” from “research‐ based.” Consider our earlier description of how most, if not all, EBPs were developed based on years of scientific research on what program components, such as content and activities, are likely to work for youth
Evidence‐based programs: An overview 3 What Works, Wisconsin – Research to Practice Series, #6
and families. Because EBPs contain program components with solid empirical bases, they can safely be called “research‐based” programs. However, the reverse is not true. Not all, or even the majority, of research‐based programs fit the definition of an EBP. Just because a program contains research‐based content or was guided by research‐based information, doesn’t mean it has been proven effective. Unless it also has scientific evidence that it works, it is incorrect to call it “evidence‐based.”
Are some evidence‐based programs better than others? Programs that meet the definition of evidence‐ based are not all similarly effective or equally likely to work in a given community. For example, some EBPs have been evaluated rigorously in several large‐scale evaluations that follow participants for a long period of time. Others have only undergone one or two less rigorous evaluations (for example, those using the quasi‐experimental design described on page 2). Those programs that are shown to be effective multiple times in experimental studies are generally considered to be of a higher standard. Furthermore, many EBPs have been successfully replicated and evaluated in a variety of settings with a range of different audiences. Others have only been evaluated with a particular audience in a certain geographical area, for example. When a program has been shown to be effective in different settings and with different audiences, it is more likely that it will be effective when implemented elsewhere.
Finally, EBPs can vary in the strength of their effects. For example, one program may have evidence that it reduces delinquent acts in its participants by 10 percent over the subsequent year, while another program has evidence of reducing delinquency by 20 or 25 percent. Generally, those programs that consistently pro‐ duce a greater effect than other programs are thought to be better programs. Thus, the level of evidence for effectiveness varies across programs, and practitioners must use a critical eye when judging where on the continuum of effectiveness a program lies.
Advantages of evidence‐based programs There are numerous merits to adopting and implementing EBPs. First, utilizing an EBP in‐ creases the odds that the program will work as intended and that the public good will be enhanced. There is also greater efficiency in using limited resources on what has been proven to work as compared to what people think will work or what has traditionally been done. Instead of putting resources toward program development, organizations can select from the growing number of EBPs, which are not only known to be effective but also often offer well‐ packaged program materials, staff training, and technical assistance. Using EBPs where appropriate can thus be viewed as a responsible and thoughtful use of limited resources. The proven effectiveness that underlies EBPs can help secure resources and support from funding agencies and other stakeholders, such as policy makers, community leaders, and members of the targeted population. Increasingly, funders and policy makers are recommending, if not requiring, that EBPs be used to qualify for their financial support. Additionally, the demonstrated effectiveness of these programs can facilitate community buy‐in
Evidence‐based programs: An overview 4 What Works, Wisconsin – Research to Practice Series, #6
and the recruitment and retention of program participants. A final benefit of EBPs is that they may have cost‐benefit information available. This type of information helps to convey the potential eco‐ nomic savings that can accrue when funds are invested in a program. Cost‐benefit information can be very influential in an era where accountability and economic factors often drive public policy and funding decisions.
Disadvantages of evidence‐based programs Despite the numerous advantages of EBPs, there are some limitations that are important to consider. A major constraint is the financial resources needed to adopt and implement them. Most EBPs are developed, copyrighted, and sold at rather substantial costs. Program designers often require that organizations purchase curricula and other specially developed program materials, that staff attend specialized training, and that program facilitators hold certain degrees or certifications. Furthermore, EBPs are often intended to be im‐ plemented exactly as designed, allowing little room for local adaptation. Finally, organizations sometimes find that there are few or no EBPs that are both well‐suited to meet the needs of targeted audiences and appropriate for their organization and local community setting. This situation is especially common when it comes to the promotion of positive outcomes rather than the prevention of negative ones. Because the development of many EBPs was sponsored by federal agencies concerned with addressing specific problems, such as substance abuse, mental illness,
violence, or delinquency, there currently exist many more problem‐focused EBPs than ones designed specifically to promote positive developmental outcomes like school success or social responsibility.
Where to find evidence‐based programs Practitioners looking for an EBP to implement in their community or learn more about these programs will find the Internet to be their most useful resource. As mentioned earlier, a number of federal agencies and respected research organizations “certify” or “endorse” programs that meet the organizations’ specified standards for effectiveness. Many of these agencies have established on‐line registries, of lists of EBPs that they have identified as effective. While there are some differences in the standards used by various organizations to assess whether a program should be endorsed and thus included on their registry, most share the primary criteria regarding the need for strong empirical evidence of program effectiveness. Organizations that endorse EBPs typically limit such endorsements, and thus their program registry, to those programs that have shown an impact on specific outcomes of interest to the organization. For example, programs listed on the Office of Juvenile Justice and Delinquency Prevention’s Model Programs Guide have all been shown to have an impact on juvenile delinquency or well‐known precursors to delinquency. As previously mentioned, because the development of many EBPs was funded by federal agencies focused on specific problems, most existing registries of EBPs are problem‐ oriented. Occasionally, EBPs are categorized according to a strengths‐based orientation and address outcomes related to positive youth
Evidence‐based programs: An overview 5 What Works, Wisconsin – Research to Practice Series, #6
WHAT WORKS, WISCONSIN: RESEARCH TO PRACTICE SERIES
This is one of a series of Research to Practice briefs prepared by the What Works, Wisconsin team at the University of Wisconsin–Madison, School of Human Ecology, and Cooperative Extension, University of Wisconsin–Extension. All of the briefs can be downloaded from http://whatworks.uwex.edu. This series expands upon ideas that are discussed in What Works, Wisconsin: What Science Tells Us about Cost‐Effective Programs for Juvenile Delinquency Prevention, which is also available for download at the web address above. This publication may be cited without permission provided the source is identified as: Cooney, S.M., Huser, M., Small, S., & O’Connor, C. (2007). Evidence‐based programs: An overview. What Works, Wisconsin Research to Practice Series, 6. Madison, WI: University of Wisconsin–Madison/Extension. This project was supported, in part, by Grant Award No. JF‐04‐PO‐0025 awarded by the Wisconsin Office of Justice Assistance through the Wisconsin Governor’s Juvenile Justice Commission with funds from the Office of Juvenile Justice and Delinquency Prevention.
development, academic achievement, school readiness and family strengthening. While registries of EBPs are usually organized around the particular outcomes the programs have been found to impact, many programs, especially those focused on primary prevention, often have broader effects than this pattern would suggest. Many EBPs have been found to be effective for reducing multiple problems and promoting a number of positive outcomes. For example, a parenting program that successfully promotes effective parenting practices may not only reduce the likelihood of particular problems such as drug abuse or aggression, but may also promote a variety of positive outcomes like academic success or stronger parent‐child relationships. For this reason, you will often see the same program appear on
multiple registries that focus on different types of outcomes. Now, more than ever, practitioners have available to them a wealth of EBPs that build on the best available research on what works. Unfortunately, they are currently underused and often not well‐understood. Although EBPs do have some limitations, they can contribute to a comprehensive approach to preventing a range of social and health‐related problems and enhancing the well‐being of individuals, families and communities.
Evidence‐based programs: An overview – Appendix A 6 What Works, Wisconsin – Research to Practice Series, #6
Appendix A Evidence‐based program registries
The following websites contain registries, or lists of evidence‐based programs, that have met specific criteria for effectiveness. Program registries are typically sponsored by federal agencies or other research organiza‐ tions that endorse programs at different rating levels based on evidence of effectiveness for certain participant outcomes. The registries listed below cover a range of areas including substance abuse and violence preven‐ tion as well as the promotion of positive outcomes such as school success and emotional and social compe‐ tence. Generally, registries are designed to be used for finding programs for implementation. However, registries can also be used to learn about evidence‐based programs that may serve as models as organizations modify aspects of their own programs. Best Practices Registry for Suicide Prevention http://www.sprc.org/featured_resources/ebpp/index.asp This registry, developed by the Suicide Prevention Resource Center (SPRC) and the American Foundation for Suicide Prevention, includes two registries of evidence‐based programs. The first draws directly from a larger registry‐ that of the Substance Abuse and Mental Health Administration’s (SAMHSA) National Registry of Evidence‐Based Programs and Practices (NREPP). Users interested in finding out more about programs drawn from this registry will be directed to the NREPP site. The second registry was developed by SPRC in 2005 and lists Effective and Promising evidence‐based programs for suicide prevention. This portion has fact sheets in PDF format for users interested in learning more about the listed programs. Center for the Study and Prevention of Violence, Blueprints for Violence Prevention http://www.colorado.edu/cspv/blueprints/index.html This research center site provides information on model programs in its “Blueprints” section. Programs that meet a strict scientific standard of program effectiveness are listed. These model programs (Blueprints) have demonstrated their effectiveness in reducing adolescent violent crime, aggression, delinquency, and sub‐ stance abuse. Other programs have been identified as promising programs. Endorsements are updated regularly, with programs added to and excluded from the registry based on new evaluation findings. The Collaborative for Academic, Social, and Emotional Learning (CASEL) http://www.casel.org/programs/selecting.php The Safe and Sound report developed at CASEL lists school‐based programs that research has indicated are effective in promoting social and emotional learning in schools. This type of learning has been shown to con‐ tribute to positive youth development, academic achievement, healthy behaviors, and reductions in youth problem behaviors. Ratings are given on specific criteria for all programs listed, with some designated “Select” programs. This registry has not been updated since programs were reviewed in 2003.
Evidence‐based programs: An overview – Appendix A 7 What Works, Wisconsin – Research to Practice Series, #6
Exemplary and Promising Safe, Disciplined and Drug‐Free Schools Programs http://www.ed.gov/admins/lead/safety/exemplary01/index.html The Department of Education and the Expert Panel on Safe, Disciplined and Drug‐Free Schools identified nine exemplary and 33 promising programs for this 2001 report. The report, which can be found at this site, provides descriptions and contact information for each program. The focus is on programs that can be imp‐ lemented in a school setting whether in the classroom, in extra‐curricular activities, or as after‐school pro‐ gramming. Helping America’s Youth http://guide.helpingamericasyouth.gov/programtool.cfm This registry is sponsored by the White House and was developed with the help of several federal agencies. Programs focus on a range of youth outcomes such as academic achievement, substance use, and delin‐ quency, and are categorized as Level 1, Level 2, or Level 3 according to their demonstrated effectiveness. The registry can be searched with keywords or by risk or protective factor, and is updated regularly to incorpo‐ rate new evidence‐based programs. Northeast Center for the Application of Prevention Technology (CAPT) Database of Prevention Programs http://www.hhd.org/capt/search.asp This site features a simple or advanced search function to find substance abuse and other types of prevention programs and determine their effectiveness according to a variety of criteria. Also included is information about the sources those agencies used for their evaluations, contact information, websites, domains, relevant references, and a brief description of each program. Office of Juvenile Justice and Delinquency Prevention (OJJDP) Model Programs Guide http://www.dsgonline.com/mpg2.5/mpg_index.htm The OJJDP Model Programs Guide is a user‐friendly, online portal to prevention and intervention programs that address a range of issues across the juvenile justice spectrum. The Guide now profiles more than 200 programs – rated Exemplary, Effective, or Promising – and helps communities identify those that best suit their needs. Users can search the Guide’s database by program category, target population, risk and protec‐ tive factors, effectiveness rating, and other parameters. This registry is continuously updated and contains more programs than other well‐known registries, although many of these are Promising rather than Exem‐ plary or Effective. Promising Practices Network on Children, Families and Communities http://www.promisingpractices.net/programs.asp A project of the RAND Corporation, the Promising Practices Network website contains a registry of Proven and Promising prevention programs that research has shown to be effective for a variety of outcomes. These programs are generally focused on children, adolescents, and families. The website provides a thorough summary of each program and is updated regularly.
Evidence‐based programs: An overview – Appendix A 8 What Works, Wisconsin – Research to Practice Series, #6
Social Programs that Work, Coalition for Evidenced‐Based Policy http://www.evidencebasedprograms.org/ This site is not a registry in the conventional sense of the word in that it does not include and exclude pro‐ grams based on some criteria of effectiveness. Instead, it summarizes the findings from rigorous evaluations of programs targeting issues such as employment, substance use, teen pregnancy, and education. Some of the programs have substantial evidence of their effectiveness, while others have evaluation results suggesting their ineffectiveness. Users are welcome to sign up for emails announcing when the site is updated. Strengthening America’s Families: Effective Family Programs for Prevention of Delinquency http://www.strengtheningfamilies.org/ This registry summarizes and rates family strengthening programs which have been proven to be effective. Programs are designated as Exemplary I, Exemplary II, Model, or Promising based upon the degree, quality and outcomes of research associated with them. A program matrix is also included, which can be helpful in determining “at a glance” which programs may best meet community needs. This registry was last revised in 1999. Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) National Registry of Evidence‐Based Programs and Practices http://nrepp.samhsa.gov/ The National Registry of Evidence‐based Programs and Practices (NREPP) is a searchable database with up‐ to‐date, reliable information on the scientific basis and practicality of interventions. Rather than categorizing programs as Model, Effective, or Promising, NREPP rates the quality of the research findings separately for each outcome that has been evaluated, as well as readiness for dissemination. Users can perform customized searches to identify specific interventions based upon desired outcomes, target populations and settings. Youth Violence: A Report of the Surgeon General http://www.surgeongeneral.gov/library/youthviolence/chapter5/sec3.html This report designates programs as Model or Promising and goes further than many other registries to also include a “Does Not Work” category. General approaches and specific programs for the prevention of youth violence are described at three levels of intervention: primary, secondary and tertiary. This report has not been updated since its publication in 2001, but it is rare in that it discusses the cost‐effectiveness of the programs.