Characteristics, Challenges, and Opportunities of Evidence-Based Design
elines for Selecting an Evidence‐Based Program What Works, Wisconsin – Research to Practice Series, #3
In recent years there has been a significant increase in the number of evidence‐ based programs designed to reduce individual and family problems and promote healthy development. Because each program has undergone rigorous testing and evaluation, program practitioners can reassure potential program sponsors that the program is likely to be effective under the right conditions, with the appropriate audience and with the proper implementation. However, knowing which program is the “right” one for a particular setting and audience is not always easy to determine. When selecting a program, it is important to move beyond current fads or what the latest salesperson is selling and consider whether a program fits with the local agency’s goals and values, the community setting and the needs of the targeted audience. The long‐term success of a program depends on the program being not only a good one, but also the right one. Unfortunately, there is currently little research on how to best go about the process of selecting an evidence‐based program. Consequently, the guidelines we present in this brief are based primarily on our experiences working with community‐based organizations, the experiences of practitioners, and common sense. We have identified a number of factors that we believe should be considered when deciding which program is the most appropriate for a particular audience and sponsoring organization. These factors can be grouped into three general categories: program match, program quality and organizational resources. In order to assist with the process of program selection, we have developed a set of questions to consider when selecting an evidence‐based program for your particular agency and audience.
WHAT WORKS, WISCONSIN – RESEARCH TO PRACTICE SERIES
Guidelines for selecting an evidence‐based program: Balancing community needs, program quality,
and organizational resources
ISSUE #3, MARCH 2007 BY STEPHEN A. SMALL, SIOBHAN M. COONEY, GAY EASTMAN, AND CAILIN O’CONNOR
University of Wisconsin–Madison and University of Wisconsin–Extension
Guidelines for Selecting an Evidence‐Based Program 2 What Works, Wisconsin – Research to Practice Series, #3
Program match: Questions to ask How well do the program’s goals and objectives
reflect what your organization hopes to achieve?
How well do the program’s goals match those of your intended participants?
Is the program of sufficient length and intensity (i.e., “strong enough”) to be effective with this particular group of participants?
Are potential participants willing and able to make the time commitment required by the program?
Has the program demonstrated effectiveness with a target population similar to yours?
To what extent might you need to adapt this program to fit the needs of your community? How might such adaptations affect the effectiveness of the program?
Does the program allow for adaptation?
How well does the program complement current programming both in your organization and in the community?
The issues raised by program match, program quality and organizational resources are overlap‐ ping. Selecting a program usually requires balancing different priorities, so it’s important to have a good understanding of all three of these before determining the usefulness of a program for a particular situation.
PROGRAM MATCH A first set of factors to consider is related to how well the program will fit with your purposes, your organization, the target audience, and the com‐ munity where it will be implemented. Perhaps the most obvious factor to consider is whether the goals and objectives of a program are consistent with the goals and objectives that the sponsoring organization hopes to achieve. While this may seem apparent, it is not uncommon for sponsors to select a program because there is grant money available to support it or everyone else is doing it. Just because a program is the latest fad or
there’s funding to support it doesn’t necessarily mean it is going to accomplish the goals of the sponsoring organization or meet the needs of the targeted audience. A second aspect of program match involves whether a program is strong enough to address the level and complexity of risk factors or current problems among participants. This refers to the issue of adequate program duration and intensity. Changing existing problem behaviors or counter‐ acting a large number of risk factors in partici‐ pants’ lives requires many hours of engaging programming over a period of time. For example, a short primary prevention program designed for families facing few problems or risks may not be effective for an audience already experiencing more severe problems. Another facet of program match concerns the length of the program and whether your intended audience will be willing and able to attend the required number of sessions. Many evidence‐ based programs are of fairly long duration, involving multiple sessions over weeks or months. A common concern of program pro‐ viders is whether potential participants will make such a long‐term commitment. Because this is a realistic concern, program sponsors need to assess the targeted audience’s availability for and interest in a program of a particular length.1 The reality is, if people don’t attend, then they can’t reap the program’s benefits. However, it is also important to keep in mind that programs of longer duration are more likely to produce lasting behavior change in participants. Program sponsors sometimes need to find a compromise between the most effective program and one that will be a realistic commitment for participants. Matching a program with the values and culture of the intended audience is also critically import- ant. Some programs are intentionally designed for particular populations or cultural groups. Most are more culturally generic and designed
1 Issue #2 in this series addresses strategies for recruiting and retaining participants.
Guidelines for Selecting an Evidence‐Based Program 3 What Works, Wisconsin – Research to Practice Series, #3
Program quality: Questions to ask Has this program been shown to be effective?
What is the quality of this evidence?
Is the level of evidence sufficient for your organization?
Is the program listed on any respected evidence‐ based program registries? What rating has it received on those registries?
For what audiences has the program been found to work?
Is there information available about what adaptations are acceptable if you do not implement this program exactly as designed? Is adaptation assistance available from the program developer?
What is the extent and quality of training offered by the program developers?
Do the program’s designers offer technical assistance? Is there a charge for this assistance?
What is the opinion and experience of others who have used the program?
for general audiences.2 It’s important to consider whether the targeted audience will find the program acceptable and will want to participate. The ideal situation would be finding evidence that a program is effective for the specific pop- ulation(s) you intend to use it with. In that case, you could reasonably expect the program to be effective when it is implemented well. Unfortunately, many evidence‐based programs have only been evaluated with a limited number of populations and under a relatively narrow range of conditions. While many evidence‐based programs are effective and appropriate for a range of audiences and situations, it is rare to find a program that is suitable or effective for every audience or situation. In many cases, you will need to carefully read program materials or talk to the program’s designers to see whether adapting a program or using it with an audience for which it hasn’t been evaluated is reasonable. Depending on the design, programs may or may not be amenable to adaptation. If adapting a program to a particular cultural group is important, then program sponsors should serious- ly consider whether such changes are possible. Some program designers are willing to help you with program adaptation so that the program’s effectiveness will not be undermined by these changes.3 Finally, when considering which program to select, sponsors should consider whether the pro- gram complements other programs being offered by the sponsoring organization and by other organizations in the community. The most effective approaches to prevention and inter- vention involve addressing multiple risk and
2 Issue #1 in this series addresses the issue of culture and evidence‐based programs. 3 Issue #4 in this series will address issues of program fidelity and adaptation.
protective factors, developmental processes and settings. Any new program implemented in a community should address needs that other community programs fail to address, which will help to create the kind of multi‐pronged approach that leads to greater overall effectiveness.
PROGRAM QUALITY A second set of factors to consider when selecting a program are related to the quality of the pro‐ gram itself and the evidence for its effectiveness. The program should have solid, research‐based evidence showing that it is effective. For a pro‐ gram to be deemed evidence‐based, it must go through a series of rigorous evaluations. Such evaluations have experimental or quasi‐experi‐ mental designs – meaning they compare a group of program participants to a similar group of people who did not participate in the program to determine whether program participation is assoc‐ iated with positive changes. These kinds of eval‐
Guidelines for Selecting an Evidence‐Based Program 4 What Works, Wisconsin – Research to Practice Series, #3
TABLE 1: Selected evidence‐based program registries
Blueprints for Violence Prevention http://www.colorado.edu/cspv/blueprints/index.html This registry is one of the most stringent in terms of endorsing programs as Model or Promising. Programs are reviewed by an expert panel and staff at the University of Colorado, and endorsements are updated regularly. Programs are added and excluded from the registry based on new evaluation findings.
Helping America’s Youth http://guide.helpingamericasyouth.gov/programtool.cfm This registry was developed with the help of several federal agencies. Programs focus on a range of youth outcomes and are categorized as Level 1, Level 2, or Level 3 according to their demonstrated effectiveness. The registry is updated regularly to incorporate new evidence‐based programs.
Office of Juvenile Justice and Delinquency Prevention Model Program Guide http://www.dsgonline.com/mpg2.5/mpg_index.htm This registry is one of the largest currently available and is continuously updated to include new programs. Programs found on this registry are designated as Exemplary, Effective, or Promising.
Promising Practices Network http://www.promisingpractices.net/ A project of the RAND Corporation, this registry regularly updates its listings of Effective and Promising programs. Programs are reviewed and endorsed by project staff.
Strengthening Americaʹs Families http://www.strengtheningfamilies.org/html/ Although this registry was last revised in 1999, it is the only registry with a focus specifically on family‐based programs. Programs were reviewed by expert panels and staff at the University of Utah and the Center for Substance Abuse Prevention. They were then designated as Exemplary I, Exemplary II, Model, or Promising.
Substance Abuse and Mental Health Services Administration (SAMHSA) National Registry of Evidence‐ Based Programs and Practices http://www.nrepp.samhsa.gov This recently re‐launched site no longer categorizes programs as Model, Effective, or Promising. Instead, programs are summarized and the quality of the research findings is rated separately for each outcome that has been evaluated. SAMHSA has also introduced a “Readiness for Dissemination” rating for each reviewed program. Nominations are accepted each year for programs to be reviewed; SAMHSA funds independent consultants to review nominated programs and update the registry.
uations allow for a reasonable assumption that it was the program itself that changed people’s knowledge, attitudes or behavior. As funders and program sponsors become more committed to implementing evidence‐based pro‐ grams, program developers are increasingly likely to promote their programs as evidence‐based. However, just because a program developer ad‐ vertises a program as evidence‐based doesn’t
mean that it meets the standards discussed above. For example, a program might be “research‐ based,” but not “evidence‐based.” A research‐ based program has been developed based on research about the outcomes or processes it add‐ resses. However, it has probably not been subjected to the rigorous evaluations and real‐ world testing that are needed to designate a program as evidence‐based. The simplest way to determine evidence of a program’s effectiveness is
Guidelines for Selecting an Evidence‐Based Program 5 What Works, Wisconsin – Research to Practice Series, #3
Organizational resources: Questions to ask
What are the training, curriculum, and implementation costs of the program?
Can your organization afford to implement this program now and in the long‐term?
Do you have staff capable of implementing this program? Do they have the qualifications recommended or required to facilitate the program?
Would your staff be enthusiastic about a program of this kind and are they willing to make the necessary time commitment?
Can this program be implemented in the time available?
What’s the likelihood that this program will be sustained in the future?
Are your community partners supportive of your implementation of this program?
to examine the designations given by well‐estab‐ lished and respected evidence‐based program registries. Program registries classify programs at different levels of endorsement based on evidence of effectiveness for certain participant outcomes. See Table 1 for an annotated listing of program registries. If a program is not listed on a respected registry, then it is important to seek out scientific evidence of the program’s effectiveness. At a minimum, you should review any evaluation studies that have been conducted by the program developer and external evaluators. Ideally, these evaluations use an experimental or quasi‐experimental research design. Another sign of a high‐quality evaluation is that its results have been published in a well‐ respected, peer‐reviewed, scientific journal. An additional indicator of program quality to consider is the level of training and follow‐up support available from the program designers. Some programs have a great deal of resources available to help program implementers. These
resources can be especially important if you’re working with a unique audience and need to make adaptations or if program implementation is particularly complex. As a general rule, more in‐ tensive training and more follow‐up support from the program developer will increase the effective‐ ness and sustainability of a program over time. Some programs provide excellent technical assis‐ tance; staff members are accessible and willing to address questions that arise while the program is being implemented. Often this technical assistance is free, but sometimes program designers charge an additional fee for it. Therefore, the benefits and costs of technical assistance should be kept in mind when selecting an evidence‐based program. Finally, while the scientific literature and infor‐ mation from the program developer provide key information about program quality, don’t over‐ look the experience of practitioners who have imp‐ lemented the program. Ask whether they encountered any obstacles when implementing the program, whether they believe the program was effective, which audiences seemed to respond most positively to the program, and whether they would recommend the program for your sit‐ uation. This type of information is usually not included in scientific program evaluations but is a critically important consideration for most practitioners.
ORGANIZATIONAL RESOURCES A final set of factors to consider when selecting a program is related to the resources required for carrying out the program. Consider whether your organization has the expertise, staff, financial sup‐ port and time available to implement the pro‐ gram. Implementing evidence‐based programs is usually fairly time‐ and resource‐intensive. For example, evidence‐based programs often require facilitators to attend multi‐day trainings or call for facilitators with particular qualifications. Even if a program is a good fit for your community, if your organization doesn’t have the human or financial resources to adequately implement the program, its chances of success are limited.
Guidelines for Selecting an Evidence‐Based Program 6 What Works, Wisconsin – Research to Practice Series, #3
WHAT WORKS, WISCONSIN: RESEARCH TO PRACTICE SERIES This is one of a series of Research to Practice briefs prepared by the What Works, Wisconsin team at the University of Wisconsin–Madison, School of Human Ecology, and Cooperative Extension, University of Wisconsin–Extension. All of the briefs can be downloaded from: http://whatworks.uwex.edu This series expands upon ideas that are discussed in What Works, Wisconsin: What Science Tells Us about Cost‐ Effective Programs for Juvenile Delinquency Prevention, which is also available for download at the address above. This publication may be cited without permission provided the source is identified as: Small, S.A., Cooney, S.M., Eastman, G., & O’Connor, C. (2007). Guidelines for selecting an evidence‐based program: Balancing community needs, program quality, and organizational resources. What Works, Wisconsin Research to Practice Series, 3. Madison, WI: University of Wisconsin–Madison/Extension. This project was supported by Grant Award No. JF‐04‐PO‐0025 awarded by the Wisconsin Office of Justice Assistance through the Wisconsin Governor’s Juvenile Justice Commission with funds from the Office of Juvenile Justice and Delinquency Prevention. The authors wish to thank Mary Huser of the University of Wisconsin–Extension for her edits, comments, and suggestions in the development of this Research to Practice brief.
In addition, when selecting a program it makes sense to assess your organization’s long‐term goals and consider which programs have the best chance of being continued in the future. Programs that require significant external funding are especially prone to abandonment after the funding runs out. Some programs are more readily adopt‐ ed by existing organizations and are easier to support over the long run. Think about whether a program has a good chance of being integrated into the base programming of your organization. Can the program be continued in the future with existing staff and resources or will it always require external support? Lastly, because many evidence‐based programs are resource intensive, think about collaborating with other organizations in the community to deliver a program. Selecting a program that meets the needs of two or more agencies may allow for the pooling of resources, thus enhancing the
likelihood that the program can be adequately funded, implemented and sustained over time. Additionally, such an arrangement can lead to positive, long‐term partnerships with other com‐ munity agencies. While all three of these factors are important, some may be more crucial to your organization than others. The key to selecting the best program for your particular situation involves balancing different priorities and trade‐offs and finding a program that best meets these competing demands. By selecting a high quality program that matches the needs of your audience and com‐ munity and the resources of your organization, you greatly enhance the likelihood that you will have an effective program that will have a long‐ term impact and improve the lives of its participants.