Discussion 1: Evidence-Based Practices: How Do You Know They Are Working?

The Journal of Special Education 47(2) 71 –82 © Hammill Institute on Disabilities 2011 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0022466911420877 journalofspecialeducation .sagepub.com

One of the most critical issues in contemporary special edu- cation is the significant and persistent gap between research documenting the effectiveness of practices and the actual instruction that occurs in typical classrooms (Carnine, 1997; Cook & Schirmer, 2006). Interventions shown by reliable research to positively impact student performance are not implemented commonly in classrooms. Yet other practices shown to have little, no, or negative effects on students’ outcomes are applied frequently. For example, special edu- cation practitioners reported regularly using some practices demonstrated to be ineffective (e.g., modality instruction) but indicated implementing other practices validated by research (e.g., mnemonic strategies) relatively infrequently (Burns & Ysseldyke, 2009; see also Jones, 2009).

Traditionally, educators have used sources such as per- sonal experience, tradition, and expert opinion to discern what works in the classroom. Although these traditional sources have produced many insights and effective prac- tices, they are also prone to Type I and Type II errors (i.e., determining that ineffective practices are effective and that effective practices are ineffective, respectively), which has contributed to the research-to-practice gap in education (Cook & Smith, in press). Educational research, though far from infallible (Gallagher, 2004), can be used to control for many of the common shortcomings of traditional methods for determining what works to improve student outcomes (Cook, Tankersley, Cook, & Landrum, 2008).

Evidence-based practices (EBPs) are instructional techniques with meaningful research supporting their effectiveness that represent critical tools in bridging the

research-to-practice gap and improving student outcomes (e.g., Cook, Smith, & Tankersley, 2011; Slavin, 2002). In recent years, various organizations have begun to identify EBPs for students with disabilities (see Table 1). However, despite their significant potential to improve student out- comes, considerable misunderstanding regarding EBPs exists that may impede the advent of evidence-based spe- cial education. In this article, we (a) discuss the impor- tance of clear thinking and terminology related to EBPs and what works in education; (b) define EBPs and discuss how they are identified; (c) differentiate the term EBP from related terms such as research-based, best, and rec- ommended practices; (d) consider the purview of EBPs; and (e) present relevant caveats related to EBPs. Given the specific focus of this article, we limit our discussion to foundational issues related to thinking and communicating about EBPs and do not examine any number of other important issues in this area (e.g., the appropriate role of science in schools and classrooms; the relevance of out- come measures in EBPs; the importance of external valid- ity in determining EBPs; and the validity of specific standards related to the design, quality, quantity, and effect size required for EBPs).

420877 SED47210.1177/0022466911 420877The Journal of Special Education Cook and Cook © The Author(s) 2011

Reprints and permissions:

1University of Hawaii at Manoa, Honolulu, USA

Corresponding Author: Bryan G. Cook, University of Hawaii at Manoa, Wist Hall 206, 1776 University Avenue, Honolulu, HI 96822, USA. E-mail: [email protected]

Unraveling Evidence-Based Practices in Special Education

Bryan G. Cook1 and Sara Cothren Cook1

Abstract

Evidence-based practices (EBPs) are instructional techniques that meet prescribed criteria related to the research design, quality, quantity, and effect size of supporting research, which have the potential to help bridge the research-to-practice gap and improve student outcomes. In this article, the authors (a) discuss the importance of clear understanding and communication regarding EBPs and what works in special education; (b) define EBPs and discuss how they are identified; (c) differentiate the term EBP from related terms such as research-based, best, and recommended practices; (d) consider the purview of EBPs; and (e) present relevant caveats related to EBPs. The authors conclude by providing recommendations to facilitate special educators’ clear and effectual thinking and communication about EBPs.

Keywords

evidence-based practices, research-to-practice gap, terminology

Article

72 The Journal of Special Education 47(2)

The Need for Clear Thinking and Terminology The causes underlying the research-to-practice gap are multiple and complex. It seems that a necessary (although certainly not sufficient) condition for broadly implement- ing effective practices is establishing clearly which practices are most likely to work for whom. Accordingly, educators have used a number of approaches to identify what they believe are the most effective practices, referring to these instructional approaches with terms such as best practices, recommended practices, research-based prac- tices, practices supported by scientifically based research, and EBPs. These labels all share the same underlying purpose—to determine and denote effective practices. However, the terms have distinct meanings and imply dif- ferent standards of rigor related to their empirical support.

EBPs represent educators’ most recent efforts to reliably identify what works and, in comparison to other methods, have “upped the ante” in terms of the research support required. EBPs are unique among efforts to determine effective practices in that, as detailed below, a trustworthy body of research that meets specific standards of rigor must support these practices. Originating in medicine in the 1990s (e.g., Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996), the identification of EBPs soon spread to fields such as agriculture, nursing, psychology, and education (Slavin, 2002). As EBPs represent practices found to be effective by the most reliable research, they have significant potential to affect meaningful, positive change in education (Slavin, 2002)—especially for students who are at risk for school

failure (e.g., students with disabilities) and require the most effective instruction to reach their potentials (Dammann & Vaughn, 2001).

However, EBPs are not self-implementing mechanisms that will be embraced and used automatically as they are identified. Indeed, influencing school practices and teacher behaviors has proven to be an extraordinarily difficult endeavor (e.g., Sarason, 1993). Moreover, many educational practitioners mistrust research—the basis for EBPs, gener- ally preferring to rely on more personal sources for deter- mining what and how to teach (e.g., Boardman, Arguelles, Vaughn, Hughes, & Klingner, 2005; Nelson, Leffler, & Hansen, 2009). Thus, to abandon traditional ways of deter- mining what works, teachers will have to be convinced to trust EBPs and accept the label “evidence-based” as a reli- able indicator of what works for their students.

The phrase “evidence-based practice” has become an increasingly popular term in the educational lexicon (Cook et al., in press; Detrich, 2008; Slavin, 2002). On one hand, this may be a positive development, in that it appears to reflect educators’ recognition of the importance and prom- ise of EBPs. On the other hand, EBP is being used perhaps too frequently, in that it commonly is misapplied. At con- ferences, in pre- and in-service trainings, and (perhaps most importantly) in informal discussions in teachers’ lounges and elsewhere, some educators (e.g., practitioners, teacher educators, scholars) use the term EBP ubiquitously and inappropriately. For example, terms such as research-based practice, best practice, and recommended practice often are used synonymously with EBP—sometimes referring to

Table 1. Sources of Evidence-Based Practices in Special Education With Selected Evidence-Based Practices.

Source Selected evidence-based practices

Exceptional Children, 75(3) Identified time delay as an evidence-based practice for teaching literacy to students with severe developmental disabilities (Browder, Ahlgrim-Delzell, Spooner, Mims, & Baker, 2009) and self-regulated strategy development as an evidence-based practice for teaching writing to students with or at risk for learning disabilities (Baker, Chard, Ketterlin-Geller, Apichatabutra, & Doabler, 2009).

National Autism Center, National Standards Project (http://www. nationalautismcenter.org/affiliates/ reports.php)

Identified 12 established treatments for individuals with autism below 22 years of age, including antecedent interventions, modeling, naturalistic teaching strategies, peer training interventions, pivotal response treatment, schedules, self-management, and story-based interventions.

National Professional Development Center on Autism Spectrum Disorders (http://autismpdc.fpg. unc.edu/content/evidence-based- practices)

Identified 24 evidence-based practices for individuals with autism below 22 years of age, including antecedent-based interventions, computer-aided instruction, differential reinforcement, discrete trial training, functional behavior assessment, naturalistic intervention, parent-implemented intervention, peer-mediated instruction and intervention, picture exchange communication system, pivotal response training, social narratives, task analysis, time delay, and video modeling.

What Works Clearinghouse (http://ies. ed.gov/ncee/wwc/)

No practices identified as having evidence of positive effects for students with disabilities. Lindamood Phoneme Sequencing and Read Naturally were identified as having evidence of potentially positive effects for students with learning disabilities; Dialogic Reading and Lovaas Model of Applied Behavior Analysis was identified as having evidence of potentially positive effects in early childhood special education.

Cook and Cook 73

empirically validated practices and sometimes referring to practices that are being promoted in the absence of substan- tive research support. Moreover, considerable confusion exists regarding whether and how these terms differ, and what exactly EBP means.

Consequently, many educators may be transferring the cynicism they have developed toward terms like best prac- tice, which often have been used to refer to unsubstantiated techniques (Peters & Heron, 1993), to EBPs. Indeed, some educators likely have already decided to discount EBPs, which may appear to them to be the latest hollow catch- phrase used to promote ineffectual instructional practices. As Cook et al. (in press) noted, rather than helping to clarify which practices are effective, EBPs “may actually contrib- ute to the confusion regarding what works, what doesn’t, and how to tell the difference” due to the term’s frequent misapplication. We recognize that terminology is far from the only issue involved in promoting the broad implementa- tion of EBPs. However, we propose that educators having a full and accurate understanding of what EBP means, how it differs from related terms, the purview of EBPs, and promi- nent caveats associated with EBPs will promote an appre- ciation of EBPs that will facilitate their acceptance and implementation.

Defining and Identifying EBPs A myriad of organizations and approaches for identifying EBPs exist in education and related disciplines (Cook et al., 2011; Mayer, 2011; Slavin, 2008), each with their own take on what is necessary for a practice to be considered evi- dence based. However, approaches tend to coalesce around four issues that we consider fundamental for determining EBPs: research design, quality of research, quantity of research, and magnitude of effect of supporting studies (see Cook et al., in press). Generally, then, we define EBPs as practices that are supported by multiple, high-quality stud- ies that utilize research designs from which causality can be inferred and that demonstrate meaningful effects on student outcomes. EBPs are identified by applying specific criteria (for research design, quantity of research, quality of research, and effect size) to the empirical literature on a practice, a process often referred to as an evidence-based review. We briefly illustrate the four defining elements of EBPs by discussing standards for identifying EBPs in spe- cial education as proposed by Gersten et al. (2005) and Horner et al. (2005).

Research Design Although a wide variety of research designs play impor- tant roles in informing and improving education (Odom et al., 2005), research consumers can only infer causality from studies that utilize specific kinds of research designs

(Cook et al., 2008). For example, correlational research may show that a particular practice is associated with increased student outcomes (e.g., students attending classes in which a practice is used regularly exhibit higher achievement scores than students in other classes). However, such studies do not demonstrate that the practice caused desirable stu- dent outcomes (e.g., teachers may have been using the prac- tice because they had high achieving students in their class).

Accordingly, Gersten et al. (2005) and Horner et al. (2005) recommended that special educators consider only group experimental, group quasiexperimental, and single- subject research studies when determining EBPs. These types of research are unique in that they are designed to address whether the independent variable (e.g., an instruc- tional practice) causes change in the dependent variable(s) (e.g., student outcomes; see Cook et al., 2008). That is, when used appropriately, these research designs rule out primary alternative explanations for what may have caused changes in student outcomes other than the practice being examined.

Quality of Research Studies Regardless of the research design, if a study is poorly con- ducted, the results may not be valid. For example, even in random clinical trials—the so-called gold standard of effi- cacy research—if the intervention is implemented incor- rectly, the dependent variables are not measured reliably, severe attrition occurs, or any of a host of other method- ological shortcomings occur, study findings may be errant and misleading. Therefore, a hallmark of EBPs is that sup- porting research studies must meet prescribed standards of methodological rigor (i.e., quality indicators).

In special education, Gersten et al. (2005) recommended that high and acceptable quality group experimental and group quasiexperimental studies must meet 9 of 10 pro- posed essential quality indicators (e.g., adequate fidelity of implementation, at least one measure of generalized perfor- mance). High-quality group studies in special education must also address at least 4 of the 8 specified desirable qual- ity indicators (e.g., attrition not severe and comparable across groups and nature of instruction in control/comparison condition specified), whereas acceptable quality studies must only meet 1 desirable quality indicator. Horner et al. (2005) proposed that single-subject research studies must address all of 21 proposed quality indicators (e.g., adequate implementation fidelity and at least three demonstrations of experimental effect) to be considered in determining EBPs in special education.

Quantity of Research Studies Scientific research is, by its nature, a gradual process. Confidence in research findings is never absolute but is

74 The Journal of Special Education 47(2)

strengthened as findings from multiple studies converge. Therefore, EBPs tend not to be identified on the basis of a single study but instead require multiple high-quality studies of appropriate design to show that a practice reli- ably improves student outcomes. Gersten et al. (2005) specified that at least two high-quality or four acceptable- quality group experimental and quasiexperimental stud- ies must support a practice for it to be considered an EBP in special education. To identify EBPs in special educa- tion using single-subject research, Horner et al. (2005) recommended that at least five high-quality single-subject research studies that (a) were published in peer-reviewed journals, (b) were conducted in at least three different geographical locations, (c) were conducted by at least three different researchers, and (d) included a minimum of 20 participants across studies must support the practice. Other approaches for identifying EBPs also specify a maxi- mum number of studies showing a negative or indeterminate effect of a practice for it to be considered an EBP. For example, the What Works Clearinghouse (WWC; 2008) required that practices with positive effects (their highest rating category) can have no acceptable quality studies dem- onstrating negative effects.

Magnitude of Effect Conceptually, EBPs should have robustly positive, socially valid effects on student outcomes. Inconsequential effects that do not have functional or pedagogical significance are insufficient for a practice to be identified as an EBP. Consequently, Gersten et al. (2005) required that EBPs demonstrate a weighted effect size significantly greater than zero across high and adequate quality studies examin- ing the practice. For single-subject research in special edu- cation, Horner et al. (2005) proposed that all high-quality, single-subject research studies demonstrate that the magni- tude of change in student outcomes as a result of the inter- vention is socially important.

To foster a collective understanding of what EBPs are and appreciation of their potential for improving student out- comes, we recommend that educators exercise caution when referring to practices as EBPs. The term EBP should only denote practices supported by research studies that meet pre- scribed standards in a field regarding research design, quality, quantity, and magnitude of effect; it should not be applied to a practice merely because some research supports its effectiveness, it is recommended by an expert, or it accords with a desired theoretical or ethical perspective.

Differentiating EBPs From Related Terms In our experience, educators frequently fail to distinguish EBPs from other related terms (e.g., research-based practice,

best practice, and recommended practice), which can lead to confusion and disillusionment regarding EBPs. In this section, we describe how these terms differ meaningfully from EBPs.

Research-Based Practices Educators often use the term research-based (and related terms such as data based, research-supported, or empiri- cally validated) practice interchangeably with EBP to imply endorsement of an instructional practice by virtue of its research support. Issues that cause confusion regard- ing research-based practices, especially in relation to EBPs, include the following: (a) research-based practice is used loosely, sometimes referring to ineffective prac- tices and (b) although educators often use the terms syn- onymously, research-based practices differ meaningfully from EBPs.

Research-based practice is a particularly bewildering term because it can be used to describe practices that range from being supported by robust, rigorous research bases to those supported by meager and flawed research bases. Indeed, because seemingly “any program can find some research that supports the principles it incorporates” (Slavin, 2002, p. 18), virtually any practice can be referred to as research based. Typically, though, the term is intended to connote an objective and reliable determination of a prac- tice’s effectiveness on the basis of its research base. That is, in our experience, educators tend to refer to a practice as research based when they believe that research clearly dem- onstrates that it is an effective practice, not when some ini- tial research tentatively suggests that it might be effective. As such, identifying practices as research based to prioritize them for classroom use seems self-evidently beneficial and has been promoted widely as a critical element in bridging the gap between research and practice.

Nonetheless, the research supporting many research- based practices often is not as robust as the term seems to imply. For example, repeated reading is frequently promul- gated as an effective, research-based practice for students with and at risk for learning disabilities in reading (e.g., Council for Learning Disabilities, n.d.; LD Online, n.d.). However, Chard, Ketterlin-Geller, Baker, Doabler, and Apichatabutra (2009) reported that although repeated read- ing was supported by multiple studies, its research base did not include a sufficient number of high-quality studies that used appropriate research designs for the practice to be con- sidered an EBP for children with and at risk for learning disabilities. EBPs and research-based practices, then, are not synonymous (see Table 2 for a summary of distinctions between EBPs and research-based practices). Although all EBPs are research based, many—most, we suspect—practices considered research based do not meet the rigorous stan- dards of EBPs.

Cook and Cook 75

We have two recommendations for researchers regarding the use of the term research-based practice. First, because it can refer to practices with such varied empirical support, we suggest that providing a brief description of the research support for the practice will be more meaningful and infor- mative than simply referring to a practice as research based. Indicating that “Practice X is research-based” is not particu- larly informative. In contrast, describing a practice as being “supported by two high quality, quasi-experimental research studies with an average effect size of d = 0.50” provides a more meaningful indication of a practice’s research base. Alternatively, organizations can specify criteria for the amount and type of research that they require to consider a practice research based. Second, to avoid confusion, “research-based practice” should not be used interchange- ably with EBP. As EBP connotes a higher standard of empir- ical support, if a practice meets the rigorous standards for EBPs, it should be referred to as such—even though it is technically both evidence and research based.

Best and Recommended Practices Description and advocacy of best practices and recom- mended practices can be found in virtually every area of education. Like EBPs and research-based practices, best and recommended practices originated as an “effort by our field to integrate the large body of research that has emerged” and bring it to bear on improving practice (Peters & Heron, 1993, p. 376). However, like research-based practices, best and recommended practices have no commonly agreed-on criteria. Thus, any discussion of best or recommended prac- tices begs at least two questions: best or recommended by whom and on what basis?

Some overlap does exist between (a) best and recom- mended practices, (b) research-based practices, and (c) EBPs. Most practices supported by research are promoted as best or recommended, and practices may be deemed best or recom- mended on the basis of their research support. However, edu- cators have referred to practices as best and recommended

due to tradition, expert opinion, theory, and moral values— regardless of whether they are validated empirically. Indeed, an instructional technique can be called a best or recom- mended practice for virtually any reason, so long as some- one is willing to advocate it. Thus, conceptually, practices promoted as best or recommended may not be research based and are even less likely to be EBPs. Coteaching, for example, often is considered a best or recommended prac- tice for included students with disabilities. However, the research support for coteaching is inconclusive and clearly does not meet EBP criteria (Cook, McDuffie-Landrum, Oshita, & Cook, 2011).

The terms best and recommended practices have been overused and misapplied for so many years (see Peters & Heron, 1993) that we encourage educators not to describe practices in these ways. The phrases invite ambiguity and have taken on a negative connotation for many educators who have seen decades of questionable best and recom- mended practices come and go. Expert opinion, theory, and values are not unimportant considerations in deciding what and how to teach. But advocating for a practice on these bases should not, in our opinion, be done under the guise of best or recommended practice, as these terms oftentimes are interpreted to indicate that the practice is supported by research. If a practice is research based or an EBP, we recommend that those terms be used to indicate that empirical support exists for the practice. Practitioners should approach best and recommended practices cau- tiously (i.e., caveat emptor) and not trust that they are empirically validated without being provided with evidence to that effect.

The Purview of EBPs In addition to uncertainty regarding terminology, other sources of potential confusion regarding EBPs include (a) the meaning of “practice” in EBP and (b) whether EBP refers to a decision-making process or an empirically vali- dated practice.

Table 2. Key Distinctions Between Evidence-Based and Research-Based Practices.

Distinguishing characteristic Evidence-based practices Research-based practices

Design of supporting research studies

Supporting studies must establish causality; typically restricted to true group experiments, group quasi experiments, and single-subject research

Supporting studies might use any research design, including those that do not establish causality (e.g., correlational research, case studies, and qualitative research)

Quality of supporting research studies

Supporting studies must address rigorous indicators of methodological quality

Quality of supporting studies may not be systematically evaluated

Quantity of supporting research studies

Typically supported by >1 studies of acceptable quality and design

Might be supported by a single study

Magnitude of effect of supporting research studies

Supporting studies must meet prescribed level of effect (e.g., statistical significance and minimum effect size)

Especially for case studies and qualitative research, no criteria typically applied for minimum effect

76 The Journal of Special Education 47(2)

What is a Practice in EBPs?

Many educators lack clarity on the scope of practices that are examined as potential EBPs and may expect that broad policies (e.g., inclusion, transition), instructional frame- works (e.g., universal design for learning), and educational philosophies (constructivism, behaviorism) will be exam- ined as being evidence based or not. However, practices under consideration as EBPs must feature replicable instruc- tional behaviors that are concretely defined. Operationally defined instructional procedures are necessary for (a) schol- ars to discern which research studies to include and exclude from evidence-based reviews and (b) practitioners to apply the EBP with fidelity. Thus, behaviorism is not an appropri- ate topic for an evidence-based review because it cannot be operationalized as a discrete set of instructional procedures. However, specific practices (e.g., self-monitoring, token economy) and programs (e.g., the University of California, Los Angeles Young Autism Program, for example, Smith & Lovaas, 1998) based on behavioral principles can be exam- ined as potential EBPs.

Horner, Sugai, and Anderson (2010) recommended that in addition to having operationalized, replicable proce- dures, descriptions of practices being considered as EBPs should include the target setting(s), target population(s), qualifications of interventionists, expected outcomes, and conceptual or theoretical underpinnings. Defining these parameters of a practice provides clear guidance to review- ers as to what research studies to include (and exclude) in evidence-based reviews and provides important informa- tion for practitioners about the relevance and application of EBPs. For example, it is not particularly useful for a prac- tice to be identified as an EBP generally, as it begs the ques- tion of evidence based for whom, what, and where? Practitioners will find it much more useful to know that a practice is evidence based, for example, for improving reading fluency for elementary students with learning dis- abilities in inclusive settings.

Practices with operationalized procedures can refer to a range of educational approaches, from simple and discrete practices to large educational programs (Horner et al., 2010). The identification of EBPs in some fields (e.g., general edu- cation; see WWC, n.d.) has tended to focus on broad pro- grams, which we refer to as “macro EBPs.” Educational programs, which often encompass entire curricula, typically entail the use of specific instructional materials, instruc- tional approaches and lessons, and even type and amount of training. As such, macro EBPs characteristically provide consumers with a clear framework on the program’s imple- mentation. However, macro EBPs may correspondingly minimize practitioners’ ability to flexibly adapt the practice by necessitating that the entire program be implemented as prescribed.

In other fields, such as special education, efforts to iden- tify EBPs have focused primarily on discrete interventions

or “micro EBPs” (e.g., repeated reading, self-regulated strategy development, time delay; see Cook, Tankersley, & Landrum, 2009). As these micropractices do not constitute entire curricula, they afford educators considerable flexibil- ity in how they might be implemented. That is, in contrast to macro EBPs, micro EBPs typically are not associated with prescribed frameworks for implementation that include specific instructional materials, student groupings, or train- ing protocols. However, flexible application of micro EBPs may introduce conditions that differ meaningfully from the contexts in which the practice was validated, which might negatively impact its effectiveness (e.g., use of teacher- made materials, provision of minimal training, application in varied student groupings).

In our view, it is not critical whether EBPs are used in a micro- or macrosense; the approaches represent rela- tively minor variations on a theme (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005) with each entailing certain ben- efits and limitations. Indeed, most approaches for identify- ing EBPs can incorporate both micro- and macro EBPs using a single set of standards. However, whether educators choose to focus on discrete practices, curricular programs, or both, potential EBPs must consist of operationally defined sets of procedures and should specify various parameters of the practices such as the target population, contexts, inter- ventionists, and outcomes (Horner et al., 2010).

Decision-Making Process or Empirically Validated Intervention? Another potential source of confusion is that the term EBP can be used in two different, albeit related, ways. As indi- cated in Table 3, prominent definitions in many fields reflect the perspective that EBP is an approach to professional deci- sion making. In this sense, EBP can be thought of as an action (e.g., “that teacher doesn’t follow fads, she engages in evidence-based practice”). Eddy (2005) referred to this use of the phrase as evidence-based individual decision making, in which professionals use the best available research evidence in concert with other factors to make pedagogical decisions. In contrast, as we have been using the term throughout this article, EBP can also be used to refer to specific practices that have been validated as effec- tive by meeting prescribed standards or evidence-based guidelines. Eddy suggested that these dual meanings of EBP are interdependent. That is, the process of evidence-based decision making relies on the identification of specific EBPs, and specific EBPs have no practical effect without practitioners engaging in evidence-based decision making.

Nonetheless, to avoid misunderstanding, educators should be clear with their terms. One option is to refer to specific EBPs using a distinct term, such as evidence-based inter- vention (e.g., Detrich, 2008; U.S. Department of Education, 2003) or empirically supported treatment (e.g., Woody, Weisz, & McLean, 2005). However, we recommend that

Cook and Cook 77

educators use EBP to refer to the specific practices sup- ported by research that meets prescribed guidelines regard- ing research design, quality, quantity, and effect—as we have throughout this article. To avoid confusion, we sug- gest that educators use “evidence-based education” to describe the broader process of evidence-based decision making, rather than EBP. This is consistent with how terms such as evidence-based medicine and evidence-based nurs- ing are used in other fields.

Caveats Regarding EBPs In addition to discussing what EBPs are, it is important to clarify what EBPs are not. Although EBPs reliably indicate generally effective practices, they should not be thought of as a panacea for a number of reasons, including (a) they are not guaranteed to work for everyone, (b) they are difficult to implement on a broad scale, (c) they are not the only consideration in instructional decision making, and (d) dif- fering standards may lead to confusion regarding what is and is not an EBP.

Nonresponders No instructional intervention will work for everyone no matter its level of research support (Cook et al., 2008). In education, as in other areas of social science, causality is used in a probabilistic rather than absolute manner. Treatment resistors or nonresponders exist for even the most effective interventions. For example, Torgesen (2000) noted that the most effective early reading interventions are ineffective for between 2% and 6% of young children. Thus, all things being equal, EBPs afford the highest likelihood of increasing a targeted outcome, but educators cannot have abject faith that an EBP will work for every individual stu- dent. Educators should therefore systematically monitor the impact of their instruction on student outcomes using

procedures such as progress monitoring (e.g., Stecker & Fuchs, 2000), even when implementing an EBP.

Difficulties in Implementation Just as educators cannot assume that EBPs will work for all students, neither can researchers suppose that generating a list of EBPs will directly translate into changed practice. As Fixsen, Blase, Horner, and Sugai (2009) noted, “An evi- dence-based practice is one thing, implementation of that practice is another thing altogether” (p. 5). Indeed, many educators have expressed hostility and mistrust toward research in general and EBPs in particular (e.g., Boardman et al., 2005; Hammersley, 2005; Jones, 2009). Accordingly, in the field of medicine, Hiss (2004) suggested that although research on identifying EBPs (i.e., Phase 1 research) has received a great deal of attention, research on how to imple- ment EBPs (i.e., Phase 2 research) will be of arguably greater importance. In education, scholars are beginning to focus on this second phase of EBP research by conducting and applying research in the nascent field of implementation science (e.g., Odom, 2009).

Implementation of EBPs is a multifaceted and challeng- ing process, involving multiple stakeholders. Although identification and understanding of EBPs—the primary focus of this article—are important initial steps, implemen- tation of EBPs with fidelity and over time requires much more. Paradoxically, educational practitioners are often expected to implement EBPs using a “train and hope” approach (Stokes & Baer, 1977), which has been shown to be largely ineffective (Joyce & Showers, 2002), rather than utilizing evidence-based guidelines. Although a full treat- ment of the research literature on implementing EBPs is beyond the scope of this article, we refer interested readers to Fixsen et al.’s (2005) cross-disciplinary review of 377 sources on implementation factors. Fixsen et al. provided guidelines for organizations (e.g., schools) to create and

Table 3. Definitions for Evidence-Based Practice in Various Fields.

Source Definition

Sackett, Rosenberg, Muir Gray, Haynes, and Richardson (1996)

Evidence-based medicine: “The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.” (p. 71)

Ingersoll (2000) Evidence-based nursing practice: “The conscientious, explicit and judicious use of theory-derived, research based information in making decisions about care delivery to individuals or groups of patients and in consideration of individual needs and preferences.” (p. 152)

APA Presidential Task Force on Evidence-Based Practice (2006)

Evidence-based practice (in psychology): “The integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences.” (p. 273)

Whitehurst (2002) Evidence-based education: “Integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction.”

The Wing Institute (n.d.) Evidence-based education: “A paradigm by which educational stakeholders use empirical evidence to make informed decisions about education interventions (policies, practices, and programs).”

Reprinted from Cook and Cook (2011).

78 The Journal of Special Education 47(2)

support high-fidelity implementation of EBPs by institut- ing six core implementation components: staff selection, preservice and in-service training, ongoing consultation and coaching, evaluation, facilitative administrative sup- port, and systems interventions. Most generally, Fixsen et al. advocated switching from a personnel-centered (i.e., relying on the expertise of individual practitioners) to a practice-centered (i.e., institutionalizing practices such as the core implementation components) approach to facili- tate the implementation of EBPs.

Additional Factors in Effective Instruction We want to be careful not to suggest that EBP status and research support for a practice are the only important con- siderations in determining what and how to teach. EBPs represent practices that meet a high bar of empirical valida- tion, but they do not trump practical wisdom and common sense when making instructional decisions. That is, just because a practice is identified as an EBP does not neces- sarily mean that it should be implemented in a particular situation. For example, if an EBP has been validated as effective for improving reading fluency for elementary students without disabilities, it should not be selected and implemented by special educators seeking to improve the reading comprehension of middle school students with moderate and severe intellectual impairments just because it is an EBP.

Similarly, educators may be justified in implementing practices that have not been identified as EBPs. Law or pol- icy mandates, community and professional values, personal teaching styles, individual learner histories, and other factors represent important considerations that may take precedence over EBP status when making instructional decisions. In fact, just because a practice has not been identified as an EBP does not necessarily mean that it is ineffective. Cook and Smith (in press) noted three potential reasons why a practice may not be identified as an EBP: (a) It has not yet been the subject of an evidence-based review, (b) it has been reviewed but insufficient high-quality research using appro- priate designs has been conducted to meaningfully deter- mine its impact, or (c) high-quality research using appropriate designs has shown that it has minimal or negative effects on student outcomes. Although the first two reasons occur with the greatest frequency (e.g., many practices have yet to be the subject of an evidence-based review), only the last rea- son clearly indicates that a practice is ineffective. Indeed, all EBPs were at one point considered “non-EBPs” because an evidence-based review had not yet been conducted on them or insufficient high-quality research had been conducted to conclusively determine their effectiveness. Savvy educators will use EBPs as important tools in their instructional reper- toire but not be limited to these techniques.

We also recommend that teachers rely on their practical wisdom and common sense when deciding which practice

to implement when two or more EBPs have been identified for a targeted outcome. As all EBPs have met a high bar for empirical support, educators can be confident that they are highly likely to be effective for the target population when implemented appropriately. As such, when multiple, rele- vant EBPs are identified, educators should prioritize EBPs according to the best fit for their needs. Imagine, for exam- ple, a teacher who wishes to increase the on-task behavior of her fifth-grade students with behavioral disorders. If one EBP has been shown to improve the on-task behavior of elementary-age students in general but requires time and expertise that the teacher does not possess and another EBP has been shown to work for fifth-grade students with behav- ioral disorders specifically and does not require additional resources or expertise, it seems clear that the second EBP is the best fit and should be selected. Among EBPs with simi- lar levels of fit, we recommend that educators target the EBP with the greatest effect size (i.e., the EBP that results in the greatest level of improved outcomes) and, finally, pri- oritize EBPs according to the level of evidence (e.g., although educators can be confident in all EBPs, they can be more confident in an EBP shown to be effective by four random clinical trials and six single-subject studies than in an EBP supported by two random clinical trials).

Multiple Approaches to Identify EBPs It is important to recognize that EBPs are being identified using different procedures and standards (Cook et al., in press; Slavin, 2008), which is significant because the import of an EBP is derived from the particular standards and parameters applied in the evidence-based review. Although standards for EBPs generally include criteria related to research design, methodological quality, quantity of supporting research, and magnitude of effect size, it is perhaps not surprising that reasonable differences of opin- ion exist among stakeholders as to exactly how much and what type of research support is necessary to consider a practice evidence based. At one level, the lack of uniform EBP standards might be considered beneficial and stimulat- ing, allowing for the expression of varied perspectives on the complex issue of determining the effectiveness of instructional practices. However, such differences may also generate confusion and distrust among educators who find it vexing that a practice is considered evidence based by one organization and not another. For example, Direct Instruction is reported by the Best Evidence Encyclopedia (n.d.) to be a program with strong evidence of effectiveness (its highest category) for struggling readers, whereas it is considered to be a promising (but not proven practice) prac- tice by the Promising Practices Network (n.d.), and is reported to have no discernible effects by the WWC (2007).

To avoid such confusion, educators might desire that only one or a very few trustworthy organizations be allowed to conduct evidence-based reviews. As much as this might

Cook and Cook 79

reduce conflicting messages, it is not a practical solution— just as any individual can conduct research, any individual or group can identify and disseminate EBPs. Moreover, multiple approaches for identifying EBPs will enable different groups of educators (e.g., special educators) to use criteria that reflect their research traditions and values as well as focus on prac- tices of particular interest to them. Practically speaking, given the time and expertise involved in conducting evidence-based reviews, we expect that the vast majority will be carried out by funded organizations and centers. However, because myriad groups and individuals can, do, and will continue to identify EBPs using a variety of criteria, it is important that educators understand how each EBP is determined. To facilitate broad understanding of EBPs and the standards on which they were identified, we recommend that professional organizations make available (and systematically update) inventories of EBPs relevant to their members that include brief, accessible descriptions of the review processes, qualifications of review- ers, and criteria used by each source of the EBPs listed.

Conclusion Recent educational reforms and legislation (e.g., Individuals With Disabilities Education Act of 2004; No Child Left Behind Act of 2001) stress the important role that scientific research can and should play in determining what works and making instructional decisions. As might be expected, educa- tors have developed new terms as thinking regarding the use of research in instruction has evolved. The terms we consid- ered in this article (i.e., EBP, research-based practice, and best or recommended practice) form a hierarchy of empirical rigor that corresponds with the terms’ chronological emergence.

Educators have used terms such as best and recom- mended practices to denote preferred practices since at least the early 1980s (Peters & Heron, 1993). However, best and recommended practices can be promoted on vari- ous grounds (e.g., expert opinion, moral values) and may or may not be empirically validated. As concerns regarding the research-to-practice gap became prominent, the term research-based practices emerged as a way to recognize only those practices supported by research. However, prac- tices have been denoted as research based despite having limited or flawed research support. To guard against the likelihood that some research-based practices are actually ineffective, EBPs were introduced to identify practices shown to be effective not just by any research but by a number of high-quality studies that utilize research designs establishing causality and that demonstrate meaningful effects on student outcomes.

As each new term entered the educational lexicon, previ- ously popular phrases were not abandoned, and the meanings of the terms and differences between them were never made clear to many educators. Thus, considerable confusion and misapplication of EBPs, research-based practices, best and recommended practices, and related terms have occurred and continue to occur. This confusion has consequences, as many educators have become perplexed and disillusioned not only with the terms but also with the educational reforms and instructional practices they represent. To facilitate clear understanding of and communication related to EBPs, we defined EBPs, differentiated them from other associated terms, discussed potentially confusing aspects of EBPs, and highlighted related caveats. See Table 4 for a summative list of recommended “dos and don’ts” drawn from these

Table 4. Recommended Dos and Don’ts Regarding Evidence-Based Practices.

Don’t . . . Do . . .

Refer to practices with limited or no research support as EBPs.

Refer to practices as EBPs only when supporting body of research meets prescribed criteria related to design, quality, quantity, and effect size.

Refer to practices as research-based or best practices without providing supporting details.

Indicate the type and/or level of research support for research-based practices; specify nonempirical reasons for recommending practices (e.g., legal, ethical, theoretical).

Assume that EBPs will work for every student.

Monitor student progress to identify nonresponders to EBPs.

Assume that EBPs will be routinely implemented after dissemination and training.

Facilitate implementation of EBPs by institutionalizing core implementation components (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005).

Assume that using EBPs is the only or primary element of effective teaching.

Prioritize EBPs, implement EBPs within broader framework of effective teaching.

Assume that non-EBPs are ineffective. Find out whether a practice is not considered an EBP because (a) an evidence-based review has not been conducted on it, (b) insufficient research has been conducted to conclude whether it is an EBP, or (c) it has been shown to be ineffective.

Select randomly among EBPs that seem appropriate.

When more than one EBP fits one’s needs, prioritize first by best fit, then by effect, then by level of evidence.

Assume that EBPs are identified in the same way.

Become familiar with the criteria used by prominent sources of EBPs.

Note. EBP = evidence-based practices.

80 The Journal of Special Education 47(2)

discussions for educators to use when thinking and commu- nicating about EBPs.

Fulfilling EBPs’ potential to broadly improve students’ outcomes will require much more than clear understand- ing and careful use of terminology. Indeed, substantial and difficult work must be accomplished in such areas as con- ducting evidence-based reviews, refining and potentially standardizing criteria for EBPs, effectively disseminating EBPs, and supporting the broad adoption and maintenance of EBPs. Yet, as the Taoist proverb suggests, “The jour- ney of a thousand miles begins beneath one’s feet.” Moreover, moving toward achieving the grander goals of evidence-based special education may depend on the foundation of clear understanding and communication regarding EBPs.

Acknowledgment

The authors thank Amy Casey-Hutcherson, Lynn Fuchs, Karen Harris, Tim Landrum, Kathleen Lane, and Sam Odom for their thoughtful review and insights on previous versions of this manuscript.

Authors’ Note

This article is based on Cook and Cook (2011), a noncopyrighted white paper available at http://www.cecdr.org/

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psycholo- gist, 61, 271–285.

Baker, S. K., Chard, D. J., Ketterlin-Geller, L. R., Apichatabutra, C., & Doabler, C. (2009). The basis of evidence for self-regulated strategy development for students with or at risk for learning disabilities. Exceptional Children, 75, 303–318.

Best Evidence Encyclopedia. (n.d.). Top-rated programs. Retrieved from http://www.bestevidence.org/reading/strug/top.htm

Boardman, A. G., Arguelles, M. E., Vaughn, S., Hughes, M. T., & Klingner, J. (2005). Special education teachers’ views of research-based practices. Journal of Special Education, 39, 168–180.

Browder, D., Ahlgrim-Delzell, L., Spooner, F., Mims, P. J., & Baker, J. N. (2009). Reviewing the evidence base for using time delay to teach picture and word recognition to students with severe developmental disabilities. Exceptional Children, 75, 343–364.

Burns, M. K., & Ysseldyke, J. E. (2009). Reported prevalence of evidence-based instructional practices in special education. Journal of Special Education, 43, 3–11.

Carnine, D. (1997). Bridging the research-to-practice gap. Excep- tional Children, 63, 513–521.

Chard, D. J., Ketterlin-Geller, L. R., Baker, S. K., Doabler, C., & Apichatabutra, C. (2009). Repeated reading intervention for students with learning disabilities: Status of the evidence. Exceptional Children, 75, 263–281.

Cook, B. G., & Cook, S. C. (2011). Thinking and communicating clearly about evidence-based practices in special education. Council for Exceptional Children’s Division for Research. Retrieved from http://www.cecdr.org/

Cook, B. G., McDuffie-Landrum, K. A., Oshita, L., & Cook, S. C. (2011). Co-teaching and students with disabilities: A critical analysis of the empirical literature. In D. P. Hallahan & J. M. Kauffman (Eds.), The handbook of special education (pp. 147–159). New York, NY: Routledge.

Cook, B. G., & Schirmer, B. R. (Eds.). (2006). What is special about special education: The role of evidence-based practices. Austin, TX: PRO-ED.

Cook, B. G., & Smith, G. J. (in press). Leadership in instruc- tion: Evidence-based practices in special education. In M. L. Boscardin, J. Crockett, & B. Billingsley (Eds.), Handbook of leadership in special education. London, England: Taylor & Francis.

Cook, B. G., Smith, G. J., & Tankersley, M. (2011). Evidence-based practices in education. In S. Graham, T. Urban, & K. Harris (Eds.), APA educational psychology handbook (Vol. 3). Wash- ington, DC: American Psychological Association.

Cook, B. G., Tankersley, M., Cook, L., & Landrum, T. J. (2008). Evidence-based practices in special education: Some practical considerations. Intervention in School and Clinic, 44, 69–75.

Cook, B. G., Tankersley, M., & Landrum, T. J. (Eds.). (2009). Evidence-based practices for reading, math, writing, and behavior [Special issue]. Exceptional Children, 75.

Council for Learning Disabilities. (n.d.). CLD Infosheets: Second- ary students with learning disabilities in reading: Developing reading fluency. Retrieved from www.ldblog.com/wp-content/ docs/CLDinfosheet-RdFluency.pdf

Dammann, J. E., & Vaughn, S. (2001). Science and sanity in spe- cial education. Behavioral Disorders, 27, 21–29.

Detrich, R. (2008). Evidence-based, empirically supported, or best practice? A guide for the scientist-practitioner. In J. K. Luiselli, D. C. Russo, W. P. Christian, & S. M. Wilczynski (Eds.), Effec- tive practices for children with autism (pp. 3–25). Oxford, UK: Oxford University Press.

Eddy, D. M. (2005). Evidence-based medicine: A unified approach. Health Affairs, 24, 9–17.

Fixsen, D. L., Blase, K. A., Horner, R., & Sugai, G. (2009). Con- cept paper: Developing the capacity for scaling up the effec- tive use of evidence-based programs in state departments of education. Retrieved from http://www.fpg.unc.edu/~sisep/ resources.cfm

Cook and Cook 81

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication #231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. Retrieved from http://www.fpg.unc.edu/~nirn/resources/publications/ Monograph/pdf/Monograph_full.pdf

Gallagher, D. J. (2004). Educational research, philosophical ortho- doxy, and unfulfilled promises: The quandary of traditional research in U.S. special education. In G. Thomas & R. Pring (Eds.), Evidence-based practices in education (pp. 119–132). Columbus, OH: Open University Press.

Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experi- mental and quasi-experimental research in special education. Exceptional Children, 71, 149–164.

Hammersley, M. (2005). Is the evidence-based practice movement doing more good than harm? Reflections on Iain Chambers’ case for research-based policy making and practice. Evidence & Policy, 1, 85–100.

Hiss, R. G. (2004). Translational research—Two phases of a con- tinuum. In National Institutes for Health and the Center for Disease Control and Prevention (Ed.), From clinical trials to community: The science of translating diabetes and obesity research (pp. 11–14). Bethesda, MD: Natcher Conference Center, National Institutes of Health.

Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to iden- tify evidence-based practice in special education. Exceptional Children, 71, 165–179.

Horner, R. H., Sugai, G., & Anderson, C. M. (2010). Examining the evidence base for school-wide positive behavior support. Focus on Exceptional Children, 42(8), 1–14.

Individuals With Disabilities Education Act of 2004, 20 U.S.C. 1415 et seq. (2004).

Ingersoll, G. L. (2000). Evidence-based nursing: What it is and what it isn’t. Nursing Outlook, 48, 151–152.

Jones, M. L. (2009). A study of novice special educators’ views of evidence-based practices. Teacher Education and Special Education, 32, 101–120.

Joyce, B., & Showers, B. (2002). Achievement through staff devel- opment (3rd ed.). Alexandria, VA: Association for Supervi- sion and Curriculum Development.

LD Online. (n.d.). Research-based principles for improving the reading achievement of America’s children. Retrieved from http://www.ldonline.org/article/357

Mayer, M. J. (2011). Evidence-based standards and methodologi- cal issues in school violence and related prevention research in education and the allied disciplines. In S. R. Jimerson, A. B. Nickerson, M. J. Mayer, & M. J. Furlong (Eds.), The handbook of school violence and school safety: International research and practice (2nd ed.). New York, NY: Routledge.

Nelson, S. R., Leffler, J. C., & Hansen, B. A. (2009). Toward a research agenda for understanding and improving the use of

research evidence. Portland, OR: Northwest Regional Educa- tional Laboratory. Retrieved from http://educationnorthwest. org/webfm_send/311

No Child Left Behind Act , P. L. No. 107-110, 115 Stat. 1425 et seq. (2001).

Odom, S. L. (2009). The ties that bind: Evidence-based practice, implementation science, and outcomes for children. Topics in Early Childhood Special Education, 29, 53–61.

Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scien- tific methods and evidence-based practices. Exceptional Chil- dren, 71, 137–148.

Peters, M. T., & Heron, T. E. (1993). When the best is not good enough: An examination of best practice. Journal of Special Education, 26, 371–385.

Promising Practices Network. (n.d.). Direct instruction. Retrieved from http://www.promisingpractices.net/program. asp?programid=146

Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. British Medical Journal, 312, 71–72.

Sarason, S. B. (1993). The predictable failure of educational reform: Can we change course before it’s too late? San Francisco, CA: Jossey-Bass.

Slavin, R. E. (2002). Evidence-based education policies: Transform- ing educational practice and research. Educational Researcher, 31(7), 15–21.

Slavin, R. E. (2008). What works? Issues in synthesizing educa- tional program evaluations. Educational Researcher, 37(1), 5–14.

Smith, T., & Lovaas, I. (1998). Intensive and early behavioral intervention with autism: The UCLA Young Autism Project. Infants & Young Children, 10, 67–78.

Stecker, P. M., & Fuchs, L. S. (2000). Effecting superior achieve- ment using curriculum-based measurement: The importance of individual progress monitoring. Learning Disabilities Research & Practice, 15, 128–124.

Stokes, T. F., & Baer, D. M. (1977). An implicit technology of generalization. Journal of Applied Behavior Analysis, 10, 349–367.

Torgesen, J. (2000). Individual differences in response to early interventions in reading: The lingering problem of treatment resisters. Learning Disabilities Research & Practice, 15, 55–64.

U.S. Department of Education. (2003). Identifying and implement- ing educational practices supported by rigorous evidence: A user friendly guide. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.

What Works Clearinghouse. (2007). Intervention: Direct instruc- tion. Retrieved from http://ies.ed.gov/ncee/wwc/reports/early _ed/dir_instruct/

What Works Clearinghouse. (2008). Procedures and standards handbook (version 2.0). Institute of Educational Sciences.

82 The Journal of Special Education 47(2)

Retrieved from http://ies.ed.gov/ncee/wwc/references/idocviewer/ doc.aspx?docid=19&tocid=1

What Works Clearinghouse. (n.d.). What Works Clearinghouse. Institute of Education Sciences. Retrieved from http://ies .ed.gov/ncee/wwc/

Whitehurst, G. J. (2002). Evidence-based education (Pow- erpoint presentation at Student Achievement and School

Accountability conference). Retrieved from ies.ed.gov/direc- tor/pdf/2002_10.pdf

Wing Institute. (n.d.). What is evidence-based education? Retrieved from http://www.winginstitute.org/Evidence-Based- Education/what-is-evidence-based-education/

Woody, S. R., Weisz, J., & McLean, C. (2005). Empirically supported treatments: 10 years later. Clinical Psychologist, 58(4), 5–11.