Applied Sciences Discussion Four-Assignment Review

FreemanSugai2013RecognizingEBPfromSSR1.pdf

Home >Applied Sciences homework help >Applied Sciences Discussion Four-Assignment Review

Professional Practice

Identifying Evidence- Based Special Education Interventions From Single-Subject Research Jennifer Freeman and George Sugai

Special educators are aware of the

need to use evidence-based academic

and behavioral interventions in their

classrooms. Using research to guide

decision making in the classroom—

whether at the program, school, or dis-

trict level—can help ensure students

are effectively served. A second-grade

resource room teacher, Mark, struggles

with managing the behavior of several

students who have trouble staying on

task and completing work. Cindy, a

middle school special education

teacher, teaches a self-contained pro-

gram and is looking for a way to teach

writing skills to a seventh-grade stu-

dent with a learning disability. How

can Mark and Cindy find out about

best practices, and read reviews or

summaries of recent studies? Cur-

rently, there is no rigorous and com-

prehensive database to support educa-

tors. It also can be difficult for teach-

ers, schools, and policy makers to

interpret the results of research. Des-

pite the limitations in currently avail-

able resources, there are ways for spe-

cial education teachers and administra-

tors to identify evidence-based prac-

tices that fit their specific contexts.

6 COUNCIL FOR EXCEPTIONAL CHILDREN

The most recent authorizations of the Individuals With Disabilities Education Act (IDEA, 2006) and the No Child Left Behind Act of 2001 (2006] provide a mandate for the use of evidence-based practices (EBPs) in the areas of aca- demic and behavioral education (20 U.S.C. § 1414[b][6][B], 20 U.S.C. § 7801 [37]). However, the lack of a clear understanding of what EBPs are and the difficulty finding EBPs that address the specific needs of students can be frustrating for many teachers and administrators. In response, education- al researchers have attempted to evalu- ate the current research base and to define clear criteria for determining

and evaluating EBPs without clear criteria.

In order to address this issue, the U.S. Department of Education created the What Works Clearinghouse (WWC) as a resource for teachers. In addition, several other agencies, such as the Council for Exceptional Children (CEC), the American Psychological Association (APA), and the Promising Practices Network (PPN), also have attempted to bring together research results in a concise, user-friendly way for educators. Each organization has developed criteria for determining EBPs, and their web sites provide quick and easy access to EBPs for spe-

The body of educational research in special education is extremely

varied in both methodology and quality, often leaving special education teachers with the very difficult task of identifying and evaluating

[evidence based practices] without clear criteria.

EBPs (Odom et a l , 2005). However, the body of educational research in special education is extremely varied in both methodology and quality, often leaving special education teachers with the very difficult task of identifying

cific categories of students. However, differing criteria, a mismatch between current research practices and EBP standards, and limited access to specific research studies limit the use- fulness of these resources for special

education teachers. In particular, prior to 2010 the WWC prioritized random- ized group design studies and excluded from consideration the results of stud- ies using single-subject designs. This practice was particularly problematic for special educators due to the preva- lence of single-subject designs in spe- cial education research (Horner et a l , 2005).

In 2010, the WWC announced stan- dards that included criteria for identify- ing EBPs through the use of single-sub- ject research (Kratochwill et a l , 2010), and both the CEC and APA have also developed standards. However, the WWC standards do not yet align with the current standard of practice in the field of special education. Conse- quently, few special education prac- tices meet both WWC design and evi- dence standards.

Identilying Evidence-Based Practices

Table 1 provides a comparison of sev- eral web-based resources to assist spe- cial educators looking for EBPs. How- ever, to benefit from these resources, educators must be critical consumers; the sites use varying criteria to evalu- ate the design standards and effects of

single-subject research. In order to be able to evaluate practices based on sin- gle-subject research, educators must be fluent with the purpose and character- istics of single-subject research designs and the guidelines for evaluation and interpretation of that research.

Recognizing Well Designed Single-Subject Research

To judge the usefulness and strength of the results and implications from sin- gle-subject research, educators must understand the specific methods used, the strengths, and the limitations of single-subject research designs. Single- subject design studies are distinguished by several key characteristics:

• The individual case (individual or group of individuals), rather than a group, is the unit of analysis.

• The individual case serves as his or her own control.

• The researcher conducts frequent and repeated observations and measures outcomes over time to assess changes in level, trend, vari- ability, and immediacy of the inter- vention effect across differing condi- tions (e.g., participants, settings).

Despite the strengths of single-sub- ject research for special education (see box, "Strengths of Single-Subject Research"), determining what is and is not an EBP based on single-subject research can be difficult, because there is no standardized method for synthe- sizing results (Conroy, Dunlap, Clarke, & Alter, 2005; Mooney, Epstein, Reid & Nelson, 2003). Single-subject research designs identify causal relationships between an intervention and an observed effect in a specific setting and context and with a specific individual. Generalizable statements to larger pop- ulations of individuals are difficult to make from single-subject research.

Systematic and repeated demon- stration of the effect of the interven- tion is a key requirement for establish- ing a causal relationship between an intervention and an observed effect. In single-subject research these replica- tions occur across individual students, settings, or conditions. Common single- subject designs include ABAB, alternat- ing treatment, or multiple baseline designs. The goal is to control and account for internal threats to validity such as practice effects or an outside event that may affect the outcomes.

TEACHING EXCEPTIONAL CHILDREN | MAY/JUNE 2013 7

Table 1 . Web Resources for Identifying Evidence-Based Practices

Features

Design standards for SSR published and available

Design standards matched to current research practices

Evidence standards for SSR published and available

Provides links to intervention descriptions

Provides links or references to research used for determinations

Fee required

What Works Clearinghouse

http://ies.ed.gov/ncee/wwc/

Resource iiMiiM|iMi[MimM

Council for Exceptional Children http://www.cec.sped.org

only includes research

commissioned by CEC

American Psychological Association

Iittp://www.apa.org/education/kl2/index.aspx

Promising

Practices Network

http://www.promisingpractices.net/

group designs only

Note. SSR = single-subject research.

Traditionally, the effects of single- subject research are evaluated using visual analysis. Researchers graph and evaluate outcome data for changes in trend (direction), level (mean/median level), and variability (fluctuation) within and across phases. Because decision rules have not been sufflcient- ly tested and established to guide visu- al analysis and interpretation of single- subject research data, inaccurate, inconsistent, or limited interpretations can result.

In addition to visual analysis, sin- gle-subject researchers often calculate an effect size, typically reported in standardized units that make compar- isons of effect easier across studies and participants. The most prevalent reported effect size involves an exami- nation of non-overlapping data points (54.65%) followed by a calculation of the standardized mean difference (18.6%; Maggin, 2011). There is cur- rently no agreed-upon method for cal-

culating an effect size in single-subject research, which makes it difflcult to synthesize research results.

In sum, single-subject research designs have a number of strengths and useful characteristics that make them appealing to conducting studies that identify EBPs in special education. However, because of (a) a reliance on visual analysis procedures, (b) a lack of an effect size metric and methodolo- gy, and (c) a need for standardized cri- teria for effective implementation and interpretation, special educators also need guidance on how to systematical- ly evaluate the existing research evi- dence base.

Evaluating and Interpreting Research

Given the signiflcant but insufflcient advances associated with using single- subject research methodologies to iden- tify EBPs, teachers and administrators require additional guidance as they

attempt to interpret research and judge

practices that are examined with sin-

gle-subject research methods. Table 2

lists six steps to guide decision making;

Figure 1 illustrates how to evaluate and

implement interventions.

1. Identify the Problem. The flrst

step in the decision-making process is

to have a specific, measurable, and

complete identification of the problem

or need. After determining whether the

primary area of concern is academic,

social, or both, assess

• The nature of the problem (fre-

quency, duration, intensity, etc.).

• The context in which it is occurring

(instructional curriculum, environ-

mental conditions, etc.).

• Characteristics of the learner (dis-

abilities, learning history, etc.).

• Other factors that might affect

intervention responsiveness and

effectiveness.

8 COUNCIL FOR EXCEPTIONAL CHILDREN

This specification would also include stating what a successful outcome would look like if an effective inter- vention could be identified. A variety of formal and informal assessment tools and procedures exist to structure this specification and identification step.

For Mark, the process of clearly identi- fying the problem included writing a clear description of each problem behavior, keeping a record of the fre- quency of these behaviors, as well as the time and the activity that was going on when the problem behaviors occurred. Cindy needed information about the specific part of the writing process that was proving difficult for her student. Using curriculum-based measurement tools, she was able to determine that her student needed specific instruction on spelling and grammar.

2. Identify Possible Interventions. After establishing an understanding of the problem, need, and context, the next step is to identify a list of inter- vention practices that might address the intended outcome. Although the utility and comprehensiveness of the WWC is not currently sufficient with respect to special education EBPs, it is an excellent starting place because of (a) high acceptability standards, (b) detailed procedural descriptions, (c) access to supporting research, and (d) thorough research reviews. An EBP selection made at the WWC is support- ed by high research standards and can be used with confidence by special educators; however, the number and/or range of choices might be limited.

Although Mark and Cindy both began their search for EBPs at the WWC web site, neither was able to find an EBP that had been rated effective for the special education populations and problems they needed to address. They continued their search by looking at the CEC, APA, and PPN databases, and general Internet searches. Both teach- ers put together a short list of inter- vention ideas but needed to learn more about the evidence base behind

Strengths of Single-Subject Research

Single-subject designs are ideal for the field of special education for several reasons (Horner et al., 2005). First, in special education the number of students with specific disabilities is relatively small, making large group design studies more difficult and often impractical. Second, in single-sub- ject research designs, researchers are able to examine the specific condi- tions under which a particular student is responding to an intervention. Having specific information at the individual student level is critical in special education for the delivery of individualized intervention plans and evaluation of student outcomes. Finally, students with disabilities have a diverse range of learning characteristics, and single-subject research designs allow individual students to serve as their own controls.

them. Mark's list included interven- tions such as student self-monitoring (Briere & Simonsen, 2011; Reid, Trout, & Schwartz, 2005) and greeting stu- dents and providing a behavioral prompt prior to the start of class (Allday & Pakurar, 2007; Simonsen, Fairbanks, Briesch, Myers, & Sugai, 2008). Cindy was interested in the use of peer tutoring to improve her stu- dent's spelling and grammar (Burks, 2004).

3. Consider the Target Population. After increasing the specification of the problem or need, and identifying a list of possible interventions, consider the fit between the target population and the intervention:

• Was the intervention designed for a student or group of students that share the characteristics of the tar- get population?

• How difficult might it be to general- ize the findings from the research subjects to the target population of students?

• How similar/different is the research setting from the setting where the intervention will be implemented?

• Can the intervention be adapted to the characteristics (e.g., learning, cultural, linguistic) of the target population of students?

Mark explored the research related to both intervention practices and discov- ered that the research in both cases was conducted with middle school stu- dents. He decided that self-monitoring

might not be a good match for his sec- ond-graders, but that greeting and prompting students at the beginning of a class period might be a good fit for his younger group. Cindy discovered that the research on peer tutoring was conducted with fifth-graders with learning disabilities, and decided this would be a good match for her sev- enth-grade student who was several years behind in writing skills.

4. Consider the Scope of the

Decision. The size of the research base should match the level of the decision being considered. Large-scale deci- sions—such as adopting a new evi- dence-based intervention for an entire special education program—are more difficult to change and require more resources than individual-level inter- ventions. Special education teachers have more freedom and opportunities to change with individual and small- group intervention decisions. With program- and schoolwide decisions, the stakes are higher, and educators must have the most and best empiri- cal evidence to support their interven- tion decisions, which also might include group design studies, general education or other students, and pro- gram-level intervention evaluations. When making school- or program- level decisions, educators should consider:

• Does the design of the research study or studies demonstrate a clear relationship between the interven- tion and the effect?

TEACHINC EXCEPTIONAL CHILDREN MAY/JUNE 2013 9

Table 2 . How to Identify Evidence-Based Practices From Single-Sub¡ect Researcb

1. Identify the problem

2. Identify possible interventions

3. Consider the target population

4. Consider the scope of the decision

5. Consider the quality of the research design.

6. Consider the effect of the intervention.

Clearly identify an area of academic or social need, using reliable and valid assessments.

Identify a list of possible intervention practices related to the specific need and the research articles associated with those practices.

Determine if the research studies were conducted with populations with similar characteristics to the target population.

Match the size of the research base to the level of decision (classroom, school, or program level).

Assess the evidence of effect for the studies in the research literature base; design should allow for the determination of causal relationship between intervention and outcome(s).

Assess the extent to which the observed effects can be attributed to the intervention.

• Has the effect of the intervention been demonstrated in a variety of settings wide enough with a variety of participants to support a range of practice uses?

• Is the total research evidence base sufficient enough to support high stakes decision making related to large numbers of students or severe problem need?

Both Mark and Cindy were looking for intervention practices to use at the

their attention on the research designs and the evidence of effect for each of the studies in the research literature base. Examining design standards prior to evidence of effects is important because the value of the evidence of effects is directly linked to the quality of the design and integrity of the analysis.

• Are student characteristics described in sufficient detail to enable com- parisons with target population?

Special education teachers have more

freedom and opportunities to change with individual

and small-group intervention decisions.

classroom level, so an individual study with positive results was an appropri- ate evidence base from which to start. Mark also was reassured to see the results of a literature review (Simon- sen et al., 2008) that indicated that providing prompts to students prior to behavioral difficulties was an effective strategy. This finding allowed Mark to feel more confident that the results from the middle school study might generalize to his second graders.

S. Consider the Quality of the Research Design. Educators must focus

• Are descriptions of the setting and intervention specific and detailed?

• Are instructions and procedures for implementing a practice or interven- tion detailed enough to support direct replication?

• Are data collection procedures described in sufficient detail to enable data collection in the applied setting?

Additionally, evaluate the design to see if it allows for the determination of causal relationship between the inter- vention and the outcomes. Although

the standards of the WWC are higher, the consensus in the research commu- nity is that studies must have at least three data points in each phase and allow for at least three demonstrations of effect in order to clearly show a functional relationship between the intervention and the outcome. The most common ways to demonstrate this functional relationship is using withdrawal/reversal (ABAB) designs or multiple-baseline designs (Hammond &. Gast, 2010). In an ABAB design, educa- tors should look for a clear base- line-intervention-baseline-intervention pattern. In a multiple-baseline design, demonstrations of effect can take place across settings or participants. Phase changes should occur at different points in time. Studies that are designed to allow for at least three demonstrations of effect can be used to determine the presence of a causal relationship between the intervention and the outcome. Educators should feel comfortable that the demonstrated effects are a direct result of the inter- vention, rather than some other vari- able or event.

The student characteristics and inter- vention descriptions were described explicitly for both Mark's prompting strategy and Cindy's peer-tutoring strategy and both teachers felt confi-

10 COUNCIL FOR EXCEPTIONAL CHILDREN

Figure 1. Evaluating and Implementing an Intervention

Identify intervention practice designed to address target

problem

tA/ \/ \ . IstheX / intervention a \

^ good fit for ^ your target

\ student/ / \ students? /

\ /

YYes 1

^ \ / l . Is the^\

/ research base ^y / broad enough ^

\ to guide ^^ decision /

^v making? /

\ /

YYes *

/ \/ Is tbe \ ^ / intervention ̂ \

/^ considered \ evidenced

\ ^ based by y/ \ WWC /

\ /

Yes

Use intervention and continue to collect data to monitor effect

Identifying Special Practices From

K ^ ^ ^ and \ ' ^ ° ^ ''«turn to list of ' intervention 7

practice£^^^

Stop and ' ^ return to list of

No intervention ^ practices

\ ^ / iiiiinmniiiiiiniii"

4 f/

Review research studies related to the intervention

1 No / *

/ / \ / / \ / / ^- '^'^^ \

y / the study \ . "^ / design allow for ^

\ the determination \^ of a functional /

^^ relationship?/

Yes 1

/ \ / \/ \

^v^^ Yes / 4. Does the \ ^ \ ^ ^ / study show ^

^v positive \ effects? /

\ /

Education Evidence-Based Single-Subject Research

No stop and \ ^ return to list of / intervention

practices -^

ĵ p Stop and V return to list of y ^" intervention

practices

1. Is the intervention a good fit for your target student/students? A) Was the intervention designed for

use by students of similar age or ability level?

B) Is the intervention a good match for your setting (ie., group size, time, or resources)?

C) Is the intervention a good match for the disability of your target student or population?

2. Is the research base broad enough to guide decision making? A) For single students or small groups

are there at least 1 or 2 promising studies?

B) For programs or schools are there 3-5 promising studies?

3. Does the study design allow for the determination of a functional relationship? A) Is the intervention clearly defined

to allow for replication? B) Are there at least 3 data points in

each phase? C) Are there at least 3 demonstrations

of effect?

4. Does the study show positive effects? A) Is a change noted in the level of

the data between baseline and intervention?

B) Is a change noted in the trend of the data between baseline and intervention?

C) Is a change noted in the variability of the data between baseline and intervention?

4) Are any non-effects noted?

Note. WWC = What Works Clearinghouse.

dent that they could replicate the strategies in their classrooms. Next, Mark and Cindy evaluated the design of each of their studies. Mark's study was a clear multiple-baseline single- subject design. In each case there were at least three data points per phase and the design allowed for three demonstrations of effect. Mark felt confident that this study would allow him to determine if this intervention resulted in changes in the students' behavior. Cindy's study on peer tutor-

ing was slightly different. It was an ABA design where baseline data was collected prior to and after the strategy implementation. Although the article reported at least three data points per phase, because the study design did not include a second intervention phase that would allow for a third demonstration of effect Cindy was not able to confidently determine if the reported results are a direct function of the intervention. Cindy could not con-

sider peer tutoring an EBP based on this study.

6. Consider the Effect of the Inter- vention. In the final step, examine the effects of the intervention, in particular the extent to which the observed effects could be attributed to the inter- vention and whether the size of the effects is educationally important or relevant. As indicated, single-subject researchers rely on visual analysis pro- cedures primarily and statistical tests

TEACHING EXCEPTIONAL CHILDREN [ MAY/JUNE 2013 11

secondarily (Maggin, 2011) to judge the effects of the intervention. Consider:

• Are graphs prepared accurately, appropriately, and clearly?

• Are changes in level, trend, variabil- ity, and immediacy of effect described within and across phases adequately?

• Are changes aligned with the intro- duction of or manipulation of the intervention?

• Is the effect of intervention demon- strated sufficiently?

• Do statistical test results support the visual analysis descriptions?

Mark reviewed the results from the prompting study and concluded that for all three students in the study the prompting intervention resulted in clear and positive effects on the stu- dents' behavior. He decided to try the intervention in his classroom and monitor the effects for his students. Cindy reviewed the results of the peer- tutoring study and also determined that the effects for students were clear and positive for all the students although graphs were not provided, and she had to make these conclu- sions from individual data points and means reported in a table. Because the study design did not clearly allow for three demonstrations of effect, she was not confident that the results that were reported were a direct result of the peer-tutoring intervention. Because she was making an individual student- level decision and she felt that peer tutoring might be motivating for her student, Cindy decided to try the inter- vention while carefully monitoring her student's progress. In addition, she decided to return to the research to see if she could find a strategy with an evidence base she could be more con- fident about for her student.

Final Tiiouglils

Regardless of whether a strong or weak research database exists, special educa- tors must be smart consumers. Regard- less of their confidence in their EBP decisions, educators also must con- tinuously monitor their intervention implementation fidelity and student

responsiveness to the intervention. Implementation of every intervention can be "tweaked" to improve its effec- tiveness over time, and the responsive- ness of every student can change over time and conditions.

Mark and Cindy were both able to make informed decisions about EBP for their classrooms despite the fact that the body of special education research is varied and clear standards for EBP are not yet clearly defined. Several organizations have attempted to com- pile intervention resources on a large scale; however, at this time none of these is mature enough to serve as a primary comprehensive and practical guide for special educators. Single-sub- ject research designs and studies can be a valuable tool for guiding the selec- tion of EBP, even though standards for evaluating interventions and outcomes based on single-subject research designs have not been established (Horner et al, 2005). Special educators can improve their identification of EBP by having solid background knowledge about the purpose, characteristics, and requirements of single-subject research designs; and a systematic process to guide their evaluation of single-subject research design, quality, and demon- strations of meaningful effects.

References Allday, R. A., & Pakurar, K. (2007). Effects

of teacher greetings on student on-task behavior. Journal of Applied Behavior Analysis, 40, 317-320. http://dx.doi.org /lO. 1901/jaba.2007.86-06

Briere, D. E., Ill, & Simonsen, B. (2011). Self-monitoring interventions for at-risk middle school students: The importance of considering function. Behavioral Disor- ders, 36, 129-140.

Burks, M. (2004). Effects of classwide peer tutoring on the number of words spelled correctly by students with LD. Interven- tion in School & Clinic, 39, 301-384. http://dx.doi.org/10.1177/1053451204039 0050701

Conroy, M. A., Dunlap, G., Clarke, S., & Alter, P. J. (2005). A descriptive analysis of positive behavioral intervention research with young children with chal- lenging behavior. Topics in Early Child- hood Special Education, 25, 157-166. http://dx.doi.org/10.1177/0271121405025 0030301

Hammond, D., & Gast, D. L. (2010). Des- criptive analysis of single-subject

research designs: 1983-2007. Education and TYaining in Autism and Develop- mental Disabilities, 45, 187-202.

Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special educa- tion. Exceptional Children, 71, 165-179.

Individuals With Disabilities Education Act, 20U.S.C. §4301 et seq. (2006).

Kratochwill, T. R., Hitchcock, J. J., Horner, R. H., Levin, J. R., Odom, S. L., Rind- skopf, D., & Shadish,W.R. (2010, June). Single-case design technical documenta- tion: Version 1.0 (Pilot). Washington, DC: Institute of Education Sciences, What Works Clearinghouse. Retrieved from http://ies.ed.gov/ncee/WWC/documents um.aspx?sid = 229

Maggin, D. (2011, February 14). Identifying evidence through synthesis of single-sub- ject research [PowerPoint slides]. Univer- sity of Connecticut, Storrs.

Mooney, P., Epstein, M. H., Reid, R., & Nelson, J. R. (2003). Status of and trends in academic intervention research for stu- dents with emotional disturbance. Remedial and Special Education, 24, 273-287. http://dx.doi.org/10.1177 /07419325030240050301

No Child Left Behind Act of 2001, 20 U.S.C. §6301 et seq. (2006).

Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special educa- tion: Scientific methods and evidence- based practices. Exceptional Children, 71, 137-148.

Reid, R., Trout, A. L., & Schwartz, M. (2005). Self-regulation interventions for children with attention deficit/ hyperactivity disorder. Exceptional Children, 71, 361-377.

Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai, G. (2008). Evidence- based practices in classroom manage- ment: Considerations for research to practice. Education and Treatment of Children, 31, 351-380. http://dx.doi.org /10.1353/etc.0.0007

Jennifer Freeman (Maine CEC), Graduate Assistant: and George Sugai (Connecticut CEC), Professor of Special Education and Director, Center for Behavioral Education & Research, University of Connecticut, Storrs.

Address correspondence concerning this article to Jennifer Freeman, 249 Glen- brook Road, Storrs, CT 06269 (e-mail: [email protected]).

TEACHING Exceptional Children, Vol. 45, No. 5, pp. 6-12.

12 COUNCIL FOR EXCEPTIONAL CHILDREN

may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express

written permission. However, users may print, download, or email articles for individual use.