Discussion 2: Strategies for Best Practice in Integrating Special Education Within an MTSS

Journal of Positive Behavior Interventions 14(3) 142 –152 © 2012 Hammill Institute on Disabilities Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/1098300712436844 http://jpbi.sagepub.com

The three-tiered Positive Behavioral Interventions and Sup- ports (PBIS) model aims to prevent disruptive behavior by developing Tier 1 (universal), Tier 2 (targeted group), and Tier 3 (intensive) systems of positive behavior support (Sugai & Horner, 2006; Walker et al., 1996). The PBIS universal system creates improved systems (e.g., discipline, reinforce- ment, and data management) and procedures (e.g., office referral, training, leadership) to promote positive change in staff and student behavior. It is anticipated that approximately 80% of the student population will respond positively to the universal PBIS model. Consistent with a Response to Inter- vention (RtI) approach to preventing behavior problems (Hawken, Vincent, & Schumann, 2008), children who do not respond to the universal level of PBIS require assessment of their concerns and more intensive group or individual pre- ventive interventions to meet their needs.

Because most of the schools trained in PBIS only imple- ment the universal aspects of the three-tiered model, there is a great need for additional research on the types of pro- grams and services implemented to help students who do not respond adequately to school-wide PBIS (SWPBIS; Sugai & Horner, 2006; also see Barrett, Bradshaw, and

Lewis-Palmer, 2008). The current paper describes the pro- grams and services that schools trained in the SWPBIS model are using to meet the needs of students not respond- ing to Tier 1. We focus on schools that have not yet received formal training in Tier 2 or 3 supports, in an effort to better understand their training and support needs and to inform professional development related to their efforts to address a continuum of social-emotional and behavioral needs.

Secondary and Tertiary Support Systems

Although the three-tiered PBIS model encourages the use of Tier 2 and 3 support systems for children not responding adequately to SWPBIS, many schools struggle to develop a coordinated support system without formal training. States

436844PBI14310.1177/1098300712436844Debnam et al.Journal of Positive Behavior Interventions © 2012 Hammill Institute on Disabilities

Reprints and permission: sagepub.com/journalsPermissions.nav

1Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

Corresponding Author: Katrina J. Debnam, Johns Hopkins Bloomberg School of Public Health, Baltimore, 624 N. Broadway Room 841, Baltimore, MD 21205, USA Email: [email protected]

Action Editor: Don Kincaid

Secondary and Tertiary Support Systems in Schools Implementing School-Wide Positive Behavioral Interventions and Supports: A Preliminary Descriptive Analysis

Katrina J. Debnam, MPH1, Elise T. Pas, PhD1, and Catherine P. Bradshaw, PhD, MEd1

Abstract

More than 14,000 schools nationwide have been trained in School-Wide Positive Behavioral Interventions and Supports (SWPBIS), which aims both to reduce behavior problems and to promote a positive school climate. However, there remains a need to understand the programs and services provided to children who are not responding adequately to the universal level of support. Data from 45 elementary schools implementing SWPBIS were collected using the School-wide Evaluation Tool and the Individual Student Systems Evaluation Tool (I-SSET) to assess the use of school-wide, Tier 2, and Tier 3 support systems. The I-SSET data indicated that nearly all schools implemented federally mandated Tier 2 and Tier 3 supports (e.g., functional behavioral assessment, student support teams), but few schools implemented other evidence-based programs for students with more intensive needs. School-level demographic characteristics were correlated with the implementation of some aspects of universal SWPBIS, but not with the Tier 2 or 3 supports. Implications of these findings for professional development are discussed.

Keywords

School-Wide Positive Behavioral Interventions and Supports (SWPBIS), secondary supports, tertiary supports, functional behavioral assessment, evidence-based programs

Debnam et al. 143

and districts increasingly encourage the use of a student support team (SST; Crone & Horner, 2003) model, which provides a structure for collaborative decision making to ensure that children are successful in school. SSTs are com- posed of a variety of stakeholders (e.g., administrators, teachers, and mental health providers) who meet regularly to develop intervention plans for students identified as in need of additional supports. In a typical school setting, a classroom teacher “refers” a student for an academic or behavioral concern and then meets with the SST to collab- oratively assess the concern and identify potential aca- demic and/or behavioral strategies that will improve the student’s performance (Crone & Horner, 2003). These interventions are often composed of small student groups, targeting a specific skill or goal, and are implemented by the teacher or staff member. In its ideal form, the SST monitors and evaluates the selected strategies to determine their success, with the expectation that noneffective inter- ventions will be discontinued and replaced with effective programs (Crone & Horner, 2003; Hawken et al., 2008).

One increasingly popular intervention, Check In/Check Out (CI/CO; Crone, Horner, & Hawken, 2004; Todd, Campbell, Meyer, & Horner, 2008), provides a structure for students to receive positive, individual contact, feedback, and support for appropriate behavior throughout the day from their teachers. The program is tied to the school-wide behavioral expectations, and has been shown to produce positive outcomes (e.g., reduction in office discipline refer- rals) in rigorous evaluation studies (Filter et al., 2007; Hawken, MacLeod, & Rawlings, 2007; Todd et al., 2008). Consistent with the tiered PBIS model, the success of tar- geted interventions should be monitored and modified by the SST if behavior does not improve (Crone & Horner, 2003).

Functional behavioral assessment (FBA) is another strategy commonly used by schools implementing PBIS (Crone & Horner, 2003). Through FBA, the “function,” or purpose, of the student’s behavior is assessed in relation to the context (e.g., environment, motivation) in which it occurs, to allow school staff to predict future occurrences of the behavior and thus “pre-correct” for the occurrence of an appropriate behavior. FBA information is used to iden- tify appropriate interventions to address the specific pur- pose of the behavior (O’Neill et al., 1997). FBAs are usually conducted by members of the SST for students who exhibit chronic behavior problems (Scott et al., 2005). This approach has been shown effective for various student behaviors and settings (e.g., Lane et al., 2007). There is an increasing emphasis on the use of FBAs to guide the imple- mentation of function-based interventions before a special education referral (Scott et al., 2005).

The process of providing targeted group and individual preventive interventions may be more challenging when the school lacks a solid SWPBIS model (Sugai & Horner,

2006). Other contextual factors may also challenge the school’s organizational capacity to provide valuable sup- port services. For example, schools that experience a high student-to-teacher ratio, a large student body, a high rate of student mobility or discipline problems, or a high concen- tration of student poverty may also struggle to implement school-based services (Domitrovich et al., 2008). In fact, research suggests that high rates of “disorder” within the school can impede successful implementation of programs (G. D. Gottfredson, Gottfredson, Payne, & Gottfredson, 2005; Bradshaw, Koth, Thornton, & Leaf, 2009). Although not well researched, other characteristics of the school, such as the concentration of students receiving special education services and academic performance, may also be related to the extent of support services provided. Specifically, we hypothesized that schools with high concentrations of stu- dents receiving special education services, and therefore more staff who have pre-service training and expertise in targeted and intensive support services, would have enhanced Tier 2 and 3 services. We also expected that school-level indicators of high academic performance would be indicative of greater academic and Tier 2 and 3 supports. This exploratory area of research fills a current gap in our understanding of how contextual factors specifi- cally relate to Tier 2 and 3 supports. It may also identify future areas of research that should be conducted.

Furthermore, given the prior research suggesting that schools are typically implementing multiple programs (Bradshaw, Mitchell, & Leaf, 2010), often without formal training (Gottfredson & Gottfredson, 2001), and that rela- tively few are using evidence-based models (Gottfredson & Gottfredson, 2002), we examined the characteristics of the targeted support services implemented. We were particularly interested in the implementation of the Tier 2 and 3 supports implemented by SWPBIS schools that had not yet received formal training in targeted or intensive services, as this would provide useful information regarding program plan- ning and data-based decision making. Consistent with the work of Gottfredson and Gottfredson (2002), we expected that these schools would have implemented relatively few “packaged” and evidence-based Tier 2 and 3 prevention programs.

Overview of the Current Study The first aim of the study was to describe the types and fea- tures of Tier 1, 2, and 3 support systems in place at elementary schools already trained in and actively implementing SWPBIS. We purposefully focused on schools that were implementing the universal supports system, but had not yet been provided formal training on the implementation of Tier 2 or 3 supports, in order to inform program planning and technical assistance. We expected that schools would natu- rally begin to provide some Tier 2 and 3 supports independent

144 Journal of Positive Behavior Interventions 14(3)

of receiving formal training, based on student need. The sec- ond aim of the study was to examine variation in the level of Tier 2 and 3 services provided in relation to the fidelity of the SWPBIS model and to a set of school-level demographic characteristics, which previous research suggests are com- monly linked with poorer implementation of prevention pro- grams (Gottfredson et al., 2005). Our third goal was to describe intervention attributes of the three most commonly used Tier 2 programs. Together, these findings will provide an enhanced understanding of the types and features of supports that are commonly used by schools implementing SWPBIS. These findings may also indicate areas for future research and which could be enhanced through professional development and technical assistance to improve behavior support systems in schools.

Method Participating Schools

Data for the present study come from the baseline data collec- tion of a large-scale study of secondary supports and services provided to schools already implementing SWPBIS. A total of 45 public elementary schools from six Maryland school districts volunteered to participate in the study. Eligible schools had been trained in the universal system of SWPBIS by the Maryland State Leadership Team (Barrett et al., 2008), had implemented SWPBIS for at least 1 year (M = 2.9 years, SD = 1.72, range = 1–7), had received at least an 80% on the SWPBIS fidelity measure (i.e., the School-wide Evaluation Tool [SET], see description below) in the prior spring, and had expressed a desire for training in targeted and intensive support services. Although the schools were not selected at random from the districts, the participating schools represent between 12.5% and 62.5% of the districts’ elementary schools implementing SWPBIS. It is important to note that the state had not developed a system for providing coordinated train- ing in targeted or intensive programs and that only select school personnel hired to conduct FBAs are provided district- supported training related to Tier 2 and 3 supports (Barrett et al., 2008). As illustrated by the school-level demographic data presented in Table 1, the participating schools were diverse and were located in different geographic locations. The Institutional Review Board at the researchers’ university approved this study.

Data School Demographic Information. Baseline school-level

characteristics were obtained from the Maryland State Department of Education regarding student enrollment, student-to-teacher ratio, student mobility, percentage of stu- dents receiving free and reduced-price meals (FARMs), percentage of students receiving special education services,

percentage of Caucasian students, percentage of suspen- sions (total number of suspensions divided by the enroll- ment), and student math and reading performance (see Table 1).

School-wide Evaluation Tool (SET). The SET (Sugai, Lewis- Palmer, Todd, & Horner, 2001) was developed to assess the degree to which schools implement the key features of SWPBIS (Horner et al., 2004). It is typically completed annually by a trained external observer who conducts brief interviews, tours the school, and reviews materials to assess the extent to which the following seven key features of SWP- BIS are in place at the school: (a) Expectations Defined; (b) Behavioral Expectations Taught; (c) System for Rewarding Behavioral Expectations; (d) System for Responding to Behavioral Violations; (e) Monitoring and Evaluation; (f) Management; and (g) District-Level Support (see Horner et al., 2004). Each item is scored on a 3-point scale (0 = not implemented, 1 = partially implemented, and 2 = fully imple- mented). It yields seven subscale scores (ranging 0–100%), with higher scores indicating greater program fidelity. An overall summary score was computed by averaging all seven scores (referred to as the Overall SET score), which also ranges 0 to 100% (Cronbach’s alpha [α] = .72). An 80% or higher on the Overall SET score is considered high fidelity (Horner et al., 2004; Sugai et al., 2001).

Individual Student Systems Evaluation Tool (I-SSET). A new measure, the I-SSET (version 1.2; Lewis-Palmer, Todd, Horner, Sugai, & Sampson, 2005), was developed to docu- ment the characteristics of Tier 2 and Tier 3 support services provided in schools implementing SWPBIS. Minor modifi- cations were made to the original I-SSET to make the instrument consistent with Maryland terminology (e.g., FBA, SST). Similar to the SET, a trained external observer conducts brief interviews at the school and reviews inter- vention planning materials. In the current study, the I-SSET and SET were conducted during a single school visit, thereby providing information regarding both SWPBIS and the targeted and intensive support programs. The I-SSET is composed of 23 items organized into three subscales: (a) Foundations (α = .50; e.g., procedures for referring students to SST); (b) Targeted Interventions (α = .64; e.g., written intervention instructions); and (c) Intensive Individualized Interventions (α = .52; e.g., elements of the FBA and quali- fications of SST members). Each item is scored on a 3-point scale (0 = not implemented, 1 = partially implemented, and 2 = fully implemented). The nine items on the Targeted Interventions subscale are derived mostly from a series of questions regarding the features of the three most com- monly used Tier 2 and 3 interventions. Specifically, the SST leader is asked to provide the name of programs imple- mented and answers a series of eight questions regarding each program identified, one of which can be an academic intervention (the other two are behavioral or social-emotional). The responses to these questions, which are scored on a

Debnam et al. 145

2-point scale (0 = no and 2 = yes), are then totaled across the three programs to generate the eight I-SSET item scores for that school (see items 12–19 on Table 2). An Overall I-SSET score was created by averaging the three subscale scores (α = .72). Each I-SSET subscale is represented by a single score (0–100%), where higher scores indicate stronger sup- port systems. Because the I-SSET is a relatively new mea- sure, there are no published studies reporting data from the I-SSET; furthermore, the psychometric properties of the I-SSET have not been previously examined. The Cron- bach’s alphas are based on a larger pool of cases (n = 132) from the larger study. The current study is the first, to our knowledge, to report data from the I-SSET.

Procedure Training of SET/I-SSET Assessors. A total of eight SET/I-

SSET assessors were hired by the project, seven of whom had previous experience conducting SETs. Each assessor conducted between 2 and 13 SET/I-SSETs (mode = 5). The assessors were primarily bachelor’s- and master’s-level professionals (e.g., teachers, special educators, school counselors, educational trainers) who were working part- time or had recently retired from full-time work in an edu- cational setting. After reviewing the written training materials, each assessor attended an initial half-day didactic group SET/I-SSET training session, which was conducted by the lead SET/I-SSET staff trainer, and then shadowed a lead SET/I-SSET staff trainer in conducting a full SET/I- SSET in a nonproject SWPBIS elementary school. All assessors conducted a second SET/I-SSET with a second

lead trainer at another nonproject school to determine interobserver agreement. The interobserver agreement for each set of pairs was calculated (range of item-level Kappas for the SET was .64 to 1.00 [M = .82] and .84 to 1.00 [M = .92] for the I-SSET).

Administration of the SET/I-SSET. After completing this three-stage training process, the assessor independently conducted the SET/I-SSET in a project school. Both mea- sures were completed during a single school visit by the assessor. Brief interviews were conducted with an adminis- trator (approximately 30 minutes) and the SST leader (approximately 20 minutes) regarding the types of pro- grams and supports provided to students not responding adequately to SWPBIS. The assessors also collected infor- mation about the PBIS procedures, policies, and positive behavior standards by interviewing a minimum of eight teachers and four support staff members for approximately 3 to 5 minutes each, and a minimum of 12 students from each grade level for approximately 1 to 2 minutes each. The measures were conducted in the fall (i.e., first month of their participation in the study).

Analyses To address our first research aim, we conducted descriptive analyses on the SET/I-SSET item-level data in SPSS 17.0. These analyses enabled us to determine the level of imple- mentation reported by schools and to identify the areas of strength and weakness. Our second aim was to examine variation in I-SSET scores by SET scores and school char- acteristics. Therefore, we conducted correlational analyses

Table 1. Correlations Among the I-SSET Subscales and School Demographics (n = 45 Schools)

M (SD) Range Foundations Targeted

interventions

Intensive individualized interventions

I-SSET overall score

I-SSET score M 68.1% 78.3% 93.9% 80.1% SD 15.8 14.5 17.0 11.3 School demographics Correlations School enrollment 461.07 (142.54) 194–867 .038 .036 –.134 –.034 Student-to-teacher ratio 20.77 (3.76) 14.60–29.92 .163 .033 –.364* –.092 Free/reduced-price meals (%) 44.99 (20.43) 6.80–80.40 .181 .235 .095 .233 Special education students (%) 14.47 (6.17) 6.00–35.00 –.185 –.061 .097 –.064 Caucasian students (%) 32.20 (31.08) 0.00–93.66 .021 –.032 .138 .065 Student mobility (%) 32.57 (24.24) 3.70–158.20a –.086 –.026 .102 .000 Suspension rate (%) 9.14 (6.89) 0.30–34.56 –.040 .161 .022 –.040 Math performance (%) 73.47 (10.67) 49.00–92.70 –.154 –.261 –.012 –.189 Reading performance (%) 75.10 (10.6) 58.80–93.50 –.218 –.187 –.007 –.185

Note. This table reports sample demographic characteristics as well as descriptives and correlations for the I-SSET. aIndicates that mobility rate exceeded 100% because the sum of the percentage of students who entered and exited the school during the school year exceeded 100% of the student body. *p < .05.

146 Journal of Positive Behavior Interventions 14(3)

to examine the association between the SET and I-SSET subscale and overall scores. We then conducted correla- tional analyses to examine the extent to which implementa- tion of the SWPBIS, Tier 2, and Tier 3 systems varied systematically by the school contextual factors; this enabled us to determine whether certain school factors were associ- ated with the implementation of these supports. Effect sizes are reported in the correlation tables and results. Finally, we conducted descriptive analyses on the types of Tier 2 sup- ports implemented. Specifically, we conducted descriptive analyses to examine the features of the three most com- monly used programs indicated on the I-SSET to determine whether schools were using evidence-based programs (Gottfredson & Gottfredson, 2002).

Results Descriptive Analyses

SET Data. We found that 93% of the schools (42 of 45) achieved an 80% or higher implementation level on the Overall SET score. Schools tended to score the highest on

the Monitoring and Decision Making subscale, with a mean score of 96.9% (SD = 6.05). In contrast, the System for Responding to Behavioral Violations subscale tended to have the lowest scores (M = 86.44%, SD = 12.81). In only 31.1% of schools, staff agreed with administration on the method of notification of an extreme emergency, whereas all of the schools’ team members reported teaching behavioral expectations, which is a key component of the SWPBIS framework. All schools reported that their PBIS team includes representation from all staff members.

I-SSET Data. The percentage of schools that received the maximum score (2) for each item on the I-SSET is reported in Table 2. With regard to the Foundations subscale, all but one school reported having a team that receives requests from teachers, consistent with a statewide requirement that all schools have an SST process (see I-SSET no. 1 on Table 2). Approximately half (51.1%) of the schools reported discuss- ing issues related to culturally responsive teaching with staff in the past year. Only 26.7% of schools indicated that the staff and the SST leader agree about the proper process for SST referrals. Just 2 of the 45 schools (4.4%) had a comprehensive form for referring students to the SST. Examination of the

Table 2. Percentage and Number of Schools With the Highest Possible Score on I-SSET Items (n = 45 Schools)

I-SSET item Number of schools (%)

Foundations 1. School has a Student Support Team (SST) 44 (97.8%) 2. Culturally responsive teaching has been discussed this year 23 (51.1%) 3. Process for including family in SST process 35 (77.8%) 4. SST meets at least twice a month 28 (62.2%) 5. System for staff to refer students to SST 37 (82.2%) 6. SST referral form lists pertinent information 2 (4.4%) 7. Response to SST referral takes no more than 3 days 21 (46.7%) 8. Process for monitoring student progress through data 30 (66.7%) 9. Staff agree with administration on SST referral process 12 (26.7%) 10. FBA intervention form lists pertinent information 32 (71.1%) Targeted interventions 11. Written process for selecting evidence-based interventions for individual students 29 (64.4%) 12. Interventions link to school-wide behavioral expectations 44 (97.8%) 13. Intervention continuously available to students 42 (93.3%) 14. Intervention is implemented within 3 days 20 (44.4%) 15. Data is used to monitor intervention 33 (73.3%) 16. Student receives positive feedback pertaining to intervention 44 (97.8%) 17. Intervention requires no more than 10 min per day 35 (77.8%) 18. Written instructions for how to implement intervention 17 (37.8%) 19. Description of intervention is provided to teacher 17 (37.8%) Intensive individualized interventions 20. Staff member trained to conduct FBAs 43 (95.6%) 21. Student’s teacher is on FBA team 43 (95.6%) 22. Staff with FBA knowledge is on FBA team 42 (93.3%) 23. Process used to lead FBA 41 (91.1%)

Note. The individual I-SSET items are abbreviated for reporting in table. FBA = functional behavioral assessment.

Debnam et al. 147

SST referral forms indicated that nearly all of the schools were lacking essential components needed for the SST pro- cess. Specifically, 95.6% of schools were missing information about the antecedents of the behavioral concern, 91.1% were missing information on the setting events, and 91.1% were missing information about the perceived function of the stu- dent’s behavior. Inspection of the items on the Intensive Indi- vidualized Interventions subscale indicated that a large proportion of the schools had high scores in several areas related to individual support systems. The majority of schools (91.1%) reported using an FBA to select intensive interven- tions. Nearly all schools reported that the team that develops FBAs is composed of one of the student’s teachers (95.6%) and that a member is trained in the FBA process (93.3%).

Correlations Between the SET, I-SSET, and School Demographic Characteristics There were no significant correlations between the SET sub- scales and I-SSET subscales (see Table 3). However, the three schools that did not meet the 80% overall score on the

SET generally received slightly lower scores on the I-SSET (i.e., 66%, 78%, and 84%). The intercorrelations between the school-level factors revealed associations in the expected directions between school demographic characteristics (see Table 4). Specifically, the rates of FARMs, suspensions, and mobility were negatively associated with student achieve- ment. The percentage of Caucasian students also was related inversely to student achievement. The correlations between the SET subscale scores and school demographics revealed several significant associations, which were all small to mod- erate in size. Specifically, about one quarter of all correla- tions conducted were significant, including the Management subscale and the percentage of students who received special education services (r = –.376, p < .05; Table 5), the FARMs rate (r = –.360, p < .05), and math achievement (r = .303, p < .05). Monitoring and Evaluation also was positively cor- related with math achievement (r = .312, p < .05) and nega- tively correlated with the percentage of Caucasian students (r = .313, p < .05) and suspensions (r = –.353, p < .05). Suspensions were also significantly negatively correlated with Expectations Defined (r = –.373, p < .05) and the Overall

Table 3. Correlations Among the SET and I-SSET Subscales

SET and I-SSET subscales 2 3 4 5 6 7 8 9 10 11 12

1. Expectations defined .234 .156 .469** .273 .096 .062 .621** .200 –.046 –.040 .053 2. Behavioral expectations taught – .081 .155 .276 .384** –.018 .501** .216 .188 .060 .211 3. System for rewarding behavioral

expectations – .073 .091 .144 .245 .493** .169 .057 .043 .124

4. System for responding to behavioral violations

– .296* .248 .045 .584** .224 .186 .055 .211

5. Monitoring and evaluation – .484** .209 .558** .221 –.018 –.048 .071 6. Management – .281 .582** .042 .124 –.025 .060 7. District-level support – .543** .015 .089 –.143 –.026 8. SET overall score – .270 .155 –.033 .176 9. Foundations – .554** .125 .765**

10. Targeted interventions – .166 .769**

11. Intensive individualized interventions

– .630**

12. I-SSET overall score –

*p < .05. **p < .01. ***p < .001.

Table 4. Correlations Among School Demographic Characteristics

2 3 4 5 6 7 8 9

1. School enrollment .51** –.51** –.19 .22 –.40** –.19 .06 .23 2. Student-to-teacher ratio – –.21 –.26 –.02 –.26 –.23 –.21 –.05 3. Free/reduced-price meals (%) – .15 –.67** .51** .35* –.56** –.68**

4. Special education students (%) – .05 .07 .03 .07 –.08 5. Caucasian students (%) – –.42** –.37* .69** .69**

6. Student mobility (%) – .31* –.23 –.37*

7. Suspension rate (%) – –.34* –.30*

8. Math performance (%) – .85**

9. Reading performance (%) –

*p < .05. **p < .01.

148 Journal of Positive Behavior Interventions 14(3)

SET score (r = –.296, p < .05). None of the other school demo- graphic variables were statistically significantly associated with SET subscale scores. The correlations between each I-SSET scale and school demographics revealed a signifi- cant relationship between the Intensive Individualized Interventions subscale and the student-to-teacher ratio (r = –.364, p < .05; Table 1). However, we are cautious to interpret this relationship, as it demonstrated a relatively small effect among a series of nonsignificant correlations. No other school demographics were significantly corre- lated with the I-SSET subscale scores.

Characteristics of Most Commonly Used Tier 2 Interventions As described above, the Targeted Interventions subscale captures information regarding three specific programs that the school frequently uses to support nonresponders to SWPBIS. The interventions most commonly listed by the schools were Check In/Check Out (n = 23 schools, 51.1%) and behavior charts/contracts (n = 20 schools, 44.4%; see Figure 1). The other interventions used most often were social skills groups (n = 12 schools, 26.7%), various reading interventions (n = 13 schools, 28.9%), and other academic interventions held outside of school hours (n = 8 schools, 17.8%). All but one school reported that the interventions were linked directly to school-wide expectations (97.8%) and resulted in the student’s receiving positive feedback from staff (97.8%). The majority of schools also reported that these programs were continuously available for student participation (93.3%) and that data were used to monitor their impacts (73.3%). However, fewer than half the schools

reported that interventions were implemented within 3 days (44.4%), or reported having intervention plans that included instructions for implementation (37.8%) or a written descrip- tion of the intervention (37.8%). Approximately one third of the schools reported not using a standardized process to identify evidence-based interventions for students. Only half of the schools (i.e., those using CI/CO) reported using a program with a published evidence base.

Discussion The current study describes the types of targeted and inten- sive supports implemented in SWPBIS schools. As a requirement for enrollment in the study, schools must have implemented the critical features of SWPBIS, as indicated by an Overall SET score of 80% or higher in the prior school year. However, 3 of the 45 schools did not achieve an 80% when assessed for the current study. Additional research is needed to better understand patterns of sustain- ability within a single school year and across multiple school years (Bradshaw, Reinke, Brown, Bevans, & Leaf, 2008). Furthermore, no significant correlations were found between the SET and I-SSET subscales. This is not surpris- ing, given that the SET measures the implementation of the critical features of the SWPBIS, whereas the I-SSET mea- sures the features and types of additional support provided to those students not responding adequately to the SWPBIS. In terms of the implementation of Tier 2 supports, a high proportion of the schools followed state and federally man- dated processes, such as teams to address student concerns (i.e., SST) and FBAs within the team framework. In con- trast, schools tended to lack a comprehensive form for

Table 5. Correlations Among the SET Subscales and School Demographics

Expectations defined

Behavioral expectations

taught

System for rewarding behavioral

expectations

System for responding to

behavioral violations

Monitoring and

evaluation Management

District- level

support

SET overall score

SET score M 93.9% 91.1% 93.9% 86.4% 96.9% 92.1% 93.3% 92.5% SD 14.3 12.3 12.6 12.8 6.1 7.9 17.2 6.6 School demographics School enrollment .259 .092 –.062 .284 .002 –.022 –.237 .075 Student-to-teacher ratio .084 .081 .114 .133 .006 –.220 .025 .088 Free/reduced-price

meals (%) –.151 –.201 –.139 –.227 –.226 –.360* .032 –.281

Special education students (%)

–.066 –.216 .132 –.207 –.370* –.376* –.085 –.245

Caucasian students (%) .216 .211 .210 .274 .313* .250 .005 .343*

Student mobility (%) –.083 –.101 .016 –.105 –.073 –.093 .105 –.064 Suspension rate (%) –.373* –.132 –.179 –.160 –.353* –.017 –.007 –.296*

Math performance (%) .232 –.055 –.055 .259 .312* .303* .105 .277 Reading performance (%) –.191 –.154 –.154 –.062 –.004 –.022 –.021 –.098

*p < .05.

Debnam et al. 149

referring students to the SST, which, in turn, may hamper the team’s ability to efficiently address concerns. Specifically, inspection of the schools’ SST referral forms indicated that most were missing information on the behav- ioral antecedents, setting events, and perceived function of the behavior. Without these critical elements, it is difficult to determine why the behavior is occurring and to choose an intervention that can adequately address it (Scott et al., 2005). Despite the increasing emphasis on cultural compe- tence and concern regarding the disproportionate represen- tation of racial and ethnic minorities in special education and school discipline (Bradshaw, Mitchell, O’Brennan, & Leaf, 2010), only half of the schools reported providing professional development training for staff in this area. Additional research is needed to identify evidence-based models of cultural proficiency training.

There is some research to suggest that school contextual factors may hinder schools from providing high-quality Tier 2 and 3 supports (Domitrovich et al., 2008). Data from the current study suggested that school contextual factors were significantly correlated with about half of the SET scale scores, but the effect sizes were in the small to moder- ate range. As hypothesized, higher rates of problem behav- ior (i.e., suspensions) in the year preceding implementation data collection were generally associated with lower imple- mentation, whereas higher academic achievement was associated with higher SET scores. Because of the correla- tional nature of the current study, we cannot assume a causal

association between the SET scores and school contextual factors. However, prior research using randomized con- trolled trial designs does suggest that SWPBIS is associated with reductions in suspensions and improvements in aca- demic achievement (Bradshaw, Mitchell, & Leaf, 2010; Horner et al., 2009).

A unique feature of the I-SSET is the assessment of the three most commonly used Tier 2 interventions in the school. Here schools identified a wide variety of interventions (see Figure 1). The most commonly used intervention, CI/CO, provides students with increased positive feedback from school staff (Crone et al., 2004). It includes the use of a behavior report card and requires targeted students to “check in” at the beginning of the school day, receive feedback from teachers throughout the day, and “check out” at the end of the day, during which they receive feedback from and inter- act with a specified staff member. CI/CO was the only inter- vention clearly identified by the schools that has an evidence base to support its use. It is possible that the behavior charts/ contracts identified as the second most commonly used intervention may be a “watered down” version of the behav- ior report card used with CI/CO or another empirically based intervention. In fact, six schools reported both using CI/CO and behavioral contracts as two separate interventions. The remaining “programs” named were nonspecific practices or strategies (e.g., tutoring or counseling). Without a specifica- tion of a program name or framework, it is difficult to deter- mine if there is an empirical base for their use, or their

11.1 17.8

11.1

44.4 51.1

17.8 11.1 11.1

15.6

28.9

8.9

26.7

13.3 15.6

0

10

20

30

40

50

60

70

A ca

de m

ic (S

m al

l G ro

up ,

O ne

o n

O ne

)

A ca

de m

ic (O

ut si

de o

f Sc

ho ol

H ou

rs )

A ng

er M

an ag

em en

t/ C

on fl

ic t R

es ol

ut io

n

B eh

av io

r C ha

rt s/

C on

tr ac

t

C he

ck -i

n/ C

he ck

-o ut

G ro

up C

ou ns

el in

g

C om

m un

ity M

en ta

l H

ea lth

S er

vi ce

s

L ea

rn in

g L

ab

M en

to ri

ng

R ea

di ng

In te

rv en

tio ns

Sc ho

ol -S

pe ci

fi c

Pr og

ra m

s

So ci

al S

ki lls

T ut

or in

g

U nd

ef in

ab le%

o f S

ch oo

ls R

ep or

ti ng

U se

o f P

ro gr

am

Targeted Program

Figure 1. Percentage of schools that reported using each targeted program as assessed on the I-SSET. Note. School staff (e.g., SST leader, administrator, school psychologist) reported on the I-SSET the three most commonly used programs for children not responding adequately to the universal SWPBIS program. The programs were grouped by the researchers to facilitate review. “Undefinable” indicates programs that did not fit within the general categories listed above.

150 Journal of Positive Behavior Interventions 14(3)

structure and intensity (Gottfredson & Gottfredson, 2001). Administers of the I-SSET should be sure to request the specific name of programs implemented, as was done in the current study, to enhance precision.

These data suggest there is room for improvement on the SST referral forms, the response to referrals, and the regular occurrence of meetings. In addition to these concerns with the SST process, the majority of the schools also did not implement the more intensive programs within 3 days of developing the intervention plan. Few schools reported that there was a clear, written description of the intervention or instructions for student’s classroom teacher on implementa- tion, perhaps limiting the level of fidelity. Taken together, these findings suggest that although the schools had SSTs in place, these teams lacked consistent processes for developing and implementing interventions. These data also suggested that most of the targeted interventions lacked a standard structure and varied in the condition and consistency of their implementation. This finding is consistent with prior research indicating that most of the interventions used in schools (i.e., outside of research studies) are not evidence based and are implemented with questionable fidelity (Domitrovich et al., 2008; Gottfredson & Gottfredson, 2002).

Finally, there was only one statistically significant cor- relation between the I-SSET and school demographics. The only association that reached statistical significance was between the student-to-teacher ratio and the Intensive Individualized Interventions. This finding needs to be inter- preted with caution, as the statistical significance of this correlation may have been due to Type I error, given that this was the only significant finding in a series of analyses (Perneger, 1998). Although we had hypothesized that school-level factors such as student mobility, school size, and high rates of student discipline problems would be associated with poorer implementation, this was not the case. It is promising that no other school-level factors were significantly correlated with the I-SSET scores. However, a previous randomized trial of SWPBIS suggested that schools that have lower levels of organizational health before implementation of SWPBIS tend to take longer to implement the universal system with high fidelity, but tend to improve the most (with regard to organization) following implementation of SWPBIS (Bradshaw et al., 2009). Further longitudinal research is needed to determine if a similar association exists for Tier 2 supports.

Limitations It is important to consider some limitations when reviewing these findings. Both the SET and I-SSET had relatively low Cronbach’s alpha coefficients (.72 for both measures). Although adequate for research purposes (Henson, 2001), there was limited variability in the items, because most schools received very high scores, especially on the SET. Limited variability in the item-level responses likely affected

the alpha. The restricted range of both SET and I-SSET scores may also have affected the strength of the correlations observed with the school-level contextual factors, thereby resulting in smaller than expected associations for some variables. The I-SSET is a relatively new tool, and its psycho- metric properties have not been thoroughly investigated. Findings from the current study showed low to adequate alpha coefficients for the I-SSET subscales (.50–.64), thus precluding the ability to make conclusive statements about I-SSET subscale analysis. A close examination of the psycho- metric properties of the I-SSET may necessitate the inclusion of additional items to capture the quality of schools’ Tier 2 and 3 supports. Although we provided data regarding the adequate level of interobserver agreement from the SET and I-SSET training sessions, data on the interobserver agreement for all 45 SETs and I-SSETs administered in the current study are not available. Similarly, we lack another source of infor- mation on the implementation of the Tier 2 and 3 supports, suggesting a need for other measures to document these pro- cesses. Relatedly, we do not have data on the outcomes of particular interventions implemented and thus are unable to conclude the efficacy of a particular intervention. Additional research is needed on the I-SSET with a larger sample of schools, including schools with formal training in Tier 2 and 3 supports. Given the different models of SWPBIS used across the United States, it is unknown the extent to which these findings will generalize to schools in other states, which may use other models of SWPBIS training and support. Additional research also is needed to examine the implemen- tation of SWPBIS, Tier 2 and 3 systems of support in middle and high schools, where training and support needs may be greater. As noted above, there were some potential concerns regarding the number of tests conducted. Because of the rela- tively small sample size, we did not apply a Bonferroni adjustment to correct for multiple tests but rather focused on findings that were both consistent across multiple related constructs and were theoretically and conceptually defensible (Nakagawa, 2004; Perneger, 1998).

At the time of data collection, Maryland had not devel- oped a coordinated Tier 2 or 3 model of support through the statewide PBIS initiative (Barrett et al., 2008). Although the overall I-SSET scores reached 80% on average, the scores were likely inflated somewhat by high scores on the Intensive Individualized Interventions scale, which mainly assessed state and federally mandated processes, like FBA and SST. Like most states (Individuals with Disabilities Education Improvement Act, 2004), Maryland requires schools to use these processes and provides some guidance to districts in the training and implementation of those pro- cesses. Therefore, school districts provide similar trainings related to SST and FBA, which suggests that the training provided to the schools across the six districts was likely similar. The PBIS teams from these schools also attended annual SWPBIS booster events hosted by the state; these trainings focused primarily on SWPBIS but did provide

Debnam et al. 151

brief overviews of how to integrate more intensive sup- ports, like CI/CO and FBA, within the PBIS framework. Yet, the schools’ scores were lower for the Foundations and Targeted Interventions scales of the I-SSET, which do not relate as closely to mandated processes, and thus suggest a need for more professional development activities that focus on connecting Tier 2 with the SWPBIS system of sup- port to promote sustainable and consistent delivery systems. A statewide PBIS initiative, which promotes integration and coordination of services and provides complementary professional development and technical assistance on evi- dence-based programs, would likely result in high-quality, sustainable systems of support (Sugai & Horner, 2006).

Implications and Future Research The present study is an initial attempt to understand the types of programs provided to children not responding ade- quately to SWPBIS. The findings indicated that most schools have SSTs in place to address student behavior concerns and actively use FBA and interventions linked directly to school-wide expectations. However, the dearth of student information captured on the referrals to the SST, as well as the absence of a documented process for selecting evidence-based interventions for children, suggest a need for further training for schools in these areas. Schools may need a more defined system for collecting and sharing infor- mation about student needs during team meetings, as well as a strategic process for identifying, implementing, and evalu- ating evidence-based interventions selected for nonre- sponders (Scott et al., 2005). This is a challenge often faced by schools as they attempt to integrate an RtI approach into the special education identification process and reflects the more general shift to prevention through schools (Hawken et al., 2008). Although beyond the scope of the current study, future research should consider how evidence-based interventions are selected and implemented by SSTs, and the extent to which their use is based on the perceived function of the student’s behavior problem, rather than merely avail- ability and familiarity (Scott et al., 2005). Similarly, we still lack sufficient evidence to determine which programs are most effective for different students.

The I-SSET appears to be a useful tool for documenting the features and processes of Tier 2 and 3 supports. To our knowledge, there are few, if any, general fidelity measures that can be used to assess multiple programs. Most fidelity measures are program specific, and thus the I-SSET is unique in this way. It would be useful to have a single mea- sure that could document the core elements of different pro- grams (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005). This seems particularly important, given that schools are generally implementing multiple programs simultaneously and with varying degrees of fidelity (G.D. Gottfredson & Gottfredson, 2001; D.C. Gottfredson & Gottfredson, 2002).

Additional research is therefore needed to determine the extent to which the I-SSET is sensitive to the core features of quality implementation of multiple programs.

A primary aim of the study was to describe the Tier 2 and 3 programs and services that SWPBIS elementary schools use when they have not received formal training in these supports. The SET and I-SSET provide an efficient method for collecting information about these services and identify- ing areas of needed support for schools. The findings sug- gest that elementary schools may still struggle with addressing the needs of nonresponders to SWPBIS. At the school level, areas of weakness can be targeted through staff training and professional development. These data also suggest a need for providing school staff with an enhanced understanding of data-based decision-making and problem- solving strategies. For example, training should focus on how to identify the functions of behavior and how to use that information to select an intervention approach (Crone & Horner, 2003; Hershfeldt, Rosenberg, & Bradshaw, 2011). Professional development should also cover the crit- ical features of the SST referral forms and how that data can be used to inform the collaborative problem-solving pro- cess (Scott et al., 2005). More consistent and detailed meth- ods and materials are needed to increase schools’ ability to conduct valid FBAs and subsequently create effective inter- vention plans (Crone & Horner, 2003). Finally, additional support is needed regarding evidence-based interventions and the process for selecting an appropriate program to meet the student’s particular pattern of needs.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Support for this project comes from the Institute of Education Sciences (R324A07118 and R305A090307), the Centers for Disease Control and Prevention (1U49CE 000728-011 and K01CE001333-01), and the National Institute of Mental Health (T32 MH19545-11).

References

Barrett, S., Bradshaw, C. P., & Lewis-Palmer, T. (2008). Maryland state-wide PBIS initiative: Systems, evaluation, and next steps. Journal of Positive Behavior Interventions, 10, 105–114.

Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J. (2009). Altering school climate through school-wide Positive Behav- ioral Interventions and Supports: Findings from a group-ran- domized effectiveness trial. Prevention Science, 10, 100–115.

Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of School-Wide Positive Behavioral Interventions and Supports on student outcomes: Results from a randomized

152 Journal of Positive Behavior Interventions 14(3)

controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133–148.

Bradshaw, C. P., Mitchell, M. M., O’Brennan, L. M., & Leaf, P. J. (2010). Multilevel exploration of factors contributing to the overrepresentation of Black students in office disciplinary referrals. Journal of Educational Psychology, 102, 508–520.

Bradshaw, C. P., Reinke, W. M., Brown, L. D., Bevans, K. B., & Leaf, P. J. (2008). Implementation of school-wide Positive Behavioral Interventions and Supports (PBIS) in elementary schools: Observations from a randomized trial. Education & Treatment of Children, 31, 1–26.

Crone, D. A., & Horner, R. H. (2003). Building positive behav- ior support systems in schools: Functional behavioral assess- ment. New York, NY: Guilford.

Crone, D. A., Horner, R. H., & Hawken, L. S. (2004). Responding to problem behavior in schools: The behavior education pro- gram. New York, NY: Guilford.

Domitrovich, C. E., Bradshaw, C. P., Poduska, J., Hoagwood, K., Buckley, J., Olin, S., . . . Ialongo, N. (2008). Maximiz- ing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion: Training and Practice, Research and Policy, 1(3), 6–28.

Filter, K. J., McKenna, M. K., Benedict, E. A., Horner, R. H., Todd, A. W., & Watson, J. (2007). Check In/Check Out: A post-hoc evaluation of an efficient, secondary-level targeted intervention for reducing problem behaviors in schools. Edu- cation and Treatment of Children, 30, 66–84.

Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida.

Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school- based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39, 3–35.

Gottfredson, G. D., & Gottfredson, D. C. (2001). What schools do to prevent problem behavior and promote safe environments. Journal of Educational and Psychological Consultation, 12, 313–344.

Gottfredson, G. D., Gottfredson, D. C., Payne, A. A., & Gottfredson, N. C. (2005). School climate predictors of school disorder: Results from a national study of delinquency prevention in schools. Journal of Research in Crime and Delinquency, 42, 412–444.

Hawken, L. S., MacLeod, K. S., & Rawlings, L. (2007). Effects of the behavior education program (BEP) on office disciplin- ary referrals of elementary school students. Journal of Positive Behavior Interventions, 9, 94–101.

Hawken, L. S., Vincent, C. G., & Schumann, J. (2008). Response to intervention for social behavior: Challenges and opportunities. Journal of Emotional and Behavioral Disorders, 16, 213–225.

Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177–189.

Hershfeldt, P., Rosenberg, M., & Bradshaw, C. P. (2011). Function- based thinking: A systematic way of thinking about function

and its role in changing student behavior problems. Beyond Behavior, 19, 12–21.

Horner, R., Todd, A., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. (2004). The School-wide Evaluation Tool (SET): A research instrument for assessing school-wide positive behavior supports. Journal of Positive Behavior Intervention, 6, 3–12.

Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide Positive Behavior Support in elementary schools. Journal of Positive Behavior Interventions, 11, 133–144.

Individuals with Disabilities Education Improvement Act of 2004. (2003). H.R. 1350-108th Congress. In GovTrack.us (database of federal legislation). Retrieved from http://www.govtrack. us/congress/bill.xpd?bill=h108-1350

Lane, K. L., Rogers, L. A., Parks, R. J., Weisenbach, J. L., Mau, A. C., Merwin, M. T., & Bergman, W. A. (2007). Function-based interventions for students who are nonresponsive to primary and secondary prevention efforts: Illustrations at the elemen- tary and middle school levels. Journal of Emotional and Behavioral Disorders, 15, 169–183.

Lewis-Palmer, T., Todd, A. W., Horner, R. H., Sugai, G., & Sampson, N. K. (2005). Individual student systems evaluation tool, version 1.2. Eugene: Educational and Community Supports, University of Oregon.

Nakagawa, S. (2004). A farewell to Bonferroni: The problems of low statistical power and publication bias. Behavioral Ecol- ogy, 15, 1044–1045.

O’Neill, R. E., Horner, R. H., Albin, R. W., Sprague, J. R., Storey, K., & Newton, J. S. (1997). Functional assessment and program development for problem behavior: A practical handbook (2nd ed.). Pacific Grove, CA: Brooks/Cole.

Perneger, T. V. (1998). What’s wrong with Bonferroni adjust- ments. British Medical Journal, 316, 1236–1238.

Scott, T. M., McIntyre, J., Liaupsin, C., Nelson, C. M., Conroy, M., & Payne, L. D. (2005). An examination of the relation between functional behavior assessment and selected intervention strat- egies with school-based teams. Journal of Positive Behavior Interventions, 7, 205–215.

Sugai, G., & Horner, R. (2006). A promising approach for expand- ing and sustaining the implementation of school-wide positive behavior support. School Psychology Review, 35, 245–259.

Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R. (2001). School-wide evaluation tool (SET). Eugene: Center for Posi- tive Behavioral Supports, University of Oregon.

Todd, A. W., Campbell, A. L., Meyer, G. G., & Horner, R. H. (2008). The effects of a targeted intervention to reduce problem behav- iors: Elementary school implementation of Check In Check Out. Journal of Positive Behavior Interventions, 10, 46–55.

Walker, H., Horner, R. H., Sugai, G., Bullis, M., Sprague, J., Bricker, D., & Kaufman, M. J. (1996). Integrated approaches to preventing antisocial behavior patterns among school-age children and youth. Journal of Emotional and Behavioral Dis- orders, 4, 194–209.