discussion 5
Journal of Positive Behavior Interventions 13(4) 208 –218 © 2011 Hammill Institute on Disabilities Reprints and permission: http://www. sagepub.com/journalsPermissions.nav DOI: 10.1177/1098300710385348 http://jpbi.sagepub.com
Each year, schools around the world expend considerable resources implementing evidence-based practices, and most will not sustain beyond a few years (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Fullan, 2005; Latham, 1988). This resulting cycle of implementing new practices each year, as opposed to sustaining effective ones, is clearly not a new phenomenon (Berman & McLaughlin, 1976; Sarason, 1971), but it has come to represent business as usual in school systems, particularly in this age of increased accountability.
This cycle has two distinct types of costs. First, the fre- quent change in programs costs the system resources. There are the obvious monetary costs regarding the purchase and training associated with new programs, but there are also costs to staff willingness to implement new programs. When implementation is abandoned, there is a draining effect on enthusiasm for implementing change, and this energy can be replaced with cynicism when the next pro- gram is introduced (Elliott, Witt, Kratochwill, & Stoiber, 2002). Eventually, hesitant staff realize that if they wait long enough, it is only a matter of time before the new pro- gram will join the others in a virtual graveyard of discontin- ued innovations (Latham, 1988).
A second cost comes in terms of diminished student outcomes. When considering an evidence-based practice such as School-Wide Positive Behavior Support (SWPBS; Sugai & Horner, 2009), an initial investment in implemen- tation can lead to important, valued outcomes. These out- comes include decreased problem behavior (Bradshaw, Mitchell, & Leaf, 2010), improved academic achievement (Horner et al., 2009; Lassen, Steele, & Sailor, 2006; McIntosh, Chard, Boland, & Horner, 2006), staff and admin- istrator time regained as a result of fewer discipline referrals (Scott & Barrett, 2004), and student instructional time regained as a result of more time spent in class and on task (Algozzine & Algozzine, 2007). Failure to sustain SWPBS would result in the loss of these benefits for students and school personnel.
385348PBI13410.1177/1098300710385348McInto sh et al.Journal of Positive Behavior Interventions © 2011 Hammill Institute on Disabilities
Reprints and permission: http://www. sagepub.com/journalsPermissions.nav
1University of British Columbia, Vancouver, Canada 2U.S. Department of Education, Washington, DC, USA 3University of Oregon, Eugene, OR, USA
Corresponding Author: Kent McIntosh, University of British Columbia, 2125 Main Mall, Vancouver, BC V6T 1Z4 Email: kent.mcintosh@ubc.ca
Development and Initial Validation of a Measure to Assess Factors Related to Sustainability of School-Wide Positive Behavior Support
Kent McIntosh1, Leslie D. MacKay1, Amanda E. Hume1, Jennifer Doolittle2, Claudia G. Vincent3, Robert H. Horner3, and Ruth A. Ervin1
Abstract
Sustainability of effective practices in schools is a critical area for research in any domain. The purpose of this article is to describe and evaluate the validity and reliability of a recently developed research instrument designed to evaluate schools’ capacity to sustain school-wide positive behavior support (SWPBS) efforts at the universal tier. The School-Wide Universal Behavior Sustainability Index–School Teams (SUBSIST) was created to assess factors (of the context, implementer practices, and outcomes) that enhance or prevent sustainability of SWPBS. Content of the web-based survey was identified through literature review, and initial validation analyses included ratings of content validity by an expert panel (n = 21) and assessment of internal consistency, test–retest reliability, interrater reliability, and concurrent validity (with SWPBS fidelity of implementation data) through a pilot study (n = 25). Results indicated strong psychometric properties for assessing sustainability. The authors discuss the results in terms of future research in enhancing SWPBS sustainability.
Keywords
school-wide positive behavior support, systems change, sustainability, behavior assessment
Action Editor: Don Kincaid
McIntosh et al. 209
Given that SWPBS is currently being implemented in more than 13,000 schools across the United States and Canada (Center on Positive Behavioral Interventions and Supports, 2010), it is important to explore how school, district, and state/ provincial teams can enhance the sustainability of SWPBS. McIntosh, Horner, and Sugai (2009) outlined a research agenda to study this phenomenon and apply results directly to schools. This agenda involves identifying factors related to sustainability, exploring the mechanisms by which sustain- ability occurs, and designing and testing large-scale interven- tions designed to enhance sustainability.
Fortunately, there has been recent progress in this area in the form of research exploring the sustainability of SWPBS in particular. Doolittle (2006) completed a study of 285 schools identifying what critical features of SWPBS implementation predicted (a) initial implementation (full implementation within 3 years) and (b) sustained implemen- tation (full implementation for 5 years). Her results indi- cated that systems for teaching expectations and use of school-wide data were significant predictors of initial implementation, as measured by the School-Wide Evaluation Tool (SET; Sugai, Lewis-Palmer, Todd, & Horner, 2001). In contrast, effective administrator and school team leader- ship and an active student reward system were significant predictors of sustained implementation.
More recently, Bambara, Nonnemacher, and Kern (2009) completed a qualitative interview study of factors affecting sustainability of individual student support systems within SWPBS. The authors identified five factors critical to sus- tainability: school culture, building administrator support, time efficiency, capacity building, and stakeholder involve- ment. These factors were identified by respondents as poten- tially both enablers and barriers. For example, a school culture based on an understanding of the concepts of SWPBS and priority for its use was considered an enabler, but a cul- ture in which a large number of staff members were opposed to SWPBS would be viewed as a barrier. Many of these themes echoed the factors, such as staff culture and building leadership, generated by Kincaid, Childs, Blase, and Wallace (2007) and Lohrmann, Forman, Martin, and Palmieri (2008), who studied enablers and barriers to initial SWPBS imple- mentation. Their research also identified important aspects within the broad areas of school culture and administrator support. Hence, there does appear to be some overlap bet- ween factors affecting initial implementation and sustain- ability, but the extent of the shared variance has yet to be studied extensively.
A Model of Sustainability From the literature on sustainability, McIntosh, Horner, and Sugai (2009) proposed a model of sustainability. This model described the theorized mechanisms by which school-based practices might sustain, as well as four factors
contributing to sustainability. These factors include prior- ity, effectiveness, efficiency, and continuous regeneration, a factor including the use of data to readapt the practice over time. These factors were proposed to affect fidelity of implementation, the critical component of sustainability.
A model describing how these factors relate to sustain- ability is included in Figure 1. In this figure, the effect of SWPBS on long-term student outcomes (i.e., improved social competence and academic achievement, reduced problem behavior) is mediated by fidelity of implementa- tion. Sustained fidelity of implementation is affected by the four sustainability factors: priority, effectiveness, effi- ciency, and continuous regeneration. As such, these factors indirectly affect SWPBS outcomes by acting on fidelity of implementation.
However, this model is as of yet untested, and given that so many schools in the United States and Canada are cur- rently implementing SWPBS, there is a tremendous oppor- tunity to study sustainability on a large scale. Research examining SWPBS sustainability could validate the model and provide as a result specific actions that school teams could take to enhance sustainability. Such results could enhance sustainability of not only SWPBS but school-based practices in general.
A Measure to Assess Factors Related to Sustainability With this goal in mind, the authors developed a measure to assess the importance of factors and critical features of SWPBS on its sustained implementation. A number of studies on sustainability factors have used extensive inter- views to generate themes for sustainability (Bambara et al., 2009; Hieneman & Dunlap, 2001), and the goal of creating this measure was to test elements of these themes on a large, international scale. As such, a survey measure was
Distal student
outcomes
Improved social
competence
Improved academic
achievement
Reduced problem behavior
SWPBS
Effectiveness Continuous
regeneration EfficiencyPriority
SUSTAINABILITY FACTORS
Sustained fidelity of
implementation
Figure 1. A model of School-Wide Positive Behavior Support sustainability
210 Journal of Positive Behavior Interventions 13(4)
envisioned to identify which variables most significantly predicted sustainability and failure to sustain. Results could then be used to help long-term and newly implementing SWPBS schools take research-validated steps to enhance the sustainability of their systems.
The School-Wide Universal Behavior Sustainability Index–School Teams (SUBSIST; McIntosh, Doolittle, Vincent, Horner, & Ervin, 2009; please contact the first author for a copy of the measure) is an instrument designed to assess the variables that enhance or prevent sustainabil- ity of universal tier SWPBS. The measure is administered online as a web-based survey and is intended to be com- pleted at the school level, by school team members or per- sonnel familiar with the school in question (e.g., external coaches).
The measure is composed of 50 items, or statements about the SWPBS systems, such as “A vast majority of school personnel (80% or more) support SWPBS” and “There is regular measurement of fidelity of implementa- tion (e.g., Team Checklist, SET, Benchmarks of Quality),” and perceived barriers, such as “There are high levels of turnover of school personnel who served as key leaders (‘champions’) of SWPBS.” These items are organized into eight broad subscales, including priority, building leader- ship, external leadership, effectiveness, efficiency, use of data, capacity building, and potential barriers, plus a set of open-ended and demographic questions.
When completing the SUBSIST, respondents are asked four questions per item. Two questions assess the extent to which each item is true for the school (a) during the 1st year of implementation and (b) at the time of administra- tion. Two questions assess perceived importance of the item for (a) initial implementation and (b) sustainability. Responses for each question are indicated through a Likert-type scale. As a result, the SUBSIST generates individual item scores as well as total scores for initial implementation, sustainability, and perceived importance at two points in time. The measure takes 40 to 60 minutes to complete.
Test and Item Development The SUBSIST measure was created from a bank of items regarding critical features (of the context, the practice, and the outcomes) hypothesized to affect sustainability. These items were derived through an extensive literature review (see McIntosh, Horner, & Sugai, 2009) and discussion among the authors based on their experience with sustain- ability, measurement, and a previous sustainability study (Doolittle, 2006). Priority was placed on identifying mal- leable variables (e.g., those that can be changed; Biglan, 2004). Inclusion of items in the bank was determined based on convergence of the literature and consensus of the authors.
These items were then mapped onto the sustainability model factors and sorted into eight measure subscales. Continuous regeneration, an important factor, was divided into two subscales, use of data and capacity building. Administrative leadership, a recurring theme in the litera- ture, was identified as a distinct area for sustainability and further divided into building and external (e.g., district and state/provincial) leadership subscales, reflecting the differ- ences in influence between these two leadership structures identified in the literature (Bambara et al., 2009; Fixsen et al., 2005; Fullan, 2005; Ransford, Greenberg, Domitrovich, Small, & Jacobson, 2009). Items hypothesized to be barri- ers to sustainability were included in an eighth subscale and reverse scored, so that higher scores in each subscale would indicate increased potential for sustainability.
Finally, the survey itself was constructed in accordance with recommended survey design practices (Alreck & Settle, 1995; Dillman, 1983). These recommendations include a brief introduction before the questions begin, common response formats that vary little across questions, inclusion of a few open-ended responses, and asking demo- graphic questions at the end of the survey. In addition, as the survey was web-based, care was taken to present items on each page to keep the response choices in the same area of the screen and minimize the need to scroll down to answer, thereby reducing the need for extensive mouse navigation.
Validation It was of interest to complete research determining how well the SUBSIST assesses factors related to sustainabil- ity of SWPBS. The remainder of this article describes initial validation and piloting of the SUBSIST measure through two investigations. First, the authors convened a review of the instrument by an expert panel to assess con- tent validity. Second, the measure was pilot tested with SWPBS school team leaders and coaches. Pilot study results were used to finalize wording, obtain reliability information, and identify concurrent validity with extant fidelity of implementation data. The article is organized into two studies: the content validity study and the pilot study.
Content Validity Study The first study examined the content validity of the SUBSIST. The authors convened an expert panel to review the mea- sure and completed a series of analyses as outlined by Rubio, Berg-Wagner, Tebb, Lee, and Rauch (2003) to assess the validity of the items, response format, and over- all measure. The analyses explored the reliability and level of expert panel members’ ratings of the measure’s content validity.
McIntosh et al. 211
Method
Participants. Forty-one individuals were identified and invited to serve on an expert panel to evaluate the content validity of the measure and its items. These individuals were selected based on one or more of the following crite- ria: (a) extensive experience with sustained implementation of SWPBS as a district, provincial, or state coordinator (21 individuals); (b) current provision of training and coor- dination as part of the National PBIS Center (13 individu- als); or (b) experience conducting and publishing research on general systems change in schools in the past 5 years (7 individuals). As noted by Grant and Davis (1997), an adequate number of content experts depends on the desired level of expertise and diversity of knowledge needed. A range of 2 to 20 experts has been suggested in the literature (Gable & Wolf, 1993; Lynn, 1986; Sheatsley, 1983; Walz, Strickland, & Lenz, 1991). Forty-one experts were sent paper copies of the SUBSIST measure and a content valid- ity scale. Twenty-one completed the scale, a response rate of 51%, far above the commonly held criterion for mail sur- vey response rates (30%; Dillman, 1983).
Measure. A content validity scale was designed to obtain information from the expert panel on each item and the SUBSIST measure as a whole. The scale included questions on the extent to which the measure assessed sustainability and was clear to understand and complete. Specifically, each of the 52 original items was associated with two ques- tions (one assessing whether the item represented sustain- ability overall and one assessing whether the item represented the subscale in particular). There were also eight additional questions, two questions about the overall measure and six questions about the survey format (e.g., question, response, and anchors for the Likert-type scale), for a total of 112 questions. All answers were provided through a 4-point Likert-type scale (from strongly disagree to strongly agree). The scale also included space for open-ended comments and suggestions to add, remove, or reword specific items.
Data analysis. The authors followed the procedures for assessing content validity specified by Rubio and colleagues (2003). Two sets of analyses were performed. As a prelimi- nary analysis, expert panel reliability was calculated to determine the agreement among expert panel members on their ratings. This analysis was used not to assess the reli- ability of the SUBSIST measure itself but rather how simi- larly the expert panel members rated each item and the measure as a whole. High expert panel reliability indicates that the panel members agreed on the extent to which items were important for sustainability, and low reliability indi- cates disagreement on the importance of items for sustain- ability. The 4-point scale was dichotomized into agree or disagree, in accordance with previous research (Davis, 1992; Grant & Davis, 1997; Lynn, 1986; Rubio et al., 2003). Expert panel reliability was calculated for each individual
question. To calculate the overall expert panel reliability for the scale, the number of questions with a reliability score of at least .80 was divided by the total number of questions on the scale. This approach is recommended for studies that involve a sample of experts that exceeds five (Lynn, 1986).
The second analysis involved calculating the Content Validity Index (CVI), used to quantify results regarding the extent to which each item and the total measure represented the construct of sustainability (Rubio et al., 2003). It was calculated based on the representativeness of the measure. The CVI for each item was computed by counting the num- ber of experts who rated the item as assessing or strongly assessing sustainability and dividing that number by the total number of experts. The final CVI number is the pro- portion of experts who deemed the item as content valid. The CVI for the measure was estimated by calculating the average CVI across the items. A CVI of .80 is recommended for new measures (Davis, 1992).
Results Regarding expert panel reliability, expert ratings for 105 of the 112 questions were above the .80 criterion. The overall expert panel reliability for the scale was .94. These results indicate that the experts provided highly similar ratings of the importance of each item for measuring sus- tainability, and the CVI results could be interpreted with confidence.
The CVI for each item was calculated to determine the proportion of experts who deemed each item as content valid. Of the 112 items on the SWPBS measure, all but one overall question and the questions associated with four indi- vidual items were rated above the .80 criterion. The CVI for the measure as a whole was .95.
Changes to Measure Of the four individual items that fell below the content validity criterion, two of the items were deleted, and two items were reworded based on panel feedback to improve their clarity and specificity. The expert panel reliability and CVI were recalculated with the two deleted items omitted. The revised expert panel reliability was .97, and the revised CVI score remained at .95.
Summary The expert panel reliability data indicated that the experts’ responses were consistent, and they rated the SUBSIST, including its items, questions, and response format, to be a valid measure of critical features related to sustainability. Feedback from the experts was used to revise the survey, resulting in a reduction of two items (from 52 to 50) and rewording of two retained items, as well as revision of the
212 Journal of Positive Behavior Interventions 13(4)
order of items. These changes were incorporated into the SUBSIST measure before the pilot testing took place.
Pilot Study The pilot study was conducted to examine the psychometric properties of the SUBSIST measure. Respondents from schools implementing universal tier SWPBS completed the SUBSIST. Specifically, the following information was generated: (a) reliability (internal consistency, test–retest, and interrater reliability) and (b) concurrent validity (with a SWPBS fidelity of implementation measure).
Method Settings and participants. Personnel serving 14 public
SWPBS schools in five states (Maryland, Michigan, Minnesota, New Hampshire, and Oregon) participated in the pilot study. Of these, 11 were elementary schools, 1 was a middle school, and 2 were secondary schools. Most schools (79%) reported an enrolment between 300 and 749. The majority of schools were in suburban regions (36%) or small to medium cities (36%), and all were public schools. Most schools (57%) reported the proportion of students receiving free or reduced-price lunch to be between 20% and 49%. The average number of years implementing SWPBS was 4.9, ranging from 2 to 10 years. Fidelity of implementation data, from either the SET (Sugai et al., 2001) or Benchmarks of Quality (BoQ; Kincaid, Childs, & George, 2005), was available for 11 of the 14 schools. Of these schools, 9 met or exceeded criteria for adequate implementation (80% on the SET; 70% on the BoQ).
Twenty-five participants in the 14 SWPBS schools com- pleted the surveys. Fourteen participants served as school SWPBS team leaders (also known as internal coaches), and 11 served as coaches (also known as district or external coaches) for the school SWPBS teams. As a result, the sam- ple included an intact dyad of school team leader and coach from 11 schools implementing SWPBS. Of the 25 total par- ticipants, 21 (84%) opted to complete the survey twice, allowing for test–retest reliability to be calculated.
Measures. The SUBSIST measure, as revised through the content validity study, was administered to pilot study par- ticipants. The pilot survey allowed participants to complete the survey and rate its overall appropriateness and ease of completion. The survey also included open fields after each item for respondents to suggest specific rewording of items. For rating items “currently in place,” the minimum average score possible was 1 (not true) and the maximum score pos- sible was 4 (very true). For the sample, the mean average SUBSIST scores for items currently in place was 3.51, with a standard deviation of 0.24 and a range of 3.05 to 3.91.
As there are no existing research validated measures of factors related to sustainability, the SUBSIST measure was
instead compared to schools’ fidelity of implementation for the current school year, the core indicator of sustained implementation (Han & Weiss, 2005; McIntosh, Horner, & Sugai, 2009). The SET (Sugai et al., 2001) is an external evaluation of a school’s universal SWPBS system that includes a site visit, observations, and interviews of admin- istrators, staff, and students. Administration takes 2 hours and produces the percentage of critical features imple- mented in seven subscales, plus an overall implementation percentage. Research on the SET has identified that it is sensitive to implementation of SWPBS and that an 80% overall implementation criterion leads to significantly improved student outcomes (Bradshaw et al., 2010; Horner et al., 2009). Psychometric data indicates strong evidence of construct and concurrent validity, interrater and test–retest reliability, and internal consistency (Horner et al., 2004). Concurrent validity and internal consistency data have since been revalidated on a separate, larger sample of schools (Vincent, Spaulding, & Tobin, 2010).
Extant SET data were available for 7 of the 14 schools in the sample for the 2008–2009 school year, the same year the SUBSIST was administered. These schools had a mean overall SET score of 0.87 and a standard deviation of 0.08. Five schools were above the 80% criterion, and two schools were below the criterion.
Procedure. The study targeted school and district person- nel implementing SWPBS in K-12 public schools. The authors contacted state SWPBS coordinators in five states (Maryland, Michigan, Minnesota, New Hampshire, and Oregon) and asked them to recommend experienced school team leaders and coaches with experience in sustainability and interest in participating in a pilot study. The state coordinators identified the schools and participants for participation.
The participants were asked to complete the survey twice; once at their earliest convenience and again approxi- mately 2 weeks after their initial completion. The option was given for the survey to be completed online or in paper format, but all participants chose to complete the survey online.
Data analysis. To assess multiple forms of reliability, internal consistency, test–retest, and interrater reliability scores were calculated. To assess internal consistency reli- ability, Cronbach’s alpha coefficients were computed based on responses to the questions “To what extent is [the item] true for your school right now?” and “To what extent was [the item] true for your school during the first 12 months of implementation?” For the 21 participants who completed the SUBSIST twice, test–retest reliability was calculated using the total scores obtained at Times 1 and 2 by dividing the lower score by the higher score. The dyads consisting of a team leader and coach (n = 11 dyads) completed the survey. Interrater agreement was calculated by dividing the lower score by the higher score.
McIntosh et al. 213
To assess concurrent validity, SUBSIST scores for items currently in place (M = 3.61, SD = 0.22) were correlated with extant SET data for the 2008–2009 school year (M = .87, SD = .08), the same year the SUBSIST was administered. Scores for schools above and below the 80% SET criterion were compared with an independent samples t test. In addi- tion, correlations for coaches and team leaders were compared to see if the strengths of the relationships differed by role.
Results Reliability. Cronbach’s alpha coefficients were calculated
based on responses to the questions “To what extent is [the item] true for your school right now?” and “To what extent was [the item] true for your school during the first 12 months of implementation?” The type of respondent was also con- sidered, and reliabilities for both team leaders and coaches were calculated (see Table 1). Alpha coefficients ranged from .77 to .94, indicating strong internal consistency for a survey measure. The test–retest reliability was .96 for both team leaders and coaches. Between team leaders and coaches, an average interrater reliability of .95 was found.
Concurrent validity. Correlations for the concurrent valid- ity analyses are presented in Table 2. The overall correla- tion between the two measures was moderate and statistically significant (r = .68, p < .05). As seen, there was a wide range of correlations between the subscales of the two mea- sures. The SUBSIST priority subscale was significantly correlated with the expectations taught, reward system, and management subscales and the SET average score. The two SUBSIST leaderships subscales were strongly and signifi- cantly correlated with the district support subscale. Effec- tiveness was significantly correlated with district support and the SET average, and efficiency was significantly cor- related with expectations taught, management, and the SET average score. Use of data, capacity building, and potential barriers were not significantly related to any of the SET subscales, indicating the possibility that they may measure domains of sustainability not captured by the SET. Overall, these results show a moderate relation with fidelity of implementation for the year assessed. Results indicate that the SUBSIST measures a construct that is similar to but not exactly the same as that measured by the SET.
In addition, the difference in average scores between schools with SET scores at or above 80% (M = 3.65, SD = 0.23) and schools with SET scores below 80% (M = 3.46, SD = 0.09) was not statistically significant (t = 1.58, p = .14), though the small sample size indicated insufficient power to detect a significant difference. Given the low power, the relation between the SUBSIST and SET was assessed in a visual format through the scatterplot shown in Figure 2. As seen, the best-fit line indicates a linear relation between the mea- sures, with scores from coaches and team leaders clustering around this line. In keeping with the reliability data, correla- tions between the SUBSIST and SET average scores for coaches (r =.73, n = 7) and team leaders (r = .61, n = 6) were not significantly different.
Discussion The purpose of these studies was to examine the validity and reliability of the SUBSIST, a measure assessing fac- tors related to sustainability of universal SWPBS sys- tems. The measure’s content validity was assessed through an expert panel with experience in sustained SWPBS implementation and published research in the area of school systems change. The expert raters consistently indicated that the measure was a valid tool for assessing the important factors that affect sustainability. Results from the pilot study indicated high levels of three indices of reliability, and comparison to fidelity of implementa- tion data indicated that the SUBSIST is sufficiently related to external evaluations of fidelity of implementa- tion. In sum, these results provide an initial indication that the SUBSIST is a valid and reliable measure for assessing factors influencing sustainability of SWPBS. The SUBSIST shows promise as a research measure for understanding how and why school SWPBS efforts are sustained or abandoned.
Results from the content validity study were both prom- ising and useful for further measure development. Overall, the experts expressed strong support for the SUBSIST as a sustainability measure. They rated nearly all of the items as valid critical features for sustainability, and those that were rated below the criterion were removed or revised. The experts also provided extensive descriptive comments that were used to revise the measure’s organization of questions and layout. Given these ratings and the changes made, there is adequate evidence of content validity.
After this refinement of the SUBSIST, pilot testing pro- vided encouraging data regarding use with its intended respondents. Results indicated that the measure has ade- quate internal consistency, or in other words, that the sus- tainability items are related to one another. Moreover, scores were stable across both time and respondents. Although the data from the pilot study could potentially have identified whether team leaders or coaches were better
Table 1. Internal Consistency Results for the SUBSIST (in Terms of Cronbach’s Alpha)
Question Team leader Coach
Item in place currently .84 .77 Item in place first 12 months .94 .92 Both questions .94 .93
Note. : N = 25; n (team leaders) = 14; n (coaches) = 11. SUBSIST = School - Wide Universal Behavior Sustainability Index—School Teams.
T a b
le 2
. C
o rr
el at
io ns
A m
o ng
S U
B SI
ST a
nd S
ET S
ub sc
al es
SU B
SI ST
S ub
sc al
es SE
T S
ub sc
al es
Pr
io ri
ty B
ui ld
in g
le ad
er sh
ip Ex
te rn
al
le ad
er sh
ip Ef
fe ct
iv en
es s
Ef fic
ie nc
y U
se o
f da
ta C
ap ac
it y
bu ild
in g
Po te
nt ia
l ba
rr ie
rs SU
B SI
ST
av er
ag e
Ex pe
ct at
io ns
de
fin ed
Ex pe
ct at
io ns
ta
ug ht
R ew
ar d
sy st
em V
io la
ti o n
sy st
em M
o ni
to ri
ng a
nd
de ci
si o n
m ak
in g
M an
ag em
en t
D is
tr ic
t su
pp o rt
SE T
av
er ag
e
Pr io
ri ty
– .2
5 .2
4 .6
1* .6
3* .5
1 −.
08 .1
1 .6
5* .2
2 .6
7* .6
7* .1
4 −.
01 .6
2* .1
1 .6
7*
B ui
ld in
g le
ad er
sh ip
– .8
2* *
.4 8
.0 6
.0 0
−. 01
−. 28
.5 1
−. 34
−. 12
−. 16
.0 1
−. 41
.1 1
.7 3*
* .2
4
Ex te
rn al
le ad
er sh
ip –
.4 4
.1 4
.1 0
.2 6
−. 28
.6 3*
−. 10
−. 06
−. 10
.0 1
−. 50
.0 2
.7 1*
* .3
1 Ef
fe ct
iv en
es s
– .4
1 .4
4 .4
0 .3
0 .8
4* *
−. 11
.3 1
.5 0
.0 4
−. 00
.3 5
.5 7*
.6 1*
Ef fic
ie nc
y –
.6 9*
* .0
0 .3
5 .6
5* .1
9 .6
6* .5
4 .3
3 −.
12 .6
2* .2
9 .7
4* *
U se
o f da
ta –
.3 1
.2 5
.6 9*
* .1
9 .4
0 .2
6 .2
5 −.
27 .3
5 .3
2 .5
3 C
ap ac
it y
bu ild
in g
– .2
6 .5
0 −.
07 −.
06 .0
1 −.
13 −.
15 −.
05 .2
6 .0
8 Po
te nt
ia l b
ar ri
er s
– .2
8 .0
9 .1
9 .3
3 .0
7 .3
6 .0
7 −.
17 .1
3 SU
B SI
ST a
ve ra
ge –
.0 1
.3 8
.3 9
.1 4
−. 28
.4 1
.6 3*
.6 8*
Ex pe
ct at
io ns
d ef
in ed
– .4
0 .4
6 .2
0 .0
3 −.
17 −.
45 .2
9 Ex
pe ct
at io
ns t
au gh
t –
.8 7*
* .5
6* .3
4 .7
8* *
−. 16
.8 5*
* R
ew ar
d sy
st em
– .2
8 .3
8 .6
2* −.
18 .7
7* *
V io
la ti o n
sy st
em –
.5 4
.2 2
.0 4
.5 9*
M o ni
to ri
ng a
nd
de ci
si o n
m ak
in g
– .0
6 −.
49 .1
2
M an
ag em
en t
– .1
1 .6
7* D
is tr
ic t
su pp
o rt
– .3
4 SE
T a
ve ra
ge –
N ot
e. :
n =
16 . S
U B
SI ST
= S
ch oo
l-W id
e U
ni ve
rs al
B eh
av io
r Su
st ai
na bi
lit y
In de
x— Sc
ho ol
T ea
m s;
SE T
= S
ch oo
l-W id
e Ev
al ua
tio n
To ol
( Su
ga i e
t al
., 20
01 ).
*p <
.0 5.
* *p
< .0
1.
214
McIntosh et al. 215
respondents for the SUBSIST, the results indicate that respondents in both roles provide valid, reliable responses for measuring sustainability.
Concurrent validity results were also promising. Given that there are currently no sustainability measures for com- parison, fidelity of implementation for the year assessed provided a best alternative, as durable, high fidelity of implementation over time is the defining quantitative mea- sure of sustained implementation. Results indicated that the SUBSIST and SET assess related but unique constructs. The differences in the scores according to SET criteria and the scatterplot show a moderate relation between the mea- sures, showing that high SUBSIST scores are associated with high SET scores. The priority, effectiveness, and effi- ciency subscales were most related to the SET average. However, some SUBSIST subscales, such as use of data, capacity building, and potential barriers, were not related to the SET in this sample. One might expect the SUBSIST to be more strongly correlated with the SET, but the SET was designed to measure actual fidelity of implementation, briefly, and at one point in time. In contrast, the SUBSIST measures a broader range of features, including conditions of the environment that influence fidelity of implementa- tion, and assesses some constructs more comprehensively. For example, the SET monitoring and decision making subscales focus exclusively on ODR data, whereas the SUBSIST use of data subscale measures a variety of out- comes and fidelity data. Although there are some items that overlap (e.g., building administrator regularly attends team
3
3.25
3.5
3.75
4
75 80 85 90 95 100
S U
B S
IS T
A v e ra
g e
SET Average
Figure 2. Scatterplot of SUBSIST and SET scores with best-fit line N o te. Squares = coaches; Diamonds = team leaders; SUBSIST = School- Wide Universal Behavior Sustainability Index–School Teams; SET = School-Wide Evaluation Tool.
meetings), only 12% of the SUBSIST’s items are similar to those of the SET. Given these points, the moderate correla- tion was to be expected. A possible conclusion is that fidel- ity of implementation can be seen as an outcome of the presence of sustainability factors as measured by the SUBSIST, though future studies will be needed to examine this relation in more detail.
Limitations There are some limitations in these studies that are worth noting. First, the sample size of the pilot study was small, particularly for the concurrent validity analyses, and did not include schools that had completely abandoned SWPBS. Moreover, the sample was not large enough to assess more complex questions, such as whether differential experience with SWPBS implementation affected either the responses provided or reliability, or whether some items were more or less associated with fidelity of implementation. Planned future research with the SUBSIST will examine these ques- tions with a sample of both sustaining and nonsustaining schools. Finally, a factor or principal components analysis with a larger sample will be critical to validate the organi- zation of the items into the existing eight subscales. As such, these findings should be viewed as tentative until a larger study can validate its conclusions.
Implications for Practice The SUBSIST was designed first and foremost as a research instrument to identify principles and factors affecting sus- tainability. Its comprehensiveness and assessment of initial implementation and perceptions data have the potential to answer important questions regarding sustainability, but its length makes it impractical for regular use by practitioners. Given the widespread interest in assessing and enhancing the sustainability of SWPBS across North America, we have adapted the SUBSIST into a self-assessment tool for use by school personnel. The SUBSIST Checklist (McIntosh, 2010; please contact the first author for a copy of the mea- sure) is based on the format of the self-assessment from the School-Wide PBS Implementer’s Blueprint (Sugai et al., 2005) and is intended for school teams and coaches to assess critical features and create action plans to enhance sustain- ability. It takes approximately 30 minutes to complete. There are 50 items on the checklist; they are the same as those on SUBSIST, but only the “in place” questions are asked for each item, reducing the number of questions by 75%. In addition, the potential barriers subscale was revised to assess the presence of strategies to address specific bar- riers, as opposed to solely the barriers themselves. This change provides teams with options, as opposed to “admiring the problem” (Curtis, Castillo, & Cohen, 2008). However, it should be noted that because the format and some items
216 Journal of Positive Behavior Interventions 13(4)
of SUBSIST were modified to create the checklist, the checklist’s own psychometric properties are unknown.
Future Directions Much of what we know about sustainability comes from qualitative and case study research. However, the SUBSIST presents an opportunity to use the information obtained from these studies and test hypotheses on a large scale. Results may lead to information on overall factors affecting sustainability and regional variations based on different external support provided at the district or provincial/state level. Rather than relying on measures based solely on theory or hypothesized factors, school and district teams could focus their attention on the most critical aspects of sustainability.
There are currently no evidence-based interventions at the school level to sustain implementation after initial training or to set the conditions so that student outcomes are maintained or further improved. To address this significant gap in both research and practice, it will be necessary to identify and develop interventions to promote sustainability. It is hoped that the use of the SUBSIST measure will identify the most impor- tant areas to target for intervention. Such research could help improve the sustainability of any evidence-based practice in schools, and by extension, improve important outcomes for more students.
Acknowledgments
The authors thank the expert panel and pilot study participants for sharing their time and experience. The authors also thank the action editor, reviewers, and Dr. Bruno Zumbo for their helpful suggestions.
Declaration of Conflicting Interests
The authors declared no conflicts of interests with respect to the authorship and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research and/or authorship of this article:
This research was supported by a grant from the UBC Hampton Endowment Fund. Opinions expressed herein do not necessarily reflect the policy of the organization, and no official endorsement should be inferred.
References
Algozzine, K., & Algozzine, B. (2007). Classroom instructional ecology and school-wide positive behavior support. Journal of Applied School Psychology, 24, 29–47.
Alreck, P. L., & Settle, R. B. (1995). The survey research hand- book: Guidelines and strategies for conducting a survey. New York, NY: McGraw-Hill.
Bambara, L., Nonnemacher, S., & Kern, L. (2009). Sustain- ing school-based individualized Positive Behavior Support:
Perceived barriers and enablers. Journal of Positive Behavior Interventions, 11, 161–178.
Berman, P., & McLaughlin, M. W. (1976). Implementation of educational innovations. Educational Forum, 40, 345–370.
Biglan, A. (2004). Contextualism and the development of effec- tive prevention practices. Prevention Science, 5, 15–21.
Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010). Examining the effects of school-wide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133–148.
Center on Positive Behavioral Interventions and Supports. (2010). Schools that are implementing SWPBIS. Retrieved from www .pbis.org
Curtis, M. J., Castillo, J. M., & Cohen, R. M. (2008). Best practices in system-level change. In A. Thomas & J. P. Grimes (Eds.), Best practices in school psychology (Vol. 5, pp. 887–901). Bethesda, MD: National Association of School Psychologists.
Davis, L. (1992). Instrument review: Getting the most from your panel of experts. Applied Nursing Research, 5, 194–197.
Dillman, D. A. (1983). Mail and other self-administered question- naires. In P. Rossi, J. D. Wright, & A. B. Anderson (Eds.), The handbook of survey research (pp. 359–377). New York, NY: Academic Press.
Doolittle, J. H. (2006). Sustainability of positive behavior supports in schools (Unpublished doctoral dissertation). University of Oregon, Eugene.
Elliott, S. N., Witt, J. C., Kratochwill, T. R., & Stoiber, K. C. (2002). Selecting and evaluating classroom interventions. In M. R. Shinn, H. M. Walker, & G. Stoner (Eds.), Interven- tions for academic and behavior problems II: Preventive and remedial approaches (pp. 243–294). Bethesda, MD: National Association of School Psychologists.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: Synthesis of the literature (FMHI Publication No. 231). Tampa, FL: Univer- sity of South Florida, Louis de la Parte Florida Mental Health Institute, the National Implementation Research Network.
Fullan, M. (2005). Leadership and sustainability. Thousand Oaks, CA: Corwin Press.
Gable, R. K., & Wolf, J. W. (1993). Instrument development in the affective domain: Measuring attitudes and values in corporate and school settings. Boston, MA: Kluwer Academic.
Grant, J. S., & Davis, L. L. (1997). Selection and use of content experts for instrument development. Research in Nursing & Health, 20, 269–274.
Han, S. S., & Weiss, B. (2005). Sustainability of teacher imple- mentation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679.
Hieneman, M., & Dunlap, G. (2001). Factors affecting the outcomes of community-based behavioral support: II. Factor category importance. Journal of Positive Behavior Interventions, 3, 67–74.
Horner, R. H. (2009, March). Expanding the science, values and vision of Positive Behavior Support. Paper presented at the
McIntosh et al. 217
Sixth International Association for Positive Behavior Support Conference, Jacksonville, FL.
Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11, 133–144.
Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. (2004). The School-wide Evalua- tion Tool (SET): A research instrument for assessing school- wide positive behavior support. Journal of Positive Behavior Interventions, 6, 3–12.
Kincaid, D., Childs, K., Blase, K. A., & Wallace, F. (2007). Iden- tifying barriers and facilitators in implementing schoolwide positive behavior support. Journal of Positive Behavior Inter- ventions, 9, 174–184.
Kincaid, D., Childs, K., & George, H. (2005). School-wide bench- marks of quality (Unpublished instrument). University of South Florida, Tampa.
Lassen, S. R., Steele, M. M., & Sailor, W. (2006). The relationship of school-wide positive behavior support to academic achieve- ment in an urban middle school. Psychology in the Schools, 43, 701–712.
Latham, G. (1988). The birth and death cycles of educational inno- vations. Principal, 68, 41–43.
Lorhmann, S., Forman, M., Martin, S., & Palmieri, S. (2008). Social context and personal belief barriers that impede the adoption of PBS at the universal level implementation. Journal of Positive Behavior Interventions, 10, 256–259.
Lynn, M. (1986). Determination and quantification of content validity. Nursing Research, 35, 382–385.
McIntosh, K. (2010). SUBSIST sustainability checklist. Vancouver, Canada: University of British Columbia.
McIntosh, K., Chard, D. J., Boland, J. B., & Horner, R. H. (2006). Demonstration of combined efforts in school-wide academic and behavioral systems and incidence of reading and behav- ior challenges in early elementary grades. Journal of Positive Behavior Interventions, 8, 146–154.
McIntosh, K., Doolittle, J. D., Vincent, C. G., Horner, R. H., & Ervin, R. A. (2009). School-wide universal behavior sustain- ability index: School teams. Vancouver, Canada: University of British Columbia.
McIntosh, K., Horner, R. H., & Sugai, G. (2009). Sustainability of systems-level evidence-based practices in schools: Current knowledge and future directions. In W. Sailor, G. Dunlap, G. Sugai, & R. H. Horner (Eds.), Handbook of positive behavior support (pp. 327–352). New York, NY: Springer.
Ransford, C. R., Greenberg, M. T., Domitrovich, C. E., Small, M., & Jacobson, L. (2009). The role of teachers’ psychological expe- riences and perceptions of curriculum supports on the imple- mentation of a social and emotional learning curriculum. School Psychology Review, 38, 510–532.
Rubio, D. M., Berg-Wagner, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying content validity: Conducting a content
validity study in social work research. Social Work Research, 27, 94–104.
Sarason, S. B. (1971). The culture of the school and the problem of change. Boston, MA: Allyn & Bacon.
Scott, T. M., & Barrett, S. B. (2004). Using staff and student time engaged in disciplinary procedures to evaluate the impact of school-wide PBS. Journal of Positive Behavior Interventions, 6, 21–27.
Sheatsley, P. B. (1983). Questionnaire construction and item writ- ing. In P. Rossi, J. D. Wright, & A. B. Anderson (Eds.), The handbook of survey research (pp. 195–230). New York, NY: Academic Press.
Sugai, G., & Horner, R. H. (2009). Defining and describing schoolwide positive behavior support. In W. Sailor, G. Dunlap, G. Sugai, & R. H. Horner (Eds.), Handbook of positive behav- ior support (pp. 307–326). New York, NY: Springer.
Sugai, G., Horner, R. H., Sailor, W., Dunlap, G., Eber, L., Lewis, T. J., . . . Nelson, C. M. (2005). School-wide positive behavior support: Implementers’ blueprint and self-assessment. Eugene: University of Oregon.
Sugai, G., Lewis-Palmer, T. L., Todd, A. W., & Horner, R. H. (2001). School-wide Evaluation Tool (SET). Eugene, OR: Educational and Community Supports. Retrieved from http:// www.pbis.org
Vincent, C. G., Spaulding, S. A., & Tobin, T. J. (2010). A reexamination of the psychometric properties of the School- wide Evaluation Tool (SET). Journal of Positive Behavior Interventions, 12, 161–179.
Walz, C. F., Strickland, O., & Lenz, E. (1991). Measurement in nursing research (Vol. 2). Philadelphia, PA: F.A. Davis.
About the Authors
Kent McIntosh, PhD, is an assistant professor of School Psychology at the University of British Columbia. His current research interests include response to intervention systems for academic and social behavior, systems change, and culturally responsive positive behavior support.
Leslie D. MacKay, MA, a doctoral candidate at the University of British Columbia, has research experience with early literacy and curriculum evaluation. Her current research interests include school-wide positive behavior support and response to interven- tion related to curriculum-based assessment in schools.
Amanda E. Hume is an MA student in school psychology at the University of British Columbia. Her current research interests include positive behavior support, consultation, and response to intervention systems.
Jennifer Doolittle, PhD, is an education program specialist in the Office of Special Education Programs in the U.S. Department of Education. She is the program lead for the State Personnel Development Grants Program. Before coming to the U.S. Department of Education, she was a program specialist for the Oregon
218 Journal of Positive Behavior Interventions 13(4)
Department of Education, focusing on statewide initiatives for response to intervention and positive behavioral interventions and supports.
Claudia G. Vincent, PhD, is a research assistant in Educational and Community Supports at the University of Oregon. Her current interests include culturally responsive implementation of school- wide positive behavior support.
Robert H. Horner, PhD is professor of special education at the University of Oregon, and codirector of the OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports.
Ruth A. Ervin, PhD, is an associate professor at the Univer sity of British Columbia. Her teaching and research interests include consul- tation, systems change, and prevention and intervention strategies to address learning and social-behavioral problems.