Differential Reinforcement of Alternative Behavior

profileMonet013
Raulston2019_Article_InterventionsForRepetitiveBeha2.pdf

Vol.:(0123456789)1 3

Journal of Autism and Developmental Disorders (2019) 49:3047–3059 https://doi.org/10.1007/s10803-019-04023-y

ORIGINALPAPER

Interventions for Repetitive Behavior in Young Children with Autism: A Survey of Behavioral Practices

Tracy J. Raulston1,3 · Sarah G. Hansen2,3 · Wendy Machalicek3 · Laura Lee McIntyre3 · Amarie Carnett4

Published online: 27 April 2019 © Springer Science+Business Media, LLC, part of Springer Nature 2019

Abstract Children with autism spectrum disorder (ASD) display social-communication deficits and present with rigid and repetitive patterns of behavior and/or interests (RRBIs). Compared to interventions for social-communication skills, less attention has been given to RRBIs, especially with regard to interventions for young children. We surveyed 128 behavior analysts who implemented interventions for young children with ASD on their use of 16 practices and one assessment for the treatment of RRBIs. The majority of our sample perceived the practices to be effective in producing sustainable behavior change. Behavior analysts generally responded in the same way to items about reinforcement-based practices, punishment-based practices, and a group of commonly packaged antecedent and consequence-based package components. Implications and future directions are discussed.

Keywords Autism spectrum disorder · Repetitive behavior · Restricted interests · Stereotypy · Young children · Behavioral treatment

Core features of autism spectrum disorder (ASD) include deficits in social-communication skills and the presence of clinically significant rigid and repetitive behavioral pat- terns (American Psychiatric Association 2013). Repetitive behaviors are maladaptive patterns of behavior such as ste- reotypy, insistence on sameness, and restricted interests. For instance, a child with ASD may attend only to certain parts of toys (e.g., car wheels, doll eyes) or insist that toys be manipulated in a very specific order or manner (Cun- ningham and Schreibman 2008). Compared to the literature for social-communication skills, the research base focused

on repetitive behavior in children with ASD is less well- established, not only in terms of available evidence-based interventions, but also in the field’s understanding of the etiology and developmental trajectories (Boyd et al. 2011).

Typically-developing infants and toddlers engage in a variety of repetitive behavior including stereotyped motor movements, rigidity or adherence to routine, perseverative interests, and unusual sensory interests (Evans et al. 1997; Leekam et al. 2007). These behaviors are hypothesized to be adaptive (e.g., self-soothing) during early developmen- tal periods (Cunningham and Schreibman 2008; Evans et al. 1997; Harrop et al. 2014). By the end of toddlerhood, typically-developing young children usually no longer exhibit these topographies of repetitive behavior (Barber et al. 2012). Young children with global developmental delays also display repetitive behavioral patterns, yet some researchers have found that children with ASD display higher rates and different types of repetitive behavior than those with global developmental delays (Goldman et al. 2009; Harrop et al. 2014; Honey et al. 2007; Richler et al. 2010; Wolff et al. 2014).

* Tracy J. Raulston [email protected]

1 Department of Educational Psychology, Counseling, and Special Education, The Pennsylvania State University, 125 CEDAR Building, University Park, PA 16802, USA

2 Department of Learning Sciences, Georgia State University, 30 Pryor Street South West, Atlanta, GA 30306, USA

3 Department of Special Education and Clinical Sciences, University of Oregon, HEDCO Education Building, Eugene, OR 97403, USA

4 Department of Educational Psychology, University of Texas at San Antonio, 501 West Cesar East Chavez Boulevard, San Antonio, TX 78207, USA

3048 Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

Topographies of Repetitive Behavior in ASD

Rigid and repetitive patterns of behavior and/or interests (RRBIs) in ASD encompass four different topographical categories: (a) stereotypy; (b) insistence on sameness, inflexible adherence to routines, or ritualized patterns of verbal or nonverbal behavior; (c) perseverative inter- ests; and (d) hyper- or hyporeactivity to sensory input or unusual interest in sensory aspects of the environment (American Psychiatric Association 2013). Stereotypy includes repetitive motor movements (e.g., rocking, toe walking), object use (e.g., lining up of toys, flipping pages of a book), as well as repetitive vocal responses (e.g., echolalia, scripting, idiosyncratic phrases). Insistence on sameness includes frequent instances in which a child experiences difficulty with changes in routines and may be exceptionally difficult to redirect or comfort during these changes. Further, the child may engage in ritualized behav- ior chains, experience difficulty with transitions, need to take the exact same route, or eat the same food every day. The third category, perseverative interests, involves preoc- cupation with unusual objects or circumscribed interests in topics, and in particular, those that are not age appropri- ate. Finally, hyper- and hyporeactivity involves unusual reactions to sensory stimuli such as over sensitivities or adverse reactions to smells, tastes, and textures, or a fasci- nation with light, reflections, or movement. Conversely, a child may display under reactivity to sensory experiences such as an unusually high pain tolerance or no apparent sense of temperature.

Impact of RRBIs

Aberrant RRBIs in ASD present early in a child’s devel- opment and impede social, occupational, or other critical areas of functioning (American Psychiatric Association 2013). Chronic RRBIs are often associated with co- occurring challenging behavior (e.g., self-injury, aggres- sion) and other conditions including anxiety and mood disorders, which can impede a child’s social learning opportunities and negatively affect family quality of life (Machalicek et al. 2016). For example, a child with ASD may become physically aggressive if a ritual is interrupted (Rispoli et al. 2014). In fact, parents have rated repeti- tive behavior as one of the most difficult aspects of ASD (South et al. 2005).

RRBIs can also hinder children’s learning opportuni- ties (Koegel and Covert 1972; Varni et al. 1979; Pierce and Courchesne 2001) or compromise socialization with peers (Loftin et al. 2008; Nadig et al. 2010). While some

repetitive behaviors are common in toddlerhood (e.g., insistence on sameness during routines), failure to develop behavioral flexibility during early childhood can interfere with learning opportunities and result in aberrant levels or increased severity of challenging behavior (Dominick et al. 2007). As such, repetitive behaviors are often tar- geted for reduction, especially in behavior analytic treat- ment for young children with ASD (Raulston and Macha- licek 2018).

Interventions for RRBIs in ASD

There is a small, but growing, literature base focused on the treatment of RRBIs in ASD (Boyd et al. 2012; Lanovaz and Sladeczek 2012; Patterson et al. 2010; Rapp and Vollmer 2005). Focused interventions (i.e., practices targeting one or a few behaviors as opposed to comprehensive treatments) include antecedent-based strategies, such as environmental enrichment and exercise, as well as consequence-based strat- egies, such as various types of differential reinforcement and punishment procedures (e.g., response interruption and redi- rection). Little is known regarding which interventions prac- titioners commonly implement in practice to treat RRBIs. As most intervention studies for RRBIs in children with ASD have been single-case studies and short in duration (Raulston and Machalicek 2018), the durability and sustainability of these focused interventions is also unknown.

The Current Study

We utilized an implementation science conceptual frame- work to design the current study. Implementation science is the study of factors that influence the adoption and inte- gration of evidence-based interventions into practice (The National Implementation Research Network 2015). In order to understand more about more about current status of implementation of evidence-based practices for young children with ASD, an initial step is to survey practition- ers. As such, we gathered data on what practices are cur- rently being implemented to address RRBIs in infants and young children, defined as birth to 8 years old. Because in most states children with medical ASD diagnoses receive applied behavior analytic therapy (National Conference of State Legislatures 2018), we chose to survey Board Certi- fied Behavior Analysts (BCBAs) who indicated that they supervised and/or implemented interventions for children with ASD. Specifically, we were interested in answering the following research questions: (1) What evidence-based and emergent practices are BCBAs implementing to treat repetitive behavior in infants and young children with ASD; (2) What are the most and least frequently used behavioral

3049Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

practices implemented by BCBAs to treat repetitive behavior in infants and young children with ASD; (3) How effec- tive do BCBAs perceive the available behavioral practices to produce durable, sustainable behavior change (4) Are there significant relations between behavioral practice use and perceived effectiveness; and (5) Is there a greater likeli- hood of practice use or perceived effectiveness depending on demographic factors including: educational background (e.g., psychology, special education), training and supervi- sion (e.g., home, center, hospital), and geographical location (rural versus urban)?

Method

Item Creation

The first author conducted a selective literature review using ERIC and PsycNET online library databases, pair- ing various key terms related to repetitive behavior (e.g., stereotyp*, rigid*) with auti* (see Raulston and Machalicek 2018). Studies that included young children with ASD were included in this selective review. From the results of this selective review, 16 focused interventions and 1 assessment (functional analysis) were identified as evidence-based or emerging practices to reduce RRBIs in young children with ASD. Specifically, the following practices were identified: antecedent-based embedded perseverative interest, conse- quence-based embedded perseverative interest, differential reinforcement of incompatible behavior (DRI), differential reinforcement of other or zero rates of behavior (DRO), dif- ferential reinforcement of variable behavior (DRV), envi- ronmental enrichment, functional communication training (FCT), non-contingent or time based schedules of reinforce- ment (NCR), overcorrection, physical exercise, response blocking, response cost, response interruption and redirec- tion (RIRD), sensory extinction, skill enrichment, and visual and/or verbal cues. Additionally, because sensory integra- tion therapy (SIT) is a commonly used treatment, albeit not evidence-based, and sensory differences were added as a subcategory or RRBIs in the Diagnostic Statistical Manual of Mental Disorders—Fifth Edition, we also included an item related to SIT in the current survey.

Expert Review of Items

The first author developed brief definitions of each afore- mentioned practice. The third and fourth authors, who held PhDs and BCBA-Ds (doctoral designation) and each had more than 15 years of experience in clinical practice and intervention research for young children with ASD, inde- pendently reviewed the practices and definitions. Next, the first and second authors solicited feedback from members

at the Oregon Association for Behavior Analysis (ORABA) quarterly meeting by providing a paper copy of the practices and definitions and receiving written feedback (e.g., confus- ing language) from the meeting attendees, who were mostly BCBAs working in the field with children with ASD. The definitions were revised based on feedback. Table 1 displays the final brief definitions for each of the 18 practices.

Frequency of Use and Perceived Effectiveness

For each practice, we asked respondents to rate the fre- quency with which they used the practice as well as their perception of its effectiveness. In the instance where the respondent rated never using the practice, we still asked about the perceived effectiveness.

For each practice, respondents were asked to indicate on a four-point Likert scale (never = less than approximately 10% of cases, rarely = approximately 10–25% of cases, some- times = approximately 25–50% of cases, or often = more than approximately 50% of cases) the frequency with which they implemented or supervised interventions for RRBIs with infants, toddlers, and young children with ASD (birth to age eight) for each practice. Next, respondents were asked to rate how effective they perceived each of the listed interven- tions to produce durable, sustainable behavior change on a four-point Likert scale (1 = highly ineffective, 2 = ineffective, 3 = effective, 4 = highly effective).

Procedure

Distribution

Following development and conformation of face validity through expert review, revised questionnaires were distrib- uted to behavior analysts of all levels, who included BCBA- Ds (i.e., doctoral designation), BCBAs at the master’s level, and BCaBAs at the assistant level. An email invitation was distributed by the Behavior Analyst Certification Board (BACB) to certificants who indicated that they worked within the following emphasis areas: autism, behavior anal- ysis, behavior therapy, children, counseling, developmental disabilities, education, health, infants, mental health, posi- tive behavior support, psychology or social work.

An accurate response rate was not possible to calcu- late because the exact number of people who received the email invitation is unknown. A Qualtrics® link to the anonymous online survey was embedded within the email invitation. Qualtrics® is a survey hosting website available to faculty that allows for de-identified responses. Qualtrics® indicated that the survey should have taken approximately 10 min to complete. The survey link was active for 2 weeks in January of 2017. A reminder email was sent via Qualtrics® 1 week after the initial email.

3050 Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

Participants were asked to provide an email address if they wished to be entered into a lottery for a $20 Amazon gift card. Emails were only retained if the participant elected to submit their email for a chance to win a gift card. At the end of the 2-week period, 50 emails (from completed surveys) were randomly chosen using a random number generator. Amazon gift cards in the amount of $20 were sent to these 50 email addresses.

Survey Design

The Qualtrics® welcome page included a brief description of the study and contact information of the first, third, and fourth authors. Participants were asked: “Do you work with infants, toddlers, or children with autism spectrum disorder ages birth to eight?” If “yes” was selected, the participant provided informed consent within Qualtrics®. Participants

Table 1 Brief practice definitions

This table displays the brief definitions provided to respondents for each practice

Practice Description

Assessment  Functional analysis A functional analysis is an experimental analysis that involves manipulating the antecedents and/or conse-

quences to identify the function of a behavior Antecedent-based practices  Antecedent-based embedded PI Antecedent-based embedded perseverative interest interventions involve embedding a perseverative interest

into a task or activity to increase the child’s motivation to engage in a desirable behavior  Environmental enrichment Environmental enrichment involves providing non-contingent access to appropriate, competing sources of

reinforcement, such as preferred objects  FCT Functional communication training (FCT) involves teaching an alternative communication behavior to

obtain the same reinforcer as the repetitive behavior  NCR Non-contingent reinforcement or time-based schedules of reinforcement involve providing access to a sen-

sory stimulus (e.g., music) at set intervals to reduce the need for the repetitive behavior  Physical exercise Physical exercise involves having the child engage in an exercise routine prior to an activity that has been

associated with high rates of repetitive behavior  Skill enrichment Skill enrichment involves teaching the child more adaptive skills to replace the need to engage in repetitive

behavior  Visual and/or verbal cues Visual and/or verbal cues involve providing a warning about a change in activity or allowing the child to

engage in a calming or highly preferred activity before a less preferred one Consequence-based practices  Consequence-based embedded PI Consequence-based embedded perseverative interest interventions involve delivering a perseverative inter-

est or perseverative interest-based item (e.g., token) contingent on a desired behavior  DRI Differential reinforcement of incompatible behavior (DRI) involves reinforcing a behavior that the child is

not able to engage in at the same time as the repetitive behavior  DRO Differential reinforcement of other or zero rates of behavior (DRO) involves reinforcing zero rates at set

intervals of a repetitive behavior  DRV Differential reinforcement of variability (DRV) involves reinforcing the child varying his/her behavior with

the reinforcement linked to novel behavior  Overcorrection Overcorrection involves requiring the child to repeatedly practice an alternative behavior to reduce the

repetitive behavior  Response blocking Response blocking involves physically blocking a child from engaging in a repetitive behavior  Response cost Response cost involves the removal of a desired stimulus contingent upon repetitive behavior  RIRD Response interruption and redirection (RIRD) involves physically or verbally interrupting a repetitive

behavior and redirecting the child to engage in a different behavior  Sensory extinction Sensory extinction involves disrupting the contingency between the stereotyped response and the sensory

effects it produces (e.g., gloves to block repetitive skin touching) Non-evidence-based practice  Sensory integration therapy Sensory integration therapy (SIT) involves the use of play and sensory-enhanced interactions to enhance

a child’s sensory processing and motor planning skills that activate the vestibular and somatosensory systems.

3051Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

who answered “no” to this question were sent directly to a thank you page that exited them from the survey. The next page of the survey described the definition of RRBIs as outlined in the DSM-V (American Psychiatric Association 2013) (i.e., descriptions for stereotypy, insistence on same- ness, perseverative interests, and sensory differences were provided). The purpose of this was to orient the behavior analyst to the topographical behavioral profile of interest in the current survey.

The next section of the survey displayed one practice per page with the brief definition at the top. The respondent was asked to indicate how often they implemented or supervised the implementation of the practice to treat an RRBI using a Likert scale for frequency of use (allowed to select one). Next, the respondent was asked to rate the perceived effec- tiveness (allowed to select one) of the practice to produce desirable, sustainable change in behavior. The Likert scales were presented vertically with the aforementioned anchors listed next to the descriptor. Each practice was presented on a separate page and followed the same predictable format. One exception to this was for the one assessment (functional analysis), wherein the respondent was asked to select the most common behavioral function revealed from the func- tional analysis in cases where the RRBI was considered a challenging behavior. Finally, the demographics questions were presented at the end of the survey.

Participants

A total of 128 behavior analysts completed surveys. Descrip- tive statistics were calculated for demographic questions to yield frequency, mean, and range for (a) education (b) gender (c) race/ethnicity, and (d) training and practice envi- ronments (i.e., clinic, school, hospital). Our sample was comprised of mostly Caucasian (81.25%) female (85.94%) respondents. The majority of respondents were certified at the BCBA level (82.81%) working in schools (63.28%), homes (58.59%), or clinic/center based (36.72%) settings. Respondents were most likely to have an undergraduate degree in psychology (57%), although some reported under- graduate degrees in education (12.5%), special education (8%), and unrelated undergraduate degrees (25%). Eighty- four percent of respondents had a master’s level degree in a related field. Of those respondents, master’s degrees were in psychology (29%), special education (23%), education (14%), ABA (10%) and school psychology (8.5%). Table 2 displays demographic information.

Data Analysis

Resulting data were transferred to Statistical Package for Social Sciences (SPSS, 24th Edition) and cleaned by two trained research assistants. Eighteen test and incomplete

survey responses were eliminated, and data were reorganized for analysis. Data evaluation and analysis was completed by the first and second authors.

Frequency of Use

Similar descriptive statistics were collected for participants’ reported use of practices. Frequencies were calculated for

Table 2 Participant demographic characteristics

N = 128 (%)

Participant sex  Female 85.94  Male 12.50

Participant race  White/Caucasian 81.25  Asian 4.69  American Indian or Alaska Native 3.91  Multiracial 3.13  Black or African American 1.56

Participant ethnicity  Non-Latino/a or Hispanic 86.72  Latino/a or Hispanic 4.69

BCBA level  Board Certified Behavior Analyst (BCBA) 82.81  Board Certified Behavior Analyst-Doctoral (BCBA-D) 12.50  Board Certified Assistant Behavior Analyst (BCaBA) 2.34

Undergraduate Degree 75.00  Psychology 57.00  Education 12.50  Special education 8.00

Master’s Degree 84.00  Psychology 29.00  Special education 23.00  Education 14.00  Applied behavior analysis 10.00  School psychology 8.50

PhD 13.00  Psychology 5.00  Special education 5.00  Education 3.00

Setting  Urban (density of at least 1,000 per square mile) 71.09  Rural (all population, housing, and territory not

included within an urban area) 42.19

Practice setting  School 63.28  Home-based 58.59  Clinic or center-based 36.72  Other 1.56  Hospital 0.78

3052 Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

each rating per practice. Additionally, responses never used and rarely used as well as sometimes used and often used were grouped respectively, and frequencies were recalcu- lated to obtain a percentage of respondents who used the practice. Percentages were calculated by dividing the num- ber of respondents who reported using the practice by the number of total respondents and multiplying by 100.

Perceived Effectiveness Ratings

Regardless of whether the respondent reported using the practice, the respondent rated the effectiveness of each prac- tice using a Likert scale. Frequency counts for each effec- tiveness rating per practice were calculated, and percentages were calculated by dividing the number of respondents who rated the practice for each effectiveness rating by the number of total respondents and multiplying by 100.

Relations Between Use and Effectiveness

In order to evaluate whether there were significant relations between frequency of use and effectiveness ratings, we recoded both variables into bivariate variables. To do this “have not used” and “rarely used” were collapsed, “some- times used” and “often used” were collapsed, “highly inef- fective” and “ineffective” were collapsed, and “effective” and “highly effective” were collapsed. Chi-squared tests were then run on these values for each practice using the cross-tabs function in SPSS.

Principal Component Analysis

Basic Pearson’s r correlations were calculated between fre- quency of use and effectiveness ratings. In order to assess the core features of the questionnaire, a principal compo- nent analysis (PCA) was conducted on the results. Results of the PCA yielded a three-factor model, indicating similar responding patterns for items about reinforcement-based strategies, punishment-based strategies, and common com- ponents of packaged interventions.

Odds Ratio Calculations

Odds ratios, or the increase or decrease in likelihood of occurrence B given occurrence A, were calculated using data from contingency tables of demographic variables (i.e., education, training, practice site) and frequency of use rat- ings across the three types of practices: antecedent-based, reinforcement-based, and punishment-based.

Results

Descriptive Results

Frequency of Use

Forty-two respondents (33%) reported conducting func- tional analyses sometimes (approximately 25–50% of cases) or often (more than approximately 50% of cases). Of those 42 respondents, the most common behavio- ral function revealed was automatic (55%); followed by social positive (i.e., attention or tangible, 24%); and social negative (i.e., escape, 22%). Respondents reported using all 16 evidence-based practices. More than half of our sample reported sometimes or often using the following practices: skill enrichment (81.4%), visual and/or verbal cues (75.2%), FCT (75.2%), environmental enrichment (68.3%), RIRD (65.9%), DRI (62.8%), consequence-based embedded perseverative interests (57.4%), and response blocking (56.2%). More than half of our sample reported often using the following three practices: FCT (62.8%), skill enrichment (60.5%), and visual and/or verbal cues (52.7%). Less than half of our sample reported sometimes or often using the following eight practices: NCR (48.9%), antecedent-based PI (43.4%), DRO (43.4%), physical exercise (21.8%), response cost (13.2%), overcorrection (9.4%), sensory extinction (9.4%), and DRV (7.0%). Fig- ure 1 displays a visual representation of the most com- monly used (i.e., respondents reported using sometimes or often) practices.

Some portion of our sample reported never or rarely using the 16 evidence-based practices. More than half of our sample reported never or rarely using the following eight practices: DRV (92.2%), sensory extinction (90.7%), overcorrection (89.9%), response cost (86.0%), physical exercise (77.5%), antecedent-based perseverative interests (55.8%), DRO (55.8%), and NCR (50.4%). More than half of our sample reported never using the following four strat- egies: DRV (73.6%), sensory extinction (57.4%), response cost (51.9%), and physical exercise (50.4%). Less than half of our sample reported rarely or never using the following eight strategies: response blocking (48.9%), consequence- based perseverative interests (41%), DRI (36.4%), RIRD (33.3%), environmental enrichment (29.4%), FCT (24.0%), visual and/or verbal cues (24.0%), and skill enrichment (17.8%).

Perceived Effectiveness Ratings

More than half of our sample rated the following 15 prac- tices as effective or highly effective: FCT (89.9%), skill

3053Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

enrichment (89.2%), DRI (84.5%), consequence-based embedded perseverative interests (84.5%), DRV (79.0%), RIRD (79.0%), RIRD (78.3%), visual and/or verbal cues (77.5%), environmental enrichment (76.0%), antecedent- based embedded perseverative interests (72.0%), NCR (69.0%), DRO (69.0%), response cost (59.7%) response blocking (59.7%), sensory extinction (55.8%), and over- correction (51.2%). Less than half of our sample rated physical exercise (48.8%) as effective or highly effective. No practice was rated as highly effective by more than half of the sample, although FCT (49.6%) came close. More than a quarter of our sample rated RIRD (26.4%) as highly

effective. The remaining 14 practices were rated as highly effective by less than a quarter of our sample. Figure 2 displays a visual representation of the practices most com- monly rated as effective or highly effective.

More than half of our sample rated the following three practices as ineffective or highly ineffective: physical exer- cise (50.4%), overcorrection (48.1%), and sensory extinc- tion (43.5%). More than a quarter of our sample rated the following five practices as ineffective or highly ineffective: response blocking (39.6%), response cost (39.5%), NCR (30.2%), DRO (30.2%), and antecedent-based embedded perseverative interests (26.4%). Less than a quarter of our

Fig. 1 Percentage of respond- ents who reported sometimes or often using each evidence-based or emergent practice

Fig. 2 Percentage of respond- ents who rated each evidence- based or emergent practice as effective or highly effective

3054 Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

sample rated the following eight practices as ineffective or highly ineffective: visual and/or verbal cues (21.7%), RIRD (21.0%), consequence-based embedded perseverative inter- ests (20.1%), DRV (20.1%), environmental enrichment (19.4%), DRI (14.8%), skill enrichment (10.1%), and FCT (9.3%). The practices rated as highly ineffective most fre- quently were: response cost (15.5%), overcorrection (14.0%), physical exercise (12.4%), consequence-based embedded perseverative interests (11.6%), DRV (11.6%), and sensory extinction (10.9%).

Frequency of Use and Perceived Effectiveness Rating for SIT

In addition to the 16 evidence-based strategies, we also asked respondents about their frequency of use and per- ceived effectiveness of SIT, as it is commonly used in inter- vention for young children with ASD (Mandell et al. 2005). The majority of our sample reported never using SIT (64%). Fourteen percent of our sample reported rarely using SIT, 12% reported sometimes using SIT, and 9% reported often using SIT. We asked respondents to rate their perceived effectiveness of SIT regardless of whether or not they reported implementing it or not. Twelve percent of respond- ents reported SIT as highly effective, 21% as effective, 41% as ineffective, and 28% as highly ineffective.

Relations Between Use and Effectiveness

Results of the Chi-squared tests indicated significant rela- tions between respondents’ frequency of use and effective- ness ratings for all practices. That is, respondents were more likely to report using a practice if they reported rating it as effective and more likely to rate it as effective if they also reported using it.

Primary Component Analysis

In order to gather initial information about the content validity of the survey tool, we conducted a Primary Com- ponent Analysis (PCA) on the respondent data for strategy effectiveness. Results of the PCA indicated three initial sur- vey components: (a) reinforcement-based practices (e.g., consequence-based embedded perseverative interests); (b) punishment-based practices (e.g., RIRD); (c) and com- monly packaged components likely used in tandem with a group of practices (e.g., visual verbal cues plus NCR; skill enrichment plus FCT). The results of the PCA indicated that respondents answered similarly on these groups of practices. All except seldomly-used practices (i.e., exercise and SIT) loaded strongly on one factor, indicating the likely three fac- tor model.

Odds Ratios

We calculated odds ratios on practice use and demographic factors to better understand the predictors of frequency use of certain practices. SIT was not included in the odds ratio analyses because it is not an evidence-based practice, and functional analysis was not included as it is an assessment and does not fall into antecedent, reinforcement, or pun- ishment categories. An odds-ratio demonstrates the rate at which likelihood is increased, where an odds-ratio result of 1.0 indicates the group is two times as likely to use the strategy, an odds ratio of 2.0 indicates the group is 4 times more likely, and so forth. Results indicated that antecedent strategies were used more often than other groups of strate- gies in clinical settings, and in both urban and rural settings. Reinforcement-based strategies were more than three times as likely to be used than other categories of strategies by behavior analysts working in urban settings. Finally, pun- ishment-based procedures were more likely to be used more

Fig. 3 Odds ratios for antecedent-based, reinforcement, and punish- ment practices

3055Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

than other groups of practices in home and urban settings. Overall, all three groups of practices were more likely to be used in urban settings compared to rural settings (Fig. 3).

Discussion

We surveyed 128 behavior analysts implementing or super- vising interventions with young children with ASD on their use of treatments for RRBIs. Overall, our findings support the hypothesis that behavior analysts are implementing a variety of evidence-based and emergent strategies with the most frequently utilized practices being skill enrichment, visual and/or verbal cues, FCT, environmental enrichment, RIRD, DRI, consequence-based embedded perseverative interests, and response blocking. The least utilized practices were DRV, sensory extinction, overcorrection, response cost, physical exercise, antecedent-based perseverative interests, DRO, and NCR. Fifteen of the 16 evidence-based and emer- gent practices were reported to be perceived as effective or highly effective by more than half of our sample. Physical exercise was the only evidence-based strategy reported to be perceived as effective or highly effective by less than half (48.8%), albeit barely, of our sample. Thus, the majority of the behavior analysts we surveyed perceived the evidence- based practices to be effective in producing sustainable behavior change for RRBIs in young children with ASD. Of note, physical exercise, overcorrection, and sensory extinc- tion were rated as ineffective or highly ineffective by more than half of our sample. Further, the following six strategies were rated as highly ineffective: response cost, overcorrec- tion, physical exercise, consequence-based embedded per- severative interests, DRV, and sensory extinction.

Respondents were significantly more likely to report using a practice if they reported rating it as effective and vice versa. A PCA indicated that practitioners generally responded in the same way to items about reinforcement- based practices, punishment-based practices, and a group of commonly packaged antecedent and consequence-based package components. Infrequently used practices included both the non-evidence-based practice (i.e., SIT) and prac- tices that may be difficult to implement with young children, such as DRO.

Implications and Future Research

First, it is not surprising that behavior analysts reported using a variety of behavioral interventions to treat the broad category of RRBIs, which encompasses a range of topo- graphical behavioral categories including motor and object stereotypy, vocal stereotypy, insistence on sameness, preoc- cupation and perseveration, and sensory differences. It is

interesting to note that a range of effectiveness ratings were given, with some respondents reporting these practices as effective in producing sustainable behavior change, and oth- ers indicating the contrary. Given that these are perceptions and that these practices have mainly been evaluated via sin- gle-case studies that are inherently short in duration, there is a need to evaluate distal outcomes of these interventions. It could be that some of the practices work well to suppress an RRBI in the short term, but do not necessarily produce dura- ble and desirable behavior change. Future research should measure the long-term outcomes of the available evidence- based behavioral practices on both decreasing undesirable RRBIs and increasing behavioral flexibility, more generally. Moreover, implementing antecedent-based strategies (e.g., environmental enrichment, functional communication train- ing) is recommended in early childhood (Division for Early Childhood 2014), but such approaches might not systemati- cally target behavioral flexibility, the ASD symptom of inter- est. As a field, we may not be applying the same rationale as is the case for intervention for social-communication skill deficits in young children with ASD, which aim to activate social motivation and systematically shape social behaviors by carefully programming caregivers to deliver reinforce- ment such as frequent, contingent attention for eye contact, communicative gestures, and other early social behaviors (Hansen et al. 2018).

The majority of our sample (73.6%) reported never using differential reinforcement of variability (DRV). DRV involves shaping and increasing children’s novel engagement with repetitive behavior stimuli and is designed to target perseverative interests (Boyd et al. 2011). Seventy-nine per- cent of respondents rated DRV as effective or highly effec- tive. It is unclear why this practice is not implemented more often, especially given that the majority of our sample rated it favorably. One reason could be that it is not one of the more common forms of differential reinforcement, such as DRI, and thus, may not be explicitly covered in behavior analysts’ course work (Cooper et al. 2007). Future studies should aim to understand more about the lack use of this promising practice. Additionally, because invariant behav- ioral patterns are a core symptom of ASD, more studies test- ing the efficacy of DRV and other interventions targeting behavioral flexibility, rather than suppression of RRBIs for children with ASD, are warranted. This approach may yield additional effective and contextually valid interventions for young children with or at risk for ASD.

Another interesting finding was that more than half of our sample (50.4%) rated physical exercise as ineffective or highly ineffective. Physical activity and exercised-based interventions have been associated with improvements in cognitive, social, and behavioral characteristics of ASD (Tan et al. 2016). Reductions in stereotypy for brief periods of time following exercise is one of the most commonly

3056 Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

reported beneficial outcomes reported (Lang et al. 2010; Neely et al. 2015). These reductions appear to be tempo- rary, which may be a reason that the behavior analysts in our sample tended to rate them as ineffective; however, lit- tle is known about the potential long-term distal benefits of continued physical exercise on RRBIs. Thirty-three per- cent of our sample reported conducting functional analyses (including rarely, sometimes, and often) when the RRBI was deemed challenging. Of those who did conduct functional analyses, the most common function revealed was automatic (55%), meaning that the reinforcer maintaining the behav- ior was non-social (i.e., sensory seeking or avoiding). The underlying behavioral mechanism of physical exercise is arguably also automatic reinforcement. Engaging children in physical exercise in the early years may be a powerful replacement behavior for some RRBIs, especially motor ste- reotypies and hyperreactivity to sensory input. Exercise is likely to have collateral benefits such as preventing obesity, which children with ASD are at an increased risk of develop- ing (Curtin et al. 2014).

It is also interesting to note that 11.6% of respondents rated consequence-based embedded perseverative interests as highly ineffective. Embedding perseverative interests into interventions (e.g., incorporating a child’s persevera- tive interest into a token economy system) has been found to increase on-task behavior and decrease problem behavior (Carnett et al. 2014; Ninci et al. 2018). However, as sug- gested by Ninci et al. practitioners should closely monitor outcomes when incorporating perseverative interests into interventions to avoid inadvertently reinforcing persevera- tion. Studies examining distal outcomes of embedding per- severative interests into interventions for children with ASD would be useful.

The only two practices rated as highly effective by more than a quarter of our sample were RIRD (26.4%) and FCT (49.6%). It is likely that behavior analysts are combining these practices, along with others focused on social-com- munication skills, to individualize behavioral programs for children with ASD. Comprehensive treatment models and packaged curricula that explicitly guide practitioners to assess and choose curricular targets to address behavio- ral flexibility for individual children, but also for small and whole groups would be beneficial. These packages exist for social-communication skills and pre-academics (e.g., Early Start Denver Model, Pivotal Response Training). Given that these packages are specially designed for children with ASD, incorporating lesson plans for behavioral flexibility would potentially enhance these models. Additionally, assessments for young children with ASD would benefit from being more comprehensive (Gould et al. 2011), which includes providing practitioners with (a) validated assessment points that flag when an RRBI is developmentally aberrant, and (b) linking assessment results of RRBI behavioral excesses

to a developmentally appropriate curricular target. Such efforts could include teaching children skills to self-soothe and manage intense emotions. Adapting age-appropriate mindfulness-based curricula, such as The Kindness Cur- riculum (Flook et al. 2015) for children with ASD, who may have complex communication needs, is a logical next step. Further, mindfulness training could be applied within the context of caregiver-child interactions—caregivers broadly encompassing parents, teachers, paraprofessionals, or behavior therapists—by focusing on increasing attention to the breath and sensory experiences (e.g., tastes, sounds, textures). This type of training may serve as a means to tar- get requisite skills for psychological flexibility needed later on when children are working through difficult emotions. Further, teaching overt (i.e., observable) mindful behaviors within the context of children’s natural routines (e.g., parents and children practicing labeling sensations together) may be beneficial in programming behavioral flexibility.

Results from several comprehensive reviews have con- cluded that SIT does not have sufficient evidence to sup- port its use with children with ASD (Barton et al. 2014; Lang et al. 2012; National Autism Center 2015). Yet, 21% of our sample indicated that they implemented or supervised SIT sometimes or often, and 33% of respondents rated SIT as effective or highly effective. This was a surprising find- ing that warrants further inquiry. It may be the case that practitioners do not have ample behavioral treatments spe- cifically designed for sensory sensitivities. Results from the odds ratio calculations indicated that practitioners operating in urban clinics were more likely to use all practices, sug- gesting that resources available to practitioners may influ- ence selected practices (i.e., higher resourced settings such as clinics in urban areas may be better equipped to imple- ment best practice). These findings may highlight a need for improvement in dissemination and implementation of practices in lower-resourced areas.

Finally, there is little known about parent and stake- holder preference and the social validity of the available evidence-based practices for RRBIs. Future research should aim to understand more about how parents and other caregivers view these practices to be acceptable, fea- sible, and developmentally appropriate. Finally, few stud- ies have investigated parent-implemented interventions for RRBIs (cf. Boyd et al. 2012; Grahame et al. 2016; Lin and Koegel 2018). The available evidence-based practices arguably lack developmental appropriateness for young children (Raulston and Machalicek 2018). One reason for this may be that most studies have included participants from various ages, and as repetitive behavior is not unique to ASD (i.e., other individuals with intellectual and devel- opmental disabilities also commonly display repetitive behavior), the focus has often been on developing effica- cious interventions to reduce RRBIs within the context of

3057Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

challenging behavior. However, it is also important that behavior analysts consider developmental appropriateness and understand that some stereotypies and other RRBIs are age appropriate to avoid suppressing behaviors that may actually be adaptive for young children.

Limitations

There are several limitations worth noting. First, the odds ratio calculations were limited by multiple responses, which may have inflated relative differences (i.e., high resource settings with highly trained professionals likely selected a range of evidence-based practices, making it difficult to determine the moderating effect of setting on results). Additionally, high rates of correlation between strategy use may indicate these respondents are using packages of interventions. Future research should explore how practitioners are combining practices for maximum behavior change, and what child, family, or system vari- ables are considered when deciding which practices to use. An understanding of practitioner use of combined practices may support the development of curriculum or efficient treatment packages that would allow for interven- tion on RRBIs to occur across settings.

Our sample size was small, meaning there were likely many shared characteristics between respondents. Further, we purposefully surveyed a homogenous group of behavior analysts on their use of primarily emerging and evidence- based behavioral interventions. Certainly, our findings are not representative of practices being implemented by other disciplines, including early intervention and early child- hood (EI/ECSE) practitioners, speech-language patholo- gists, occupational therapists, psychologists, psychiatrists, or developmental-behavioral pediatricians for the treatment of RRBIs in young children with ASD. Various fields con- ceptualize the etiology and functions of these behaviors dif- ferently, and as such, treat them differently. For example, a speech-language pathologist may treat vocal stereotypy, or an occupational therapist might treat hyperreactivity to sensory stimuli differently than a behavior analyst. Future research should survey these aforementioned specialists to shed light on commonalities and differences in the practices with an eye toward understanding more about best interdis- ciplinary practice.

To more fully address thebroad range of behaviors encompassed within the RRBI section of the DSM-V, we did not ask respondents to rate each practice based on operant function. It is imperative to consider function when treating challenging behaviors, which could explain why FCT was the only practice where almost half of our sample rated as highly effective.

Conclusion

In sum, we surveyed 128 behavior analysts their use of 16 practices and one assessment for the treatment of RRBIs in young children with ASD. The majority of our sample per- ceived the practices to be effective in producing sustainable behavior change for RRBIs in young children with ASD, yet some practices were reported to be implemented less often than expected. Future research should emphasize treating behavioral flexibility from a developmentally appropriate lens.

Author Contributions TJR, WM, and LLM conceptualized and designed the study. TJR and SGH created and validated the survey, ran the analyses, and wrote the manuscript. AC assisted in data analysis and manuscript preparation.

Funding The first author received funding through the Engaging New Leaders in Implementation Science Training Leadership Grant of the United States Department of Education Office of Special Education Programs during survey item creation and data collection. The third author received faculty research funds from the College of Education at the University of Oregon for subject honorarium. The second author presented these data at a symposium session at the 44th annual Asso- ciation for Behavior Analysis in San Diego, California in May 2018.

Compliance with Ethical Standards

Conflicts of interest There are no potential conflicts of interest to re- port.

Research involving human participants All procedures performed in studies involving human participants were in accordance with the ethi- cal standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent Informed consent was obtained from all individual participants included in the study.

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington: American Psy- chiatric Publishing.

Barber, A.B., Wetherby, A.M., & Chambers, N.W. (2012). Brief report: Repetitive behaviors in young children with autism spectrum dis- order and developmentally similar peers: A follow up to Watt et al. (2008). Journal of Autism and Developmental Disorders, 42, 2006–2012.

Barton, E. E., Reichow, B., Schnitz, A., Smith, I. C., & Sherlock, D. (2014). A systematic review of sensory-based treatments for chil- dren with disabilities. Research in Developmental Disabilities,37, 64–80.

Boyd, B. A., McDonough, S. G., & Bodfish, J. A. (2012). Evidence- based behavioral interventions for repetitive behaviors in autism. Journal of Autism and Developmental Disabilities,42, 1236–1248.

3058 Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

Boyd, B. A., McDonough, S. M., Rupp, B., Khan, F., & Bodfish, J. W. (2011). Effects of a family-implemented treatment on the repetitive behavior of children with autism. Journal of Autism and Developmental Disorders,41, 1330–1341.

Carnett, A., Raulston, T., Lang, R., Tostanoski, A., Lee, A., Sigafoos, J., & Machalicek, W. (2014). Effects of perseverative interest- based token economy on challenging and on-task behavior in a child with autism. Journal of Behavioral Education,23, 368–377.

Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis. New York: Macmillan.

Cunningham, A. B., & Schreibman, L. (2008). Stereotypy in autism: The importance of function. Research in Autism Spectrum Dis- orders,2, 469–479.

Curtin, C., Jojic, M., & Bandini, L. G. (2014). Obesity in children with autism spectrum disorders. Harvard Review of Psychiatry,22, 93–103.

Division for Early Childhood. (2014). DEC recommended practices in early intervention/early childhood special education 2014. Retrieved from https ://www.dec-sped.org/dec-recom mende d-pract ices.

Dominick, K. C., Ornstein Davis, N., Lainhart, J., Tager-Flusberg, H., & Folstein, S. (2007). Atypical behaviors in children with autism and children with a history of language impairment. Research in Developmental Disabilities,28, 145–162.

Evans, D. W., Leckman, J. F., Carter, A., Reznick, J. S., Henshaw, D., King, R. A., et al. (1997). Ritual, habit, and perfectionism: The prevalence and development of compulsive-like behavior in nor- mal young children. Child Development,68, 58–68.

Flook, L., Goldberg, S. B., Pinger, L., & Davidson, R. J. (2015). Pro- moting prosocial behavior and self-regulatory skills in preschool children through a mindfulness-based kindness curriculum. Developmental Psychology,51, 44–51.

Goldman, S., Wang, C., Salgado, M. W., Greene, P. E., Kim, M., & Rapin, I. (2009). Motor stereotypies in children with autism and other developmental disabilities. Developmental Medicine & Child Neurology,51, 30–38.

Gould, E., Dixon, D. R., Najdowski, A. C., Smith, M. N., & Tarbox, J. (2011). A review of assessments for determining the content of early intensive behavioral intervention programs for autism spectrum disorders. Research in Autism Spectrum Disorders,5, 990–1002.

Grahame, V., Brett, D., Dixon, L., McConachie, H., Lowry, J., Rodg- ers, J., Steen, N., &Couteur, A.L. (2016). Managing repetitive behavior in young children with autism spectrum disorder (ASD): Pilot randomized controlled trial of new parent group intervention. Journal of Autism and Developmental Disorders,45, 3168–3182.

Hansen, S. G., Carnett, A., Tullis, C. A. (2018). Defining early social communication skills: A systematic review and analysis. Advances in Neurodevelopmental Disorders, 2, 116–128.

Harrop, C., McConachie, H., Emsley, R., Leadbitter, K., & Green, J. (2014). Restricted and repetitive behaviors in autism spectrum dis- orders and typical development: Cross-sectional and longitudinal comparisons. Journal of Autism and Developmental Disorders,44, 1207–1219.

Honey, E., Leekam, S., Turner, M., & McConachie, H. (2007). Repeti- tive behavior and play in typically developing children and chil- dren with autism spectrum disorders. Journal of Autism and Developmental Disorders,37, 1107–1115.

Koegel, R. L., & Covert, A. (1972). The relationship of self-stimula- tion to learning in autistic children. Journal of Applied Behavior Analysis,5, 381–387.

Lang, R., Koegel, L. K., Ashbaugh, K., Regester, A., Ence, W., & Smith, W. (2010). Physical exercise and individuals with autism spectrum disorders: A systematic review. Research in Autism Spectrum Disorders,4, 565–576.

Lang, R., O-Reilly, M., Rispoli, M., Lydon, H., Streusand, W., Davis, T., Kang, S., Sigafoos, J.,Lancioni, G., Didden, R., & Giesbers, S. (2012). Sensory integration therapy for autism spectrum dis- orders: A systematic review. Research in Autism Spectrum Dis- orders,6, 1004–1018.

Lanovaz, M. J., & Sladeczek, I. E. (2012). Vocal stereotypy in indi- viduals with autism spectrum disorders: A review of behavioral interventions. Behavior Modification,36, 146–164.

Leekam, S. R., Nieto, C., Libby, S. J., Wing, L., & Gould, J. (2007). Describing the sensory abnormalities of children and adults with autism. Journal of Autism and Developmental Disorders,37, 894–910.

Lin, C. E., & Koegel, R. (2018). Treatment of higher-order restricted repetitive behaviors (H-RRB) in children with autism spectrum disorder. Journal of Autism and Developmental Disorders.,48, 894–910. https ://doi.org/10.1007/s1080 3-018-3637-3.

Loftin, R. L., Odom, S. L., & Lantz, J. F. (2008). Social interaction and repetitive motor behaviors. Journal of Autism and Developmental Disorders,38, 1124–1135.

Machalicek, W., Raulston, T., Knowles, C., Ruppert, T., Carnett, A., & Alresheed, F. (2016). Challenging behavior. In J. Matson (Ed.), Comorbid conditions among children with autism spectrum dis- orders. New York: Springer.

Mandell, D. S., Novak, M. M., & Levy, S. (2005). Frequency and cor- relates of treatment use among a community sample of children with autism. CA: San Diego.

Nadig, A., Lee, I., Singh, L., Bosshart, K., & Ozonoff, S. (2010). How does the topic of conversation affect verbal exchange and eye gaze? A comparison between typical development and high- functioning autism. Neuropsychologia,48, 2730–2739.

National Autism Center (2015). Findings and conclusions: National standards project, phase 2. Retrieved from https ://www.natio nalau tismc enter .org/resou rces

National Conference of State Legislatures. (2018, August 8). Autism and insurance coverage: State laws. Retrieved from: https ://www. ncsl.org/resea rch/healt h/autis m-and-insur ance-cover age-state -laws.aspx

National Implementation Research Network. (2015). Implementation science defined. Retrieved from https ://nirn.fpg.unc.edu/learn -imple menta tion/imple menta tion-scien ce-defin ed

Neely, L., Rispoli, M., Gerow, S., & Ninci, J. (2015). Effects of ante- cedent exercise on academic engagement and stereotypy during instruction. Behavior Modification,39, 98–116.

Ninci, J., Rispoli, M., Burke, M., & Neely, L. C. (2018). Embedding interests of individuals with autism spectrum disorder: A quality review. Review Journal of Autism and Developmental Disorders,5, 15–28.

Patterson, S. Y., Smith, V., & Jelen, M. (2010). Behavioural interven- tion practices for stereotypic and repetitive behavior in individuals with autism spectrum disorder: A systematic review. Developmen- tal Medicine & Child Neurology,52, 318–327.

Pierce, K., & Courchesne, E. (2001). Evidence for a cerebellar role in reduced exploration and stereotyped behavior in autism. Biologi- cal Psychiatry,49, 655–664.

Rapp, J. T., & Vollmer, T. R. (2005). Stereotypy I: A review of behav- ioral assessment and treatment. Research in Developmental Dis- abilities,26, 527–547.

Raulston, T., & Machalicek, W. (2018). Early intervention for repeti- tive behaviorin autism spectrum disorder: A conceptual model. Journal of Developmental and Physical Disabilities, 30, 89–109.

Richler, J., Huerta, M., Bishop, S. L., & Lord, C. (2010). Developmen- tal trajectories of restricted and repetitive behaviors and interests in children with autism spectrum disorders. Developmental Psy- chopathology,22, 55–69.

Rispoli, M., Camargo, S., Machalicek, W., Lang, R., & Sigafoos, J. (2014). Functional communication training in the treatment of

3059Journal of Autism and Developmental Disorders (2019) 49:3047–3059

1 3

problem behavior by access to rituals. Journal of Applied Behav- ior Analysis,47, 580–593.

South, M., Ozonoff, S., & McMahon, W. M. (2005). Repetitive behav- ior profiles in Asperger syndrome and high-functioning autism. Journal of Autism and Developmental Disorders,14, 42–54.

Tan, B. W., Pooley, J. A., & Speelman, C. P. (2016). A meta-analytic review of the efficacy of physical exercise interventions on cog- nition in individuals with autism spectrum disorder and ADHD. Journal of Autism and Developmental Disorders,46, 3126–3143.

Varni, J. W., Lovaas, O. I., Koegel, R., & Everett, N. (1979). An analy- sis of observational learning in autistic and normal children. Jour- nal of Abnormal Child Psychology,7, 31–43.

Wolff, J. J., Botteron, K. N., Dager, S. R., Elison, J. T., Estes, A. M., Gu, H., … Piven, J. (2014). Longitudinal patterns of repetitive behavior in toddlers with autism. The Journal of Child Psychology and Psychiatry,55, 945–953.

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

  • Interventions for Repetitive Behavior in Young Children with Autism: A Survey of Behavioral Practices
    • Abstract
    • Topographies of Repetitive Behavior in ASD
    • Impact of RRBIs
    • Interventions for RRBIs in ASD
    • The Current Study
    • Method
      • Item Creation
        • Expert Review of Items
        • Frequency of Use and Perceived Effectiveness
      • Procedure
        • Distribution
        • Survey Design
      • Participants
      • Data Analysis
        • Frequency of Use
        • Perceived Effectiveness Ratings
        • Relations Between Use and Effectiveness
        • Principal Component Analysis
        • Odds Ratio Calculations
    • Results
      • Descriptive Results
        • Frequency of Use
        • Perceived Effectiveness Ratings
        • Frequency of Use and Perceived Effectiveness Rating for SIT
        • Relations Between Use and Effectiveness
      • Primary Component Analysis
      • Odds Ratios
    • Discussion
    • Implications and Future Research
    • Limitations
    • Conclusion
    • References