Evidence base practice
Journal of Positive Behavior Interventions 15(1) 5 –15 © 2013 Hammill Institute on Disabilities Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/1098300712440453 http://jpbi.sagepub.com
Teachers typically enter the field with inadequate training in behavioral instruction and classroom management (Begeny & Martens, 2006). Therefore, school leaders and special- ized support staff (e.g., administrators, school psycholo- gists, special educators) need to identify effective and efficient ways to support teachers’ use of evidence-based classroom management practices. Multiple strategies have been explored for teacher training, including didactic train- ing, prompting, modeling, role playing, feedback, and rein- forcement (Allen & Forman, 1984). Across studies, the consensus is that training alone does not result in changes in teacher behavior (Allen & Forman, 1984; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005).
Instead, research suggests that performance feedback, in combination with training, results in desired increases in teachers’ use of classroom management practices (e.g., Abbott et al., 1998; Jeffrey, McCurdy, Ewing, & Polis, 2009; Noell, Witt, Gilbertson, Rainer, & Freeman, 1997; Simonsen, Myers, & DeLuca, 2010). Performance feedback involves collecting data on an individual’s behavior and providing feedback about that behavior (Noell et al., 2005). Although effective, performance feedback is time intensive, and typical school resources often limit its feasibility. Rather than relying on another individual to observe, collect data, and provide feedback, it may be possible to train teachers to monitor, record, and provide feedback on their
own behavior. Thus, self-management may be a potential solution to the training to practice gap.
According to Skinner (1953), individuals manage their own behavior in the same manner as they manage anyone else’s—“through the manipulation of variables of which behavior is a function” (p. 228). That is, individuals manip- ulate the antecedents and consequences of their own behav- ior, and they engage in other (self-management) behaviors to make target behaviors more or less likely. Over the past 10 years, researchers have studied self-management in vari- ous populations of adults, including adults who are obese (Donaldson & Normand, 2009), have asthma (Caplin & Creer, 2001; Creer, Caplin, & Holroyd, 2005; Ngamvitroj & Kang, 2007), have depression (Rokke, Tomhave, & Jocic, 2000), and are experiencing insomnia (Creti, Libman, Bailes, & Fichten, 2005). Generally, studies have found that self-management interventions are related to desired behav- ior changes in adults.
XXX10.1177/1098300712440453Simonsen et al.Journal of Positive Behavior Interventions © 2011 Hammill Institute on Disabilities
Reprints and permission: http://www. sagepub.com/journalsPermissions.nav
1University of Connecticut, Storrs, CT, USA
Corresponding Author: Brandi Simonsen, University of Connecticut, Educational Psychology, 249 Glenbrook Rd., Unit 2064, Storrs, CT 06269-2064, USA Email: brandi.simonsen@uconn.edu
Action Editor: V. Mark Durand
The Effects of Self-Monitoring on Teachers’ Use of Specific Praise
Brandi Simonsen, PhD1, Ashley S. MacSuga, MA1, Lindsay M. Fallon, MA1, and George Sugai, PhD1
Abstract
Teachers typically enter the field with limited training in classroom management, and research demonstrates that training alone does not result in improved practice. Typically, researchers have relied on time-intensive training packages that include performance feedback to improve teachers’ use of classroom management practices; however, initial evidence suggests that self-management may be an effective and efficient alternative. In this study, the authors directly compared the effects of three different self-monitoring conditions (tally, count, and rate) and no self-monitoring on five middle school teachers’ rate of specific praise using an alternating treatments design. The authors also included baseline and follow-up phases to descriptively explore the effects of self-monitoring across time. Results indicate that noting each instance of specific praise by either tallying or using a counter resulted in optimal performance, and teachers preferred using a counter. Additional study results, limitations, and implications are discussed.
Keywords
classroom management, specific praise, teacher training, teacher self-management, teacher self-monitoring
Articles
6 Journal of Positive Behavior Interventions 15(1)
Researchers have also begun to explore the use of self- management with teachers. For example, Browder, Liberty, Heller, and D’Huyvetters (1986) found that teachers made bet- ter instructional decisions (i.e., choices about maintaining or changing instructional practices based on students’ academic performance) when they were trained to self-monitor. Self- monitoring is noting the presence, absence, or level of a spe- cific behavior and is one example of self-management (Cooper, Heron, & Heward, 2007). Similarly, Allinder, Bolling, Oats, and Gagnon (2000) found that teachers who self-monitored made better instructional decisions that resulted in better stu- dent performance than teachers who did not self-monitor.
Researchers have also started to examine the effects of self-management on teachers’ use of praise (e.g., Keller, Brady, & Taylor, 2005; Sutherland & Wehby, 2001; Workman, Watson, & Helton, 1982). Praise is an empiri- cally supported classroom management practice (Simonsen, Fairbanks, Briesch, Myers, & Sugai, 2008) that can be effective if contingent, credible, and specific (Brophy, 1981). That is, saying “Thank you for raising your hand” immediately after a student raised her or his hand would be more effective in increasing the likelihood of hand raising than providing general feedback, such as “good job,” min- utes after the desired behavior. Research has shown, for example, that increases in teachers’ specific praise are asso- ciated with increases in students’ on-task behavior (Chalk & Bizo, 2004; Sutherland, Wehby, & Copeland, 2000).
Given the importance of specific praise, Sutherland and Wehby (2001) investigated the effects of teachers’ self- evaluation on their use of specific praise with students with emotional and behavioral disorders. They trained a group of teachers to use praise, audio record a segment of instruc- tion, review the tape later in the day, and self-evaluate (i.e., calculate praise rates, set goals, deliver self-praise, and graph their progress). When they compared the praise rates of these teachers with teachers in a control group, they found that (a) teachers in the self-evaluation group demon- strated higher levels of praise and lower levels of repri- mands and (b) their students gave higher levels of correct responses. Keller et al. (2005) conducted a similar study (using self-evaluation of audiotapes) with student teaching interns. They also found increases in preservice teachers’ rates of specific praise as a result of self-evaluation. Together, these studies demonstrated the positive effects of self-evaluation on teachers’ use of praise; however, the self- evaluation methods required time outside of instruction (i.e., daily review of taped instruction), and researchers did not explore alternative methods for self-evaluation.
In the present study, we directly compared the effective- ness of three simple and efficient self-monitoring condi- tions (tally, count, and rate) and no self-monitoring on teachers’ use of specific praise. Specifically, this study addressed the following research question: Which self- monitoring strategy is associated with the highest rate of
specific praise during teacher-directed instruction for indi- vidual middle school teachers? In addition, we explored which strategy resulted in the highest fidelity of implemen- tation (including measures of both adherence and accuracy) and was preferred by each teacher. The findings of this study add to the limited literature on teachers’ self-management of evidence-based classroom management strategies.
Method Setting and Participants
This study took place in an urban middle school in New England. The year before this study, approximately half (range 44%–66%) of the student body (N = 926 students, Grades 5–8) scored below proficient in reading, writing, math, or science as measured by statewide tests (http://www. greatschools.net); and more than half (75%) of the student body was eligible for free or reduced-price lunch (http://nces. ed.gov). Student ethnicity was described as 62.7% Hispanic, 29.9% White, 5.6% Black, 1.5% Asian, and 0.4% American Indian (http://nces.ed.gov).
The staff at this school (including 87.8 full-time equiva- lent teachers) had been exposed to positive behavior sup- port strategies across a variety of staff in-service and professional development events. Despite this training, some teachers continued to struggle with implementing evidence-based classroom management strategies. The pri- mary researchers (the first two authors) approached teach- ers during team meetings and presented this study as an opportunity to receive feedback and support with classroom management. Interested teachers provided researchers with a preferred mode of contact (e.g., email), and researchers scheduled individual meetings to explain the scope of the study and obtain written informed consent. Ultimately, five female teachers volunteered to participate in this study.
Teacher 1 earned a BA, was certified in math (Grades 7–12), and taught eighth grade math. Teacher 2 earned her BA, was certified in elementary education (K–8), and taught fifth grade language arts. Teacher 3 earned a MA, was certified in special education, and taught reading, lan- guage arts, and math to fifth through eighth grade students with disabilities who were also English language learners. Teacher 4 completed her master’s degree plus 15 credits, was certified in elementary education (K–8) and science (Grades 4–8), and taught seventh grade science. Teacher 5 earned a MA, was certified in elementary education (K–8), and taught fifth grade math. The teachers ranged in experi- ence: At the time of the study, Teachers 1, 2, 3, 4, and 5 had accrued 3, 2, 13, 28, and 4 years of teaching experience, respectively.
Each participating teacher selected the class period in which she experienced the greatest challenges with class- room management, and she identified a 15-min segment of
Simonsen et al. 7
that period when she provided teacher-directed instruction to serve as the focus of intervention and data collection.
Dependent Measures Teachers’ use of specific praise was the primary dependent variable in this study. In addition, we explored teachers’ fidelity of implementation and the social validity for each self-monitoring strategy, as described in the next subsections.
Systematic direct observation (SDO). SDO data were col- lected during 15-min observations of teacher-directed instruction during the selected class period. Trained data collectors recorded the frequency with which teachers delivered specific praise by making a tally mark any time the teacher provided audible, specific, and positive verbal feedback to one or more students contingent on behavior (e.g., “Thank you for raising your hand”). The frequency of specific praise was converted to rate by dividing by number of minutes observed.
Four data collectors (two PhD and two MA students) were trained to collect SDO data across a series of training activities. First, data collectors met with the lead researcher (first author) to review operational definitions and proce- dures for data collection. Then, the graduate student project coordinator and lead data collector (second author) pro- vided additional practice using video segments and in vivo observations in each teacher’s classroom. Training activi- ties continued until all data collectors met or exceeded 85% interobserver agreement (IOA). In addition, retraining meetings (to review operational definitions and discuss areas of disagreement) were held in the event of decreases in IOA (below criterion levels).
IOA was assessed during 40% of (100 of 249) sessions. Agreement for teachers’ use of specific praise was calcu- lated by dividing the smaller number of praise statements recorded (agreements) by the larger number of praise state- ments recorded by observers during the same 15-min obser- vation (opportunities for agreement) and multiplying by 100%. IOA was acceptable across the study (M = 85.2%, SD = 8.8%) and for each teacher (Teacher 1: M = 82.7%, SD = 11.5%; Teacher 2: M = 85.5%, SD = 8.8%; Teacher 3: M = 85.74%, SD = 7.7%; Teacher 4: M = 86.5%, SD = 5.5%; Teacher 5: M = 86.2%, SD = 9.0%).
Indicators of implementation fidelity. To determine whether teachers were self-monitoring with fidelity, we collected data on both the adherence to and accuracy of teachers’ self- monitoring across conditions. To assess adherence, trained observers noted whether the teacher was implementing the assigned self-monitoring strategy (fully, partially, or not at all) or the incorrect strategy by checking the appropriate box on the data sheet for each observation. Specifically, observers recorded that the teacher was implementing (a) fully if she consistently used the assigned self-monitoring strategy throughout the observation, (b) partially if she
implemented the assigned strategy for part of the time or implemented some (but not all) of the features of the strat- egy, (c) not at all if she did not use any self-monitoring strategy for any period of time, or (d) the incorrect strategy if she implemented a different strategy than assigned for that observation (e.g., tallied when she should have rated).
To assess accuracy, observers recorded the self-monitoring data each teacher collected during the observation. That is, observers noted the total number of praise statements recorded by the teacher using the assigned self-monitoring strategy (i.e., total tallies, total count, or rating) at the end of each observation. We calculated agreement between the teacher and observer by dividing the smaller number (agree- ments) by the larger number (opportunities for agreement) of praise statements recorded and multiplying by 100%.
Social validity measures. The first author adapted ques- tions on the Intervention Rating Profile-15 (IRP-15; Mar- tens, Witt, Elliott, & Darveaux, 1985) to collect descriptive data on the acceptability of each self-monitoring strategy from the teachers’ perspectives. The adapted IRP-15 con- sisted of five questions that prompt responses on a 1 (strongly disagree) to 6 (strongly agree) scale, and a sixth open-ended question prompts teachers to share any com- ments or concerns. Scores on the IRP-15 have been found to be reliable indicators of intervention acceptability (Martens et al., 1985), and researchers have used it to assess the acceptability of academic and behavioral interventions by teachers (e.g., Reynolds & Kelley, 1997).
Design and Procedures We used a modified alternating treatments design (e.g., Barlow & Hayes, 1979; Gast, 2010), with baseline, alter- nating treatments, optimal treatment, and follow-up phases, to explore the relative effectiveness of different self-monitoring strategies on five teachers’ use of specific praise during teacher-directed instruction. All five teachers progressed through baseline, alternating treatments, and indicated treatment phases. Teachers whose data were either stable or demonstrated clear increasing or decreasing trends pro- gressed to one of two possible follow-up phases: (a) main- tenance (for teachers who demonstrated high stable levels or increasing trends during the indicated treatment phase) or (b) performance feedback (for teachers who demon- strated low levels or decreasing trends during the indicated treatment phase). Each phase is described in the following sections.
Baseline phase. During baseline, we observed and recorded each teacher’s rate of specific praise before any training was delivered. After a stable pattern was docu- mented (i.e., three or more consecutive data points with minimal variability or a clear trend), each teacher received a brief scripted training on how to provide specific praise. The training comprised discussion (definition, rationale,
8 Journal of Positive Behavior Interventions 15(1)
examples, and critical features of specific praise), applica- tion activity (scripting contextually appropriate specific praise statements), introduction to self-monitoring (defini- tion of self-management, explanation of three self-monitoring strategies, and instructions on how to use each), and a brief summary of study purpose. Fidelity of training was mea- sured by an observer completing a rating of the extent to which each component was delivered with no, partial, or full fidelity during the training of each teacher. All training com- ponents were delivered with 100% fidelity to all teachers.
Alternating treatments phase. Following the brief training, the alternating treatments phase of the study began. During this phase, the teacher’s behavior was observed during the following four self-monitoring “treatments” or conditions.
1. Tally of specific praise statements. Teachers were instructed to record a tally each time they provided specific praise to one or more students during teacher-directed instruction (i.e., during observation). Most teachers used a Post-It note or clipboard so that they could carry the tally sheet around the classroom with them.
2. Count of specific praise statements (using coun- ter). Teachers were instructed to press a button to advance a small yellow golf counter each time they provided specific praise to one or more stu- dents during teacher-directed instruction.
3. Rating of specific praise statements (using brief rating scale). Teachers were instructed to rate their use of praise during teacher-directed instruc- tion by estimating the number of specific praise statements they provided per minute on a 0–4 times per minute scale.
4. Day off. To directly evaluate the effects of the three conditions relative to the absence of self- monitoring, teachers were also given “days off” (i.e., no self-monitoring).
The tally and count conditions were designed to compare the effects, fidelity, and social validity of two different methods of frequency recording; the rate condition was designed as a comparison condition that may require less effort than fre- quency counting.
Teachers were also instructed to record their data daily. For the tally and count conditions, teachers (a) recorded the total number of specific praise statements, (b) recorded the number of minutes of data collection (typically 15), (c) cal- culated their rate of specific praise (number of specific praise statements divided by number of minutes on sum- mary sheet), and (d) graphed their specific praise rate on the summary sheet we provided. For the rate condition, teach- ers recorded and graphed their rating of specific praise on the summary sheet we provided.
Each condition was implemented once a day, during the same 15-min period of teacher-directed instruction observed during baseline. Condition order was randomly scheduled by drawing condition names without replacement, such that each condition was implemented once every 4 days. Condition order was communicated to teachers via a written schedule, which was located on the summary sheet where they recorded their self-monitoring data. If additional days needed to be scheduled (e.g., if data were unstable and the teacher needed to remain in the alternating treatments phase), additional conditions were communicated in writing via email and with an updated summary sheet. In addition, teachers were offered an email reminder of condition order, and all teachers were emailed if schedule changes (e.g., half days, snow days) altered the planned schedule. Emails did not contain any feedback on their use of specific praise.
Data collection continued until a stable pattern of behav- ior and separation among the conditions was documented for each teacher. In addition, observers collected data on the fidelity with which each teacher implemented the selected self-management strategy each day. If teachers did not implement the correct strategy, they received a reminder from the data collector either in person or via email about condition order (with no performance feedback).
At the end of this phase, primary researchers (first two authors) met with each teacher to (a) review the components of the initial training (i.e., each component was quickly rein- troduced and teachers were given the opportunity to ask questions), (b) inform each teacher which condition was considered optimal for her, and (c) give each teacher the opportunity to complete the social validity questionnaires (based on the IRP-15) for each self-monitoring strategy.
Optimal treatment phase. During this phase, each teacher continued to implement the self-management strategy asso- ciated with her best performance. The optimal self-monitoring strategy was selected using one of the following decision rules (in order of preference): the strategy associated with the highest (a) level or increasing trend of specific praise (visual analysis), (b) mean rate, (c) mean accuracy (agree- ment between teacher and data collector), or (d) mean adherence (rated by the data collector) during the observed teacher-direction instruction activities. For example, if visual analysis did not reveal a clear optimal strategy (deci- sion rule “a”), then researchers selected the strategy with the highest mean rate (decision rule “b”). During this phase, observers continued to collect SDO data on teacher behav- ior and record the fidelity (accuracy and adherence) with which each teacher used the self-management strategy.
Teachers remained in the optimal treatment phase until a stable pattern of responding or clear trend emerged. If highly variable performance was observed, a teacher remained in this phase through the end of the study. Otherwise, a teacher was moved to a follow-up phase.
Simonsen et al. 9
Follow-up phases. Based on data, teachers were moved into one of two follow-up phases: maintenance (weekly data probes) or performance feedback (daily data updates and suggestions for using specific praise). Specifically, if a teacher demonstrated either a high stable level or clearly increasing trend in specific praise rate, she was moved into maintenance. That is, the lead data collector (second author) informed her that her performance indicated that she was ready to move into maintenance (weekly data probes). Dur- ing each probe, the teacher self-monitored with the optimal strategy and observers continued to collect SDO data.
If a teacher demonstrated either a low stable or clearly decreasing trend in specific praise rate during the optimal treatment phase, she was moved into performance feed- back. That is, researchers met with her and provided verbal and graphic performance feedback using a one-page sheet that summarized the critical features of specific praise, shared contextually appropriate examples of specific praise statements for her classroom, and presented summary data (bullet points summarizing means and a graph of specific praise rates across conditions and phases). Throughout this phase, observers continued to collect daily data, and the lead data collector emailed the teacher an updated perfor- mance feedback sheet, which included each additional day of data, other examples, and summary statements, such as, “You increased/decreased your rate of specific praise to X today. Remember your goal is at least Y times per minute.” Teachers were asked to respond via email that they had received and reviewed the performance feedback sheet.
At the end of the study, researchers met with each teacher to (a) provide feedback about their performance throughout the study, which included a one-page data summary with suggestions for on-going improvement in classroom man- agement after the study, and (b) thank them for their partici- pation with a $50 gift card.
Results
In the following sections, results are summarized for imple- mentation fidelity, SDO of teachers’ specific praise rates, and social validity.
Fidelity of Self-Monitoring In general, teachers adhered to the self-monitoring condi- tions across phases (Table 1). All teachers fully refrained from self-monitoring during the “day off” (no intervention) condition, and all teachers rated their specific praise during the rating condition. The count and tally conditions required a greater response effort throughout the 15-min observation and were associated with higher variability across teachers during the alternating treatments, optimal condition, and follow-up phases. The accuracy of teachers’ self-monitoring varied among the teachers and across conditions (Table 2).
SDO Data (Teacher Praise Rates) SDO data for teachers’ specific praise rates were graphed and analyzed visually within and across phases for each teacher (Figure 1). In addition, means and standard devia- tions were calculated for teachers’ specific praise rates (Table 3). Given the concerns with these measures of cen- tral tendency and spread for auto-correlated data (i.e., repeated measures), these data should be interpreted with caution. Results are summarized for each teacher by phase.
Teacher 1. During baseline, Teacher 1 demonstrated low and generally stable levels of specific praise. During the alternating treatments phase, Teacher 1 demonstrated an increase in level of specific praise across all conditions, and the count condition was associated with the greatest level of specific praise, with the exception of the final data point.
Table 1. Mean and Standard Deviation Rating of Adherence to Self-Monitoring (0 = Not at All, 1 = Partially, 2 = Fully) for Each Condition (Alternating Conditions, Optimal Condition, and Follow-Up Phases) Across Teachers (1–5)
Alternating condition Optimal Follow-up
No intervention Count Tally Rate Count or tallya Maintenance or
feedbackb
M SD M SD M SD M SD M SD M SD
1 2.00 0.00 2.00 0.00 1.80 0.45 2.00 0.00 2.00 0.00 2.00 0.00 2 2.00 0.00 2.00 0.00 1.75 0.50 2.00 0.00 1.95 0.22 — 3 2.00 0.00 1.60 0.55 1.67 0.82 2.00 0.00 1.94 0.25 1.75 0.71 4 2.00 0.00 2.00 0.00 1.83 0.41 2.00 0.00 2.00 0.00 1.29 0.95
5 2.00 0.00 1.50 0.84 1.20 0.84 2.00 0.00 2.00 0.00 2.00 0.00
a. Count was the optimal condition for Teachers 1, 2, and 5; tally was the optimal condition for Teachers 3 and 4. b. Teacher 1 was moved into maintenance, and Teachers 3, 4, and 5 received performance feedback during follow-up phases.
10 Journal of Positive Behavior Interventions 15(1)
During the optimal treatment phase, Teacher 1 demon- strated a clear increasing trend, resulting in a high level of specific praise. Given the high level and increasing trend of specific praise during the optimal treatment phase, Teacher 1 was moved into maintenance (weekly probes), and she maintained a high and stable level of specific praise.
Teacher 2. During baseline, Teacher 2 demonstrated a low and stable specific praise rate. During the alternating treatments phase, Teacher 2 demonstrated variable specific praise rates across conditions. Because count, tally, and rate conditions were associated with similarly high levels of specific praise, the count condition was selected as her opti- mal condition as it was associated with the highest level of accuracy. During the optimal treatment phase, Teacher 2 demonstrated highly variable specific praise rates, which were generally higher than her rates in previous phases and increased in trend throughout the phase. Because of high variability, data collection continued in this phase, and she was not moved to a follow-up phase.
Teacher 3. Teacher 3’s specific praise rates increased throughout the baseline phase. Unlike other teachers, she received training, but she asked not to begin the alternating treatments phase until a later date (after statewide testing concluded). Thus, she received training during baseline (as indicated by the arrow in Figure 1). During the alternating treatments phase, an immediate increase in level was observed for all self-monitoring conditions. Throughout this phase, specific praise rates were variable and generally decreased in trend across all conditions. The tally condition was associated with the highest average specific praise rate and was selected as her optimal condition. During the opti- mal treatment phase, Teacher 3 maintained a relatively high specific praise rate. However, her data were variable and gen- erally lower than data for the same condition (tally) during the alternating treatments phase. Therefore, we provided per- formance feedback during the follow-up phase. When
receiving performance feedback, Teacher 3’s specific praise rates were more variable and slightly lower than in the opti- mal treatment phase. In other words, daily performance feed- back was not associated with greater improvements in praise rate than self-monitoring with the optimal strategy (tally).
Teacher 4. Teacher 4 engaged in low and stable rates of specific praise throughout the baseline phase. During the alternating treatments phase, Teacher 4’s specific praise rates remained relatively low with variability across conditions. Tally and count conditions were both associated with similar specific praise rates; therefore, tally was selected as the opti- mal condition based on her accuracy. During the optimal treatment phase, Teacher 4 demonstrated a slight increase in specific praise rate, but her data were variable and demon- strated a slight decreasing trend. As a result, performance feedback was provided during the follow-up phase, and her specific praise rates increased in level and trend.
Teacher 5. Teacher 5 provided low and stable rates of spe- cific praise during baseline. Her specific praise rates were highly variable, and overlap was noted among conditions throughout the alternating treatments phase. Both tally and count conditions were associated with the highest average rate, and she implemented both with a similar level of accu- racy. Therefore, the count strategy was selected because it was associated with the highest level of adherence. During the optimal treatment phase, Teacher 5 increased her average specific praise rate, but her data were still variable and rela- tively low in comparison with other teachers. Therefore, per- formance feedback was provided during the follow-up phase. The introduction of daily performance feedback was associated with a slight increase in level and trend.
Social Validity In general, teachers found self-monitoring strategies accept- able (Figure 2). Relative to other strategies, teachers indicated
Table 2. Mean and Standard Deviation Accuracy of Self-Monitoring (Agreement Between Teacher and Observer) for Each Condition (Alternating Conditions, Optimal Condition, and Follow-Up Phases) Across Teachers (1–5)
Alternating conditiona Optimal Follow-up
Count Tally Rate Count or tallyb Maintenance or
feedbackc
M SD M SD M SD M SD M SD
1 0.69 0.19 0.62 0.20 0.62 0.26 0.77 0.16 0.74 0.07 2 0.73 0.14 0.63 0.23 0.48 0.28 0.59 0.15 — 3 0.58 0.38 0.76 0.17 0.57 0.40 0.76 0.18 0.66 0.36 4 0.24 0.16 0.35 0.27 0.20 0.20 0.45 0.20 0.23 0.25
5 0.50 0.32 0.52 0.37 0.15 0.12 0.49 0.19 0.79 0.10
a. Accuracy data were not collected during the no intervention condition as teachers did not record data. b. Count was the optimal condition for Teachers 1, 2, and 5; tally was the optimal condition for Teachers 3 and 4. c. Teacher 1 was moved into maintenance, and Teachers 3, 4, and 5 received performance feedback during follow-up phases.
Simonsen et al. 11
that self-monitoring with the counter resulted in greater decreases in students’ inappropriate behavior and greater increases in students’ appropriate behavior, that it was easier and less effortful, and that they were more likely to recom- mend it to others.
Discussion
In this study, we examined the effects of three self-monitoring strategies (tally, count, and rate) and no self-monitoring on five middle school teachers’ use of specific praise during
Figure 1. Specific praise rate (per minute) across phases and conditions for Teachers 1–5
12 Journal of Positive Behavior Interventions 15(1)
teacher-directed instruction. In general, we found that (a) teachers adhered to all self-monitoring conditions, but recorded their praise rates with varying levels of accuracy across conditions; (b) teachers’ specific praise rates were higher during self-monitoring conditions than baseline or the no self-monitoring condition, with either count or tally considered optimal; and (c) teachers preferred the count strategy. Therefore, self-monitoring may be a promising strategy for increasing teachers’ use of specific praise. In the following sections, we discuss study results, limitations, and implications in more detail.
Discussion of Study Results All teachers engaged in low and stable rates of specific praise during baseline, and the introduction of the three
self-monitoring strategies during the alternating treatments phase was associated with an increase in level, trend, or both across teachers, with the exception of the rating strat- egy for Teacher 3. Count and tally conditions were found to be optimal because they were associated with the highest levels of specific praise, accuracy of recording, or adher- ence to the strategy for participating teachers. Both condi- tions required teachers to note each time they delivered specific praise and differed only in the mode of recording (counter vs. paper and pencil). Teachers preferred the coun- ter condition to the other self-monitoring strategies and indicated that the counter was the easiest to implement, resulted in the most desired outcomes, and required an acceptable level of effort. For example, Teacher 1 com- mented that the counter may have served as a prompt because holding it “reminded [her] to make praise state- ments,” whereas she “forgot to tally and praise.” Similarly, Teacher 2 commented that the counter was “easier to have with her for the whole 15 min,” unlike the tally sheet, which she often left on her desk. In other words, teachers considered the counter the most efficient and effective strategy.
Following the alternating treatment phase, two addi- tional phases were implemented: optimal treatment and follow-up. Because neither phase was implemented in a staggered fashion across teachers, experimental control was not achieved and the following results are descriptive in nature. During the optimal treatment phase, Teacher 1 clearly increased the level and trend of specific praise, Teacher 2 increased the level of specific praise, but her per- formance remained variable, and Teachers 3–5 engaged in inconsistent levels of specific praise.
When Teacher 1 was moved into the maintenance phase, she maintained her level of praise across three weekly probes. She commented that she appreciated learning how to effectively praise, and she used this skill throughout the selected period and across her other classes. Because of
Table 3. Mean and Standard Deviation Rate of Specific Praise Statements per Minute for Each Condition (Baseline, Alternating Conditions, Optimal Condition, and Follow-Up Phases) Across Teachers (1–5)
Baseline Alternating condition Optimal Follow-up
No intervention No intervention Count Tally Rate Count or
tallya Maintenance or
feedbackb
M SD M SD M SD M SD M SD M SD M SD
1 0.15 0.21 0.53 0.20 1.11 0.46 0.55 0.32 0.75 0.34 1.23 0.58 1.51 0.38 2 0.02 0.04 0.19 0.11 0.69 0.28 0.62 0.19 0.67 0.54 1.08 0.59 — 3 0.62 0.36 1.00 0.83 1.34 0.57 1.53 0.91 0.61 0.54 1.07 0.35 0.79 0.49 4 0.01 0.03 0.20 0.19 0.32 0.20 0.32 0.19 0.23 0.21 0.38 0.20 0.74 0.60
5 0.15 0.04 0.18 0.18 0.40 0.26 0.47 0.23 0.31 0.18 0.52 0.15 0.66 0.35
a. Count was the optimal condition for Teachers 1, 2, and 5; tally was the optimal condition for Teachers 3 and 4. b. Teacher 1 was moved into maintenance, and Teachers 3, 4, and 5 received performance feedback during follow-up phases.
Figure 2. Average ratings of acceptability on the Intervention Rating Profile–15 Note. High scores are desired on Items 1, 2, 3, and 5; low scores are desired on Question 4.
Simonsen et al. 13
high variability, Teacher 2 was not moved into a follow-up phase. However, her average praise rate was higher during the optimal treatment phase than any of the conditions dur- ing the previous alternating treatments phase.
When performance feedback was introduced for Teachers 3–5, findings were inconsistent. Teacher 3 did not appear to respond as her specific praise rate initially decreased and then returned to previous levels. She commented that it was difficult to tally on various days during this phase. Teacher 4 did not appear to respond initially, but her use of specific praise increased toward the end of the phase. It is interesting that this increase in praise corresponded to a decrease in adherence to self-monitoring; therefore, another variable (e.g., end of school year pressure to widely distribute school- wide behavior coupons paired with praise) may better explain changes in her specific praise rate. Teacher 5 gradu- ally increased her specific praise during the performance feedback, but all data points overlapped with those from the previous phase. In sum, most teachers engaged in their opti- mal specific praise rates during the optimal treatment phase, and performance feedback did not result in substantial gains over self-monitoring.
Therefore, self-monitoring appears to be an effective tool to increase teachers’ use of specific praise. This finding adds to existing literature demonstrating that self-monitoring (Allinder et al., 2000; Browder et al., 1986) and self-evaluation (Keller et al., 2005; Sutherland & Wehby, 2001) are associated with increases in teachers’ use of evidence-based practices (i.e., data-based decision making and specific praise, respectively). Results of this study also suggest that teach- ers’ use of simple and efficient self-monitoring strategies employed while teaching may be effective and may reduce the need for more time-intensive performance feedback and self-management procedures, such as reviewing and evalu- ating audio recordings of instruction.
Limitations Study results should be viewed in light of the following limitations related to the scope, context, measurement, and design of this study. First, we explored the effects of vari- ous self-monitoring strategies on five middle school teach- ers’ use of specific praise. As these teachers volunteered to participate in a study on classroom management, they may have responded differently to self-monitoring and increas- ing praise than other teachers. Generalizations of findings to other populations of teachers and to other classroom management practices (e.g., prompts) are premature. The effects of self-monitoring on other populations of teachers and other classroom management practices should be sys- tematically studied.
Second, although we scheduled direct observations during the times each teacher identified as teacher-directed instruc- tion, variability existed among instructional conditions within and across teachers. In addition to providing direct instruction,
teachers worked with individual students, facilitated indepen- dent seatwork, and delivered other types of instruction. Also, four of the classrooms were typical general education class- rooms, and one classroom (Teacher 3) was a small-group spe- cial education setting. This variability in instructional practices and class composition may have influenced teachers’ specific praise rates.
Third, observers may not have captured all instances of specific praise delivered by teachers. Because instructional conditions varied, observers may have had difficulty hear- ing and recording teachers’ specific praise statements dur- ing certain instructional conditions. For example, if a teacher walked around the room and quietly provided feed- back to students, observers may not have had heard all instances of praise. Similarly, Teacher 3 delivered instruc- tion in both English and Spanish. Although her style was to repeat information in both languages, observers may have missed some instances of praise in Spanish.
Finally, although the study design allowed direct com- parison of self-monitoring strategies and no self-monitoring during the alternating treatments phase, the introduction of the optimal treatment and follow-up phases were not stag- gered; therefore, a functional relationship between the opti- mal strategy or follow-up (e.g., performance feedback) and teacher behavior was not documented. In addition, research- ers were directly involved in providing brief trainings between phases and delivering feedback during the perfor- mance feedback phase.
Implications Although preliminary, the results of this study suggest that simple and efficient self-monitoring strategies may be related to increases in teachers’ use of specific praise. In particular, recording each instance of specific praise using either a counter or tally resulted in optimal praise rates for all teachers, and all teachers preferred using the counter. Therefore, school administrators and others involved in supporting teachers may consider asking teachers to use a simple self-monitoring strategy to record their use of spe- cific practices, like praise, to increase their implementation of that practice.
In addition, this study clearly highlights a need for addi- tional research in the use of simple strategies to increase teachers’ use of evidence-based classroom management practices. First, researchers should use experimental designs (e.g., multiple baseline, withdrawal, group experimental) to continue to study the effects of self-monitoring on teachers’ use of classroom management skills like specific praise. Second, if self-monitoring is functionally related to increases in specific classroom management skills, research- ers should explore the conditions under which self-monitoring may be used. For example, it would be useful to examine (a) how many behaviors teachers can effectively and effi- ciently monitor at one time; (b) whether self-monitoring
14 Journal of Positive Behavior Interventions 15(1)
effectiveness is similar under other instructional contexts, such as transitions, student-led activities, and teacher lec- ture; (c) what “dose,” or length and intensity, of self- monitoring is required to sustain the desired level of teacher behavior; and (d) how to fade self-monitoring while maintaining desired levels of teacher behavior.
Third, given the variability in teacher characteristics with respect to years of experience, prior training, skill fluency, and other characteristics, we would expect general strategies, like self-monitoring, to be effective with some, but not all, teach- ers. Therefore, future researchers should examine what addi- tional supports might be needed if simple self-monitoring is ineffective. Finally, researchers should explore the effective- ness of self-monitoring under typical school conditions to establish the ecological validity of this practice (e.g., Carr et al., 2002). In this study, researchers provided training, feed- back, and prompting. An important question is whether simi- lar implementation fidelity can be achieved when support is provided by school administrators, school psychologists, and peer mentors under typical work conditions.
In sum, if teachers are to benefit from the use of effective practices, they must be able to implement that practice with fidelity. Performance feedback can enhance implementation fidelity; however, obtaining useful and meaningful feedback may be difficult when resources are limited. The findings from this study suggest that self-monitoring may be strategy for teachers to obtain information about their implementa- tion in a relevant, efficient, and effective manner.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The development of this article was supported in part by Grant H029D40055 from the Office of Special Education Programs (OSEP), U.S. Department of Education for the OSEP Center on Positive Behavioral Interventions and Supports (www.pbis.org). Opinions expressed herein are the authors’ and do not necessarily reflect the position of the U.S. Department of Education, and such endorsements should not be inferred.
References
Abbott, R. D., O’Donnell, J., Hawkins, J. D., Hill, K. G., Kosterman, R., & Catalano, R. F. (1998). Changing teach- ing practices to promote achievement and bonding to school. American Journal of Orthopsychiatry, 68, 542–552.
Allen, C. T., & Forman, S. G. (1984). Efficacy of methods of training teachers in behavior modification. School Psychology Review, 13, 26–32.
Allinder, R., Bolling, R., Oats, R., & Gagnon, W. (2000). Effects of teacher self-monitoring implementation of curriculum-based measurement and mathematics computation achievement of students with disabilities. Remedial and Special Education, 21, 219–226.
Barlow, D. H., & Hayes, S. C. (1979). Alternating treatments design: One strategy for comparing the effects of two treat- ments in a single subject. Journal of Applied Behavior Analy- sis, 12, 199–210.
Begeny, J. C., & Martens, B. K. (2006). Assessing pre-service teachers’ training in empirically-validated behavioral instruc- tion practices. School Psychology Quarterly, 21, 262–285.
Brophy, J. (1981). Teacher praise: A functional analysis. Review of Educational Research, 51, 5–32.
Browder, D., Liberty, K., Heller, M., & D’Huyvetters, K. K. (1986). Self-management by teachers: Improving instructional decision making. Professional School Psychology, 1, 165–175.
Caplin, D., & Creer, T. (2001). A self-management program for adult asthma, Part III: Maintenance and relapse of skills. Jour- nal of Asthma, 38, 343–356.
Carr, E. G., Dunlap, G., Horner, R. H., Koegel, R. L., Turnbull, A. P., Sailor, W., . . . Fox, L. (2002). Positive behavior support: Evolution of an applied science. Journal of Positive Behavior Interventions and Supports, 4, 4–16, 20.
Chalk, K., & Bizo, L. A. (2004). Specific praise improves on-task behavior and numeracy enjoyment: A study of year four pupils engaged in numeracy hour. Educational Psychology in Prac- tice, 20, 335–351.
Cooper, J., Heron, T., & Heward, W. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson.
Creer, T., Caplin, D., & Holroyd, K. (2005). A self-management program for adult asthma, Part IV: Analysis of context and patient behaviors. Journal of Asthma, 42, 455–462.
Creti, L., Libman, E., Bailes, S., & Fichten, C. (2005). Effectiveness of cognitive-behavioral insomnia treatment in a community sample of older individuals: More questions than conclu- sions. Journal of Clinical Psychology in Medical Settings, 12, 153–164.
Donaldson, J., & Normand, M. (2009). Using goal setting, self- monitoring, and feedback to increase calorie expenditure in obese adults. Behavioral Interventions, 24, 73–83.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication #231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Insti- tute, National Implementation Research Network.
Gast, D. L. (Ed.). (2010). Single subject research methodology in behavioral sciences. New York, NY: Routledge.
Jeffrey, J. L., McCurdy, B. L., Ewing, S., & Polis, D. (2009). Class- wide PBIS for students with EBD: Initial evaluation of an integ- rity tool. Education and Treatment of Children, 32, 537–550.
Keller, C. L., Brady, M. P., & Taylor, R. L. (2005). Using self- evaluation to improve student teacher interns’ use of specific
Simonsen et al. 15
praise. Education and Training in Developmental Disabilities, 40, 368–376.
Martens, B. K., Witt, J. C., Elliott, S. M., & Darveaux, D. X. (1985). Teacher judgments concerning the acceptability of school- based interventions. Professional Psychology: Research and Practice, 16, 191–198.
Ngamvitroj, A., & Kang, D. (2007). Effects of self-efficacy, social support and knowledge on adherence to PEFR self- monitoring among adults with asthma: A prospective repeated measures study. International Journal of Nursing Studies, 44, 882–892.
Noell, G. H., Witt, J. C., Gilbertson, D. N., Rainer, S. D., & Freeland, J. T. (1997). Increasing teacher intervention imple- mentation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77–88.
Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., . . . Duhon, G. J. (2005). Treatment imple- mentation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87–106.
Reynolds, L. K., & Kelley, M. L. (1997). The efficacy of a response cost-based treatment package for managing aggres- sive behavior in preschoolers. Behavior Modification, 21, 216–230.
Rokke, P., Tomhave, J., & Jocic, Z. (2000). Self-management therapy and educational group therapy for depressed elders. Cognitive Therapy and Research, 24, 99–119.
Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai, G. (2008). A review of evidence based practices in classroom management: Considerations for research to practice. Educa- tion and Treatment of Children, 31, 351–380.
Simonsen, B., Myers, D., & DeLuca, C. (2010). Providing teach- ers with training and performance feedback to increase use of three classroom management skills: Prompts, opportunities to respond, and reinforcement. Teacher Education in Special Education, 33, 300–318. doi:10.1177/0888406409359905
Skinner, B. F. (1953). Science and human behavior. New York, NY: MacMillan.
Sutherland, K. S., & Wehby, J. H. (2001). The effect of self- evaluation on teaching behavior in classrooms for students with emotional and behavioral disorders. Journal of Special Education, 35, 161–171.
Sutherland, K. S., Wehby, J. H., & Copeland, S. R. (2000). Effects of varying rates of behavior-specific praise on the on-task behavior of students with EBD. Journal of Emotional and Behavioral Disorders, 8, 2–8.
Workman, E. A., Watson, P. J., & Helton, G. B. (1982). Teachers’ self- monitoring of praise vs. praise instructions: Effects on teachers’ and students’ behavior. Psychological Reports, 50, 559–565.