ABA 527 M11 Project
zanuyekeh
Socialvalidity.pdf
EMPIRICAL REPORT
Social Validity Assessment of Training Methods to Improve Treatment Integrity of Special Education Service Providers
Craig Strohmeier & Christina Mulé & James K. Luiselli
Published online: 29 April 2014 # Association for Behavior Analysis International 2014
Abstract We report the results of a social validity assessment that was administered to special education service providers (N=44) to document the acceptability and effectiveness rat- ings of several treatment integrity training methods. The par- ticipants judged performance feedback as the most likely method to improve their treatment integrity, followed by avoidance (negative reinforcement) of supervision meetings, online training, and financial incentive. Performance feedback was also rank-ordered as most effective among the four train- ing methods. We discuss the merits of social validity assess- ment in designing programs for enhancing treatment integrity among practitioners within educational and clinical settings.
Keywords Treatment integrity . Social validity assessment .
Human services organizations
Treatment integrity refers to the degree to which intervention procedures are implemented as planned, designed, or intended (McIntyre et al. 2007; Solomon et al. 2012). Research has shown that high treatment integrity is associated with positive therapeutic outcomes (Fiske 2008; Noell et al. 2002; Wilder et al. 2006). Unfortunately, and despite receiving pre-service training, many practitioners do not maintain acceptable treat- ment integrity over time (DiGennaro et al. 2007; Noell et al. 2000). With regard to clinical practice, it is not possible to properly evaluate the effectiveness of intervention plans if practitioners apply procedures inaccurately. Furthermore,
replication of successful plans may fail if intervention integ- rity is not closely monitored.
Assessing treatment integrity entails direct observa- tion of practitioners implementing procedures with the people in their care (Noell et al. 2005; DiGennaro Reed and Codding 2011). Typically, observers use a recording form with steps that correspond to procedures that are listed on a written instructional or behavior support plan. The form allows the observer to document wheth- er a practitioner implemented each procedure properly or deviated from expected criteria. A common training strategy is to follow direct observation with verbal per- formance feedback in which the observer praises accu- rately implemented procedures and corrects procedures that were misapplied (Codding et al. 2005; Luiselli 2013; Ricciardi 2005). Performance feedback may also include written correspondence, graphic display of inter- vention integrity data, and in vivo practice sessions (DiGennaro Reed and Codding 2011).
DiGennaro Reed and Codding (2011) described several other procedures for enhancing treatment integrity. For exam- ple, a supervisor and practitioner can schedule directed re- hearsal meetings to review and practice procedures that re- quire correction. Video modeling is another training strategy in which a practitioner views “a video of a model, often an experienced colleague, demonstrating accurate implementa- tion of all individualized steps with either the actual client or with an actor” (DiGennaro Reed and Codding 2011, p. 46). A video modeling procedure can either be used by itself or combined with conventional performance feedback methods. Finally, allowing practitioners to cancel directed rehearsal meetings and similar contacts with a supervisor contingent on achieving optimal treatment integrity may also have a positive effect on procedural fidelity (DiGennaro et al. 2005).
Although research endorses the benefits of addressing and improving intervention integrity among service providers,
C. Strohmeier Kennedy Krieger Institute, Baltimore, MD, USA
C. Mulé Tufts University Medical Center, Boston, MA, USA
J. K. Luiselli (*) May Institute, 41 Pacella Park Drive, Randolph, MA 02368, USA e-mail: [email protected]
Behav Analysis Practice (2014) 7:15–20 DOI 10.1007/s40617-014-0004-5
there has been limited study of practitioners’ attitudes about the methods that are used to enhance performance. Reid (2004), for example, noted that “success in working with staff to promote proficient application of behavior plans can be seriously affected by the degree to which supervisory prac- tices are accepted by staff” (p. 86). Some practitioners, in fact, do not approve of systematically monitoring their work per- formance (Reid and Parsons 1995). Accordingly, it would be valuable to evaluate the social validity (Kazdin 1977; Wolf 1978) of treatment integrity training methods. From the per- spective of staff performance improvement, such assessment would enable supervisors and program administrators to select performance management procedures that have high appeal (Luiselli 2013; Reid 2004).
In the present study, we surveyed practitioners at a special education setting about their experiences implementing behavior support plans and recommenda- tions that could improve treatment integrity. Specifically, we constructed a questionnaire that listed different treat- ment integrity training methods and asked the practi- tioners to rate each method according to perceived ef- fectiveness, satisfaction, and acceptability. The study illustrates how social validity assessment can be employed by human services organizations to inform decisions about staff training procedures that are intended to strengthen and support treatment integrity.
Method
Participants and Settings
The participants were 44 direct-service staff at a day and residential school serving children and youth who had autism spectrum disorder (ASD) and related devel- opmental disabilities. There were 36 (82 %) females and 8 (18 %) males between 18–39 years old. Approximate- ly, 7 % of the participants had a high school diploma, 71 % had a bachelor’s degree, and 23 % had a master’s degree. The participants functioned as primary teachers and teacher assistants within classrooms at the school. Their educational teaching experiences ranged from 0– 2 years (48 %), 3–5 years (32 %), 6–8 years (11 %), 9– 10 years (2 %), and 10 or more years (7 %).
As educational staff in the school, the participants implemented instructional and behavioral support proce- dures with the students in their classrooms. The proce- dures were specified and described in written, student- specific plans that all staff was expected to follow. Several school administrators supervised the participants by ob- serving them periodically, conducting classroom meetings, and providing general feedback about plan implementa- tion. However, the supervisors did not perform treatment
integrity assessment or deliver systematic performance feedback.
Questionnaire Construction and Distribution
The participant questionnaire was comprised of two parts and 15 items. Part 1 requested the following demographic infor- mation: (1) gender, (2) age, (3) ethnicity, (4) highest degree earned, (5) years of teaching experience, (6) experience with students who had ASD, and (7) experience implementing behavior support plans. Part 2 of the questionnaire, shown in Fig. 1, asked the participants to answer seven questions about behavior support plans and training methods to improve treat- ment integrity according to a four-point Likert scale that was specific to each question. Questions 1–3 elicited information about the behavior support plans the participants were expect- ed to implement: how well they understood the plans, how effective current feedback procedures were for incorrect pro- cedural implementation, and how effective current feedback procedures were for correct procedural implementation. The objective of questions 4 and 5 were to acquire information about two instructional methods, performance feedback and online training, for increasing the ability to correctly imple- ment behavior support plans. Questions 6 and 7 asked for the ratings of two consequence-based methods, positive and neg- ative reinforcement, for increasing correctly implemented plans.
One additional item in part 2 had the participants rank- order the perceived effectiveness of the four treatment integ- rity training methods that were presented in questions 4–7. Using a four-point scale, the ratings ranged from 1 (least effective) to 4 (most effective).
We distributed the questionnaire to approximately 110 classroom staff at the school. Staff was informed that their participation was voluntary, and they would be eligible for a gift card drawing if they completed the questionnaire. To maintain anonymity but allow for the gift card drawing, staff wrote their names on a blank ticket that was stapled to the questionnaire. Each classroom had a large envelope for collecting completed questionnaires and a second large enve- lope for depositing the tickets. Five days following distribu- tion, we collected the envelopes containing the completed questionnaires and the staff tickets from each classroom. One ticket was drawn, and the gift card winner was announced.
The first and second authors coded the completed ques- tionnaires into an electronic spreadsheet. Responses to the seven questions in part 1 and questions 1–7 in part 2 of the questionnaire were summarized as the percentage of partici- pants who endorsed the available choices per question. Data from question 8 in part 2 of the questionnaire were computed as the mean effectiveness ratings of the four treatment integ- rity training methods.
16 Behav Analysis Practice (2014) 7:15–20
Results
Beyond the part 1 demographic data reported previously (gender, ethnicity, age, degree, years of teaching experience), 64 % of participants reported that they had “a lot of experi- ence,” 27 % had “some experience,” and 9 % had “no expe- rience” teaching students who had ASD. Concerning behavior
support plan implementation, 55 % of participants reported that they had “a lot of experience,” 34 % had “some experi- ence,” and 11 % had “no experience.” The participants also indicated that 66 % of them understood the behavior support plans of their students “very well” and 34 % of them “some- what well.” Their ratings about the current feedback proce- dures they received were varied, with 16 % reporting “very
1. How well do you understand the behavior support plans of the students with whom you work?
Very poor Somewhat poor
Somewhat well
Very well
2. How effective are the current feedback procedures for incorrect behavior support plan implementation?
Very ineffective
Somewhat ineffective
Somewhat effective
Very effective
3. How effective are the current feedback procedures for correct behavior support plan implementation?
Very ineffective
Somewhat ineffective
Somewhat effective
Very effective
4. How well would you receive performance feedback (i.e., discussion about your areas of strength and weakness) to increase your ability to correctly implement behavior support plans?
Very poor Somewhat poor
Somewhat well
Very well
5. How likely would you be to complete online training modules to increase your ability to correctly implement behavior support plans?
Very unlikely Somewhat unlikely
Somewhat likely
Very likely
6. How likely would you be to increase correct implementation of a behavior support plan to avoid ongoing performance feedback meetings?
Very unlikely Somewhat unlikely
Somewhat likely
Very likely
7. How likely would you be to increase correct implementation of a behavior support plan if given financial incentive (e.g., a gift card)?
Very unlikely Somewhat unlikely
Somewhat likely
Very likely
8. Please rank the following methods in order of most effective to least effective (1 = most effective; 4 = least effective):
______ Online training modules
______ Performance feedback
______ Escape from future performance feedback meetings
______ Access to financial incentive
Fig. 1 Social Validity Questionnaire
Behav Analysis Practice (2014) 7:15–20 17
effective,” 48 % “somewhat effective,” 34 % “somewhat ineffective,” and 2 % “very ineffective.”
Figure 2 (top panel) shows that 77 % of the partic- ipants reported that they would respond “very well,” and 23 % “somewhat well,” to receiving performance feedback. The bottom panel of Fig. 2 reveals that for the other three treatment integrity training methods, 52 % of participants reported that they would “very likely” and 30 % would “somewhat likely” increase correct implementation of behavior support plans to avoid performance feedback meetings. For online train- ing, 32 and 36 % of participants would “very likely” and “somewhat likely” increase correct implementation of behavior support plans, respectively. The results for receiving a financial incentive to increase correct imple- mentation of behavior support plans were 52 % of participants reporting “very likely” and 16 % reporting “somewhat likely.”
When requested to rank-order effectiveness of the treat- ment integrity training methods (Fig. 3), the participants en- dorsed performance feedback as most effective (M=3.59, SD=.81), followed by online training (M=2.27, SD=1.00), financial incentive (M=2.24, SD=.97), and avoiding perfor- mance feedback meetings (M=1.90, SD=.92).
Discussion
All of the participants in this study responded “somewhat well–very well” when queried about how they would receive performance feedback through the discussion of their strengths and weaknesses in implementing behavior support plans.With regard to the other training methods for improving treatment integrity, avoiding ongoing performance feedback meetings was rated ahead of completing online training and receiving a financial incentive. Relative to rank-ordered effec- tiveness ratings, the participants judged performance feedback as most effective and avoiding performance feedback meet- ings as least effective. Thus, performance feedback was the most acceptable and highly rated training procedure for im- proving treatment integrity among these service providers.
As noted previously, the rationale for social validity assess- ment of training methods to improve treatment integrity is selecting procedures that appeal to practitioners and, in con- sequence, are likely to produce the most positive effect. Re- cent literature suggests that performance feedback is the most effective method for improving treatment integrity (DiGennaro Reed and Codding 2011; DiGennaro Reed et al. 2013). It appears, then, that positive practitioner ratings of performance feedback, like those reported in this study, links high acceptability to procedural effectiveness. Of course, it is possible that other treatment integrity training methods could be successful regardless of staff opinions. That is, some prac- titioners might prefer methods that are ineffective or, con- versely, dislike methods with demonstrated effectiveness. A reasonable approach in this regard is to have individuals select the training method for improving treatment integrity they would like to experience, followed by direct measurement to evaluate whether preferences, in fact, are associated with desirable treatment integrity. This approach also addresses the concern of relying exclusively on verbal report via rating scales as the sole dependent measure of social validity assessment.
Concerning the questionnaire itself, we noted earlier that questions 4–5 were constructed to gather information
0
10
20
30
40
50
60
70
80
90
Very Well Somewhat Well Somewhat Poor Very Poor
P er
ce nt
ag e
of P
ar tic
ip an
ts
Ratings
Performance Feedback
0
10
20
30
40
50
60
Avoid Feedback Meetings
Online Training
Financial Incentive
P er
ce nt
ag e
of P
ar tic
ip an
ts
Training Method
Very Likely
Somewhat Likely
Somewhat Unlikely
Very Unlikely
Fig. 2 Percentage of participants endorsing ratings for performance feedback (top panel) and avoiding feedback meetings, completing online training, and receiving a financial incentive (bottom panel)
0
0.5
1
1.5
2
2.5
3
3.5
4
Performance Feedback
Online Training Financial Avoid Feedback MeetingsIncentive
M ea
n R
at in
g
Training Method
Fig. 3 Mean effectiveness ratings of four treatment integrity training methods
18 Behav Analysis Practice (2014) 7:15–20
about treatment integrity instructional methods, while ques- tions 6–7 concerned treatment integrity consequence methods. In retrospect, these questions could have been stronger if we wrote them similarly and had the same rating choices. Notably, questions 4–5 assessed a partici- pant’s acceptance of the respective treatment integrity training method, and questions 6–7 asked about the pre- dicted success of the respective treatment integrity training method. Question 8 aligned the four methods according to perceived effectiveness. Thus, these qualitatively dissimilar questions must be considered when interpreting the social validity assessment data and implications for improving treatment integrity.
Another limitation is that the study had a small sample size of service providers at a special education setting, making it difficult to generalize findings to other practi- tioners and human service organizations. Most of the participants also had college degrees and multi-year expe- rience as teachers—therefore, these findings could vary with different population demographics. Note too, the possibility of a response bias affecting results because less than half of the available direct-service staff completed the questionnaire. One additional qualification is that our ques- tionnaire included only a few of the possible intervention integrity training methods.
We suggest that the training methods for improving treatment integrity that were endorsed in this question- naire appear viable for most human service organiza- tions. Certainly, scheduled supervisory observations with accompanying performance feedback are recognized as essential methods of staff training and management (DiGennaro Reed et al. 2013; Luiselli 2013). Online instruction and contingency arrangements for avoiding meetings and receiving performance “rewards” can also be adapted to different service settings as long as there are sufficient personnel and financial resources. Certain methods also have unique features. For example, if prac- titioners judge avoiding feedback meetings as a desirable treatment integrity training method, supervisors have to ensure that they do not become an aversive stimulus that is avoided at other times. Regardless of the methods to address treatment integrity, completing routine social va- lidity assessment of service providers will hopefully en- able organizations to institute systems that are judged acceptable and can be supported based on available re- sources and allocation of administrative and supervisory personnel.
Acknowledgments This study was conducted at the May Center School for Autism and Developmental Disabilities, Randolph,MA. Craig Strohmeier is currently affiliated with the Kennedy Krieger Institute, Baltimore, MD; Christina Mulé is currently affiliated with Tufts Univer- sity Medical Center, Boston, MA.
References
Codding, R. S., Feinberg, A. B., Dunn, E. K., & Pace, G. M. (2005). Effects of immediate performance feedback on implementation of behavior support plans. Journal of Applied Behavior Analysis, 38, 205–219.
DiGennaro Reed, F. D., & Codding, R. S. (2011). Intervention integrity assessment. In J. K. Luiselli (Ed.), Teaching and behavior support for children and adults with autism spectrum disorder: a practi- tioner’s guide (pp. 38–47). New York: Oxford University Press.
DiGennaro Reed, F. D., Hirst, J. M., & Howard, V. J. (2013). Empirically supported staff selection, training, and management strategies. In D. D. Reed, F. D. DiGennaro Reed, & J. K. Luiselli (Eds.), Handbook of crisis intervention and developmental disabilities (pp. 71–85). New York: Springer.
DiGennaro, F. D., Martens, B. K., & McIntyre, L. L. (2005). Increasing treatment integrity through negative reinforcement: effects of teach- er and student behavior. School Psychology Review, 34, 220–231.
DiGennaro, F. D., Martens, B. K., & Kleinmann, A. E. (2007). A comparison of performance feedback procedures on teachers’ treat- ment implementation integrity and students’ inappropriate behavior in special education classrooms. Journal of Applied Behavior Analysis, 40, 447–461.
Fiske, K. E. (2008). Treatment integrity of school-based behavior analytic interventions: a review of the research. Behavior Analysis In Practice, 1, 19–25.
Kazdin, A. E. (1977). Assessing the clinical or applied importance of behavior change through social validation. Behavior Modification, 1, 427–452.
Luiselli, J. K. (2013). Peer review. In D. D. Reed, F. D. DiGennaro Reed, & J. K. Luiselli (Eds.), Handbook of crisis intervention and devel- opmental disabilities (pp. 27–48). New York: Springer.
McIntyre, L. L., Gresham, F.M., DiGennaro, F. D., &Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis 1991–2005. Journal of Applied Behavior Analysis, 40, 659–672.
Noell, G. H., Witt, J. C., LaFleur, L. H., Mortenson, B. P., Ranier, D. D., & LeVelle, J. (2000). Increasing intervention implementation in general education following consultation: a comparison of two follow-up strategies. Journal of Applied Behavior Analysis, 33, 271–284.
Noell, G. H., Gresham, F. M., & Gansle, K. A. (2002). Does treatment integrity matter? A preliminary investigation of instructional imple- mentation and mathematics performance. Journal of Behavioral Education, 11, 51–67.
Noell, G. H.,Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L.,Williams, K. L., & Duhon, G. J. (2005). Treatment implementation following behavioral consultation in schools: a comparison of three follow-up strategies. School Psychology Review, 34, 87–106.
Reid, D. H. (2004). Training and supervising direct support personnel to carry out behavioral procedures. In J. L. Matson, R. B. Laud, & M. L. Matson (Eds.), Behavior modification for persons with develop- mental disabilities: Treatments and supports (pp. 73–97). Kingston, NY: NADD Press.
Reid, D. H., & Parsons, M. B. (1995). Motivating human services staff: supervisory strategies for maximizing work effort and work enjoyment. Morganton, NC: Habilitative Management Consultants Inc.
Ricciardi, J. N. (2005). Achieving human services outcomes through competency based training: a guide for practitioners. Behavior Modification, 29, 488–507.
Solomon, B. G., Klein, S. A., & Politylo, B. C. (2012). The effect of performance feedback on teachers’ treatment integrity: a meta- analysis of the single-case literature. School Psychology Review, 41, 160–175.
Behav Analysis Practice (2014) 7:15–20 19
Wilder, D. A., Atwell, J., &Wine, B. (2006). The effects of varying levels of treatment integrity on child compliance during a three-step prompting procedure. Journal of Applied Behavior Analysis, 39, 369–373.
Wolf, M. M. (1978). Social validity: the case for subjective measurement of how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203–214.
20 Behav Analysis Practice (2014) 7:15–20
