ABA 527 M11 Project
zanuyekeh
BehaviorRatingscale.pdf
JmwlofSchdP&do~, Vol. 29, pp. 43-51, 1991 oozz-4405/91/s3.00 + .oa Pcrgamon Press pk. Printed in the USA. Q 1991 The Journal of School Psychology, Inc.
The Behavior Intervention Rating Scale: Development and Validation of a Pretreatment
Acceptability and Effectiveness Measure
Stephen N. Elliott University of Wisconsin-Madison
Mary Van Brock Treuting buisiansr State University
The construct validity of the Behavior Intervention Rating Scale (BIRS) was investi- gated to develop an instrument to measure teachers’ perceptions of treatment accept- ability and perceived effectiveness of classroom interventions. An oblique factor analy- sis was utilized, and three factors, Acceptability, Effectiveness, and Time of Effect, were determined. Correlational analysis demonstrated moderate to strong relations among the three factors. The BIRS was shown to be a vahd measure of two general constructs important to treatment evaluation, acceptability and effectiveness.
The social validity of psychological assessment and treatment service has be- come increasingly important to researchers and practitioners alike (Kazdin, 1977, 1980; Martens, Witt, Elliott, 8r Darveaux, 1985; Wolf, 1978). Research- ers currently are concerned with the factors that influence the implementation and success of psychological treatments in specific situations. Behavioral tech- nology has provided a number of potentially effective techniques for correcting behavior problems; however, to use these treatments within a natural environ- ment, we must often rely on consumers (e.g., parents, teachers) to do the actual implementation. Therefore, a method is needed to assess consumer attitudes concerning different treatments.
A number of factors have been isolated under analogue conditions to influ- ence consumers’ acceptance of treatments. For example, Kazdin (1980) found that the severity of problems to be addressed by a given treatment affected its acceptability ratings (i.e., the more severe the problem, the more acceptable the treatment). In his study, Kazdin also found adverse side effects were important considerations. Kazdin, French, and Sherick (1981) found positive reinforcement of incompatible behaviors to be the most acceptable treatment (when compared with positive practice, medication, and time-out) in a hospi- tal setting. Research in schools on treatment acceptability has indicated that a
Received October 1, 1989; final revision received July 17, 1990. Address correspondence and reprint requests to Stephen N. Elliott, Department of Educational
Psychology, University of Wisconsin-Madison, 1025 West Johnson Street, Madison, WI 53706.
43
44 Journal of School Psychology
number of variables influence teachers’ evaluations of treatment procedures (Clark & Elliott, 1988; Elliott, Witt, Galvin, & Peterson, 1984; Martens et al., 1985; Turco & Elliott, 1990; Von Brock & Elliott, 1987; Witt, Elliott, & Martens, 1984). For example, teachers’ acceptability ratings appear to be influenced by the amount of a teacher’s time, the severity of a child’s problem, the type of interventions (i.e., positive versus reductive), and the perceptions of treatment effectiveness.
The purpose of the present study was to develop an instrument that mea- sures teachers’ perceptions of treatment acceptability and treatment effective- ness. The Behavior Intervention Rating Scale (BIRS) consists of a revision and extension of the Intervention Rating Profile (IRP-15) (Martens et al., 1985). The IRP-15 is a 15-item single-factor scale that has been demonstrated to assess treatment acceptability. The IRP-15’s reported reliability measured as Cronbach’s alpha is .98. To develop the BIRS, nine new items were added to the 15 IRP items. The items were generated from the treatment effectiveness literature and covered the dimensions of rate of behavior change, level of behavior change, maintenance of behavior change, generalization to other behaviors and settings, and peer comparisons. The 15 IRP items were used to operationalize the construct of treatment acceptability; the nine new items operationalized the construct of treatment effectiveness and were labeled the Effectiveness Rating Profile (ERP). The 24 BIRS items were scaled on a 6- point Likert format, ranging from Strongly d&gree to Strongly agree. The present study outlined the preliminary psychometric properties of the BIRS, which has been utilized in four recently published treatment evaluation studies, of which two were analogue investigations (Clark & Elliott, 1988; Von Brock & Elliott, 1987) and two (Sheridan, Kratochwill, & Elliott, 1990; Turco & Elliott, 1990) were actual treatment programs.
METHODS
Subjects
The subjects were 2 16 teachers attending graduate courses at two state univer- sities in Louisiana. The subjects ranged from 22 to 63 years of age (mean = 34). The average length of their teaching experience was 8.4 years. Female5 accounted for 81% of the subjects. The racial breakdown was 86% white, 13 % black, and 1% other. The majority of the teachers were regular education teachers (82%); the remaining teachers were special education teachers (18%). The participants were surveyed in groups during regularly scheduled classes; they were informed that the research was an investigation of teachers’ percep. tions of common classroom interventions. They were also informed that al responses were anonymous and participation was voluntary.
Elliott and Treuting 45
Materials
Each subject was given a packet including instructions, demographics, a case description, the BIRS, and the Semantic Differential (Osgood, Suci, & Tan- nenbaum, 1957). The case descriptions included problem behaviors, interven- tions, and information concerning the effectiveness of the interventions. These materials were modeled after previous acceptability studies (e.g., Elliott et al., 1984; Witt et al., 1984). The subjects received one of two problem behavior descriptions, which were typical of classroom problems.’ Both problems con- sisted of out-of-seat behavior. The less severe problem description included day-dreaming, and failing to pay attention. The more severe problem included talking out of turn, taking things, and pushing others.* The interventions included a token reinforcement, a response cost, and a time-out procedure. There were three types of effectiveness information paragraphs: No informa- tion, consumer satisfaction information, or research-based outcome informa- tion was provided. Each subject received only one problem description, one intervention, and one type of effectiveness information paragraph. For more details concerning these materials and the results of multivariate analyses of these data the reader is referred to Von Brock and Elliott (1987). The descrip- tive case information was rated by the subject with the BIRS and the Semantic Differential, which was included for validity purposes. A complete copy of the BIRS is displayed as Table 1.
Procedures and Analyses
The procedures used to investigate the psychometric properties of the BIRS included both an orthogonal and an oblique factor analysis to distinguish between the constructs of acceptability and effectiveness, correlational com- parisons to establish concurrent validity between the BIRS and the Semantic Differential, and partial correlations to determine the importance of each BIRS factor. In addition, coefficient alphas were computed to determine the reliability of each factor.
tThe vast majority of analogue investigations of treatment acceptability have involved pencil-
and-paper descriptions of problem behaviors. In an early investigation by Martens, Witt, Elliott,
and Darveaux (1985), a pencil-and-paper format was directly compared with videotape presenta-
tion of children misbehaving. The results indicated no differences in ratings of problem severity or
treatment acceptance for the two formats for presenting problems. Thus, because the pencil-and-
paper format proved more efficient than the videotape, subsequent investigators have used the
pencil-and-paper format almost exclusively. ZJudgments about the severity of any misbehaviors are often rather subjective, yet they play an
important role in the interpretation of much of the treatment acceptability research. Thus, as part
of the development of this program of research back in 1984, Witt, Elliott, and Martens piloted a list of 15 behaviors of elementary school children with a sample of over 300 teachers. The teachers
were asked to rate each description of a behavior on a 4-point scale ranging from 1 = No problem to 4 = V@y scunepobh. These pilot data on the teachers’ perceptions of problem severity have been
used to guide selection of mild, moderate, and severe problem behaviors in many of our studies.
46 Journal of School Psychology
Tbble 1 Behavior Intervention Rating Scale
You have just read about a child with a classroom problem and a description of an intetventi for improving the problem. Please evaluate the intervention by circling the number which b describes your agreement or disagreement with each statement. You mutt answer each question.
1.
2,
3.
4. 5.
6.
7. a.
9.
10.
11.
12. 13. 14.
15. 16. 17.
la.
19.
20.
21.
22.
23.
24.
This would be an acceptable intervention for the child’s problem behavior. Most teachers would lind this intervention appropriate for behavior problems in addition to the one described. The intervention should prove effective in changing the child’s problem behavior. I would suggest the use of this intervention to other teachers. The child’s behavior problem is severe enough to warrant use of this intervention. Most teachers would fmd this intervention suitable for the behavior problem described. I would be willing to use this in the classroom setting. The intervention would auf result in negative side-effects for the child. The intervention would be appropriate intervention for a variety of children. The intervention is consistent with those I have used in classroom settings. The intervention was a fair way to handle the child’s problem behavior. The intervention is reasonable for the behavior problem described. I like the procedures used in the intervention. This intervention was a good way to handle this child’s behavior problem. Overall, the intervention would be beneficial for the child. The intervention would quickly improve the child’s behavior. The intervention would produce a lasting improvement in the child’s behavior. The intervention would improve the child’s behavior to the point that it would not noticeably deviate from other classmates’ behavior. Soon after using the intervention, the teacher would notice a positive change in the problem behavior. The child’s behavior will remain at an improved level even after the intervention is discontinued. Using the intervention should not only improve the chid’s behavior in the classroom, but also in other settings (e.g., other classrooms, home). When comparing this child with a well-behaved peer before and after use of the intervention, the child’s and the peer’s behavior would be more alike after using the intervention. The intervention should produce enough improvement in the child’s behavior so the behavior no longer is a problem in the classroom. Other behaviors related to the problem behavior also are likely to be improved by the intervention.
1
1
1 1
1
1 1
1
1
1
1 1 1
1 1 1
1
1
1
1
1
1
1
1
2
2
2 2
2
2 2
2
2
2
2 2 2
2 2 2
2
2
2
2
2
2
2
2
3
3
3 3
3
3 3
3
3
3
3 3 3
3 3 3
3
3
3
3
3
3
3
3
4
4
4 4
4
4 4
4
4
4
4 4 4
4 4 4
4
4
4
4
4
4
4
4
5
5
5 5
5
5 5
5
5
5
5 5 5
5 5 5
3
E
C
I:
I
I
I
,
Elliott and Treuting 47
RESULTS
The two subscales of the BIRS, the original IRP and the ERP (composed of the nine effectiveness items) were initially factor-analyzed independently by varimax rotation. Each of these scales yielded a single factor with eigenvalues greater than unity. When analyzed by Pearson correlation, treatment effective- ness and treatment acceptability were found to be highly correlated (. 79). This high correlation demonstrated the need for a nonorthogonal or oblique rota- tion of the entire BIRS. Both the oblique and varimax rotations were per- formed and both yielded a three-factor scale. The oblique rotation, however, provided much cleaner factor loadings, since the intercorrelations were mini- mized. The same items loaded on the same three factors regardless of the type of rotation. The oblique rotation, however, did meet more rigorous criteria in respect to simplicity of structure. Table 2 presents the factor loadings for the varimax and oblique rotations.
The oblique rotation results are used for the following report of findings for this study. The resulting three factors accounted for 73.6% of the total vari- ance. Factor 1, Acceptability, consisted of all 15 IRP items and accounted for 63% of the variance. Finding a single factor for the IRP is consistent with previous factor analysis findings with the IRP scale. Factor loadings were all greater than .50 on this factor and less than .27 on the other two factors.
Factor 2, Effectiveness, included 7 of the 9 items on the ERP, and accounted for an additional 6% of the variance. This factor was related to the effective- ness dimensions of level of change, maintenance and generalization of change, and peer comparisons. All factor loadings were greater than .50 on this factor and less than .30 on the other factors. The highest factor loadings were for the items concerning generalization and maintenance; however, this factor did appear to account for other dimensions as well and was labeled the general Effectiveness factor.
Factor 3, Time of Effectiveness (Time), consisted of two items concerning how effective an intervention was in terms of rate of change (e.g., behavior will quickly improve; soon after implementation a positive change will be noted). This factor accounted for an additional 4.3% of the variance. Both factor loadings were greater than .60 on this factor and less than .27 on the other two factors.
Following the oblique factor rotation, a factor correlation matrix was com- puted for the three BIRS factors. The Acceptability factor was found to have a high positive relation with the Effectiveness factor (. 79) and a moderate corre- lation with the Time factor (.65). The Effectiveness factor and the Time factor correlated moderately (.63).
Partial correlation, with the Time factor partialed from the Effectiveness factor, between Acceptability and Effectiveness was .49. When the Time factor was partialed from both Acceptability and Effectiveness factors, the resulting correlation was .64. This demonstrated the importance of the two-item Time factor.
T ab
le 2
F
ac to
r M
at ri
ce s:
O
rt h
og on
al
an d
O bl
iq u
e R
ot at
io n
s of
t h
e B
eh av
io r
In te
rv en
ti on
R
at in
g S
ca le
( B
IR S
) It
em s
It em
O bl
iq u
e ro
ta ti
on
O rt
h og
on al
r ot
at io
n
fa ct
or
fa ct
or
1 2
3 1
2 3
1.
2.
3.
4.
5.
6.
7.
8.
9.
&
10 .
11 .
12 .
13 .
14 .
15 .
17 .
18 .
20 .
21 .
22 .
23 .
24 .
16 .
19 .
A cc
ep ta
bl e
fo r
pr ob
le m
.7
5 .1
3 .0
9 .7
2 .3
9 .3
3 A
pp ro
pr ia
te f
or o
th er
.6
2 .2
4 -.
21
.6 2
.4 3
.0 2
E ff
ec ti
ve in
c h
an gi
n g
.5 8
.2 6
.1 2
.6 1
.4 5
.3 4
W ou
ld s
u gg
es t t
o ot
h er
s .8
3 .ll
.0
2 .7
8 .3
9 .2
8 P
ro bl
em i
s se
ve re
e n
ou gh
.8
2 -.
04
.0 4
.7 2
.2 6
.2 7
In te
rv en
ti on
is s
u it
ab le
.7
7 .0
9 -.
38
.7 0
.3 4
-. 13
W
il li
n g
to u
se in
c la
ss
1. 00
-.
19
.0 2
.8 3
.1 9
.2 7
N o
n eg
at iv
e si
de e
ff ec
ts
.6 8
.0 5
.1 3
.6 4
.3 0
.3 3
A pp
ro pr
ia te
f or
v ar
ie ty
o f
ch il
dr en
.7
6 .0
5 .0
9 .7
0 .3
2 .3
2 C
on si
st en
t w
it h
w h
at is
u se
d .8
9 -.
13
.0 7
.7 6
.2 0
.3 0
F ai
r w
ay t
o h
an dl
e pr
ob le
m
.8 8
.0 2
.0 8
.8 0
.3 4
.3 4
R ea
so n
ab le
f or
t h
is p
ro bl
em
.8 2
.0 3
.1 4
.7 5
.3 2
.3 8
L ik
ed p
ro ce
du re
s .8
8 .0
6 .0
2 .8
1 .3
6 .2
8 G
oo d
w ay
t o
h an
dl e
pr ob
le m
.7
9 .1
4 .0
8 .7
6 .4
1 .3
3 B
en ef
ic ia
l f or
c h
il d
.7 4
.1 7
.1 0
.7 2
.4 2
.3 4
P ro
du ce
d la
st in
g im
pr ov
em en
t .1
5 .6
9 .1
3 .3
8 .6
9 .3
2 N
ot n
ot ic
ea bl
y de
vi an
t .1
8 .5
2 .3
0 .3
6 .5
6 .4
6 C
h an
ge w
il l
be m
ai n
ta in
ed
-. 12
.9
7 -.
O l
.2 2
.a 4
.1 7
C h
an ge
w il
l ge
n er
al iz
e -.
04
.8 7
-. lO
.2
5 .7
8 .0
8 W
il l
co m
pa re
f av
or ab
ly t
o pe
er s
.1 8
.6 1
.1 7
.3 8
.6 3
.3 5
E n
ou gh
i m
pr ov
em en
t .1
6 .7
2 .0
6 .3
9 .7
2 .2
6 R
el at
ed b
eh av
io rs
i m
pr ov
ed
.2 0
.6 5
.0 7
.4 1
.6 7
.2 7
Q u
ic k
ly i
m pr
ov e
be h
av io
r .1
7 .1
6 .7
3 .2
7 .2
4 .8
2 C
h an
ge s
oo n
a ft
er im
pl em
en te
d .2
6 .1
7 .6
5 .3
4 .2
8 .7
6 E
ig en
va lu
es
15 .2
0 1.
47
1. 01
15
.2 0
1. 47
1.
01
P er
ce n
ta ge
v ar
ia n
ce
63 .3
0 6.
10
4. 20
63
.3 0
6. 10
4.
20
C u
m u
la ti
ve p
er ce
n ta
ge v
ar ia
n ce
63
.3 0
69 .4
0 73
.6 0
63 .6
0 69
.4 0
73 .6
0
Elliott and Treuting 49
The Semantic Differential (SD) had been utilized in a number of previous studies on treatment acceptability and was included in this study for concur- rent validation purposes. The findings, which were consistent with previous studies comparing treatment acceptability measures and the evaluation factor of the Semantic Differential (Kazdin, 1980), were a high correlation between the Acceptability factor and the Evaluation factor of the SD (. 78), and a high correlation with the Effectiveness factor (.67) and a moderate correlation with the Time factor (. 52).
A coefficient alpha was used to determine the reliability of each scale. The total BIRS (24 items) yielded an alpha of .97. The three factors of the BIRS, Acceptability, Effectiveness, and Time, yielded alphas of .97, .92, and .87, respectively.
In summary, these analyses demonstrated that three distinct factors were reliably derived from the BIRS. This scale allowed for the differentiation and measurement of two important pretreatment considerations, acceptability and perceived effectiveness.
DISCUSSION
The factors of treatment effectiveness and treatment acceptability were clearly differentiated as a result of these factor analysis and correlational findings. The factor analysis of the BIRS resulted in three distinct factors, supporting the claim that the scale was measuring different constructs. The first factor, Acceptability, replicated research using the IRP-15 with several different sam- ples of teachers (Elliott, Turco, & Gresham, 1987; Martens et al., 1985). The new items, devised for the measurement of treatment effectiveness, demon- strated effectiveness ratings that consisted of two factors, labeled Effectiveness and Time. Although the Time factor consisted of only two items concerning the rate of effect, the clarity of the factor structure pointed to its distinct aspect of effectiveness. Logic would dictate that the time requirement of an effect would have a salient place in the evaluation of any treatment. In other words, a teacher or parent faced with a problem wants it changed and changed as soon as possible. A high Cronbach alpha also demonstrated the internal consistency of this measure.
Owing to the strength of the factor structure of the BIRS and the reliability of the subscales resulting from each factor, it was concluded that the BIRS provided for meaningful differentiation of treatment effectiveness and treat- ment acceptability. The BIRS appears to have content and construct validity, and to be a useful instrument. There are a number of implications for the use of this scale. First, the BIRS provides a means for advancing an understanding of the construct of social validity The BIRS also adds a new dimension (i.e., perceived effectiveness) with which to evaluate subjective judgments concern- ing treatments. Second, the BIRS can have important value in consultation with both parents and teachers. Both pretreatment and posttreatment mea-
50 Journal of School Psychology
sures will provide insight into consumers’ perceptions of a particular treat-
ment, which may influence outcome. One final implication for the BIRS is its
potential use with other consumers besides teachers (e.g., parents, hospital
staff). Although the wording of the scale items are in terms of a classroom
setting, a minor modification of wording would provide for easy adaptation to
other settings. However, the BIRS is clearly limited to evaluation of treatment
of a child in a group setting. The generalization of these findings are also
limited in regard to the interventions, the problem behaviors, and the research
subjects utilized in this and the Clark and Elliott (1988) investigation.
Since the conclusion of the present study, several investigators have used the
BIRS in small-N treatment studies as a pretreatment acceptability measure
(e.g., Sheridan et al., 1990; Turco & Elliott, 1990). In all cases, the researchers
found that the instrument provided useful information about consumers’
(teachers’) reactions to treatment plans that allowed for adjustments to various
implementation steps and specific therapeutic procedures, thus saving the
investigators much possible frustration and reducing consumer resistance to
the treatments.
The BIRS was developed from a behavioral orientation, but its use is not
limited to behavioral interventions. A review of the items demonstrates that
many types of treatments could be evaluated with this scale. The BIRS is a
new tool for those interested in evaluating consumers’ pretreatment and post-
treatment reactions to interventions.
REFERENCES
Clark, L., & Elliott, S. N. (1988). The influence of treatment strength information OI knowledgeable teachers’ evaluation of two social skills training methods. Pm+ima School Psychology, 3, 241-251.
Elliott, S. N., Turco, T. L., & Gresham, F. M. (1987). Consumers’ and clients pretreatment acceptability ratings of classroom-based group contingencies. Jownal q school Psyclrology, 25, 145-154.
Elliott, S. N., Witt, J. C., Galvin, G. A., & Peterson, R. (1984). Acceptability a positive and reductive behavioral interventions: Factors that influence teacherr decisions. Jouml ofSchool Psychology, 22,353-360.
Kazdin, A. E. (1977). Assessing the clinical or applied significance of behavior chang through social validation Behavbr Modification, 1,427-452.
Kazdii, A. E. (1980). Acceptability of alternative treatments for deviant child behav ior. Journal of Applied Behovin Analysis, 13, 259-273.
Kazdin, A. E., French, N. H., & Sherick, R. B. (1981). Acceptability of alternativ treatments for children: Evaluations of inpatient children, parents, and staff. J&m of Gmrultbg and Clinkal P&o&, 49,900-907.
Martens, B. K., Witt, J. C., Elliott, S. N., & Darveaux, D. X. (1985). Tea& judgments concerning the acceptability of school-based interventions. Pro&w& Psychology: Rmmh andPm&e, 16, 191-198.
Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). Mmnrrmenf of awzn& Urbana: University of Illinois Press.
Elliott and Treuting 51
Sheridan, S. M., Kratochwill, T. R., & Elliott, S. N. (1990). Behavioral consultation with parents and teachers: Delivering treatment for socially withdrawn children at home and school. School Psychology Review, 19,33-52.
Turco, T. L., & Elliott, S. N. (1990). Acceptability and effectiveness of group con- tingencies for improving spelling achievement. J0urna.l OfSchool Psychology, 28, 27-38.
Von Brock, M. B., & Elliott, S. N. (1987). The influence of treatment effectiveness information on the acceptability of classroom interventions. J&m& of School Psycholo-
gy, 25, 131-144. Witt, J. C., Elliott, S. N., & Martens, B. K. (1984). Acceptability of behavioral
interventions used in classrooms: The influences of amount of time, severity of behavior problems, and types of intervention. Behavtial Disordm, 9, 95-104.
Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavim Analysis, 11
203-214.
