Evidence and Non-Evidence Based Treatment Options
tashrifie
1
Rationale and Standards of
Evidence in Evidence-Based
Practice
OLIVER C. MUDFORD, ROB MCNEILL, LISA WALTON, AND KATRINA J. PHILLIPS
What is the purpose of collecting evidence to
inform clinical practice in psychology con-
cerning the effects of psychological or other
interventions? To quote Paul’s (1967) article
that has been cited 330 times before
November 4, 2008, it is to determine the answer
to the question: “What treatment, by whom, is
most effective for this individual with that
specific problem, under which set of circum-
stances?” (p. 111). Another answer is pitched at
a systemic level, rather than concerning indi-
viduals. That is, research evidence can inform
health-care professionals and consumers about
psychological and behavioral interventions
that are more effective than pharmacological
treatments, and to improve the overall quality
and cost-effectiveness of psychological health
service provision (American Psychological
Association [APA] Presidential Task Force on
Evidence-Based Practice, 2006). The most
general answer is that research evidence can be
used to improve outcomes for clients, service
providers, and society in general.
The debate about what counts as evidence
of effectiveness in answering this question
has attracted considerable controversy
(Goodheart, Kazdin, & Sternberg, 2006;
Norcross, Beutler, & Levant, 2005). At one
end of a spectrum, evidence from research on
psychological treatments can be emphasized.
Research-oriented psychologists have pro-
moted the importance of scientific evidence in
the concept of empirically supported treat-
ment. Empirically supported treatments
(ESTs) are those that have been sufficiently
subjected to scientific research and have been
shown to produce beneficial effects in well-
controlled studies (i.e., efficacious), in more
natural clinical environments (i.e., effective),
and are the most cost-effective (i.e., efficient)
(Chambless & Hollon, 1998). The effective
and efficient criteria of Chambless and Hollon
(1998) have been amalgamated under the term
“clinical utility” (APA Presidential Task Force
on Evidence-Based Practice, 2006; Barlow,
Levitt, & Bufka, 1999). At the other end of the
spectrum are psychologists who value clinical
expertise as the source of evidence more
highly, and they can rate subjective impres-
sions and skills acquired in practice as pro-
viding personal evidence for guiding treatment
(Hunsberger, 2007). Kazdin (2008) has
asserted that the schism between clinical
researchers and practitioners on the issue of
evidence is deepening. Part of the problem,
which suggests at least part of the solution, is
that research had concentrated on empirical
evidence of treatment efficacy, but more needs
c01 20 April 2012; 12:43:29
3 Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
to be conducted to elucidate the relevant par-
ameters of clinical experience.
In a separate dimension from the evidence–
experience spectrum have been concerns
about designing interventions that take into
account the uniqueness of the individual cli-
ent. Each of us can be seen as a unique mix
of levels of variables such as sex, age,
socioeconomic and social status, race,
nationality, language, spiritual beliefs or per-
sonal philosophies, values, preferences, level
of education, as well as number of symptoms,
diagnoses (comorbidities), or problem behav-
ior excesses and deficits that may bring us into
professional contact with clinical psycholo-
gists. The extent to which there can be prior
evidence from research or clinical experience
to guide individual’s interventions when these
variables are taken into account is question-
able, and so these individual differences add to
the mix of factors when psychologists delib-
erate on treatment recommendations with an
individual client.
Recognizing each of these three factors
as necessary considerations in intervention
selection, the APA Presidential Task Force on
Evidence-Based Practice (2006, p. 273) pro-
vided its definition: “Evidence-based practice
in psychology (EBPP) is the integration of
the best available research with clinical
expertise in the context of patient characteris-
tics, culture, and preferences.” The task force
acknowledged the similarity of its definition
to that of Sackett, Straus, Richardson,
Rosenberg, and Haynes (2000) when they
defined evidence-based practice in medicine as
“the integration of best research evidence with
clinical expertise and patient values” (p. 1).
Available research evidence is the base or
starting point for EBPP. So, in recommending
a particular intervention from a number of
available ESTs, the psychologist, using clin-
ical expertise with collaboration from the cli-
ent, weighs up the options so that the best
treatment for that client can be selected. As we
understand it, clinical expertise is not to be
considered as an equal consideration to
research evidence, as some psychologists have
implied (Hunsberger, 2007). Like client pref-
erences, the psychologist’s expertise plays an
essential part in sifting among ESTs the clin-
ician has located from searching the evidence.
Best research evidence is operationalized as
ESTs. Treatment guidelines can be developed
following review of ESTs for particular
populations and diagnoses or problem behav-
iors. According to the APA (APA, 2002; Reed,
McLaughlin, & Newman, 2002), treatment
guidelines should be developed to educate
consumers (e.g., clients and health-care sys-
tems) and professionals (e.g., clinical psych-
ologists) about the existence and benefits of
choosing ESTs for specific disorders over
alternative interventions with unknown or
adverse effects. Treatment guidelines are
intended to recommend ESTs, but not make
their use mandatory as enforceable profes-
sional standards (Reed et al., 2002). The
declared status, implications, misunderstand-
ing, and misuse of treatment guidelines based
on ESTs continue to be sources of controversy
(Reed et al., 2002; Reed & Eisman, 2006).
Our chapter examines the issues just raised
in more detail. We start with a review of the
history and methods of determining evidence-
based practice in medicine because the evi-
dence-based practice movement started in that
discipline, and has led the way for other
human services. Psychologists, especially
those working for children and young
people, tend to work collaboratively with
other professionals and paraprofessionals.
Many of these groups of colleagues subscribe
to the evidence-based practice movement
through their professional organizations. We
sample those organizations’ views. The gen-
eralizability to psychology of methods for
evidence-based decision making in medicine
is questionable, and questioned. Next we
examine criteria for determining the strength
of evidence for interventions in psychology.
These criteria are notably different to those
employed in medicine, particularly concern-
ing the relative value to the evidence base of
4 Overview and Foundational Issues
c01 20 April 2012; 12:43:29
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
research in psychology that has employed
methods distinct from medicine (e.g., small-N
design research). The controversies concern-
ing treatment guidelines derived from ESTs
are outlined briefly. The extent to which spe-
cial considerations exist regarding treatment
selection for children and adolescents is then
discussed. Finally, we highlight some of the
aspects of EBPP that require further work by
researchers and clinicians.
EVIDENCE-BASED MEDICINE
Evidence-based practice can be perceived as
both a philosophy and a set of problem-solving
steps, using current best research evidence to
make clinical decisions. In medicine, the
rationale for clinicians to search for best evi-
dence when making diagnostic and treatment
decisions is the desire or duty, through the
Hippocratic Oath, to use the optimal method to
prevent or cure physical, mental, or social
ailments and promote optimal health in indi-
viduals and populations (Jenicek, 2003). This
section of the chapter provides an overview of
the history of evidence-based practice in
medicine (EBM), as well as a description of
the current methodology of EBM. A critical
reflection on the process and evidence used for
EBM is also provided, leading to an intro-
duction of the relevance of evidence-based
practice (EBP) to other disciplines, including
psychological interventions.
History of Evidence-Based
Practice in Medicine
The earliest origins of EBM can be found in
19th-century Paris, with Pierre Louis (1787–
1872). Louis sought truth and medical cer-
tainty through systematic observation of
patients and the statistical analysis of observed
phenomena; however, its modern origins and
popularity are found much more recently in the
advances in epidemiological methods in
the 1970s and 1980s (Jenicek, 2003). One of
the key people responsible for the emergence
and growth of EBM, the epidemiologist Archie
Cochrane, proposed that nothing should be
introduced into clinical practice until it was
proven effective by research centers, and
preferably through double-blind randomized
controlled trials (RCTs). Cochrane criticized
the medical profession for its lack of rigorous
reviews of the evidence to guide decision
making. In 1972, Cochrane reported the results
of the first systematic review, his landmark
method for systematically evaluating the
quality and quantity of RCT evidence for
treatment approaches in clinical practice. In an
early demonstration of the value of this
methodology, Cochrane demonstrated that
corticosteroid therapy, given to halt premature
labor in high-risk women, could substantially
reduce the risk of infant death (Reynolds,
2000).
Over the past three decades, the methods and
evidence used for EBM have been extended,
refined, and reformulated many times. From
the mid-1980s, a proliferation of articles has
instructed clinicians about the process of
accessing, evaluating, and interpreting med-
ical evidence; however, it was not until 1992
that the term evidence-based medicine was
formally coined by Gordon Guyatt and the
Evidence-Based Working Group at McMaster
University in Canada. Secondary publication
clinical journals also started to emerge in the
early to mid-1990s, with the aim of summar-
izing original articles deemed to be of high
clinical relevance and methodological rigor
(e.g., Evidence-Based Medicine, ACP Journal
Club, Evidence-Based Nursing, Evidence-
Based Mental Health).
Various guidelines for EBM have been
published, including those from Evidence-
Based Working Group (Guyatt et al., 2000),
the Cochrane Collaboration, the National
Institute for Clinical Excellence (NICE),
and the British Medical Journal Clinical
Evidence group, to name but a few. David
Sackett, one of the strong proponents and
authors in EBM, describes it as an active
Rationale and Standards of Evidence in Evidence-Based Practice 5
c01 20 April 2012; 12:43:29
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
clinical decision-making process involving
five sequential steps: (1) convert the patient’s
problem into an answerable clinical question;
(2) track down the best evidence to answer
that question; (3) critically appraise the evi-
dence for its validity—closeness to truth;
impact—size of the effect; and applicability—
usefulness in clinical practice; (4) integrate the
appraisal with the practitioner’s clinical
expertise and the patient’s unique characteris-
tics, values, and circumstances; and (5) evaluate
the change resulting from implementing the
evidence in practice, and seek ways to improve
(Sackettetal.,2000,pp.3–4).Theguidelinesfor
EBM are generally characterized by this sort of
systematic process for determining the level
of evidence for treatment choices available to
clinicians, while at the same time recognizing
the unique characteristics of the individual’s
characteristics, situation, and context.
It is not difficult to see the inherent benefits of
EBM, and health professionals have been quick
to recognize the potential benefit of adopting it
as standard practice. Citing a 1998 survey of
UK general practitioners (GPs), Sackett and
colleagues (2000) wrote that most reported
using search techniques, with 74% accessing
evidence-based summaries generated by
others, and 84% seeking evidence-based prac-
tice guidelines or protocols. The process of
engaging in EBM requires some considerable
understanding of research and research
methods, and there is evidence that health
professionals struggle to use EBM in their
actual practice. For example, Sackett et al.
(2000) found that GPs had trouble using the
rules of evidence to interpret the literature, with
only 20% to 35% reporting that they understood
appraising tools described in the guidelines.
Clinicians’ ability to practice EBM also may be
limited by lack of time to master new skills and
inadequate access to instant technologies
(Sackett et al., 2000). In addition to these
practical difficulties, there have also been some
criticisms of EBM’s dominant methodology.
An early criticism of EBM, and nearly all
EBP approaches, is that it appears to give
greatest weight to science with little attention
to the “art” that also underlies the practice of
medicine, nursing, and other allied health
professions. For example, Guyatt and col-
leagues cited attention to patients’ humanistic
needs as a requirement for EBM (Evidence-
Based Medicine Working Group, 1992).
Nursing and other health-care disciplines note
that EBP must be delivered within a context of
caring to achieve safe, effective, and holistic
care that meets the needs of patients (DiCenso,
Cullum, Ciliska, & Guyatt, 2004).
Evidence-based practice is also criticized as
being “cookbook care” that does not take the
individual into account. Yet a requirement of
EBP is that incorporating research evidence
into practice should consistently take account
of the patient’s unique circumstances, prefer-
ences, and values. As noted by Sackett et al.
(2000), when these three elements are inte-
grated to inform clinical decision making,
“clinicians and patients form a diagnostic and
therapeutic alliance which optimizes clinical
outcomes and quality of life” (p. 1).
One of the key issues in the use of EBM is
the debate around what constitutes best evi-
dence from the findings of previous studies and
how these various types of evidence are
weighted, or even excluded, in the decision-
making process. The following section first
outlines the way in which evidence currently
tends to be judged and then provides a more in-
depth analysis of each type of evidence.
Levels and Types of Evidence
Partly through the work of Archie Cochrane,
the quality of health care has come to be
judged in relation to a number of criteria:
efficacy (especially in relation to effective-
ness), efficiency, and equity. Along with
acceptability, access, and relevance, these
criteria have been called the “Maxwell Six”
and have formed the foundation of decision
making around service provision and funding
in the United Kingdom’s National Health
Service (Maxwell, 1992). Other health systems
6 Overview and Foundational Issues
c01 20 April 2012; 12:43:29
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
around theworld havealso adopted these criteria
to aid in policy decision making, in both publicly
and privately funded settings. Despite this there
has been a tendency for evidence relating to
effectiveness to dominate the decision-making
process in EBM and, in particular, effectiveness
demonstrated through RCTs.
There has been considerable debate about
the ethical problems arising from clinicians
focusing too much on efficacy and not enough
on efficiency when making decisions about
treatment (Maynard, 1997). In any health
system with limited resources, the decision to
use the most effective treatment rather than the
most efficient one has the potential to impact
on the likelihood of having sufficient resources
to deliver that treatment, or any other, in the
future. By treating one person with the most
effective treatment the clinician may be taking
away the opportunity to treat others, especially
if that treatment is very expensive in relation to
its effectiveness compared to less expensive
treatment options.
According to current EBM methods, the
weight that a piece of evidence brings to
the balance of information used to make a
decision about whether a treatment is supported
by evidence can be summarized in Table 1.1.
Although there are subtle variations in this
hierarchy between different EBM guidelines
and other publications, they all typically start
with systematic reviews of RCTs at the top and
end with expert opinion at the bottom.
Before discussing systematic reviews, meta-
analyses, and clinical guidelines, it is import-
ant to understand the type of study that these
sources of evidence are typically based on:
the RCT. RCTs are a research design in
which the participants are randomly assigned
to treatment or control groups. RCTs are
really a family of designs, with different
components such as blinding (participants
and experimenter), different randomization
methods, and other differences in design.
Analysis of RCTs is quantitative, involving
estimation of the statistical significance or
probability of the difference in outcomes
observed between the treatment and control
groups in the study. The probabilities obtained
are an estimate of the likelihood of that size
difference, or something larger, existing in the
population.
There are a number of international data-
bases that store and provide information
from RCTs, including the Cochrane Central
Register of Controlled Trials (CENTRAL;
mrw.interscience.wiley.com/cochrane/cochrane
_clcentral_articles_fs.html), OTseeker (www
.otseeker.com), PEDro (www.pedro.fhs.usyd
.edu.au), and the Turning Research Into Prac-
tice (TRIP; www.tripdatabase.com) database.
Another effort to increase the ease with which
clinicians can access and use evidence from
RCTs has come through efforts to standardize
the way in which they are reported, such
as the CONSORT Statement and other efforts
from the CONSORT Group (www.consort-
statement.org).
The major strength of well-designed RCTs
and other experimental designs is that the
researcher has control over the treatment given
to the experimental groups, and also has
control over or the ability to control for any
confounding factors. The result of this control
TABLE 1.1 Typical Hierarchy of Evidence for EBM
Level of
Evidence Type of Evidence
1 (High) Systematic reviews or meta-analysis of
randomized controlled trials (RCTs)
OR
Evidence-based clinical practice guidelines
based on systematic reviews of RCTs
2 At least one well-designed RCT
3 Well-designed quasi-experimental studies
4 Well-designed case control and cohort
studies
5 Systematic reviews of descriptive and
qualitative studies
6 A single descriptive or qualitative study
7 (Low) The opinion of authorities and/or expert
committees
Source: Adapted from Melnyk and Fineout-Overholt
(2005, p. 10)
Rationale and Standards of Evidence in Evidence-Based Practice 7
c01 20 April 2012; 12:43:29
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
is the ability, arguably above all other designs,
to infer causation—and therefore efficacy—
from the differences in outcomes between the
treatment and control groups.
RCTs, and other purely experimental
designs, are often criticized for having poor
ecological validity, where the conditions of
the experiment do not match the conditions
or populations in which the treatment might
be delivered in the real world. Low ecological
validity can, although does not necessarily,
lead to low external validity where the findings
of the study are not generalizable to other
situations, including the real world. Another
issue with RCTs is that they are relatively
resource intensive and this leads to increased
pressure to get desired results or to suppress
any results that do not fit the desired outcome
(Gluud, 2006; Simes, 1986).
In social sciences, RCTs are not always
possible or advisable, so many systematic
reviews and meta-analyses in these areas have
less restrictive inclusion criteria. Commonly,
these include studies that compare a treated
clinical group with an untreated or attention
control group (Kazdin & Weisz, 1998;
Rosenthal, 1984). In these situations, a quasi-
experimental design is often adopted. There is
also evidence that RCTs are not necessarily any
more accurate at estimating the effect of agiven
treatment than quasi-experimental and obser-
vational designs, such as cohort and case con-
trol studies (Concato, Shah, & Horwitz, 2008).
Systematic reviews provide a summary of
evidence from studies on a particular topic. For
EBM this typically involves the formation of
an expert committee or panel, followed by the
systematic identification, appraisal, and syn-
thesis of evidence from relevant RCTs relating
to the topic (Melnyk & Fineout-Overholt,
2005). The result of the review is usually some
recommendation around the level of empirical
support from RCTs for a given diagnostic tool
or treatment. The RCT, therefore, is seen as the
gold standard of evidence in EBM.
Systematic review and meta-analysis are
closely related EBP methods to evaluate
evidence from a body of sometimes contra-
dictory research to assess treatment quality.
There is a great deal of overlap between
methods, stemming in part from their different
origins. The systematic review arises from
EBM and the early work of Cochrane and more
recently Sackett and colleagues (2000). Meta-
analysis originated in the 1970s, with the
contributions of Glass (1976) in education and
Rosenthal (1984) in psychology being central.
Meta-analyses are a particular type of sys-
tematic review. In a meta-analysis, measures
of the size of treatment effects are obtained
from individual studies. The effect sizes from
multiple studies are then combined using a
variety of techniques to provide a measure of
the overall effect of the treatment across all
of the participants in all of the studies included
in the analysis.
Both approaches use explicit methods to
systematically search, critically appraise for
quality and validity, and synthesize the litera-
ture on a given issue. Thus, a key aspect is the
quality of the individual studies and the jour-
nals in which they appear. Searches ideally
include unpublished reports as well as pub-
lished reports to counteract the “file drawer”
phenomenon: Published findings as a group
may be less reliable than they seem because
studies with statistically nonsignificant find-
ings are less likely to be published (Rosenthal,
1984; Sackett et al., 2000).
The principal difference between systematic
review and meta-analysis is the latter includes
a statistical method for combining results of
individual studies that produces a larger sam-
ple size, reduces random error, and has greater
power to determine the true size of the inter-
vention effect. Not only do these methods
compensate for the limited power of individual
studies resulting in a Type I error, the failure to
detect an actual effect when one is present,
they can also reconcile conflicting results
(Rosenthal, 1984; Sackett et al., 2000).
The criticisms of meta-analysis mostly relate
to the methodological decisions made during
the process of conducting a meta-analysis,
8 Overview and Foundational Issues
c01 20 April 2012; 12:43:29
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
often reducing the reliability of findings. This
can result in meta-analyses of the same topic
yielding different effect sizes (i.e., summary
statistic), with these differences stemming
from differences in the meta-analytic method
and not just differences in study findings
(Flather, Farkouh, Pogue, & Yusuf, 1997).
Methodological decisions that can influence
the reliability of the summary statistic pro-
duced from a meta-analysis include the coding
system used to analyze studies, how inclusive
the study selection process was, the outcome
measures used or accepted, and the use of raw
effect sizes or adjusted sample sizes (Flather
et al., 1997). Some meta-analyses are con-
ducted without using a rigorous systematic
review approach, and this is much more likely
to produce a mathematically valid but clinic-
ally invalid conclusion (Kazdin & Weisz,
1998; Rosenthal, 1984).
There are also some criticisms of the
meta-analysis approach, specifically from
child and adolescent psychology literature.
Meta-analyses can obscure qualitative differ-
ences in treatment execution such as investi-
gator allegiance (Chambless & Hollon, 1998;
Kazdin & Weisz, 1998) and are limited by
confounding among independent variables such
as target problems, which tend to be more evi-
dent with certain treatment methods (Kazdin &
Weisz, 1998).
Evidence-based clinical practice guidelines
are also included in this top level of evidence,
with the caveat that they must be primarily
based on systematic reviews of RCTs. The
purpose of these practice guidelines is to pro-
vide an easy-to-follow tool to assist clinicians
in making decisions about the treatment that is
most appropriate for their patients (Straus,
Richardson, Glasziou, & Haynes, 2005).
There are some issues with the publication
and use of evidence-based clinical practice
guidelines. One problem is that different
groups of experts can review the same data and
arrive at different conclusions and recom-
mendations (Hadorn, Baker, Hodges, & Hicks,
1996). Some guidelines are also criticized for
not being translated into tools for everyday
use. One of the major drawbacks of clinical
guidelines is that they are often not updated
frequently to consider new evidence (Lohr,
Eleazer, & Mauskopf, 1998).
In addition to individual researchers and
groups of researchers, there are a large number
of organizations that conduct systematic
reviews, publish clinical practice guidelines,
and publish their findings in journals and in
databases on the Internet, including the
Cochrane Collaboration (www.cochrane.org/),
the National Institute for Clinical Evidence
(NICE; www.nice.org.uk/), the Joanna Briggs
Institute (www.joannabriggs.edu.au/about
/home.php), and the TRIP database (www
.tripdatabase.com/index.html). For example,
the Cochrane Collaboration, an organization
named after Archie Cochrane, is an inter-
national network of researchers who conduct
systematic reviews and meta-analyses and
provide the results of these to the research,
practice, and policy community via the
Cochrane Library.
The overall strengths of systematic reviews,
meta-analyses, and clinical practice guidelines
relate partly to the nature of the methods used
and partly to the nature of RCTs, which were
discussed earlier. The ultimate goal of scien-
tific research is to contribute toward a body of
knowledge about a topic (Bowling, 1997).
Systematic reviews and clinical practice
guidelines are essentially trying to summarize
the body of knowledge around a particular
treatment, which is something that would
otherwise take a clinician a very long time to
do on their own, so it is easy to see the value in
a process that does this in a rigorous and sys-
tematic way.
There are a number of overall weaknesses
for this top level of evidence for EBM. For
example, the reliability of findings from sys-
tematic reviews has been found to be sensitive
to many factors, including the intercoder reli-
ability procedures adopted by the reviewers
(Yeaton & Wortman, 1993). It is possible for
different people using the same coding system
Rationale and Standards of Evidence in Evidence-Based Practice 9
c01 20 April 2012; 12:43:29
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
to come up with different conclusions about
the research evidence for a particular treat-
ment. The selection of comparable studies is
also a major issue, and especially in matching
comparison groups and populations. The con-
trol and treatment groups in RCTs and other
studies are usually not identical and this raises
issues, particularly in the use of meta-analyses,
where the treatment effects are being com-
bined (Eysenck, 1994). The mixing of popu-
lations can also cause problems, where the
treatment may only be effective in one or more
specific populations but the review has
included studies from other populations where
the treatment is not effective. The overall
effect of mixing these populations is to
diminish the apparent effectiveness of the
treatment under review (Eysenck, 1994).
Other issues with this top level of evidence
relate to the information not included. Once a
review or guideline is completed it must be
constantly updated to check for new evidence
that may change the conclusions. There are
also issues around the information not
included in the review process, including non-
RCT studies, gray literature, such as technical
reports and commissioned reports, and
unpublished studies. Studies that do not get
into peer-reviewed journals, and therefore
usually get excluded from the review process,
are often those that did not have significant
results. The effect of this is that reviews will
often exaggerate the effectiveness of a treat-
ment through the exclusion of studies where
the treatment in question was found to be
ineffective. For this reason there has been a
recent move to introduce what is called the
“failsafe N” statistic. This is the hypothetical
number of unpublished (or hidden) studies
showing, on average, no effect that would be
required to overturn the statistically significant
effects found from review of the published (or
located) studies results (Becker, 2006).
Quasi-experimental designs are the same
as RCTs but the groups are not randomly
assigned, there is no control group, or they lack
one or more of the other characteristics of an
RCT. In many situations randomization of
participants or having a control group is not
practically and/or ethically possible. The
strengths of quasi-experimental designs come
from the degree of control that the research has
over the groups in the study, and over possible
confounding variables. A well-designed quasi-
experimental design has the important char-
acteristic of being able to infer some degree
of causation from differences in outcomes
between study groups.
Case control studies identify a population
with the outcome of interest (cases) and a
population without that outcome (controls),
then collects retrospective data to try to
determine their relative exposure to factors of
interest (Grimes & Schulz, 2002). There are
numerous strengths of case control studies.
They have more ecological validity than
experimental studies and they are good for
health conditions that are very uncommon
(Grimes & Schulz, 2002). There is a relatively
clear temporal sequence, compared to lower
level evidence, which allows some degree of
causality to be inferred (Grimes & Schulz,
2002). They are also relatively quick to do,
relatively inexpensive, and can look at mul-
tiple potential causes at one time (Grimes &
Schulz, 2002). The weaknesses of case control
studies include the inability to control poten-
tially confounding variables except through
statistical manipulations and the reliance on
participants to recall information or retro-
spectively collating information from existing
data; the choice of control participants is also
difficult (Grimes & Schulz, 2002; Wacholder,
McLaughlin, Silverman, & Mandel, 1992). All
of this often leads to a lot of difficulty in cor-
rectly interpreting the results of case control
studies (Grimes & Schulz, 2002).
Cohort studies are longitudinal studies
where groups are divided in terms of whether
they receive or do not receive a treatment
or exposure of interest, and are followed
over time to assess the outcomes of interest
(Roberts & Yeager, 2006). The strengths of
cohort studies include the relatively clear
10 Overview and Foundational Issues
c01 20 April 2012; 12:43:29
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
temporal sequence that can be established
between the introduction of an intervention
and any subsequent changes in outcome vari-
ables, making the establishment of causation
possible, at least to some degree. The limita-
tions of cohort studies include the difficulty in
controlling extraneous variables, leading to a
relatively limited ability to infer causality.
They are also extremely resource intensive and
are not good where there is a large gap between
treatment and outcome (Grimes & Schulz,
2002).
Descriptive studies involve the description of
data obtained relating to the characteristics
of phenomena or variables of interest in a par-
ticular sample from a population of interest
(Melnyk & Fineout-Overholt, 2005). Correl-
ational studies are descriptive studies where the
relationship between variables is explored.
Qualitative studies collect nonnumeric data,
such as interviews and focus groups, with the
analysis usually involving some attempt at
describing or summarizing certain phenomena
in a sample from a population of interest
(Melnyk & Fineout-Overholt, 2005). Descrip-
tive studiesingeneralsit low downontheladder
of evidence quality due to their lack of control
by the researcher and therefore their inability to
infer causation between treatment and effect.
On the other hand, much research is only feas-
ible to conduct using this methodology.
Qualitative researchin health services mostly
arose from a desire to get a deeper under-
standing of what the quantitative research
finding meant for the patient and provider
(Mays & Pope, 1996). It asks questions such as
“How do patients perceive . . . ?” and “How
do patients value the options that are offered?”
Expert opinion is material written by rec-
ognized authorities on a particular topic. Evi-
dence from these sources has the least weight
in EBM, although to many clinicians the views
of experts in the field may hold more weight
than the higher level evidence outlined above.
Despite its criticisms and methodological
complexity, EBM is seen by most clinicians
as a valid and novel way of reasoning and
decision making. Its methods are widely dis-
seminated through current clinical education
programs at all levels throughout the world. As
proposed by Gordon Guyatt and colleagues
in 1992, EBM has led to a paradigm shift in
clinical practice (Sackett et al., 2000).
CURRENT STATUS OF EBP
MOVEMENTS ACROSS HEALTH
AND EDUCATION PROFESSIONS
Evidence-based practice was initially in the
domain of medicine, but now most human
service disciplines subscribe to the principles
of EBP. For example, EBP or the use of
empirically supported treatment is recom-
mended by the APA, Behavior Analyst Certi-
fication Board (BACB), American Psychiatric
Association, National Association of Social
Workers, and General Teaching Council for
England. For U.S. education professionals,
EBP has been mandated by law. The No Child
Left Behind Act of 2001 (NCLB; U.S.
Department of Education, 2002) was designed
to make the states, school districts, principals,
and teachers more answerable for the per-
formances shown by the students for whom
they were providing education services. Along
with an increase in accountability, the NCLB
requires “schoolwide reform and ensuring the
access of children to effective, scientifically
based instructional strategies and challenging
academic content” (p. 1440). The advent of the
NCLB and the resulting move toward EBP
occurred because, despite there being research
conducted on effective and efficient teaching
techniques (e.g., Project Follow Through:
Bereiter & Kurland, 1981; Gersten, 1984) and
a growing push for accountability from the
public (Hess, 2006), this seldom was translated
into actual practice. Education appears to have
been particularly susceptible to implementing
programs that were fads, based on little more
than personal ideologies and good marketing.
Much money and time has been lost by school
districts adopting programs that have no
Rationale and Standards of Evidence in Evidence-Based Practice 11
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
empirical support for their effectiveness, such
as the adoption of known ineffective substance
abuse prevention programs (Ringwalt et al.,
2002) or programs that are harmful for some
students and their families, such as facilitated
communication (Jacobson, Mulick, &
Schwartz, 1995). This leads not only to
resource wastage, but an even greater cost in
lost opportunities for the students involved.
It seems like common sense for EBP to be
adopted by other disciplines, as it is difficult to
understand why reputable practitioners, ser-
vice providers, and organizations would not
want to provide interventions that have been
shown to be the most effective and efficient.
Yet, despite this commonsense feeling, the
codes of conduct and ethical requirements, and
mandated laws, many disciplines (including
medicine) have found it difficult to bridge the
gap between traditional knowledge-based
practice and the EBP framework (Greenhalgh,
2001; Stout & Hayes, 2004). Professions such
as social work (Roberts & Yeager, 2006;
Zlotnick & Solt, 2006), speech language
therapy (Enderby & Emerson, 1995, 1996),
occupational therapy (Bennett & Bennett,
2000), and education (Shavelson & Towne,
2002; Thomas & Pring, 2004) have all
attempted to overcome the difficulties that
have arisen as they try to move EBP from a
theoretical concept to an actual usable tool
for everyday practitioners. Although some of
these disciplines have unique challenges to
face, many are common to all.
Many human services disciplines have
reported that one of the major barriers to the
implementation of EBP is the lack of sound
research. For example, speech language ther-
apy reports difficulties with regard to the
quality and dissemination of research. A
review of the status of speech language therapy
literature by Enderby and Emerson (1995,
1996) found that there was insufficient quality
research available in most areas of speech
language therapy; however, they did find
that those areas of speech therapy that
were associated with the medical profession
(e.g., dysphasia and cleft palate) were more
likely to have research conducted than those
associated with education (e.g., children with
speech and language disorders and populations
with learning disabilities). There continues to
be a lack of agreement on speech language
therapies effectiveness (Almost & Rosenbaum,
1998; Glogowska, Roulstone, Enderby, &
Peters, 2000; Robertson & Weismer, 1999).
In addition to the lack of research, Enderby
and Emerson were also concerned about the
manner in which health resources were being
allocated for speech language therapists. They
found that of the resources allocated to speech
language therapy approximately 70% were
being used with children with language
impairments, despite there being limited
quality evidence of the effectiveness of speech
language therapy with the population at that
time. They also identified that dysarthria was
the most commonly acquired disorder, but had
very little research outside of the Parkinson’s
disease population. So, again, many resources
have been allocated to programs that have no
evidence of effectiveness. This questionable
allocation of resources is seen in other fields
also. For example, a number of studies have
found that people seeking mental health
treatments are unlikely to receive an inter-
vention that would be classified as EBP and
many will receive interventions that are inef-
fective (Addis & Krasnow, 2000; Goisman,
Warshaw, & Keller, 1999).
A number of organizations and government
agencies within a variety of fields try to
facilitate the much-needed research. Examples
in education include the United States
Department for Children, Schools, and Fam-
ilies (previously Department for Education
and Skills), the National Research Council
(United Kingdom), and National Teacher
Research Panel (United Kingdom). Similarly,
in social work the National Association of
Social Workers (United States) and the Society
for Social Work and Research (United States)
both facilitate the gathering of evidence to
support the use of EBP within their field;
12 Overview and Foundational Issues
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
however, even if these organizations generate
and gather research demonstrating the effect-
iveness of interventions, this does not always
translate into the implementation of EBP.
As mentioned previously, resource allocation
does not necessarily go to those treatments
with evidence to support them due to a lack of
effective dissemination on the relevant topics.
Despite medicine being the profession with the
longest history of EBP, Guyatt et al. (2000)
stated that many clinicians did not want to use
original research or they fail to do so because
of time constraints and/or lack of understand-
ing on how to interpret the information. Rosen,
Proctor, Morrow-Howell, and Staudt (1995)
found that social workers also fail to consider
empirical research when making research
decisions. They found that less than 1% of the
practice decisions were justified by research.
Although the lack of research-based decision
making is concerning, many professions are
attempting to disseminate information in a
manner that is more user friendly to its prac-
titioners. Practice or treatment guidelines have
been created to disseminate research in a
manner that will facilitate EBP. These guide-
lines draw on the empirical evidence and
expert opinion to provide specific best-practice
recommendations on what interventions or
practices are the most effective/efficient for
specific populations (Stout & Hayes, 2004).
There are a number of guideline clearing-
houses (National Guidelines Clearinghouse
[www.guideline.gov/] and What Works
Clearinghouse [ies.ed.gov/ncee/wwc/]). Asso-
ciations and organizations may also offer
practice guidelines (e.g., American Psychiatric
Association).
Although there appears to have been
emphasis placed on EBP, there is still much
work to be done in most human service fields,
including medicine, to ensure that there is
appropriate research being conducted, that this
research is disseminated to the appropriate
people, and that the practitioners then put it
into practice. Reilly, Oates, and Douglas
(2004) outlined a number of areas that the
National Health Service Research and Devel-
opment Center for Evidence-Based Medicine
had identified for future development. They
suggest that there is a need for a better
understanding of how practitioners seek
information to inform their decisions, what
factors influence the inclusion of this evidence
into their practice, and the value placed, both
by patients and practitioners, on EBP. In add-
ition, there is a need to develop information
systems that facilitate the integration of evi-
dence into the decision-making processes for
practitioners and patients. They also suggest
that there is a need to provide effective and
efficient training for frontline professionals in
evidence-based patient care. Finally, they
suggest that there is simply a need for more
research.
EVIDENCE IN PSYCHOLOGY
Although RCT research is still held up as the
gold standard of evidence in medical science,
in other clinical sciences, especially psych-
ology, best research evidence has been less
focused on RCTs as being the only scientific-
ally valid approach. Randomized controlled
trials are still held in high regard in clinical
psychology, but it has long been recognized
that alternative designs may be preferred and
still provide strong evidence, depending on the
type of intervention, population studied, and
patient characteristics (APA Presidential Task
Force on Evidence-Based Practice, 2006;
Chambless & Hollon, 1998).
Research Methods Contributing to
Evidence-Based Practice in Psychology
We have described and discussed RCTs in the
previous section on EBM. The issues con-
cerning RCTs and their contribution to the
clinical psychology evidence base are similar.
The inclusion of evidence of treatment efficacy
and utility from research paradigms other than
RCT methods and approximations thereto has
Rationale and Standards of Evidence in Evidence-Based Practice 13
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
been recommended for clinical psychology
since the EBPP movement commenced
(Chambless & Hollon, 1998). The APA
Presidential Task Force on Evidence-Based
Practice (2006) findings support the use of
qualitative studies, single-case or small-N
experimental designs, and process-outcome
studies or evaluations, in addition to the RCT
and systematic review techniques employed in
EBM. A more in-depth discussion of some of
these methods will be provided.
Psychologists and members of other profes-
sions that are more influenced by social sci-
ence research methods than medicine may
consider that findings from qualitative
research add to their EBP database. Although
qualitative research is not accepted as legit-
imate scientific study by many psychologists,
and not usually taught to clinical psychologists
in training, it may well have its place in
assessing variables regarding clinical exper-
tise and client preferences (Kazdin, 2008).
Research questions addressable through
qualitative research relevant to EBPP are
similar to those mentioned for EBM.
Another important source of scientific evi-
dence for psychologists, educators, and other
nonmedical professionals is that derived
from small-N research designs, also known as
single-case, single-subject, or N ¼ 1 designs. These alternative labels can confuse, espe-
cially since some single-case designs
include more than one subject (e.g., multiple
baseline across subjects usually include three
or more participants). The most familiar
small-N designs are ABAB, multiple baseline,
alternating treatments, and changing criterion
(Barlow & Hersen, 1984; Hayes, Barlow, &
Nelson-Gray, 1999; Kazdin, 1982). Data from
small-N studies have been included as sources
of evidence, sometimes apparently equivalent
to RCTs (Chambless & Hollon, 1998), or at a
somewhat lower level of strength than RCTs
(APA, 2002).
Strength of evidence from small-N designs.
Small-N designs can be robust in terms of
controlling threats to internal validity;
however, external validity has often been
viewed as problematic. This is because par-
ticipants in small-N studies are not a randomly
selected sample from the whole population of
interest. The generality of findings from small-
N studies is established by replication across
more and more members of the population of
interest in further small-N studies. A hypo-
thetical example of the process of determining
the generality of an intervention goes like this:
(a) A researcher shows that a treatment works
for a single individual with a particular type of
diagnosis or problem; (b) Either the same or
another researcher finds the same beneficial
effects with three further individuals; (c)
Another researcher reports the same findings
with another small set of individuals; and so
on. At some point, sufficient numbers of indi-
viduals have been successfully treated using
the intervention that generality can be claimed.
Within a field such as a particular treatment for
a particular disorder, small-N studies can be
designed so results can be pooled to contribute
to an evidence base larger than N ¼ 1 to 3 or 4 (Lord et al., 2005).
Even if there is an evidence base for an
intervention from a series of small-N studies,
every time another individual receives the
same treatment, the clinician in scientist-
practitioner role evaluates the effects of the
intervention using small-N design methods.
Thus, every new case is clinical research to
determine the efficacy and effectiveness of this
treatment for that individual.
Clinical psychologists can be cautioned that
physicians and medical researchers have a
similar-sounding name for an experimental
design with their “N of 1 trials.” The definition
of N of 1 trials can vary somewhat but typically
they are described as “randomised, double
blind multiple crossover comparisons of an
active drug against placebo in a single patient”
(Mahon, Laupacis, Donner, & Wood, 1996,
p. 1069). These N of 1 trials are continued until
the intervention, usually a drug, in question
shows consistently better effects than its
comparison treatment or control condition.
14 Overview and Foundational Issues
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
Then the intervention is continued, or discon-
tinued if there were insufficient beneficial
effects. The N of 1 trials can be considered
top of the hierarchy of evidence in judging
strength of evidence for individual clinical
decisions, higher even than systematic reviews
of RCTs (Guyatt et al., 2000). This is because
clinicians do not have to generalize findings
of beneficial effects of the treatment to this
patient from previous researched patients.
Consistent beneficial findings from many N
of 1 trials add up to an “N of many” RCT,
thus providing evidence of the generality of
findings. An N of 1 trial is quite similar to
the alternating treatments small-N design
employed in clinical psychology, especially
applied behavior analysis; however, they are
not identical and the differences may be
overcome for some psychosocial interventions
only with careful planning, but many inter-
ventions could not be assessed by fitting them
into an N of 1 trial format. Further details are
beyond the scope of this chapter; however,
close study of the methodological require-
ments of both N of 1 designs and small-N
designs can show the similarities and differ-
ences (Barlow & Hersen, 1984; Guyatt et al.,
2000; Hayes et al., 1999; Kazdin, 1982; Mahon
et al., 1996).
Some state that small-N designs may be most
suitable for evaluating new treatments (Lord
et al., 2005), others that single-case studies are
most suitable for determining treatment effects
for individuals (APA Presidential Task Force
on Evidence-Based Practice, 2006). We do not
disagree with either, except to point out that it
has been argued that small-N studies can con-
tribute much more to EBPP than these two
advantages. In the following section, we
review how ESTs can be determined from an
evidence base consisting entirely of small-N
studies.
Criteria for Assessing Efficacy
Lonigan, Elbert, and Johnson (1998) tabulated
criteria for determining whether an intervention
for childhood disorders could be considered
well-established (i.e., efficacious) or probably
efficacious (i.e., promising). For the former,
they recommended at least two well-conducted
RCT standard studies by independent research
teams or a series of independent well-designed
small-N studies with at least nine participants
carefully classified to the diagnostic category
of interest showing that the intervention was
better than alternative interventions. The
availability of treatment manuals was recom-
mended. For “promising” treatments, the
criteria were relaxed to allow nonindependent
RCTs,comparingtreatment tonotreatment,ora
minimum of three small-N studies. These cri-
teria followed those established at the time for
psychological therapies in general (Chambless
& Hollon, 1998).
We have discussed the determination of
empirical support from RCTs already. The
next sections will examine how evidence
is derived from small-N studies: First,
how evidence is assessed from individual
research reports; and second, how the evi-
dence can be combined from a group of
research articles addressing the same topic
of interest.
Evaluating Evidence From Small-N
Research Designs
How do those assessing the efficacy of a treat-
ment from small-N designs measure the
strength of the design for the research purpose
and whether, or to what extent, a beneficial
effect has been demonstrated? Chambless and
Hollon (1998) recommended reviewers to rate
single-case studies on the stability of their
baselines, use of acceptable experimental
designs, such as ABAB or multiple base-
line designs, and visually estimated effects.
Baselines need not always be stable to provide
an adequate control phase; there are other valid
designs (e.g., changing criterion, multielement
experimental designs); and visual estimates
of effects are not necessarily reliable
(Cooper, Heron, & Heward, 2007). Validity
Rationale and Standards of Evidence in Evidence-Based Practice 15
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
and, probably, reliability of reviews using only
Chambless and Hollon’s (1998) criteria could
not be assured.
Attempts to improve reliability and validity
of judgments about the value of small-N design
studies have included establishing more well-
defined criteria. Quite detailed methods for
evaluating small-N studies have been pub-
lished. For example, Kratochwill and Stoiber
(2002) describe the basis of a method endorsed
by the National Association of School Psych-
ologists (NASP) designed for evaluating mul-
tiple research reports that used single-case
designs for establishing evidence-based rec-
ommendations for interventions in educational
settings. The system is more inclusive of
design variations beyond those recommended
by Chambless and Hollon (1998), and
reviewers are instructed to code multiple
variables, including calculating effects sizes
from graphical data. The coding form extended
over 28 pages. Shernoff, Kratochwill, and
Stoiber (2002) illustrated the assessment pro-
cedure and reported that, following extensive
training and familiarity with the coding man-
ual, they achieved acceptable agreement
among themselves. They stated that the pro-
cess took 2 hours for a single study, although,
so our own graduate students report, it takes
much longer if multiple dependent and inde-
pendent variables or hundreds of data points
had been features of the research article with
which they chose to illustrate the NASP pro-
cedure. The NASP method was constructively
criticized by Levin (2002), and is appa-
rently being revised and expanded further
(Kratochwill, 2005).
Meanwhile, others have developed what
appear to be even more labor intensive
methods for attempting to evaluate objectively
the strength of evidence from a series of small-
N designs. As an example of a more detailed
method, Campbell (2003) measured every data
point shown on published graphs from 117
research articles on procedures to reduce
problem behaviors among persons with aut-
ism. Each point was measured by using
dividers to determine the distance between the
point and zero on the vertical axis. Campbell
calculated effect sizes for three variables: mean
baseline reduction, percentage of zero data
points, and percentage of nonoverlapping data
points. Use of these statistical methods may
have been appropriate considering the types of
data Campbell examined; nonzero baselines of
levels of problem behavior followed by inter-
vention phases in which the researchers’ goal
was to produce reduction to zero (see Jensen,
Clark, Kircher, & Kristjansson, 2007, for crit-
ical review of meta-analytic tools for small-N
designs). Nevertheless, the seemingly arduous
nature of the task and the lack of generaliz-
ability of the computational methods to
reviewing interventions designed to increase
behaviors are likely to mitigate wider accept-
ance of Campbell’s (2003) method.
A final example of methods to evaluate the
strength of an evidence base for interventions
is that outlined by Wilczynski and Christian
(2008). They describe the National Standards
Project (NSP), which was designed to deter-
mine the benefits or lack thereof of a wide
range of approaches for changing the behav-
iors of people with autism spectrum disorders
(ASD) aged up to 21 years. Their methods of
quantitative review enabled the evidence from
group and small-N studies to be integrated.
Briefly, and to describe their method for
evaluating small-N studies only, their review
rated research articles based on their scientific
merit first. Articles were assessed to deter-
mine whether they were sufficiently well-
designed in terms of experimental design,
measurement of the dependent variables,
assessment of treatment fidelity, the ability
to detect generalization and maintenance
effects, and the quality of the ASD classifi-
cation of participants. If the article exceeded
minimum criteria on scientific merit, the
treatment effects were assessed as being
beneficial, ineffective, adverse, or that the
data were not sufficiently interpretable
to decide on effects. Experienced trained
reviewers were able to complete a review
16 Overview and Foundational Issues
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
for a research article in 1 hour or less with
interreviewer agreement of 80% or more.
Inevitable problems for all these systems for
review of single case designs arise from the
necessity of creating “one size fits all” rules.
For example, the evaluative methods of both
the NASP (Kratochwill & Stoiber, 2002) and
NSP projects (Wilczynski & Christian, 2008)
include consideration of level of interobserver
agreement for determining partly the quality of
measurement of the dependent variables. The
minimum acceptable level of agreement is
specified at, say, 70% or 80%, which makes it
relatively convenient for reviewers to deter-
mine from reading the published research art-
icle that they are rating; however, it has been
known for more than 30 years that an agree-
ment percentage is rather meaningless without
examination of how behaviors were measured,
what interobserver agreement algorithm was
employed, and the relative frequency or dur-
ation of the behaviors measured (Hawkins &
Dotson, 1975). Another example concerns
coding rules for evaluating the adequacy of
baseline measures of behavior. Whether the
criterion for a baseline phase of highest scien-
tific merit is a minimum of 3 points (NASP) or
5 points (NSP), there will be occasions when
the rule should be inapplicable. For example,
in Najdowski, Wallace, Ellsworth, MacAleese,
and Cleveland (2008), after more than 20
observational sessions during intervention
showing zero severe problem behavior, a return
to baseline in an ABAB design raised the rate
of problem behavior to more than four per
minute, which was higher than any points in the
first A-phase. To continue with the baseline to
meet NASP or NSP evaluative criteria would
have been unnecessary to show experimental
control and dangerous for the participant.
These examples indicate why more detailed
and codified methods for reviewing small-N
studies quantitatively are, as yet, not firmly
established. Although there may be satisfac-
tory methods for fitting RCTs’ scientific merit
and size of effects into databases for meta-
analyses, that appears to be more problematic
with small-N designs, given their flexibility in
use (Hayes et al., 1999).
Volume of Evidence From Small-N
Studies Required to Claim That an
Intervention Is Evidence Based
Having rated the strength of evidence from
individual research articles, the next step is to
determine whether a group of studies on the
same topic between them constitute sufficient
evidence to declare that an intervention is an
empirically supported treatment or a promis-
ing or emerging intervention. We discuss only
empirically supported treatment criteria here.
Consensus on what minimum criterion should
apply has yet to be reached. Chambless and
Hollon (1998) originally recommended a
minimum of two independent studies with
three or more participants (N $ 3) showing
good effects for a total of N $ 6 participants.
Lonigan et al. (1998) required three studies
with N $ 3; that is, beneficial effects shown for
N $ 9 participants. Since then, the bar has been
raised. For instance, Horner et al. (2005) pro-
posed that the criteria for determining that an
intervention is evidence-based included a
minimum of five small-N studies, from three or
more separate research groups, with at least 20
participants in total. Wilczynski and Christian
(2008) used similar criteria with $ 6 studies
of strongest scientific merit totaling N $ 18
participants with no conflicting results from
other studies of adequate design. Others have
recommended similar standards: Reichow,
Volkmar, and Cicchetti (2008) described a
method for evaluating research evidence
from both group and small-N designs, as had
Wilczynski and Christian (2008). Finally,
Reichow et al. (2008) set the criterion for an
established EBP at $ 10 small-N studies of at
least “adequate report strength” across three
different locations and three different research
teams with a total of 30 participants, or, if at
least five studies of “strong report strength”
with a total of 15 or more participants existed,
that could substitute for the 10-study criterion.
Rationale and Standards of Evidence in Evidence-Based Practice 17
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
The rationale for selecting the numerical
criteria is usually not stated. Thus, all we can
state is that some systems (e.g., Reichow et al.,
2008) are more conservative than others (e.g.,
Lonigan et al., 1998). Sometimes conservative
criteria may be appropriate, for example, when
the costs of treatment are high, and/or the
intervention is exceedingly complex requiring
highly skilled intervention agents, and/or the
benefits for the intervention are less than ideal
(i.e., it reduces problems to a more manageable
level, but does not eliminate them), and/or
some negative side effects have been observed.
On the other hand, if relatively few resources
are required to implement an effective and
rapid intervention without unwanted side
effects, fewer well-conducted studies may be
needed to persuade consumers that the inter-
vention is empirically supported, and therefore
worth evaluating with the individual patient.
This brief discussion should advise readers to
examine and consider criteria carefully when
reviewers claim that a particular intervention is
an empirically supported treatment for a par-
ticular disorder or problem for a particular
population.
The discussions on evidence from small-N
designs have left much out. For instance,
Reichow et al. (2008) and Wilczynski and
Christian (2008) developed algorithms for
assessing the strength of evidence at different
levels, although we have outlined only the
highest levels. Both groups have also reported
algorithms for determining ESTs from
mixed methods (e.g., RCTs and small-Ns).
Wilczynski and Christian (2008) report rules
for incorporating conflicting results into the
decision-making process about overall strength
of evidence (De Los Reyes & Kazdin, 2008).
Level of Specificity of Empirically
Supported Treatments
The issue to be discussed next concerns the unit
of analysis of the research evidence. We illus-
trate by examining an example provided by
Horner et al. (2005) in which they assessed the
level of support for functional communication
training (FCT; Carr & Durand, 1985). Func-
tional communication training is an approach
to reducing problem behaviors that teaches an
appropriate nonproblematic way for individ-
uals to access reinforcers for the problem
behavior that have been identified through
functional assessment. Horner and colleagues
cited eight published research reports that
included 42 participants who had benefited in
FCT studies across five research groups. The
evidence was sufficient in quantity for it to be
concluded that FCT is an empirically supported
treatment, exceeding all criteria reviewed earlier
except that two more studies would have been
needed to reach the number of studies $ 10
by the criteria of Reichow et al. (2008).
It might reasonably be asked: “For which
population is FCT beneficial?” Perusal of the
original papers cited by Horner et al. (2005)
shows that 21/42 participants’ data were
reported in one of the eight cited studies
(Hagopian, Fisher, Sullivan, Acquisto, &
LeBlanc, 1998), with the oldest participant
being 16 years old, and none reported to have a
diagnosis of autism. Thus, applying Horner
et al.’s criteria, it cannot be concluded from the
studies cited that FCT is an empirically sup-
ported treatment for participants with autism
or for participants older than 16, regardless of
diagnosis. As an aside, eight participants across
the other seven studies were reported to have
autism, and three participants in total were aged
over 20 years; however, the literature on FCT
that Horner et al. (2005) included did not
appear to have been obtained from a systematic
search, so it is possible that there has been
sufficient research to show that FCT is an
empirically supported treatment for subgroups,
and perhaps autism and adults are two of those.
TREATMENT GUIDELINES
Treatment guidelines specifically recommend
ESTs to practitioners and consumers. Alter-
native descriptors are clinical practice and
18 Overview and Foundational Issues
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
best practices guidelines (Barlow, Levitt, &
Bufka, 1999). The view of APA was that
guidelines are
not intended to be mandatory, exhaustive,
or definitive . . . and are not intended to
take precedence over the judgment of
psychologists. APA’s official approach to
guidelines strongly emphasizes professional
judgment in individual patient encounters
and is therefore at variance with that of
more ardent adherents to evidence-based
practice. (Reed et al., 2002, p. 1042)
It is apparent that many health-care organ-
izations, insurance companies, and states in the
United States interpret the purpose of lists of
ESTs and treatment guidelines differently
(Gotham, 2006; Reed & Eisman, 2006). They
can interpret guidelines as defining what treat-
ments can be offered to patients and, via
manualization, exactly how treatment is to be
administered, by whom, and for how long. The
requirement for manualizationallowsfunders of
treatment to specify a standard reimbursement
for the treatment provider. Thus, the empirically
supported treatment movement was embraced
by governments and health-care companies as it
was anticipated to be a major contributor to
controlling escalating health-care costs.
Many practicing psychologists were less
enthusiastic about the empirically supported
treatment and EBPP movements (see contri-
butions by clinicians in Goodheart et al., 2006;
Norcross et al., 2005). General concerns
included that requirements to use only ESTs
restrict professionalism by reframing psych-
ologists as technicians going by the book
mechanically; restricting client choice to
effective interventions that have been granted
empirically supported treatment status higher
than others, only because they, like drugs, are
relatively easy to evaluate in the RCT format.
Prescription of one-size-fits-all ESTs may
further disadvantage minorities, and people
with severe and multiple disorders for whom
there is scant evidence available. There
were also concerns that the acknowledged
importance of clinical expertise, such as
interpersonal skills to engage the client (child)
and significant others (family) in a therapeutic
relationship, would be ignored.
Contrary to the pronouncements from the
APA (2002, 2006), guidelines have been
interpreted or developed that “assume the force
of law” in prescribing some interventions and
proscribing others (Barlow et al., 1999, p. 155).
Compulsion of psychologists in practice to
follow treatment guidelines has been reported
to occur in the United States by giving
immunity from malpractice lawsuits to those
who use only ESTs, and increasing the vul-
nerability of those who do not to litigation and
increased professional indemnity insurance
(Barlow et al., 1999; Reed et al., 2002). Some
guidelines, especially those produced by agen-
cies or companies employing psychologists,
have been viewed as thinly veiled cost-cutting
devices justified with a scientistic gloss.
Ethical Requirements
For many human service professional organ-
izations, EBP and ESTs have become an
ethical requirement. The APA’s Ethical Prin-
ciples of Psychologists and Code of Conduct
document mentions the obligation to use some
elements of EBPP; for example, “Psycholo-
gists’ work is based upon established scien-
tific and professional knowledge of the
discipline” (American Psychological Associ-
ation, 2010, p. 5). Other professional groups
appear to be more prescriptive with regard to
EBP. The BACB’s Code for Responsible
Conduct, for example, recommends EBP with
statements such as, “Behavior analysts rely on
scientifically and professionally derived
knowledge when making scientific or profes-
sional judgments in human service provision”
(Behavior Analyst Certification Board,
2004, p. 1). The BACB also require the use
of ESTs:
a. The behavior analyst always has the
responsibility to recommend scientifically
Rationale and Standards of Evidence in Evidence-Based Practice 19
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
supported most effective treatment
procedures. Effective treatment procedures
have been validated as having both long-term
and short-term benefits to clients and society.
b. Clients have a right to effective treatment
(i.e., based on the research literature and
adapted to the individual client). c. Behavior
analysts are responsible for review and
appraisal of likely effects of all alternative
treatments, including those provided by other
disciplines and no intervention. (Behavior
Analyst Certification Board, 2004, p. 4)
As shown by the previous examples, each of
these organizations have variations in how
they have included EBP and empirically sup-
ported treatment into their codes of conduct
and ethical statements; however, they both
include the basic tenets of EBP: combining the
best research information with clinical know-
ledge and the preferences of the individuals
involved.
CHILDREN, ADOLESCENTS, AND
EVIDENCE-BASED PRACTICE
IN PSYCHOLOGY
Despite there being ESTs for a number of
disorders and problem behaviors manifesting
in children and adolescents, there are more
than 500 treatments in use with children
(Kazdin, 2008), most of which are unre-
searched. Chambless and Ollendick (2001)
listed 108 ESTs for adults, compared with 37
for children, suggesting that research on ther-
apies for young people is lacking relative to
adults. Further research with child and ado-
lescent populations has been prioritized by
APA (APA Presidential Task Force on Evi-
dence-Based Practice, 2006). Researching the
effects of treatments for children brings special
difficulties (Kazdin & Weisz, 1998). Regard-
ing practice, children do not typically self-
refer for mental, psychological, or behavioral
disorders, nor are they active seekers of
ESTs or preferred interventions. Parents
or guardians tend to take those roles, either
independently or following recommendations
from family, or health or education profes-
sionals. Children and youth cannot legally
provide informed consent for treatments or for
participation in research studies. These are
among the developmental, ethical, and legal
factors that affect consideration of EBPP with
children.
Lord et al. (2005) discussed the challenges of
acquiring rigorous evidence regarding efficacy
of treatments for children with complex,
potentially chronic behavioral/psychological
disorders. Contributors to the article were
researchers from multiple disciplines assem-
bled by the National Institutes of Health in
2002. They wrote about autism spectrum dis-
orders specifically, but acknowledged that the
issues may have relevance to other child and
youth problems.
Parents may be unwilling to consent to ran-
domization studies in case their child is
assigned to what parents perceive to be a less
preferred treatment alternative, particularly
when the intervention is long term and early
intervention is, or is perceived to be, critical,
such as early intensive behavioral intervention
for pervasive developmental disorders. Lord
et al. (2005) noted that ethical concerns may
prohibit RCTs of promising interventions
when randomization to no treatment or treat-
ment of unknown effects is required by the
evaluation protocol. Additional factors that
reduce the internal validity of group compari-
son studies of psychosocial interventions
include that parental blindness to the inter-
vention allocated to their children is nigh on
impossible, diffusion of treatment through
parent support groups is likely, parents may
choose to withdraw their children from no
treatment or treatment as usual groups and
obtain the experimental intervention or an
approximation to it from outside the study,
and children with severe disorders will often
be subjected to multiple interventions of
unknown benefit, provided with varying
20 Overview and Foundational Issues
c01 20 April 2012; 12:43:30
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
fidelity that may interact with one another to
produce uninterpretable beneficial, neutral, or
adverse effects (Smith & Antolovich, 2000).
Parents seek out information on their child’s
disorder and intervention recommendations
through Internet or parent organizations,
sometimes accepting advice from profession-
als. Mackintosh, Meyers, and Goin-Kochel
(2006) received 498 responses to a Web-based
survey of parents with children with autism
spectrum disorders and found that the most
oft-cited sources of information were books
(88%), Web pages (86%), other parents (72%),
and autism newsletters (69%). Lagging
somewhat as sources of advice were profes-
sionals other than educators or physicians
(57%). Physicians, education professionals,
and family members were cited as sources of
information by fewer than half the parents who
responded to the survey.
Multiple fad treatments have been recom-
mended for childhood onset disorders, and
their adoption by families and some profes-
sionals, including psychologists, wastes
resources and time that could have been spent
profitably by employing ESTs (Jacobson,
Foxx, & Mulick, 2005). Although some of our
examples of supported and unsupported treat-
ments have related to children with autism
spectrum disorders (Romanczyk, Arnstein,
Soorya, & Gillis, 2003), the problem of treat-
ment selection uninformed by research affects
children with other difficulties and their fam-
ilies. Some interventions for children with
developmental and other disabilities have been
found ineffective or harmful (Jacobson et al.,
2005), and the same occurs for children with
ADHD (Waschbusch & Hill, 2003). (See also
Chapter 2 of this volume for a further discus-
sion of this point by Waschbush, Fabiano, and
Pelham.) We believe that clinical psycholo-
gists working with young people ought to have
a professional ethical obligation to inform
themselves and others about empirically
unsupportable treatments as well as ESTs for
their clients.
LIMITATIONS OF THE EVIDENCE
BASE REGARDING EVIDENCE-BASED
PRACTICE IN PSYCHOLOGY
There is a relatively large body of evidence
concerning efficacy of treatments (Kazdin &
Weisz, 1998, 2003), but far less on treatment
utility, effectiveness, and efficiency. There is
evidence that the utility of some efficacious
treatments has been demonstrated, but further
study is needed before the general statements
can be made about the similarity or difference
between outcomes from controlled research
and clinical practice (Barlow et al., 1999;
Hunsley, 2007).
There is less evidence concerning dimen-
sions of clinical expertise and client charac-
teristics, culture, and preferences that are
relevant to beneficial treatment outcomes
(Kazdin, 2008). Employment of qualitative
research methods may help us to understand
clients’ experiences of psychological treat-
ments. The APA Task Force has suggested that
clinical expertise is made up of at least eight
components, including assessment and treat-
ment planning, delivery, interpersonal skills,
self-reflection, scientific skills in evaluating
research, awareness of individual and social
factors, the ability to seek additional resources
where necessary, and having a convincing
rationale for treatment strategies (APA Presi-
dential Task Force on Evidence-Based Prac-
tice, 2006). Qualitative methods may provide
evidence regarding clinical expertise also.
Improvement of two-way communication
between psychologists who are primarily
researchers and those who identify more as
practitioners would assist dissemination of
ESTs, collaboration in clinical utility studies
of efficacious treatments, and facilitate
research into clinical expertise and barriers to
the adoption of ESTs by clinicians (Kazdin,
2008).
Lilienfeld (2005) and McLennan, Wathen,
MacMillan, and Lavis (2006) recommended
further research on interventions in child
Rationale and Standards of Evidence in Evidence-Based Practice 21
c01 20 April 2012; 12:43:31
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
psychopathology on two fronts: (1) increased
research on promising or new but plausible
interventions; and (2) on research to combat
questionable, but potentially harmful, inter-
ventions. Attention is needed to increase
training of clinical psychologists in ESTs and
cessation of training to use treatments that
have been shown to be harmful, ineffective, or
less effective. Decreasing the demand and use
of treatments that have not been evaluated
scientifically or have been found ineffective or
harmful may be another strategy for helping
clinical psychologists orient more to treat-
ments that do work (i.e., ESTs).
Woody, Weisz, and McLean (2005) report
that APA accredited clinical psychologist
training programs taught and supervised interns
in fewer ESTs in 2003 than they had in 1993.
The training of clinical psychologists should
include sufficient study of research method-
ology so that career-long learning, and contri-
bution to research in practice concerning ESTs
can be enhanced (Bauer, 2007; Kazdin, 2008).
Considering the APA definition of EBPP,
trainees need supervised practice in incor-
porating clients’ preferences, values, and
cultural considerations as well as to develop
clinical expertise (Collins, Leffingwell, &
Belar, 2007). Further training for university-
based psychologist training faculty should
be studied.
Psychologists in practice may wish or feel
forced to adapt to employing EBPP, but will-
ingness or compulsion to do so is not the same
as becoming immediately competent to use an
empirically supported treatment effectively.
The typical workshop format for introducing
new techniques is as ineffective for profes-
sionals (Gotham, 2006), as it is for direct care
staff (Reid, 2004). Skill-based training can be
effective when trainees practice an interven-
tion method in the natural clinical environment
with differential feedback from the trainer on
their performance of the skill. This should
occur after workshops that introduce the
rationale and method, and include in vivo or
videotaped demonstration of the skill by the
trainer (i.e., modeling). Frequent follow-up
observations by the trainer, again with
objective feedback to the trainee, can facilitate
maintenance of the newly acquired skills
(Reid, 2004). Gotham (2006) identified bar-
riers to implementation of EBPP, and provided
an example of how to implement an empiric-
ally supported treatment statewide despite
obstacles. McCabe (2004) wrote quite opti-
mistically for clinicians about the challenges
of EBPP, offering advice in a step-by-step
form to psychologists. Reorienting and training
clinicians is an area for further clinician-
researcher collaboration that requires emphasis.
We have not included discussion of an area
of EBPP that has, to date, received less atten-
tion than evidence-based treatment, which is
“evidence-based assessment” (Kazdin, 2005;
Mash & Hunsley, 2005). An initial assessment
with clinical utility will identify what is the
disorder or problem behavior so that it points
the psychologist in the right direction for
identifying the range of available ESTs for this
client. Evidence-based assessment includes
identifying reliable and valid ongoing meas-
ures that show the effects of an intervention
with the individual client (Kazdin, 2005).
Typically, empirically supported treatment
reviews identify treatments for DSM-type
classifications, the nosological approach;
however, an idiographic functional approach
to assessment may lead to better problem-EST
match (Sturmey, 2007).
It was mentioned earlier that some profes-
sional groups are concerned at the lack of
research on the outcomes of their assessment
and treatment methods. Because clinical and
other psychologists have extensive training in
research methods, we can assist other profes-
sions to assess the evidence for their interven-
tion methods. Interdisciplinary collaborations
also may help elucidate the interaction effects
of behavioral or psychological ESTs with
interventions with presently unknown effects
delivered by other professions.
22 Overview and Foundational Issues
c01 20 April 2012; 12:43:31
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
CONCLUDING REMARKS
The issues concerning evidence-based clinical
practice of psychology for children and ado-
lescents are arguably even more complex than
for adult clients. All readers would welcome
the day when, for any referred psychological,
behavioral, or mental problem, psychologists
with clinical expertise were able to offer a wide
range of effective and safe interventions. Then
the young person, with more or less help from
those who care for them, could select the
effective treatment that suited their culture,
preferences, and resources. The literature
reviewed for this chapter suggests that, gener-
ally speaking, the health care, including clinical
psychology, and education professions are
starting on the road to addressing the aspir-
ational targets of the EBP movement. The lack
of unanimous agreement within professions
that the evidence-based practice movement is
desirable and what constitutes evidence for
beneficial interventions is unsurprising; how-
ever, with more evidence for safe, acceptable,
and effective interventions such as contained
in the present volume and more education for
consumers, professionals, and the public,
eventually the naysayers and peddlers of
unsupportable treatments may find they have
no raison d’être (and also no income!).
REFERENCES
Addis, M. E., & Krasnow, A. D. (2000). A national survey
of practicing psychologists’ attitudes toward psycho-
therapy treatment manuals. Journal of Consulting and
Clinical Psychology, 68, 331–339.
Almost, D., & Rosenbaum, P. (1998). Effectiveness of
speech intervention for phonological disorders: A
randomized controlled trial. Developmental Medicine
and Child Neurology, 40, 319–325.
American Psychological Association. (2002). Criteria for
evaluating treatment guidelines. American Psycholo-
gist, 57, 1052–1059.
American Psychological Association. (2010). Ethical prin-
ciples of psychologists and code of conduct. Retrieved
from www.apa.org/ethics/code/index.aspx#
American Psychological Association Presidential Task
Force on Evidence-Based Practice. (2006). Evidence-
based practice in psychology. American Psychologist,
61, 271–285.
Barlow, D. H., & Hersen, M. (1984). Single case
experimental designs: Strategies for studying behav-
ior change (2nd ed.). New York, NY: Pergamon.
Barlow, D. H., Levitt, J. T., & Bufka, L. F. (1999). The
dissemination of empirically supported treatments: A
view to the future. Behaviour Research and Therapy,
37, 147–162.
Bauer, R. M. (2007). Evidence-based practice in psych-
ology: Implications for research and research training.
Journal of Clinical Psychology, 63, 683–694.
Becker, B. J. (2006). Failsafe N or file-drawer number.
In H. Rothstein, A. Sutton, & M. Borenstein
(Eds.), Publication bias in meta-analysis: Prevention,
assessment and adjustments (pp. 111–125). Hoboken,
NJ: Wiley.
Behavior Analyst Certification Board. (2004). Behavior
Analyst Certification Board guidelines for respon-
sible conduct for behavior analysts. Retrieved
from www.bacb.com/Downloadfiles/BACBguidelines/
40809_ BACB_Guidelines.pdf.
Bennett, S. A., & Bennett, J. W. (2000). The process of
evidence based practice in occupational therapy:
Informing clinical decisions. Australian Occupational
Therapy Journal, 47, 171–180.
Bereiter, C., & Kurland, M. (1981). A constructive look at
Follow Through results. Interchange, 12, 1–22.
Bowling, A. (1997). Research methods in health: Inves-
tigating health and health services. Philadelphia, PA:
Open University Press.
Campbell, J. M. (2003). Efficacy of behavioral interven-
tions for reducing problem behavior in persons with
autism: A quantitative synthesis of single-subject
research. Research in Developmental Disabilities, 24,
120–138.
Carr, E. G., & Durand, V. M. (1985). Reducing
behavior problems through functional communication
training. Journal of Applied Behavior Analysis, 18,
111–126.
Chambless, D. L., & Hollon, S. D. (1998). Defining
empirically supported therapies. Journal of Consult-
ing and Clinical Psychology, 66, 7–18.
Chambless, D. L., & Ollendick, T. H. (2001). Empirically
supported psychological interventions: Controversies
and evidence. Annual Review of Psychology, 52,
685–716.
Collins, F. L., Leffingwell, T. R., & Belar, C. D. (2007).
Teaching evidence-based practice: Implications for
psychology. Journal of Clinical Psychology, 63,
657–670.
Concato, J., Shah, N., & Horwitz, R. (2008). Randomized,
controlled trials, observational studies, and the
Rationale and Standards of Evidence in Evidence-Based Practice 23
c01 20 April 2012; 12:43:31
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
hierarchy of research designs. New England Journal
of Medicine, 342, 1887–1892.
Cooper, J. O., Heron, T. E., & Heward, W. L. (2007).
Applied behavior analysis (2nd ed.). Upper Saddle
River, NJ: Pearson.
De Los Reyes, A., & Kazdin, A. E. (2008). When the
evidence says, “Yes, No, and Maybe So”: Attending
to and interpreting inconsistent findings among
evidence-based interventions. Current Directions in
Psychological Science, 17, 47–51.
DiCenso, A., Cullum, N., Ciliska, D., & Guyatt, G.
(2004). Introduction to evidence-based nursing. In
A. DiCenso, N. Cullum, D. Ciliska, & G. Guyatt
(Eds.), Evidence-based nursing: A guide to clinical
practice. Philadelphia, PA: Elsevier.
Enderby, P., & Emerson, J. (1995). Does speech and
language therapy work? Review of the literature.
London, England: Whurr.
Enderby, P., & Emerson, J. (1996). Speech and language
therapy: Does it work? British Medical Journal, 312,
1655–1658.
Evidence-Based Medicine Working Group. (1992). A
new approach to teaching the practice of medicine.
Journal of the American Medical Association, 268,
2420–2425.
Eysenck, H. (1994). Meta-analysis and its problems.
British Medical Journal, 309, 789–793.
Flather, M., Farkouh, M., Pogue, J., & Yusuf, S. (1997).
Strengths and limitations of meta-analysis: Larger
studies may be more reliable. Controlled Clinical
Trials, 18, 568–579.
Gersten, R. M. (1984). Follow Through revisited:
Reflections of the site variability issue. Educational
Evaluation and Policy Analysis, 6, 411–423.
Glass, G. (1976). Primary, secondary, and meta-analysis
of research. Educational Researcher, 5, 3–8.
Glogowska, M., Roulstone, S., Enderby, P., & Peters, T. J.
(2000). Randomised controlled trial of community
based speech and language therapy in preschool
children. British Medical Journal, 231, 923–927.
Gluud, L. (2006). Bias in clinical intervention research.
American Journal of Epidemiology, 163, 493–501.
Goisman, R. M., Warshaw, M. G., & Keller, M. B.
(1999). Psychosocial treatment prescriptions for gen-
eralized anxiety disorder, panic disorders and social
phobia, 1991–1996. American Journal of Psychiatry,
156, 1819–1821.
Goodheart, C. D., Kazdin, A. E., & Sternberg, R. J. (Eds.).
(2006). Evidence-based psychotherapy: Where prac-
tice and research meet. Washington, DC: American
Psychological Association.
Gotham, H. J. (2006). Advancing the implementation of
evidence-based practices into clinical practice: How
do we get there from here? Professional Psychology:
Research and Practice, 37, 606–613.
Greenhalgh, T. (2001). How to read a paper: The basics
of evidence based medicine. London, UK: BMJ
Publishing Group.
Grimes, D. A., & Schulz, K. F. (2002). Cohort studies:
Marching towards outcomes. The Lancet, 359,
341–345.
Guyatt, G. H., Haynes, R. B., Jaeschke, R. Z., Cook, D. J.,
Naylor, C. D., & Wilson, W. S. (2000). Users’ guides
to the medical literature: XXV. Evidence-based
medicine: Principles for applying the users’ guides to
patient care. Journal of the American Medical Asso-
ciation, 284, 1290–1296.
Hadorn, D., Baker, D., Hodges, J., & Hicks, N. (1996).
Rating the quality of evidence for clinical practice
guidelines. Journal of Clinical Epidemiology, 49,
749–754.
Hagopian, L. P., Fisher, W. W., Sullivan, M. T., Acquisto,
J., & LeBlanc, L. A. (1998). Effectiveness of func-
tional communication training with and without
extinction and punishment: A summary of 21 inpa-
tient cases. Journal of Applied Behavior Analysis, 31,
211–235.
Hawkins, R. P., & Dotson, V. A. (1975). Reliability
scores that delude: An Alice in Wonderland trip
through the misleading characteristics of interobserver
agreement scores in interval recording. In E. Ramp &
G. Semb (Eds.), Behavior analysis: Areas of research
and application (pp. 359–376). Englewood Cliffs, NJ:
Prentice Hall.
Hayes, S. C., Barlow, D. H., & Nelson-Gray, R. O.
(1999). The scientist-practitioner: Research and
accountability in the age of managed care (2nd ed.).
Boston, MA: Allyn & Bacon.
Hess, F. M. (2006). Accountability without angst? Public
opinion and No Child Left Behind. Harvard Educa-
tional Review, 76, 587–610.
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom,
S., & Wolery, M. (2005). The use of single-subject
research to identify evidence-based practice in special
education. Exceptional Children, 71, 165–179.
Hunsberger, P. H. (2007). Reestablishing Clinical Psy-
chology’s subjective core. American Psychologist, 62,
614–615.
Hunsley, J. (2007). Addressing key challenges in
evidence-based practice in psychology. Professional
Psychology: Research and Practice, 38, 113–121.
Jacobson, J. W., Foxx, R. M., & Mulick, J. A. (Eds.).
(2005). Controversial therapies for developmental
disabilities: Fad, fashion, and science in professional
practice. Mahwah, NJ: Erlbaum.
Jacobson, J. W., Mulick, J. A., & Schwartz, A. A. (1995).
A history of facilitated communication: Science,
pseudoscience and antiscience. American Psycholo-
gist, 50, 750–765.
Jenicek, M. (2003). Foundations of evidence-based
medicine. New York, NY: Parthenon.
24 Overview and Foundational Issues
c01 20 April 2012; 12:43:31
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
Jensen, W. R., Clark, E., Kircher, J. C., & Kristjansson,
S. D. (2007). Statistical reform: Evidence-based
practice, meta-analyses, and single subject designs.
Psychology in the Schools, 44, 483–493.
Kazdin, A. E. (1982). Single-case research designs:
Methods for clinical and applied settings. New York,
NY: Oxford University Press.
Kazdin, A. E. (2005). Evidence-based assessment for
children and adolescents: Issues in measurement
development and clinical application. Journal
of Clinical Child and Adolescent Psychology, 34,
548–558.
Kazdin, A. E. (2008). Evidence-based treatment and
practice: New opportunities to bridge clinical research
and practice, enhance the knowledge base, and
improve patient care. American Psychologist, 63,
146–159.
Kazdin, A. E., & Weisz, J. R. (1998). Identifying and
developing empirically supported child and adoles-
cent treatments. Journal of Consulting and Clinical
Psychology, 66, 19–36.
Kazdin, A. E., & Weisz, J. R. (Eds.). (2003). Evidence-
based psychotherapies for children and adolescents.
New York, NY: Guilford Press.
Kratochwill, T. R. (2005). Evidence-based interventions
in school psychology: Thoughts on a thoughtful
commentary. School Psychology Quarterly, 17,
518–532.
Kratochwill, T. R., & Stoiber, K. C. (2002). Evidence-
based interventions in school psychology: Conceptual
foundations of the Procedural and Coding Manual of
Division 16 and the Society for the Study of School
Psychology Task Force. School Psychology Quar-
terly, 17, 341–389.
Levin, J. R. (2002). How to evaluate the evidence of
evidence-based interventions? School Psychology
Quarterly, 17, 483–492.
Lilienfeld, S. O. (2005). Scientifically unsupported and
supported interventions for child psychopathology: A
summary. Pediatrics, 115, 761–764.
Lohr, K. N., Eleazer, K., & Mauskopf, J. (1998). Health
policy issues and applications for evidence-based
medicine and clinical practice guidelines. Health
Policy, 46, 1–19.
Lonigan, C. J., Elbert, J. C., & Johnson, S. B. (1998).
Empirically supported psychosocial interventions for
children: An overview. Journal of Clinical Child
Psychology, 27, 138–145.
Lord, C., Wagner, A., Rogers, S., Szatmari, P., Aman, M.,
Charman, T., . . . Yoder, P. (2005). Challenges in
evaluating psychosocial interventions for Autistic
Spectrum Disorders. Journal of Autism and Devel-
opmental Disorders, 35, 695–708.
Mackintosh, V. H., Meyers, B. J., & Goin-Kochel, R. P.
(2006). Sources of information and support used by
parents of children with autism spectrum disorders.
Journal on Developmental Disabilities, 12, 41–51.
Mahon, J., Laupacis, A., Donner, A., & Wood, T. (1996).
Randomised study of n of 1 trials versus standard
practice. British Medical Journal, 312, 1069–1074.
Mash, E. J., & Hunsley, J. (2005). Evidence-based
assessment of child and adolescent disorders: Issues
and challenges. Journal of Clinical Child and Ado-
lescent Psychology, 34, 362–379.
Maxwell, R. (1992). Dimensions of quality revisited:
From thought to action. Quality in Health Care, 1,
171–177.
Maynard, A. (1997). Evidence-based medicine: An
incomplete method for informing treatment choices.
The Lancet, 349, 126–128.
Mays, N., & Pope, C. (Eds.). (1996). Qualitative research
in health care. London, England: BMJ Publishing
Group.
McCabe, O. L. (2004). Crossing the quality chasm in
behavioral health care: The role of evidence-based
practice. Professional Psychology: Research and
Practice, 35, 571–579.
McLennan, J. D., Wathen, C. N., MacMillan, H. L., &
Lavis, J. N. (2006). Research-practice gaps in child
mental health. Journal of the American Academy of
Child and Adolescent Psychiatry, 45, 658–665.
Melnyk, B. M., & Fineout-Overholt, E. (Eds.). (2005).
Evidence-based practice in nursing and healthcare: A
guide to best practice. Philadelphia, PA: Lippincott
Williams & Wilkins.
Najdowski, A. C., Wallace, M. D., Ellsworth, C. L.,
MacAleese, A. N., & Cleveland, J. M. (2008). Func-
tional analysis and treatment of precursor behavior.
Journal of Applied Behavior Analysis, 41, 97–105.
Norcross, J. C., Beutler, L. E., & Levant, R. F. (Eds.).
(2005). Evidence based practices in mental
health: Debate and dialogue on the fundamental
questions. Washington, DC: American Psychological
Association.
Paul, G. L. (1967). Strategy of outcome research in
psychotherapy. Journal of Consulting Psychology, 31,
109–118.
Reed, G. M., & Eisman, E. J. (2006). Uses and misuses of
evidence: Managed care, treatment guidelines, and
outcomes measurement in professional practice. In
C. D. Goodheart, A. E. Kazdin, & R. J. Sternberg
(Eds.), Evidence-based psychotherapy: Where prac-
tice and research meet (pp. 13–35). Washington, DC:
American Psychological Association.
Reed, G. M., McLaughlin, C. J., & Newman, R. (2002).
American Psychological Association policy in con-
text: The development and evaluation of guidelines
for professional practice. American Psychologist, 57,
1041–1047.
Reichow, B., Volkmar, F. R., & Ciccetti, D. V. (2008).
Development of the evaluative method for evaluating
and determining evidence-based practices in autism.
Journal of Autism and Developmental Disorders, 38,
1311–1319.
Rationale and Standards of Evidence in Evidence-Based Practice 25
c01 20 April 2012; 12:43:32
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .
Reid, D. H. (2004). Training and supervising direct care
support personnel to carry out behavioral procedures.
In J. L. Matson, R. Laud, & M. Matson (Eds.),
Behavior modification for persons with developmental
disabilities: Treatments and supports, Volume 1
(pp. 101–129). Kingston, NY: NADD Press.
Reilly, S., Oates, J., & Douglas, J. (2004). Evidence based
practice in speech pathology. London, England: Whurr.
Reynolds, S. (2000). The anatomy of evidence-based
practice: Principles and methods. In L. Trinder &
S. Reynolds (Eds.), Evidence-based practice: A critical
appraisal (pp. 17–34). Oxford, England: Blackwell.
Ringwalt, C. L., Ennett, S., Vincus, A., Thorne, J.,
Rohrbach, L. A., & Simons-Rudolph, A. (2002). The
prevalence of effective substance use prevention
curricula in U.S. middle schools. Prevention Science,
3, 257–265.
Roberts, A. R., & Yeager, K. R. (2006). Foundations of
evidence based social work practice. New York, NY:
Oxford University Press.
Robertson, S. B., & Weismer, S. E. (1999). Effects of
treatment on linguistic and social skills in toddlers with
delayed language development. Journal of Speech,
Language, and Hearing Research, 42, 1234–1247.
Romanczyk, R. G., Arnstein, L., Soorya, L. V., & Gillis, J.
(2003). The myriad of controversial treatments
for autism: A critical evaluation of efficacy. In S. O.
Lillienfeld, S. J. Levin, & J. R. Lohr (Eds.), Science
and pseudoscience in clinical psychology (pp.
363–395). New York, NY: Guilford Press.
Rosen, A., Proctor, E. E., Morrow-Howell, N., & Staudt, M.
(1995). Rationales for practice decisions: Variations
in knowledge use by decision task and social work
service. Research on Social Work Practice, 5, 501–523.
Rosenthal, R. (1984). Meta-analytic procedures for social
research. Beverly Hills, CA: Sage Publications.
Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg,
W., & Haynes, R. B. (2000). Evidence-based medicine:
How to practice and teach EBM (2nd ed.). Edinburgh,
Scotland: Churchill Livingstone.
Shavelson, R. J., & Towne, L. (2002). Scientific research in
education. Washington, DC: National Academy Press.
Shernoff, E. S., Kratochwill, T. R., & Stoiber, K. C.
(2002). Evidence-based interventions in school
psychology: An illustration of task force coding cri-
teria using single-participant research design. School
Psychology Quarterly, 17, 390–422.
Simes, J. (1986). Publication bias: The case for an inter-
national registry of clinical trials. Journal of Clinical
Oncology, 4, 1529–1541.
Smith, T., & Antolovich, A. (2000). Parental perceptions
of supplemental interventions received by young
children with autism in intensive behavior analytic
treatment. Behavioral Interventions, 15, 83–97.
Stout, C. E., & Hayes, R. A. (2004). Evidence-based
practice: Methods, models, and tools for mental health
professionals. Hoboken, NJ: Wiley.
Straus,S.E.,Richardson,W.S.,Glasziou,P.,&Haynes,R.B.
(2005). Evidence-based medicine: How to practice
and teach EBM (2nd ed.). Edinburgh, Scotland: Elsevier.
Sturmey, P. (Ed.). (2007). Functional assessment in
clinical treatment. Burlington, MA: Academic Press.
Thomas, G., & Pring, R. (2004). Evidence-based practice
in education. Maidenhead, England: McGraw-Hill
Education.
U.S.DepartmentofEducation.(2002).NoChildLeftBehind
Act of 2001. Public Law 107-110. Retrieved from www
.ed.gov/policy/elsec/leg/esea02/107-110.pdf
Wacholder, S., McLaughlin, J., Silverman, D., & Mandel, J.
(1992). Selection of controls in case-control studies.
American Journal of Epidemiology, 135, 1019–1028.
Waschbusch, D. A., & Hill, G. P. (2003). Empirically
supported, promising, and unsupported treatments for
children with attention-deficit/hyperactivity disorder.
In S. O. Lillienfeld, S. J. Levin, & J. R. Lohr (Eds.),
Science and pseudoscience in clinical psychology
(pp. 333–362). New York, NY: Guilford Press.
Wilczynski, S. M., & Christian, L. (2008). The National
Standards Project: Promoting evidence-based practice
in autism spectrum disorders. In J. K. Luiselli, D. C.
Russo, W. P. Christian, & S. M. Wilcyznski (Eds.),
Effective practices for children with Autism: Educa-
tional and behavior support interventions that work
(pp. 37–60). New York, NY: Oxford University Press.
Woody, S. R., Weisz, J., & McLean, C. (2005). Empir-
ically supported treatments: 10 years later. The Clin-
ical Psychologist, 58, 5–11.
Yeaton, W., & Wortman, P. (1993). On the reliability of
meta-analytic reviews: The role of intercoder agree-
ment. Evaluation Review, 17, 292–309.
Zlotnick, J. L., & Solt, B. E. (2006). The institute for the
advancement of social work research: Working to
increase our practice and policy evidence base.
Research on Social Work Practice, 16, 534–539.
26 Overview and Foundational Issues
c01 20 April 2012; 12:43:33
Hersen, M., & Sturmey, P. (2012). Handbook of evidence-based practice in clinical psychology, child and adolescent disorders. John Wiley & Sons, Incorporated. Created from ashford-ebooks on 2021-11-23 18:29:39.
C o p yr
ig h t ©
2 0 1 2 . Jo
h n W
ile y
& S
o n s,
I n co
rp o ra
te d . A
ll ri g h ts
r e se
rv e d .