SWRM #11

O R I G I N A L P A P E R

Using Methodological Search Filters to Facilitate Evidence-Based Social Work Practice

Aron Shlonsky • Tobi Michelle Baker •

Esme Fuller-Thomson

Published online: 11 January 2011

� Springer Science+Business Media, LLC 2011

Abstract The process of Evidence-Based Practice (EBP)

requires clinical social workers to conduct systematic

searches of academic databases in order to ascertain current

best evidence and integrate this with client preferences/

values and clinical state/circumstances. Yet social workers

are often pressed for time, and searches for evidence dis-

regarded as too time-consuming to conduct. There is hope.

Searches of the literature can be more easily and quickly

facilitated through the use of methodological search filters.

This study introduces a new methodological search filter

created especially for social care and evaluates the extent

to which this and four other filters accurately and effi-

ciently identify known social care effectiveness studies in

two major scholarly databases (Psycinfo and Medline).

Sensitivity, specificity, and a new metric for establishing

efficiency (the AVALANCHE INDEX) are reported.

Keywords Evidence-based practice � Methodological search filters � Sensitivity � Specificity

Introduction

The use of evidence in practice is an issue of social justice.

That is, when we as social workers offer up interventions

for our clients, particularly those clients who are

involuntary, it is our ethical duty to bring current best

evidence into the decision-making context. But what evi-

dence? How do we find what is best? Evidence-Based

Practice (EBP) is a process involving several distinct steps.

As defined by Gibbs (2003), EBP is a process of ‘‘lifelong

learning that involves continually posing specific questions

of direct practical importance to clients, searching objec-

tively and efficiently for the current best evidence related

to the question’’ (p. 6), evaluating the identified literature in

terms of its methodological rigor and applicability of

findings to the client, and ‘‘taking appropriate action guided

by the evidence’’ (p. 6). In this context, posing a question

involves phrasing a question in a way that the answer can

be located using various research databases (i.e., for chil-

dren diagnosed with ADHD whose parents do not wish to

use medication, which form of psychosocial counseling

works best to improve academic functioning?).

Two of the main criticisms of EBP involve the lack of

empirical literature and the limited time practitioners have

to search for evidence (Gibbs and Gambrill 2002; Gray

et al. 2009; Rosen and Proctor 2003). This paper is focused

on evaluating strategies to improve practitioners’ skills in

the quick and efficient location of research articles evalu-

ating the effectiveness of a given intervention, in this case,

random controlled trials or RCT’s. Although there are

many valid forms of evidence, the search for studies of

effectiveness (i.e., one type of intervention compared to

another type, usual treatment, or nothing) is an appropriate

starting point. As any practitioner or student who has tried

to search for evidence is aware, a poorly specified question

and search can lead to thousands of irrelevant ‘hits’ or

results, and wading through these can be cumbersome,

inefficient, and discouraging. For instance, simply stating

the term ‘depression’ in a search engine such as Google

Scholar will return tens of millions of hits, most of which

A. Shlonsky (&) � T. M. Baker � E. Fuller-Thomson Factor-Inwentash Faculty of Social Work, University of Toronto,

Toronto, ON, Canada

e-mail: [email protected]

E. Fuller-Thomson

The Department of Family and Community Medicine

and the Faculty of Nursing, University of Toronto,

Toronto, ON, Canada

123

Clin Soc Work J (2011) 39:390–399

DOI 10.1007/s10615-010-0312-3

are not empirical studies. Even in PsycINFO and Medline,

such a search would result in hundreds of thousands of hits,

a virtual avalanche of irrelevant peer-reviewed articles.

One of the main tools used to combat this inefficiency is

the use of methodological search filters. These are com-

binations of search terms that retrieve certain types of

studies (e.g., experimental, quasi-experimental, diagnostic

and prognostic accuracy, qualitative) from scholarly dat-

abases, and they hold the promise of allowing practitioners

to search more quickly and accurately for relevant studies

that can be used to help guide their clinical decisions. For

example, Leonard Gibbs (2003) proposes using the fol-

lowing terms, in combination with subject specific search

terms, to efficiently find highly controlled studies best

suited for questions of intervention effectiveness: (Ran-

dom* OR Controlled Clinical trial* OR Control group* OR

Evaluation stud* OR Study design OR Statistical* Signif-

ican* OR Double-blind OR Double blind OR Placebo).

The use of search filters, however, is not a panacea. Search

filters are similar to diagnostic and prognostic tests in the

sense that they can be of variable quality and their ability to

correctly identify relevant studies can be measured with

similar techniques.

Specifically, filters can be measured in terms of their

sensitivity (degree to which filters can accurately identify

relevant studies—range is from 0 to 1 with 1 reflecting the

ability to capture all relevant studies) and specificity

(degree to which filters can accurately exclude non-rele-

vant studies—range if from 0 to 1 with 1 reflecting the

ability to exclude all irrelevant studies). Haynes et al.

(1994) tested medline-specific methodological search fil-

ters intended to retrieve random controlled trials (RCT’s)

related to adult general medicine. They found that the fil-

ters could be used to produce both specific and sensitive

results, with sensitivity ranging from 0.72 to 0.99

(depending on search year and combination of terms used)

and specificity ranging from 0.70 to 0.79 (in general, sen-

sitivity and specificity levels are best if they are over 0.9

for prognostic and diagnostic tests. However, such general

rules may not apply in this situation since these constructs

are being used simply to compare approaches). Leeflang

et al. (2005) examined eight articles describing 28 sets of

validated methodological search filters for studies of

diagnostic accuracy through testing the capacity of the

filters to retrieve articles identified in systematic reviews,

finding that the filters ranged widely in their ability to

correctly identify relevant articles.

Yet there are indications that methodological search

filters specific to medicine are not accurate across disci-

plines. Murphy (2002) examined effectiveness search fil-

ters for the veterinary sciences and found wide variability

in accuracy when broken down by journal and purpose of

study. In their analysis, sensitivity ranged from 0.05–0.88

and specificity from 0.10–1.00, indicating that, overall, the

search filters they used were not effective in identifying

desired articles. Similarly, Dickersin et al. (1994) reviewed

articles which identified RCT filters in ophthalmology and,

while many studies were correctly identified, the authors

felt that the achieved sensitivity levels of 51–77%, were

not satisfactory.

The accuracy and efficiency of methodological search

filters for social care has not been established. In social

work and the non-medical helping professions, Gibbs

(2003) appears to be the only author to have proposed a

unique set of methodological filters and a process for car-

rying out searches in the context of EBP. In the spirit of

inquiry that is very much in line with the EBP approach

and the legacy of Leonard Gibbs, we decided to test Gibbs’

methodological search filters in the hopes of building and

improving upon his seminal work. This study uses 12

Campbell Collaboration systematic reviews of effective-

ness to: (1) ascertain which scholarly database is most

likely to yield the largest number of relevant articles for

effectiveness questions in social care; (2) develop a new set

of social care methodological search filters; and (3) com-

pare the sensitivity and specificity of these methodological

search filters with two other commonly used filters. In the

process, we also introduce a new metric, the Avalanche

Index, for measuring the efficiency of searches. The crea-

tion of this tool and other filters like it is crucial if clinical

social workers are expected to efficiently locate evidence

within the large and growing body of academic studies.

Methods

Using the final included set of studies identified in 12

Campbell Collaboration systematic reviews (5 pilot, 7 final

study sample) as a reference standard, a new methodo-

logical search filter was developed and validated and

compared with the filters used by Gibbs (2003) and the

Medline filters referenced in the Cochrane Collaboration

Handbook for Systematic Reviews of Interventions (2009).

The Cochrane Search filter was originally designed by

Carol Lefebvre as published in Dickersin et al. (1994). The

terms have been modified, as necessary, over time.

Search Strategy

Each Campbell Collaboration systematic review is based

on an exhaustive and time-consuming search of the liter-

ature, across multiple databases, and is the most compre-

hensive process for identifying relevant and rigorous

experimental and quasi-experimental studies used to

answer a particular social welfare question. Similar to

Cochrane Collaboration systematic reviews of the medical

Clin Soc Work J (2011) 39:390–399 391

123

literature, Campbell Collaboration reviews are built upon

pre-specified and inclusive searches of the peer-reviewed

and gray (unpublished) literature. Since sensitivity (i.e.,

finding all studies for a given question) for these reviews is

of utmost importance, methodological search filters are not

used. Initial ‘Hits’ or results from Campbell Collaboration

content searches are subjected to manual screening

involving at least two raters, with studies being screened

for both content relevance and methodological rigor.

While this process is certainly meticulous and inclusive,

and the results of systematic reviews are a rich source of

evidence for practitioners, the process is time-consuming

and painstaking. It is unrealistic to expect practitioners to

ascertain answers to pressing clinical questions in this

manner. Nonetheless, due to the quality of their search

processes, Campbell Collaboration reviews can be seen as

the benchmark, or reference standard, for identifying the

highest quality effectiveness studies for a given effective-

ness question.

Final included studies from existing Campbell Collab-

oration (C2) systematic reviews in social welfare were used

as a subject-specific reference standard upon which our

stable of methodological search filters was tested, a strat-

egy that has been employed using Cochrane Collaboration

systematic reviews (see, for example, Leeflang et al. 2005).

For this study, Campbell Collaboration search strategies

from existing C2 reviews were combined with five differ-

ent sets of methodological search filters in an effort to

identify as many of the final included studies in each

review (a measure of sensitivity) while cutting down the

number of irrelevant studies (a measure of specificity). In

order to narrow down the number of reviews used to test

the methodological filters, only published social welfare

reviews available in August 2009 were selected (n = 26).

This number was further reduced (n = 14) by selecting

only reviews that exclusively contained randomized con-

trolled trials (RCTs). Although constricting the C2 sample

in this way is somewhat limiting and does not reflect

standard practice in C2 reviews (i.e., C2 reviews often

include different types of study designs), methodological

search filters tend to be method-specific. Although there is

some debate about the hierarchy of evidence (Upshur and

Tracy 2004; Rubin 2008), locating well-conducted RCTs

where they exist is imperative for any reasonable search of

effectiveness studies.

Finally, two of the reviews used the same search strat-

egies and therefore one of the systematic reviews was

removed. Thus, there were 12 systematic reviews which

met our inclusion criteria. Five were used as pilot studies

(Barlow et al. 2003; Kristjansson et al. 2007; MacDonald

and Turner 2007; Smedslund et al. 2007; Zwi et al. 2007)

and the remaining seven were used to test the filters

(Barlow and Parsons 2005; Coren and Barlow 2004;

Ekeland et al. 2005; Littell et al. 2005; MacDonald and

Turner 2007; Mayo-Wilson et al. 2008; Scher et al. 2006).

Databases

The first step in the process was to ascertain which dat-

abases contained the largest number of articles found in C2

reviews. The final list of included studies from two C2

reviews (Barlow et al. 2003; Zwi et al. 2007) was searched

using 8 databases: Social Science Abstracts, PsychINFO,

Cochrane CRCT, ERIC, Medline, EMBASE, CINAHL,

and Social Service Abstracts (Table 1). Please note that

Barlow et al. (2003) is an earlier version of the Barlow

et al. 2005) review, ‘Parent-training programmes for

improving maternal psychosocial health.’

PsycINFO and Cochrane CRCT were the top performing

databases, containing all of the articles for Barlow et al.

(2003) and a substantial proportion of studies contained in

Zwi et al. (2007). Medline also performed fairly well for

the Zwi et al. (2007) study. However, it should be noted

that Cochrane CRCT is updated with articles included in

Cochrane systematic reviews, and the two selected reviews

were co-registered with Cochrane. The remaining data-

bases did not hold many articles for Barlow et al. (2003),

ranging from one study found to nine studies found out of

22 possible, while ERIC, Social Science Abstracts, and

EMBASE held a moderate amount of articles for Zwi et al.

(2007), between 4 and 11; CINAHL held none (Table 1).

Results from this process indicate that PsycINFO is, far and

away, the most likely database to contain relevant studies

Table 1 Methodological filters

Filter set Filters

Avalanche RCT; randomi*; control* trial*; control* clinical;

clinical trial*; random* assign*; random* allocat*;

wait* list*; wait*-list*; control* group*; control*

condition*; quasi-ex*; quasi ex*; control* near

intervention; control* near treat*

Avalanche-

RCT

RCT; randomi*; control* trial*; control* clinical;

clinical trial*; random* assign*; random* allocat*

Gibbs Random*; controlled Clinical trial*; Control group*;

evaluation stud*; study design; statistical*

significanc*; double-blind; placebo

Gibbs-RCT Random*; controlled clinical trial*; control group*

Cochrane Randomized controlled trial; controlled clinical trial;

randomized; placebo; clinical trials as topic;

randomly; tria; not (animals not (humans and

animals)

* Indicates a ‘wild card’ truncation using a ‘wild card’ (a marker at

the end of a string of words prompting the database to search for all

terms containing the letters to the left of the *)

392 Clin Soc Work J (2011) 39:390–399

123

for the purpose of retrieving social care articles. In order to

simplify the validation process, we decided to use Psy-

cINFO and MEDLINE as our validation databases.

Methodological Filters

Development of EFFECTS

Efficient Methodological Filter for Experiments Comparing

Treatments in Social Care. The social care methodological

search filter terms for this study were developed in a pilot

study that involved a trial and error process using five C2

social welfare reviews as test cases. These five reviews

(Barlow et al. 2003; Kristjansson et al. 2007; MacDonald and

Turner 2007; Smedslund et al. 2007; Zwi et al. 2007) were

not used as part of the later validation phase. Individual

methodological search filter terms were selected based on

their ability to find the studies listed in each C2 review using

PsycINFO. Specifically, we began by applying the Gibbs

filter on a single review (Barlow et al. 2003) and systemat-

ically tried to improve its performance (i.e., increase its

capacity to correctly identify studies included in the C2

review while minimizing the number of false positives) by

modifying, adding, or deleting individual search terms

within the filter. For instance, the Gibbs term ‘random’ might

find studies using random sampling rather than random

assignment, while the EFFECTS term randomi* would only

find studies using the word ‘randomized’ or ‘randomization’.

Similar to validation studies of predictive instruments, there

is a danger that a tool (in this case, the filter) will be ‘overfit’

to the data at hand (in this case, the review upon which it is

being developed). That is, the filter would predict well for the

systematic review upon which it was constructed, but would

perform less well when applied to other reviews. Similar to

validation studies using a construction and validation sam-

ple, we then applied the EFFECTS filter to four other sys-

tematic reviews (Kristjansson et al. 2007; MacDonald and

Turner 2007; Smedslund et al. 2007; Zwi et al. 2007) and

adjustments were made as we proceeded.

Comparison Filters

EFFECTS terms were compared to two other sets of

methodological filters: Gibbs (2003) effectiveness meth-

odological search filters and those detailed in the Cochrane

Collaboration Handbook for Systematic Reviews of Inter-

ventions (2009). While the Cochrane filter terms were

designed to identify randomized controlled trials (RCTs),

the EFFECTS and Gibbs filter terms were designed to

retrieve both RCTs and quasi-experimental studies. Thus,

to be fair, RCT-only versions of the EFFECTS and Gibbs’

filter terms were created and tested as well. For a list of the

five sets of methodological filters, please see Table 1.

Analysis

Searches were individually run in both PsycINFO and

MEDLINE for each included review. The following search

results were recorded: the number of hits found using each

systematic review’s original search terms and the number

of hits using the review search terms in combination with

each of the methodological filters. Using the number of

reviewed articles found by the searches, the results of the

searches can be grouped into four categories:

1. True positive (TP): article found AND included in

original systematic review

2. False positive (FP): article found but NOT included in

original review

3. False negative (FN): article NOT found but was

included in original systematic review

4. True negative (TN): article NOT found and is NOT

included in original review.

The example in Table 2 is the search history in Psy-

cINFO of our analysis of Barlow and Parsons (2005)

review and the EFFECTS terms.

Evaluations of methodological search filters in the

medical sciences have used various measures including:

sensitivity (Dickersin et al. 1994; Haynes et al. 1994;

Leeflang et al. 2005; Murphy 2002); specificity (Haynes

et al. 1994; Leeflang et al. 2005; Murphy 2002); precision

(Dickersin et al. 1994; Haynes et al. 1994; Taylor et al.

2003, 2007; Watson and Richardson 1999). These are all

strategies to measure the number of desired articles

retrieved as a proportion of all articles retrieved; positive

likelihood ratio (Ingui and Rogers 2001), indicating the

proportion of desired articles retrieved as compared to the

proportion of undesired articles retrieved; and diagnostic

odds ratio (Deville et al. 2000), which is the positive

likelihood ratio divided by the negative likelihood ratio

(the proportion of desired articles that were not found as

compared to the proportion of undesired articles that were

not found). The present study uses sensitivity and speci-

ficity to evaluate the effectiveness of the methodological

filters. Sensitivity is calculated by TP/(TP ? FN) and

specificity is calculated by TN/(FP ? TN). A measure was

also created, the Avalanche Index (AI), that identifies the

number of hits one would need to read through in order to

find one of the desired studies (i.e., studies included in a

review). A similar measure was used by Bachmann et al.

(2002) ‘‘number needed to read’’, which, like the AI, is

calculated as 1/precision. In the present study, the AI is

calculated by taking the total number of hits (TP ? FP)

yielded by the full search (i.e., subject terms combined

with methodological search filter terms) and dividing by

the number of actual studies included in the review that

were found using this search (TP) or ((TP ? FP)/(TP)).

Clin Soc Work J (2011) 39:390–399 393

123

Similar to Number Needed to Treat (NNT) and Number

Needed to Harm (NNH), the AI can also be defined as the

Number Needed to Find (NNF). The AI is probably a better

tool for gauging the success of EBP searches since it

provides an overall measure of efficiency for a search

strategy whereas specificity and sensitivity can only

describe predictive capacity of a strategy as it relates to

previous searches.

Results

Visual comparisons were made for each of the seven

reviews used to validate the search filters in terms of their

sensitivity, specificity, and Avalanche Index (Table 3) for

each of the five sets of methodological search filters

(EFFECTS, EFFECTS-RCT, Gibbs, Gibbs-RCT, and

Cochrane). The number of hits retrieved in Medline using

the content search terms for each review (without the

benefit of a methodological filter) ranged from 26 (Littell

et al. 2005) to 143,084 (Scher et al. 2006) with an average

of 29,525, indicating substantial variability in terms of the

precision of the C2 content searches. There was also sub-

stantial variability in PsycINFO, with total hits ranging

from 559 (Littell et al. 2005) to 66 017 (Scher et al. 2006)

and an average of 17,542 hits. In combination with the

methodological filters, results were also varied. In Medline,

hits ranged from 14 (Littell et al. 2005) to 27,178 (Scher

et al. 2006); in PsycINFO they ranged from 50 (Coren and

Barlow 2004) to 6,239 (Scher et al. 2006).

The first step in the process of measuring each of the

filters’ capacity to retrieve the desired articles was to iso-

late only those studies from reviews that were available in

each of the databases through a title and author search. The

total number of articles that could possibly be found in

PsycINFO ranged from two (Coren and Barlow 2004) to 20

(Ekeland et al. 2005; Littell et al. 2005) with an average of

11. In Medline, possible hits ranged from one (Coren and

Barlow 2004) to 18 (Scher et al. 2006) with an average of

seven. Once the number of possible studies was found,

content terms were combined with each of the filters. Hits

from the combination of filters and review search terms in

PsycINFO ranged from zero (Coren and Barlow 2004) to

16 (Littell et al. 2005) and, in Medline, from one (Coren

and Barlow 2004) to 16 (Scher et al. 2006).

Next, we measured the sensitivity, specificity, and

Avalanche Index for each of the studies, and then calcu-

lated the average for each of these constructs across all

reviews (Table 3). Sensitivity ranged from 0 (Coren and

Barlow 2004) to 0.88 (Scher et al. 2006) in PsycINFO and

from 0.67 (Barlow and Parsons 2005; Ekeland et al. 2005)

to 1.00 (Barlow and Parsons 2005; Coren and Barlow

2004; Littell et al. 2005) in Medline. The EFFECTS filter

had the best sensitivity scores in PsycINFO, achieving the

highest in 5 out of 7 reviews, with an average of 0.67.

However, the EFFECTS-RCT filters performed the poor-

est, achieving the lowest sensitivity in 4 of the 7 reviews,

with an average of 0.48. The Cochrane, Gibbs, and Gibbs-

RCT filters did not demonstrate any clear trends for sen-

sitivity in PsycINFO, with averages of 0.51, 0.60, and 0.60

respectively. In Medline, the filters performed similarly,

achieving identical sensitivity scores in five of the seven

reviews. The EFFECTS filters performed the best in the

remaining two reviews, with an average of 0.87, however

the sensitivity scores between filters did not vary

appreciably.

Table 2 Example search strategy

Step Search terms Hits Review

articles

found

Step—1 Barlow and Parsons (2005)

search

(parent* training or parent* program* or parent* education) and (toddler or

infant or preschool or pre-school or pre school or baby or babies)

2,137 4

Step 2—EFFECTS Search (RCT or randomi* or control* trial* or control* clinical or clinical trial* or

random* assign* or random* allocat* or wait* list* or wait*-list* or control*

group* or control* condition* or quasi-ex* or quasi ex* or control* near

intervention or control* near treat*)

68,804 3

Step 3—Barlow and Parsons (2005)

terms combined with EFFECTS terms

(parent* training or parent* program* or parent* education) and (toddler or

infant or preschool or pre-school or pre school or baby or babies) and (RCT

or randomi* or control* trial* or control* clinical or clinical trial* or

random* assign* or random* allocat* or wait* list* or wait*-list* or control*

group* or control* condition* or quasi-ex* or quasi ex* or control* near

intervention or control* near treat*)

341 3

* Indicates a ‘wild card’ truncation using a ‘wild card’ (a marker at the end of a string of words prompting the database to search for all terms

containing the letters to the left of the *)

394 Clin Soc Work J (2011) 39:390–399

123

Table 3 Review summary

Filters Database

PsycInfo Medline

Sensitivity Specificity Avalanche index Sensitivity Specificity Avalanche index

Barlow and Parsons (2005). Group-based parent-training programmes for improving emotional and behavioural adjustment in 0–3 year old

children.

EFFECTS 0.75 0.84 114 1.00 0.82 49

EFFECTS-RCT 0.25 0.92 182 1.00 0.84 43

Gibbs 0.50 0.85 160 1.00 0.78 60

Gibbs-RCT 0.50 0.86 150 0.67 0.81 76

Cochrane 0.25 0.90 205 0.67 0.86 57

Coren and Barlow (2004) Individual and group based parenting for improving psychosocial outcomes for teenage parents and their children

EFFECTS 0.50 0.94 116 1.00 0.94 116

EFFECTS-RCT 0.00 0.97 N/A 1.00 0.96 79

Gibbs 0.50 0.94 115 1.00 0.89 212

Gibbs-RCT 0.50 0.94 105 1.00 0.93 132

Cochrane 0.00 0.96 N/A 1.00 0.96 80

Ekeland et al. (2005) Exercise to improve self- esteem in children and young people

EFFECTS 0.35 0.89 496 0.67 0.90 121

EFFECTS-RCT 0.25 0.94 405 0.67 0.92 92

Gibbs 0.30 0.86 720 0.67 0.88 143

Gibbs-RCT 0.30 0.88 630 0.67 0.90 121

Cochrane 0.30 0.90 516 0.67 0.93 83

Littell et al. (2005) Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10–17

EFFECTS 0.75 0.72 11 1.00 0.79 1

EFFECTS-RCT 0.70 0.81 9 1.00 0.64 1

Gibbs 0.65 0.76 11 1.00 0.71 1

Gibbs-RCT 0.65 0.80 10 1.00 0.71 1

Cochrane 0.80 0.78 9 1.00 0.86 1

MacDonald and Turner (2007) Treatment foster care for improving outcomes in children and young people

EFFECTS 0.82 0.94 32 0.75 0.96 14

EFFECTS-RCT 0.82 0.96 18 0.75 0.97 12

Gibbs 0.73 0.93 39 0.75 0.93 24

Gibbs-RCT 0.73 0.94 36 0.75 0.95 16

Cochrane 0.82 0.95 27 0.75 0.97 11

Mayo-Wilson et al. (2008) Personal assistance for adults (19–64) with physical impairments

EFFECTS 0.67 0.92 671 0.80 0.88 1,529

EFFECTS-RCT 0.67 0.94 462 0.80 0.90 1,318

Gibbs 0.67 0.90 772 0.80 0.85 1,980

Gibbs-RCT 0.67 0.91 690 0.80 0.89 1,419

Cochrane 0.67 0.93 535 0.80 0.92 1,102

Scher et al. (2006) Interventions intended to reduce pregnancy- related outcomes among adolescents

EFFECTS 0.82 0.91 439 0.89 0.89 1,016

EFFECTS-RCT 0.71 0.94 323 0.72 0.91 1,003

Gibbs 0.88 0.89 505 0.89 0.81 1,699

Gibbs-RCT 0.82 0.91 446 0.89 0.90 924

Cochrane 0.71 0.92 425 0.72 0.93 822

Clin Soc Work J (2011) 39:390–399 395

123

Specificity ranged from 0.72 (Littell et al. 2005) to 0.97

(Coren and Barlow 2004) in PsycINFO and from 0.64

(Littell et al. 2005) to 0.97 (MacDonald and Turner 2007)

in Medline. The EFFECTS-RCT terms performed best in

PsycINFO, achieving the highest specificity scores in all of

the reviews, with an average of 0.93. In Medline, the

Cochrane filters achieved the highest specificity scores in

all of the reviews, with an average of 0.92. However,

similar to specificity in PsycINFO, specificity scores in

Medline did not vary greatly and, while the EFFECTS-

RCT and Cochrane filters achieved the highest specificity,

they did not substantially outperform the other filters.

In PsycINFO the Avalanche Index (an indicator of how

many studies a practitioner would have to sift through in

order to find a highly relevant article) ranged from nine

(Littell et al. 2005) to 772 (Mayo-Wilson et al. 2008); in

Medline it ranged from one (Littell et al. 2005) to 1980

(Mayo-Wilson et al. 2008). The EFFECTS-RCT filters

achieved the best avalanche scores in PsycINFO ranking

first in 4 of the 8 studies and an average AI of 204. The

EFFECTS filters ranked second, with top indexes in two

reviews and an average AI of 239. The Gibbs filters had the

highest (most inefficient) overall Avalanche Index scores,

ranking last in 5 of the 7 reviews with an average of 233. In

Medline the Cochrane filters had the best AI in 5 of the 7

reviews, with an average of 308. Ranking second, the

EFFECTS-RCT filters had the best Avalanche Indexes in

two of the reviews, with an average of 364.

Discussion and Applications to Clinical Social Work

Practice

Clients deserve no less than our level best to provide them

with services that have a high likelihood of being suc-

cessful. But in order for EBP to become a practice reality,

caseworkers need to be able to quickly locate and evaluate

the research evidence. The results of this study indicate

that, when conducting searches for effectiveness studies in

social care, using the proposed EFFECTS-RCT and full

EFFECT filter terms would be a beneficial addition to a

practitioner’s search methodology. 1

While all of the

methodological search filter sets were able to substantially

reduce the number of hits produced by the searches found

in the reviews, the EFFECTS-RCT and full EFFECTS filter

terms were often superior at retrieving the desired articles

while reducing the total amount of hits in PsycINFO.

When comparing the EFFECTS-RCT to the full

EFFECTS filter terms, and the Gibbs-RCT to the full Gibbs

filter terms, we found that methodological search filters

specifying only RCTs yielded a much smaller number of

irrelevant hits. As such, we recommend that, when

searching for articles, the full EFFECTS filter terms should

be used if one is searching for quasi-experimental studies

or when searching for articles on topics that are not highly

researched. Similar to Gibbs’ (2003) suggested methods for

searching, the full EFFECTS terms can also be used first

and, if a large number of hits are returned, the EFFECTS-

RCT filter can be used to better limit the search.

In interpreting the results, an important caveat regarding

the Avalanche Index should be made. The Avalanche Index

is calculated by taking the number of hits retrieved using

the substantive area search terms combined with the

methodological search filter terms, and dividing this by the

number of correct articles found using these combined

terms. The number of correct or total articles found is

dependent upon the number of correct articles carried in the

database searched. All else being equal, the larger the

number of correct articles found, the smaller the Avalanche

Index becomes. Therefore, unless the total number of

Table 3 continued

Filters Database

PsycInfo Medline

Sensitivity Specificity Avalanche index Sensitivity Specificity Avalanche index

Average for all reviews

EFFECTS 0.67 0.88 268.43 0.87 0.88 406.57

EFFECTS-RCT 0.49 0.93 233.17 0.85 0.88 364.00

Gibbs 0.60 0.88 331.71 0.87 0.84 588.43

Gibbs-RCT 0.60 0.89 295.29 0.83 0.87 384.14

Cochrane 0.51 0.91 286.17 0.80 0.92 308.00

Average 0.57 0.90 282.95 0.84 0.88 410.23

Sensitivity TP/(TP ? FN), Specificity TN/(FP ? TN), Avalanche index ((TP ? FP)/(TP))

1 It should be noted that, while we use the term ‘effectiveness’ here,

most of these studies would be more accurately described as efficacy

studies—the difference being the efficacy studies test whether an

intervention works in tightly controlled settings while effectiveness

studies would test whether interventions work in more typical practice

settings.

396 Clin Soc Work J (2011) 39:390–399

123

articles retrieved and the total possible number of correct

articles within a database happen to be the same across

studies (which is unlikely), direct Avalanche Index com-

parisons can only be made within, not between, individual

reviews. That is, content searches also have greater and

lesser degrees of precision, and this will probably influence

the AI to a much greater extent than the selection of

methodological filter. Some narrowly specified searches

will yield a small number of total hits while less carefully

specified searches will yield a large number of total hits.

There is little that even a well-constructed methodological

search filter can do to overcome the avalanche of irrelevant

hits triggered by a poorly specified search.

The process of developing and testing methodological

filters prompted a number of important observations. First,

as indicated by the AI, the total number of hits yielded by

substantive area search terms strongly influences the total

number hits found when using methodological search fil-

ters. For example, Barlow et al. (2005) search yielded

5,126 hits and, when combined with the full EFFECTS

filter, yielded 612 hits (a reduction of 88%). On the other

hand, Littell et al. (2005) original content search terms

yielded 559 hits and, when combined with the EFFECTS

terms, yielded 169 hits (a reduction of 70%). While the

Barlow, Coren, and Stewart-Brown search reflects a larger

reduction in hits, the number of remaining hits is still very

large due to the review’s use of fairly broad content search

terms. This finding highlights the importance of using

efficient substantive area search terms in combination with

methodological search filter terms in order to reduce the

number of hits that one must examine. An important dis-

tinction must be made, though, between systematic review

searches and practical evidence searches. Specifically,

systematic reviews require broadly specified content sear-

ches in order to ensure that all studies in a particular area

are found, and such an approach requires a great deal of

time, effort, and person-hours. Alternatively, practitioners

in the helping professions must be able to search quickly

and efficiently in order to find methodologically rigorous

studies needed to guide their practice decisions. Both

approaches are necessary components of evidence

informed practice. Users would do well to first carefully

consider the question they are asking and then follow

guidelines for content searches established by such authors

as Gibbs (2003) and McGibbon et al. (1991).

Another important finding was the role played by the

descriptive terms used in keywords, titles, and abstracts

within journal publications. When methodological filters

failed to find an article present in a database, it was often

due to the fact that the article’s keywords, title, or abstract

did not contain any methodology descriptors. This finding

highlights the need for authors and journal editors to apply

strict guidelines with respect to the inclusion of key words

in their subject headings and abstracts that indicate that

type of methodology employed by the study. While this

appears to occur with more frequency in more recent

articles and in articles published in the health sciences,

there is surprisingly little consistency in the published

literature.

Another important consideration in using search filters is

the capacity of any single database to retrieve relevant

articles. We found that, at least for studies contained in

Campbell systematic reviews, PsycInfo consistently con-

tained the greatest number of relevant articles. Our results

are perhaps related to the findings of others who have

investigated the capacity of Social Work Abstracts (SWA)

and found it limited. Both Shek (2008) and Holden et al.

(2009) assessed the database Social Work Abstracts (SWA)

and concluded that SWA does not contain a satisfactory

number of social work articles and, even when articles are

found, they can be difficult to retrieve. Based on our

findings, we contend that it is important to ensure that

methodological filters are being employed within databases

that have the capacity to yield relevant articles. At this

point, SWA may not be up to the task.

Recommendations and Conclusion

Based on the findings of this study, we recommend that

journal editors across disciplinary boundaries find a way to

standardize their use of methodological descriptors and

require that these are added to every new journal article

submitted. Over time, searches will become far more effi-

cient and accurate. Similarly, Taylor et al. (2007) contend

that standardization of search terms is required in order to

improve the efficiency of searches. Taylor et al. (2003)

further recommend that these methodological descriptors

be clearly included in abstracts in order support ease of

searching. We could not agree more. If EBP is to become a

reality, publishers must make the search process as easy

and efficient as possible.

The following is a recommended search strategy inten-

ded to help practitioners locate high quality effectiveness

studies on their topic of interest. Step 1: Refine substantive

area search terms. This first step involves identifying

specific terms related to the area of interest. For example, if

a practitioner were interested in learning about the effects

of cognitive behavioral therapy on anxiety in school-aged

children, the following subject related search terms would

be selected for use in PsycINFO: ‘‘((cognitive behavio*

therapy or cognitive therapy or CBT) and (anxiety)).’’ Step

2: Methodological Filters. The search terms identified in

Step 1 should be combined with the appropriate set of

methodological search filter terms or MOLES. Using the

example from Step 1, the EFFECTS-RCT methodological

Clin Soc Work J (2011) 39:390–399 397

123

search filters might be appropriate since CBT and anxiety

are common research topics and the EFFECTS-RCT filter

terms are more likely to return a higher number of more

rigorous studies. Again, the set of methodological search

filters selected should be based on the amount of literature

available in the topic area of interest. If one is searching for

a more obscure set of literature, the full-set of EFFECTS

terms would be more appropriate. Step 3: Outcomes. If,

after combining the search terms from Step 1 and 2, the

number of hits remains large, search terms that specify the

outcome of interest should be included. Carrying on with

the example, outcome search terms may include: academic

achievement or academic achievement motivation. Step 4:

Demographic variables. If the number of hits is too large,

search terms specifying information regarding the popula-

tion of interest should be included. In this example, search

terms might be: elementary school student *. By following

these guidelines, practitioners should be able to quickly

locate a set of articles on their topic of interest while

avoiding what can be called an avalanche of irrelevant hits.

There are several limitations to this study. First, C2

reviews tend to have rigid inclusion criteria and may only

be representative of high quality studies that ask a partic-

ular question in a particular way. They may exclude studies

for various reasons that have nothing to do with the prac-

titioner question and, as a result, may not be representative

of all relevant RCT’s. The full EFFECTS search terms are

better suited for avoiding this type of problem than either

EFFECTS-RCT or the Medline filter. Second, we only

tested effectiveness filters. There are other types of ques-

tions that require different filters (e.g., risk/prognosis,

assessment/diagnosis, qualitative). Finally, we only tested

these effectiveness filters in two databases: PsycINFO and

Medline. This approach was taken after finding that these

two databases contained a substantial portion of the studies

included in systematic reviews. There may be excellent

studies in other databases and our filters may be less

effective than the original Gibbs (2003) filters at identify-

ing appropriate articles in those databases. Nonetheless,

our finding that PsycINFO is the best database for finding

rigorous studies in social care should serve as a wake-up

call to social services database developers. Finally, it

should be noted that the search strategies identified in the

reviews were, oftentimes, very broad and did not identify

concise subject terms. While practitioners should use more

specific search criteria, for the purpose of the present study

the search terms were taken at face value. As such, the

results produced by our searches probably contain many

more false positives or irrelevant articles than would be the

case with better specified subject searches.

In conclusion, practitioners are being asked to use evi-

dence to guide their decisions when working with con-

sumers of social services. Yet the impossibly large number

of articles in certain subject areas and the enormous effort

required to ascertain the quality of studies is daunting even

for the most seasoned and committed helping profession-

als. The information age is upon us and requires an

evolving set of tools to help us locate and use evidence

efficiently. Efficient and effective methodological search

filters, such as EFFECTS and EFFECTS-RCT can help us

meet this challenge.

References

Bachmann, L. M., Coray, R., Estermann, P., & Ter Riet, G. (2002).

Identifying diagnostic studies in MEDLINE: Reducing the

number needed to read. Journal of the American Medical Informatics Association, 9, 653–658.

Barlow, J., Coren, E., & Stewart-Brown, S. (2003). Parent-training

programmes for improving maternal psychosocial health.

Campbell Systematic Reviews, 1–101 [no longer available]. Barlow, J., & Parsons, J. (2005). Group-based parent-training

programmes for improving emotional and behavioural adjust-

ment in 0–3 year old children. Campbell Collaboration System- atic Reviews, 2, 1–56.

Barlow, J., Coren, E., & Stewart-Brown, S. (2005). Parent-training

programmes for improving maternal psychosocial health. Camp- bell Systematic Reviews, 3, 101.

Coren, E., & Barlow, J. (2004). Individual and group based parenting

for improving psychosocial outcomes for teenage parents and

their children. Campbell Collaboration Systematic Reviews, 1, 1–38.

Deville, W., Bezemer, P., & Bouter, L. (2000). Publications on

diagnostic test evaluation in family medicine journals: An

optimal search strategy. Journal of Clinical Epidemiology, 53, 65–69.

Dickersin, K., Scherer, R., & Lefebvre, C. (1994). Identifying

relevant studies for systematic reviews. British Medical Journal, 309, 1286–1291.

Ekeland, E., Heian, F., Hagen, K. B., Abbott, J., & Nordheim, L.

(2005). Exercise to improve self-esteem in children and

young people. Campbell Collaboration Systematic Reviews, 4, 1–49.

Gibbs, L. (2003). Evidence based practice for the helping profes- sions: A practical guide with integrated multimedia. Pacific Grove: Brooks Cole-Thompson.

Gibbs, L., & Gambrill, E. (2002). Evidence-based practice: Count-

erarguments to objections. Research on Social Work Practice, 12, 452–476.

Gray, M., Plath, D., et al. (2009). Evidence-based social work: A critical stance. New York: Routledge.

Haynes, R., Wilczynski, N., McKibbon, A., Walker, C., & Sinclair, J.

(1994). Developing optimal search strategies for detecting

clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association, 1, 447–458.

Higgins, J., & Green, S. (Eds.). (2009). Cochrane handbook for systematic

reviews of interventions version 5.0.2. The Cochrane Collaboration, 2009. Available from www.cochrane-handbook.org.

Holden, G., Barker, K., Covert-Vail, L., Rosenberg, G., & Cohem, S.

A. (2009). Social Work Abstracts fails again: A replication and

extension. Research on social work practice, 16(6). Ingui, B. J., & Rogers, M. A. M. (2001). Searching for clinical

prediction rules in MEDLINE. Journal of the American Medical Informatics, 8, 391–397.

398 Clin Soc Work J (2011) 39:390–399

123

Kristjansson, B., et al. (2006). School feeding for improving the

physical and psychosocial health of disadvantage students.

Campbell Systematic Reviews, 14, 1–132. Leeflang, M., Scholten, R., Rutjes, A., Reitsma, J., & Bossuyt, P.

(2005). Use of methodological search filters to identify diagnos-

tic accuracy studies can lead to the omission of relevant studies.

Journal of Clinical Epidemiology, 59, 234–240. Littell, J. H., Popa, M., & Forsythe, B. (2005). Multisystemic therapy

for social, emotional, and behavioral problems in youth aged

10–17. Campbell Collaboration Systematic Reviews, 1, 1–60. MacDonald, G. M., & Turner, W. (2007). Treatment foster care for

improving outcomes in children and young people. Campbell Collaboration Systematic Reviews, 9, 1–95.

Mayo-Wilson, E., Montgomery, P., & Dennis, J. (2008). Personal

assistance for adults (19–64) with physical impairments. Campbell Collaboration Systematic Reviews, 3, 1–36.

McGibbon, A., Eady, A., & Marks, S. (1991). PDQ evidence-based principles and practice. Ontario, Canada: BC Decker.

Murphy, S. A. (2002). Applying methodological search filters to CAB

abstracts to identify research for evidence-based veterinary

medicine. Journal of the Medical Library Association, 90, 406–410.

Rosen, A., & Proctor, E. K. (Eds.). (2003). Developing practice guidelines for social work intervention: Issues, methods, and a research agenda. New York: Columbia University Press.

Rubin, A. (2008). Practitioner’s guide to using research for evidence- based practice. New Jersey: Wiley.

Scher, L. S., Maynard, R. A., & Stagner, M. (2006). Interventions

intended to reduce pregnancy-related outcomes among adoles-

cents. Campbell Collaboration Systematic Reviews, 12, 1–70. Shek, D. T. L. (2008). Comprehensiveness of Social Work Abstracts

as a database for researchers and practitioners. Research on Social Work Practice, 18(5), 500–506.

Smedslund, G., Dalsbø, T. K., Steiro, A. K., Winsvold, A., & Clench-

Aas, J. (2007). Cognitive behavioural therapy for men who

physically abuse their female partner. Campbell Collaboration Systematic Reviews, 4, 1–57.

Taylor, B. J., Dempster, M., & Donelly, M. (2003). Hidden gems:

Systematically searching electronic databases for research pub-

lications for social work and social care. British Journal of Social Work, 33(4), 423–439.

Taylor, B., Wylie, E., Dempster, M., & Donnelly, M. (2007).

Systematically retrieving research: A case study evaluating

seven databases. Research on Social Work Practice, 17(6), 697–706.

Upshur, R., & Tracy, C. S. (2004). Legitimacy, authority, and

hierarchy: Critical challenges for evidence-based medicine. Brief Treatment and Crisis Intervention, 4, 197–204.

Watson, R. J. D., & Richardson, P. H. (1999). Identifying randomized

controlled trials of cognitive therapy for depression: Comparing

the efficiency of Embase, Medline and PsycINFO bibliographic

databases. British Journal of Medical Psychology, 72(4), 535–542.

Zwi, K., Woolfenden, S., Wheeler, D., O’Brien, T., Tait, P., &

Williams, K. (2007). School-based education programmes for

the prevention of child sexual abuse. Campbell Collaboration Systematic Reviews, 5, 1–44.

Author Biographies

Aron Shlonsky is Associate Professor and Factor-Inwentash Chair in Child Welfare at the University of Toronto, Factor-Inwentash Faculty

of Social Work.

Tobi Michelle Baker is a Research Assistant at the University of Toronto, Factor-Inwentash Faculty of Social Work.

Esme Fuller-Thomson is Professor and Sandra Rotman Chair at the University of Toronto. She is cross-appointed to the Factor-Inwentash

Faculty of Social Work, the Department of Family & Community

Medicine & the Faculty of Nursing.

Clin Soc Work J (2011) 39:390–399 399

123

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.