Using Data Presentation

FEATUREARTICLE

experienti something is a problem.

A practice may be so widespread that it has been accepted as conventional wisdom — whether or not it is

effective or true."

10 Things Every Literacy Educator and School Librarian Should Know about Research

Adapted with permission from; Duke, N. K., ft Martin, N. M. (2011). 10 things every lit-l

eracy educator should know about researeh. The Reading Teacher, 65, 9-22. '

A s educators interested in texts and .xlj-eading, literacy teachers and school librarians share a kinship. While we might approach literacy from different backgrounds and varied perspectives, we are all focused on the same goal-to support the reading development of students.

Many literacy teachers and school librarians share another disposition—a distrust of re-

search. "Research-based," "research-proven," "scientifically based," "evidenced-based"-

in the reading and school library world these days, it seems that the term research is

being used everywhere. It is also being misused and misunderstood. We are encountering

a growing number of literacy teachers and school librarians who are dismissing research

altogether, based on beliefs that research is simply a propaganda tool for those trying to

push a particular approach to instruction or based on a feeling that research will be ig-

nored by school administrators anyway.

The purpose of this article is to argue for the value of research for literacy and school

library educators, including but not limited to classroom teachers, school librarians, lit-

eracy coaches, technology facilitators, administrators, curriculum specialists, and profes-

8 T E A C H E R L I B R A R I A N 4 0 : 4

sors, and provide some information that

may help us to make better use of research

and, at the same time, guard against mis-

use as we plan for and teach literacy and

transliteracies.

Specifically, we discuss 10 things that

we believe every literacy and school library

educator should know about research:

1. What research can do.

2. What research is.

3. What research is not.

4. The difference between research-

based and research-tested.

5. Many kinds of research have valuable

contributions to make to our understand-

ing of literacy learning, development, and

education.

6. Different kinds of research are good

for different questions.

7. High-quality research has a logic of

inquiry.

8. Conclusions drawn from research are

only as sound as the research itself.

9. Where and how research is published

or presented requires particular attention.

10. Educational research proceeds

through the slow accumulation of knowl-

edge.

1. WHAT RESEARCH CAN DO

Given all there is to know about research,

and all the dangers of misrepresentation,

misuse, and disregard of research, you

might wonder whether research is worth

paying attention to at all. Educators fre-

quently tell us that they just go with what

works for them, that they trust what their

mentor does over what some research

study says, or that research does not re-

ally apply to them or their setting. School

librarians who can quote the research stud-

ies that suggest school libraries contribute

to student success may be hard-pressed to

see value in research that is consistently

ignored by their administrators. Although

we certainly recognize the value of experi-

ence and that research will never provide

all the solutions or answers, we do believe

research should be seen as an essential

guide to policy and practice. Concentrating

here on practice, the following are some of

the reasons for this belief.

OUR EXPERIENCES ALONE MAY MISGUIDE US

For a very long time, people believed the

earth was flat, and that was (and still is) a

reasonable conclusion to draw from per-

sonal, individual observations. Research,

however, eventually proved this A/iew to be

incorrect. In reading and school library ed-

ucation, we have these kinds of examples

as well, though perhaps not as dramatic.

For instance, for a long time we thought

that persistent word reading difficulty, of-

ten called "dyslexia," was primarily a vi-

sual problem. To illustrate, we thought that

reading star as rats reflected a problem

with the visual processing of print. We now

understand that the vast majority of word

reading difficulties are actually caused by

problems in phonological or auditory pro-

cessing rather than in visual processing

(Snowling, 2000).

Similarly, we frequently visit class-

rooms in which teachers assign students

to look up a list of vocabulary words in

the dictionary and write the definition and

a sentence. This is a very common-sense

practice. If we asked people on the street to

suggest a good way to teach children new

vocabulary, many would no doubt recom-

mend exactly this practice. However, as it

turns out, this practice often proves to be

inferior to other approaches that, while less

common and arguably less commonsen-

sible, actually result in greater vocabulary

learning (e.g.. Bos Ö Anders, 1990).

Likewise, in understanding the role of

school libraries in student success, prin-

cipals may be misled by personal experi-

ences. Principals' early experiences with

school libraries may be limited or nega-

tive, and little to no information about the

role of the library in the school may be

provided in postsecondary coursework or

through other professional outlets. Princi-

pals may not understand the role of school

librarians and how to best position them to

support student success, although research

indicates that principal support is crucial

to the success of a school library program

(Hartzell, 2004). Just as research has limi-

tations, so too does common sense, past

experiences, and individual judgment.

SOMETIMES WE DO NOT KNOW WHAT WE DO NOT KNOW

We may not recognize experientially that

something is a problem. A practice may

be so widespread that it has been accepted

as conventional wisdom - whether or not

it is effective or true. For example, for

many years in the United States, reading

materials used in the primary grades were

overwhelmingly stories; informational

text, among other genres, was strongly ne-

glected (Duke, Bennett-Armistead, ft Rob-

erts, 2003). Research is an important tool

for documenting phenomena such as this

(e.g.. Duke, 2000; Moss ft Newton, 2002).

Indeed, many people have shared with

us that they had not really thought about

how little experience children were get-

ting with informational text in the primary

grades. Others had thought about this but

operated with assumptions that young chil-

dren should read stories first and that read-

ing stories was more "natural" for children

than reading other genres. It turns out this

is not the case (e.g., Duke ft Kays, 1998;

Pappas, 1993), and research was again an

important tool for examining an assump-

tion that was taken for granted.

For decades, conventional wisdom

dismissed graphic novels as inferior and

unworthy of school library collections.

Research has also disproved this view. Re-

searchers have found that graphic novels

may motivate males to read, support the

achievement of English learners and stu-

dents with exceptionalities, increase read-

ing comprehension and vocabulary devel-

opment, and teach multiple literacies. (For

more information, see Gavigan, 2012.)

Finally, research has helped us to see

that fixed scheduling and isolated in-

struction—the traditional model of school

library use—is less effective than other

models. In many elementary and some

middle schools, library time for each class

is scheduled on a rigid, fixed basis. Teach-

ers may use library time to plan and pre-

pare for classroom instruction while librar-

ians read to students or teach an isolated

lesson in the library before students check

out books. Gaver (1963) found that library

programs were more effective when teach-

ers stayed with their classes. Lance, Well-

born, and Hamilton-Pennell (1993) in the

first Colorado Study demonstrated that

student success on standardized tests cor-

related with strong, well-funded school

library programs where librarians collabo-

rated with teachers. Subsequent research

replicating the first Colorado Study es-

tablished the importance of librarians and

teachers working together as instructional

partners who coUaboratively plan and

teach lessons (Asselin, 2001; Lance, 2001,

2002; Todd, 2002)

RESEARCH ALLOWS US TO TAKE A LONGER TERM VIEW THAN OUR PERSONAL EXPERIENCES MAY ALLOW

A classroom teacher typically only has one

to two years with a student, and it is dif-

ficult to systematically monitor a student's

progress after that time. This means that

the teacher cannot observe the long-term

outcomes of his or her practices. A school

librarian may see students over a greater

A P R I L 2 0 1 3

number of years, but typically has very

little time with students in any given year.

School librarians can neither systemati-

cally monitor a student's progress in other

classes nor easily see how his or her in-

dividual program data compares to other

school libraries across the state or across

the nation. Research, however, can address

these needs. Researchers can track students

as they move through the grades, for ex-

ample, by administering the same assess-

ments over multiple time points to under-

stand long-term growth. They can track

student achievement data over multiple

years from schools across the nation and

correlate it with school library programs to

make powerful determinations about what

works. Researchers can also study the re-

lationships between principals and school

librarians across many sites and over ex-

tended periods of time.

RESEARCH ALLOWS US INTO PLACES AND SITUATIONS WE MAY NOT BE ABLE TO OBSERVE OTHERWISE

The constraints of daily life mean that edu-

cators simply cannot engage in practices

such as spending long periods of time in a

child's home, observing what students do

and do not do when visiting their local li-

brary, or compare library collections across

many schools. Research can provide some

of this information for us. For example.

Perry (e.g., 2009) spent 18 months in the

homes, churches, and other settings of Su-

danese refugee families. Her research pro-

vided valuable insights about the literacy

practices in which these children were and

were not engaged and ways in which these

practices were and were not consistent with

school literacy practices. In addition, Todd

and Kuhlthau (2004) documented what

13,123 students in 39 Ohio schools be-

lieved about the role of school libraries and

learning, while Hughes-Hassell, Overberg,

and Harris (2013) found insufficiencies in

125 high school library collections with re-

spect to lesbian, gay, bisexual, transgender,

and questioning (LGBTQ)-themed literature

available for teens. In each of these in-

stances, this work would have been nearly

impossible for a full-time literacy educator

or school librarian to conduct.

RESEARCH ALLOWS US TO POOL OUR NUMBERS AND EXPERIENCES

As individuals we simply may not have

enough experience with a question to feel

confident that we have a reasonable an-

swer. For example, a classroom teacher

may not have had enough students with

disorders on the autism spectrum to feel

confident about which instructional ap-

proaches might be most efficacious, or a

literacy coach may not have had enough

experience working with teachers to know

what forms of professional development

are likely to result in the greatest change

in teacher practice. Research allows us to

pool data across many sites and settings

to address important questions about phe-

nomena and practices. To illustrate, school

librarians may have not had enough time

with students to assess whether and how

library instruction contributes to their aca-

demic success. The School Library Impact

studies (for a summary, see Kachel and the

Graduate Students of LSC 5530 School Li-

brary Advocacy, 2011) provided compre-

hensive data from across schools and states

that unequivocally correlated student suc-

cess to well-funded, well-supported school

library programs. This wide-ranging work

across school library programs validated

statistically what school librarians had

long suspected but had been otherwise un-

able to demonstrate through individual ef-

forts.

In the end, we believe that research does

have value. It is worth learning about. It

is worth paying attention to, even when

it requires sorting out misrepresentation

and misuse. The remainder of this article

focuses on knowledge that we believe can

help literacy and school library educators

make use of this valuable tool.

2. WHAT RESEARCH IS

Put simply, in our view, research is the

systematic collection and analysis of data

to address a question. Collecting and ana-

lyzing pre- and posttest data in multiple

classrooms to find out which approach to

writing instruction is more effective, col-

lecting and analyzing data on how best

to organize a school library collection or

use social media most effectively, and col-

lecting and analyzing data to learn about

how a particular family engages in literacy

practices with their children - all of these,

assuming they are done systematically, are

examples of research.

The ultimate purpose of literacy and

school library researchers is to deepen un-

derstanding of and thus improve literacy

and school library education. Of course,

this is the ultimate purpose literacy and

school library educators hold as well. In

fact, at their essence, the activities in which

literacy and school hbrary educators and

researchers engage are often not that dif-

ferent: intensive, long-term observation

of a student or students; close observation

of student learning following implemen-

tation of a new instructional approach or

materials; analysis of school demograph-

ics to assess areas of need that the library

might address; careful examination of the

reading scores of students in a school dis-

trict over the preceding years; analysis of

librarian and classroom teacher talk dur-

ing collaborative planning sessions; and so

forth. Both educators and researchers en-

gage in these activities with the end goal of

improving teaching and learning. We are,

as the expression goes, on the same team.

Despite being on the same team, teach-

ers and researchers do not collahorate

or communicate as much as we might

like. Teachers often tell us that they find

researchers intimidating, and some re-

searchers find great teachers intimidating!

However, we have observed that some of

the most compelling research arises from

collaborations between researchers and

teachers, when teachers and researchers

share insights and burning questions that

they have ahout practice and perhaps tin-

ker together toward answers. For example,

the famous research on Concept-Oriented

Reading Instruction, or CORI, began as

a collaboration between researchers at

the University of Maryland and teach-

ers in public elementary schools in Prince

1 0 T E A C H E R L I B R A R I A N 4 0 : 4

George's County (cf., Guthrie, Wigfield, Et

Perencevich, 2004).

You cannot tell whether or not some-

thing is research from where it is published.

Periodically, research journals, such as

Reading Research Quarterly, School Library

Research, or Teacher Librarian, include

pieces that are not research, such as essays

on possible directions for future research in

a particular area. Similarly, some articles in

a practitioner journal, such as The Reading

Teacher or School Library Monthly, may

report on a research study with the intent

of bringing findings and implications of

that study to a broader audience. That said,

there are a set of elements you should ex-

pect to find, in one form or another, in any

report of research, although they may not

be neatly labeled as such:

• A statement of the research question(s)

and/or purpose(s)

• A rationale for the study (i.e., infor-

mation about why the question was asked,

the problem the research was meant to in-

vestigate or address, and the research and

theory that came beforehand)

• A description of the methods used to

collect data to address the question(s)

• A description of the methods used to

analyze the data collected to address the

question(s)

• Results of these analyses

• Conclusions that the researchers have

drawn based on the results

• Implications of those conclusions for

practice

• Limitations of the study

• Directions for future research

To help identify these different compo-

nents of a report of research. Figure 1 pro-

vides a note-taking sheet. If a piece of writ-

ing or a presentation does not include most

of these elements-for example, if it ap-

pears to be exclusively one person's opin-

ion about effective practices or a memoir

of his or her teaching experience-then this

is a good indication that it probably is not

research. This does not mean that it is not

valuable, just that it is not research.

There are many kinds of research (sec

Table 1). For example, some research is fo-

cused on examining a single case, such as a

single student or classroom or library. That

research is not intended to generalize or ap-

ply to all students or classrooms or librar-

ies; its contributions lie in other areas, such

as building theory. Other research, such as

some survey and assessment research, is

conducted with nationally or internation-

ally representative samples and intended to

have findings that generalize or apply to

an entire population. No one kind of re-

search is less than research of another kind.

Different kinds of research address differ-

ent kinds of questions and provide differ-

ent kinds of information and insights. This

variety allows us to gain deeper insights

into a particular topic and to begin to solve

complex problems. We discuss this idea in

greater depth in item 5.

You may notice that the term teacher

research does not appear in Table 1. (Be-

cause school librarians are teachers, the

term teacher will be used here to include

school librarians.) This is because the term

refers to who is doing the research rather

than the kind of research that is being

done. Teachers, like researchers, can do

any of the kinds of research listed in the

table, although some kinds are undoubt-

edly more practical than others. Teachers'

goal for the research may be more local-

generally, although not always, to inform

their own practice but not to generalize to

the practice of others-but their methods

may be the same as those of researchers

who hope to impact practice more widely.

Finally, we should note that some people

use the terms scientific research or scien-

tifically based research. When probing, we

find that different people intend quite dif-

ferent meanings by these terms. In fact, we

urge educators who hear others use these

terms to ask those who use them what they

mean by them. In our view, these terms

are redundant. It is not as though, as some

seem to believe, some kinds of research are

scientific and some arc not. By (our) defini-

tion, all research is scientific.

3. WHAT RESEARCH IS NOT

Recognizing what research is not is as im-

portant as recognizing what research is.

One common misconception we encounter

is the notion that, if something is written

by a researcher, it automatically counts

as research. If a novelist writes a shop-

ping list, its authorship does not make it

a novel. If a researcher writes an article,

book, or chapter, this fact alone does not

make it research. Similarly, just because a

researcher is presenting a set of practices

docs not mean that those practices arc re-

search-tested or research-based.

Just because a researcher develops an

approach, works on an educational prod-

uct, or endorses an instructional technique

does not mean that that the result is re-

search-tested or even research-based. A

researcher may be trying something com-

pletely new, may have only very limited

influence on a product, or may be making

a less-than-informed guess about the ef-

fectiveness of an instructional technique.

Not everything a researcher touches be-

comes research or is even research-based,

so we need to interrogate writings or prod-

ucts with a researcher's imprimatur just as

we would anyone else's.

A second misconception we have en-

countered is the notion that if something is

written or developed by someone who has

a position at a university, then the work

is necessarily research or research-based.

For example, we recently heard a teacher

claim that the effectiveness of a particu-

lar approach to writing instruction must be

research-proven, because it was developed

by a person who works at a university.

In reality, there arc many different kinds

of positions at universities and colleges.

Many of these do not require position-

holders to conduct or to ever have con-

ducted research.

For some positions, a person may only

need to have conducted a single research

study: the dissertation study that originally

earned the PhD or EdD. Of course, in many

cases, professors are expected to and/or do

in fact conduct a large number of other re-

search studies, but that is by no means a

given. In light of the range of roles that

many professors play, they may at one mo-

ment be conducting a research study and at

another moment be providing advice that

as yet has little or no basis in research.

A fiip side of these points is that just

because someone is not a university-based

A P R I L 2 0 1 3 1 1

researcher or does not have a PhD or EdD

does not mean that he or she has not con-

ducted research. Quality research can be

and is conducted outside of a university

context: in educational research founda-

tions, in schools or other educational set-

tings, in public libraries and community

centers, in daycares, retirement homes,

transitional housing, leaming commons,

and even in virtual spaces such as wikis,

chat rooms, MUDs, and other virtual com-

munities. Quality research can be and is

conducted by individuals who do not have

a research degree. Similarly, it is possible

for people who are not researchers and not

in university positions to have a strong

command of research, even if they do not

conduct it themselves.

4 . THE DIFFERENCE BETWEEN RESEARCH-BASED AND RESEARCH-TESTED

When invoking the term research in rela-

tion to instmctional practices, approaches,

or products, we think it is valuable to make

a distinction between research-based and

research-tested. (As will become evident,

this is both similar to and different from

Shanahan's [2002] distinction between re-

search based and researeh proven.] We use

researeh-tcsted to mean that one or more

research studies tested the impact of that

particular practice, approach, or product.

In contrast, we use research-based to mean

that the pariicular practice, approach, or

product has not been tested in a research

study but has been designed to be consis-

tent with research findings.

Let's say that someone develops an ap-

proach to teaching reading comprehension

that combines elements of three previous,

research-tested approaches. We would say

that the new approach that has been de-

veloped is research-based, but we would

not say it is research-tested because we

have not tested the impact of that particu-

lar combination of approaches in research.

The combination could be more effective,

as effective, or less effective than the indi-

vidual approaches on which it was based.

The terms research-tested, research-

based, research-proven, evidence-based.

and others are being used by many to

elevate the status of their product or ap-

proach. However, use of these terms alone

means very little. We have to ask ques-

tions that allow us to get underneath any

individual's, organization's, or company's

use of these terms. In the remainder of this

section, we identify some of the questions

that need to be asked when a product or

approach is said to be researeh-tested or

research-based.

Research- Tested

To say that a practice, approach, or prod-

uct is research-tested, or research-proven,

sounds like a powerful endorsement or at-

tribution, but its strength really depends on

how it was tested and what the tests found.

Questions we need to ask include the fol-

lowing:

• What exactly did the research test?

For the "research-tested" claim to be valid,

it needs to be applied to a practice, ap-

proach, or product that is highly similar to

or the same as what was originally tested.

What exaetly did the researeh find? Just

because something was research-tested

does not mean the test/research found it to

be effective.

• Did the research test the practice, ap-

proach, or product against something else?

Simply trying out something provides lim-

ited information. If the practice, approach,

or product is not tested against something,

then it is difficult to infer whether any

impact we observe would have been seen

even without that practice, approach, or

product. For example, students may appear

to have better reading comprehension after

receiving a pariicular kind of instmction,

but it may well be the case that their read-

ing comprehension is simply developing

as it normally would have without the in-

tervention, or that something else entirely

is causing the observed improvement. A

school librarian may believe students are

reading more because they check out e-

readers frequently, but there is no way of

knowing whether increased e-reader cir-

culation is related to higher amounts of

reading without comparing students' read-

ings habits before and after e-readers were

purchased and collecting data about the

time students spend reading print versus

electronic text. For these reasons, having a

comparison group that differs only in that

they did not use the pariicular practice, ap-

proach, or product is imporiant to address-

ing questions about impact.

• To what exactly was the practice, ap-

proach, or product compared? When there

is a comparison, you need to ask what it

is. It is one thing for a practice, approach,

or product to be effective when compared

with recess or completing worksheets,

and quite another to show it is effective

in comparison to good, quality instmc-

tion designed to address that same area of

learning. Likewise, comparing students'

reading engagement with text displayed

on a computer screen to their engagement

with text displayed on an e-reader is quite

different from comparing these effects to

students' reading engagement with print-

based text.

• With what samplels) was the research

conducted? We have to be cautious about

applying the results of a study with one

population to another population if we think

the populations are different in consequen-

tial ways. For example, if an approach has

only been tested with native English speak-

ers, its use with English learners should be

approached with caution. Similarly, an ap-

proach tested and shown to be efficacious

in only small, homogenous classes may not

be as efficacious in large, heterogeneous

classes. Ideally, there should be a close

match between the sample or samples with

which the practice, approach, or product has

been tested and the group with whom you

plan to use it. Similarly, studies reporiing

that males were successful in summarizing

and analyzing data using tablets in the li-

brary would not necessarily be applicable to

females or to school libraries that did not

have tablets for student use.

• What outcome measures were used?

The outcome measures used in a study

should be consistent with the outcomes

you are concerned about for your pariicu-

lar purpose or context For example, sup-

pose your goal is to improve children's

fluency, or accuracy, rate, and prosody, in

reading connected text. Suppose you find

HI 1 2 T E A C H E R L I B R A R I A N 4 0 : 4

a study in which an intervention improved

students' accuracy and rate at reading lists

of nonsense words, but did not improve the

accuracy and rate with which students read

connected text. Or, consider the challenge

of trying to help students evaluate the reli-

ability of a website by using research fea-

turing only print-based materials. In such

cases, you might look for other studies or

interventions with an outcome measure

aligned more closely to your goals. When

there is a mismatch between the outcome

measures of a study and what you are try-

ing to impact, the fact that the practice, ap-

proach, or product proved to be effective

may be beside the point.

• What impact did the research find?

Another question to ask is about the de-

gree of the impact of a particular practice,

approach, or product. If a practice requires

the input of considerable resources to

implement and results in children learn-

ing 1.5 more alphabet letters on average

or increasing their end-of-grade scores by

1.90/0, then it may not be worth the invest-

ment. In contrast, if a practice required

relatively few resources to implement and

improved children's spelling development

by several stages or their comprehension

by many percentage points in a short pe-

riod of time, then it may well be worth the

investment.

• How many studies were conducted?

A practice, approach, or product that has

been tested and shown to be effective in

dozens of research studies with a wide

range of contexts should be viewed as hav-

ing stronger research support than a prac-

tice, approach, or product that has been

tested in only a single study or in a very

narrow range of contexts. Similarly, all

things being equal, a practice, approach, or

product that has been shown to be effective

in some studies and not others should be

seen as having less research support than

one that has been shown to be effective in

every study in which it was examined.

• What was the quality of the studies?

You also need to ask whether the research

meets standards of quality for that type of

research. We talk more about standards of

quality for different types of research later

in this paper (for research-tested claims.

see especially "Experimental and quasi-

experimental research" in Table 1).

Research-Based

Just as we must ask questions to get under-

neath claims that something is research-

tested or research-proven, so too must we

ask questions to get underneath claims that

something is research-based. Questions we

should ask include the following:

• How many studies were conducted,

and what were the findings?

• How similar is what was researched

and found to what was being given the

research-based label?

• What research design(s) was(were)

used, including type of research, the sam-

ple, outcome measures, and so forth?

• Of what quality were the studies?

Notably, we should not assume that

something that is research-tested is inher-

ently more supported by research than

something that is research-based. For ex-

ample, a practice that has been tested and

found to be effective in a single study (i.e.,

research-tested) is, in our view, generally

less compelling than a practice that is simi-

lar to, but not exactly the same as, a number

of practices that have been tested and shown

to be effective in a larger number and wide

range of studies (i.e., research-based).

5. MANY KINDS OF RESEARCH HAVE VALUABLE CONTRIBUTIONS TO MAKE TO OUR UNDERSTANDING OF LITERACY LEARNING, DEVELOPMENT, SCHOOL LIBRARIES, AND EDUCATION

You may have read references to the ex-

periment or randomized, controlled trial

as "gold standard" in educational research.

Some seem to think that experiments-

studies in which individuals or groups are

randomly assigned to different experiences

and then the results are compared-are the

best and most valuable kind of educational

research. We believe that this way of think-

ing is mistaken and misleading. Instead,

many kinds of research have valuable con-

tributions to make.

For example, research examining the

validity and reliability of an assessment is

valuable. Survey research telling us how

motivated, or unmotivated, U.S. students

are to read or showing that school librar-

ians are technology leaders in their schools

is valuable. Research that examines school

library practices which motivate students

to read for pleasure or that follows a single

struggling reader in and out of classrooms

over a period of years is valuable. Research

analyzing the characteristics of young

children's spelling or that looks at how the

use of mobile technologies in school li-

braries engages youth is also valuable. The

educational enterprise is far too complex

for one type of research to answer all of

our questions or meet all of our needs (e.g.,

Shavelson & Towne, 2002).

6. DIFFERENT KINDS OF RESEARCH ARE GOOD FOR DIFFERENT QUESTIONS

Although many different kinds of research

are valuable, they are not valuable for the

same reasons. For example, an experiment

is a valuable tool in learning whether one

approach to instruction is more or less ef-

fective than another, but it is not particu-

larly well suited for understanding what is

going on in the minds of readers as they

read; verbal protocols and neuroimaging

would work well for that. Similarly, al-

though instrument development is of great

value when trying how to assess or mea-

sure something, it does not, alone, help us

gain in-depth understanding of the social

and cultural context of a particular group

of learners. Likewise, although survey data

can report co-occurring patterns in school

library staffing, funding, and use and high

levels of student academic performance, it

does not help illuminate the thinking be-

hind a principal's lack of support for the

school library, why a classroom teacher re-

sists staying with her class in the library,

or why a particular student prefers graphic

novels. Interviews, case studies, and fo-

cus groups would more appropriately re-

veal a person's thinking and motivation.

The type of research design that is most

valuable depends on the research question

A P R I L 2 0 1 3 13

that one is trying to address (Shavelson ft

Towne, 2004, 2002).

7. HIGH-QUALITY RESEARCH HAS A LOGIC OF INQUIRY

According to Wilkinson and Bloome

(2008),

What matters in the evaluation of

the worth of a piece of research, of any

paradigm or intellectual tradition, is

the manner in which researchers locate

their inquiry against a background of

extant knowledge and assumptions, the

goodness of fit between research ques-

tions and methodologies, the quality of

the data collection and analysis, and the

integrity of the overall warrant for the

claims (Howe ft Eisenhart, 1990). (p.7)

In other words, a good research study

has a strong match between the research

question or purpose, the research design,

and the conclusions drawn and claims

made from the research. For instance,

consider the question; What do highly ef-

fective literacy teachers do during small-

group reading instruction? To address the

question, the researcher must develop a de-

fensible means of identifying effective lit-

eracy teachers, a systematic procedure for

collecting data about their practices during

small-group reading instruction (e.g., ob-

servation, audio-recording), and a system-

atic procedure for analyzing the data (e.g.,

coding it for specific kinds of practices,

generating descriptive statistics to identify

common practices).

To address the question, "How are stu-

dents' research projects affected by working

with the school librarian?," the researcher

might randomly assign students to carry

out research projects with and without the

guidance of their school librarian. The re-

searcher would develop a systematic pro-

cedure for observing and documenting the

development of students' research projects,

interviewing students, and analyzing stu-

dents' written work, focusing on compar-

ing projects of the two groups.

When studies are complete, the re-

searcher has to take care not to make

claims that cannot be substantiated by the

data that has been collected. For example.

the claim, "Telling readers to identify the

selection's text structure prior to reading is

a highly effective instructional practice" is

not warranted given the design described

earlier in this section. Even if all of the ob-

served teachers engaged in this practice,

we cannot be sure that this is what makes

their instruction effective. In contrast, the

claim, "Highly effective teachers regularly

tell readers to identify the selection's text

structure" might be well supported by

the design. Correspondingly, telling read-

ers that working with a school librarian

strengthens students' reading comprehen-

sion is not warranted given the design of

the school library study. A claim that "Stu-

dents think more critically about sources to

use in their research projects when working

closely with the school librarian," however,

would be appropriate.

8. CONCLUSIONS DRAWN FROM RESEARCH ARE ONLY AS SOUND AS THE RESEARCH ITSELF

It is extremely imporiant to look at the

qualify of the research design and imple-

mentation when evaluating a particular

claim or conclusion. Conclusions based on

a seriously flawed study may be seriously

flawed. A logic of inquiry, discussed pre-

viously, must be evident in the research.

Also, the nature of the inquiry will call for

different markers of qualify. In Table 1,

we present some standards of qualify for

several kinds of research. For further infor-

mation about evaluating the qualify of any

of these kinds of research, we suggest con-

sulting methods texts, such as Fundamen-

tals of Educational Research (Anderson,

1998), Educational Research: An Introduc-

tion (Gall, Gall, ft Borg, 2007) or Literacy

Research Methodologies (Duke ft Mallette,

2004; Duke ft Mallette, 2011).

9. WHERE AND HOW RESEARCH IS PUBLISHED OR PRESENTED REQUIRES PARTICULAR ATTENTION

Consider a pariicular news item and the

range of different ways it is covered, for

example, by the New York Post, The New

York Times, Newsweek, The Economist,

Fox News, or the MacNeil/Lehrer News

Hour. These sources will cover the same

story in substantially different ways. Simi-

larly, literacy and school library research in

different outlets, and by different writers,

may be reported very differently.

For example, a study reporied in The

Reading Teacher or Teacher Librarian

may provide relatively little detail about

a study's methodology and findings but

may provide considerable detail about the

targeted instructional practices; in con-

trast, a study reporied in Reading Research

Quarterly or School Library Research may

provide much more detail about the study's

methodology and findings but may pro-

vide a much briefer explanation of how to

implement the given instructional practices

(cf. Duke, Purcell-Gates, Hall, ft Tower,

2006/2007; Purcell-Gates, Duke, ft Mariin-

eau, 2007). A study reporied to an audience

primarily comprised of literacy or school

library educators may give short shrift to

institutional, fiscal, and equify issues—con-

siderations that would likely be prominent

in a report of the same study intended for

an audience primarily comprised of edu-

cation policymakers (cf Buly ft Valencia,

2002; Valencia ft Buly, 2004).

As you read or listen to research or ref-

erences to research, it is important to con-

sider the match or mismatch between your

purpose(s) and the purpose(s) for which a

given piece was written or presented in

that pariicular ouüet and by that pariicular

person. If your intent, for instance, is to

understand and evaluate the qualify of the

methodology used in the study, then an ar-

ticle in Instructor Magazine or Teacher Li-

brarian or Library Media Connection is not

the best place to go; however, if your in-

tent is to understand what a particular re-

search-tested practice looks like in a school

setting, then this magazine might be the

correct source. In the end, it would be best

to read everything you can, as presented in

a range of outlets, about any specific study

or area of research that is of great impor-

tance to you.

You may have heard or read about the

notion of peer-reviewed research. This re-

1 4 T E A C H E R L I B R A R I A N 4 0 : 4

fers to whether the research was reviewed

by a set of peers prior to its publication.

Publications like The Reading Teacher,

Reading Research Quarterly, School Library

Research, School Libraries Worldwide,

and Journal of Youth Services in Librar-

ies generally, though not always, employ a

blind peer-review process. This means that

when deciding whether or not to publish

a particular submission, the editors send

it out to a set of scholars (researchers on

the editorial board in the case of Read-

ing Research Quarterly or School Librar-

ies Worldwide), and a combination of re-

searchers, practitioners, and other scholars

in the case of The Reading Teacher, School

Library Research, and Journal of Youth

Services in Libraries] for their evaluation

ahout whether or not it should be pub-

lished and, if it is worthy of publication, to

find out what improvements could be made

to the piece prior to publication.

The adjective blind means that submit-

ted articles are sent to reviewers without

information that would clearly identify the

submission's author(s); this increases the

likelihood that reviewers are not swayed in

their judgment by their high or low opin-

ions of the submitted article's author(s).

The intent of peer review is to inform deci-

sions ahout what merits publication and to

improve the quality of what is ultimately

published. Of course, this tool is not per-

fect. For various reasons, work of dubious

quality is sometimes published, and some-

times high-quality research is not. A piece's

peer-reviewed status is just one metric you

can use to increase the likelihood that the

research you are reading is of high quality,

but you should keep in mind that it is just

one, necessarily limited, metric.

Another issue to consider with re-

spect to how the research is presented is

the quality of the writing itself. All other

things being equal, and given that our time

for reading research is always limited, it is

worth considering the quality of a piece's

writing in deciding whether or not to read

it. That said, all other things are often not

equal. A study may follow a sound logic

of inquiry as described above, meets stan-

dards of quality as described in Table 1,

and have many other markers of high qual-

ity but be poorly written. For example, im-

portant information may be missing, or the

piece's prose may be dense and difficult to

parse. Conversely, very strong writing may

make a study sound better than it actually

is. For example, the piece's prose may be

so articulate and persuasive that it is easy

to miss that its design fails to meet many

standards of quality.

These are important considerations to

bear in mind as you read research. It may

be worth plowing through an article that

is difficult and frustrating to read, and we

have to be careful not to be seduced by a

study that, although presented eloquently,

is problematic in important ways.

10. EDUCATIONAL RESEARCH PROCEEDS THROUGH THE SLOW ACCUMULATION OF KNOWLEDGE

When designing a study, there typically are

trade-offs. For example, involving more

classrooms or school libraries may increase

the generalizability of the study-that is,

the extent to which you can apply what

is learned in the study to other classrooms

or libraries-but may decrease our capacity

to study in detail how individual teachers

or school librarians are implementing the

intervention. Even if there were a perfect

study (and of course, researchers dream of

conducting a perfect, flawless study!), we

must be cautious about basing policy or

practice on any single study.

First, it is possible for any study, no

matter how well designed, to get an anom-

alous result. Second, the generalizability of

any one study is limited in terms of the

populations and contexts in which similar

results can be expected to be found. For

these reasons, it makes the most sense to

think about research as proceeding as a

slow accumulation of knowledge over time

and to read across many different studies

on a particular question or topic. Although

breakthroughs or headline-making studies

periodically appear, it is usually a mass of

related studies over a period of years that

lead to a well-accepted or durable conclu-

An important tool that can help us

understand whether practices or insights

have been widely validated is the research

review or synthesis. These syntheses pull

together individual studies to draw more

robust conclusions. These syntheses may

also be a great asset for educators who are

too busy to read a lot of studies.

There have been a number of influen-

tial research reviews in recent years, most

notably perhaps, for literacy, the National

Reading Panel Report (National Institute

of Child Health and Human Development,

2000), and, for school libraries. School Li-

braries Work (National Commission on

Libraries and Information Sciences, 2008)

and Library Power Executive Summary:

Findings from the National Evaluation

of the National Library Power Program

(Wheelock, 1999).

Other important literacy reviews in-

clude Preventing Reading Difficulties in

Young Children (Snow, Burns, ft Griffin,

1998), Writing Next: Effective Strategies to

Lmprove Writing of Adolescents in Middle

and High Schools (Graham ft Perin, 2006),

the National Literacy Panel on language-

minority children and youths (August ft

Shanahan, 2006), and the National Early

Literacy Panel repori (National Early Lit-

eracy Panel, 2008). For school libraries,

Grover and Fowler (1993) reviewed Recent

Trends in School Library Media Research

from 1987 to 1991. Clyde (2003) looked at

school library research from 1991-2000 for

31 countries, and Wirkus (2006) summa-

rized research from 1992-2004.

There are volumes that compile many

research reviews or syntheses, as in the

Handbook of Reading Research (e.g., Kamil,

Mosenthal, Pearson, ñ Barr, 2000; Kamil,

Pearson, Moje, Ö Afflerbach, 2010) and

What Research Has to Say About Read-

ing Instruction (Farstrup Et Samuels, 2002).

There are also outlets for peer-reviewed

reviews of research, such as the journal

Review of Educational Research. Finally,

for school librarians, the School Library

Media Annual, School Library Media Re-

search, School Libraries Worldwide, and

The Library Research Service (http://www.

lrs.org/) also provide reviews of research.

APRIL 2 0 1 3 1 5

However, these important and po-

tentially time-saving tools also must be

treated with caution. They are necessarily

reductive. They may omit important de-

tails about the methods used to conduct

the studies or about findings of the stud-

ies, particularly those findings that are not

directly germane to the focus of the review.

These publications may also gloss over im-

portant differences between studies, the

ways particular items have been operation-

alized, and so forth. Ideally, you yourself

would go back to all of the original papers

on which the review is based, as well as

search for and read any relevant papers

that may have been omitted. However, this

is oñen not possible. If it is not, then there

are other questions that you could ask:

• Are the authors of the review well re-

spected in the field?

• Do they represent, or at least show

openness to, a variety of theoretical per-

spectives related to the research reviewed?

• Does it appear that the authors are

indeed reviewing all the available research

rather than simply selecting research that

is consistent with their conclusion or per-

spective?

• Was their approach to locating and

reviewing the research systematic? Did it

seem thorough?

• If provided, was the list of terms used

in the literature search relevant and ex-

haustive?

• Did another person also code the stud-

ies? To what degree did he or she agree

with the authors' coding efforts?

• If you are familiar, or can make your-

self familiar, with an individual study or a

few studies reviewed, did the authors code,

characterize, and draw conclusions that

you think are appropriate from this study?

• Are the authors explicit in their expla-

nations of findings and results?

Additional standards apply in the case

of a special kind of research review called

a quantitative meta-analysis (see Table 1

for more information). Quantitative meta-

analyses can be very valuable, but it is

important to note that these analyses can

only examine studies that are correlational,

quasi-experimental, or experimental in

design, and these analyses are limited to

only those areas in which a critical mass

of sufficiently similar studies have been

conducted. For this reason, it is often not

a quantitative meta-analysis-or not solely

a quantitative meta-analysis-that will best

address your needs.

CONCLUSION

By this point, it is likely clear that being an

informed and critical reader of research is

a formidable task. Thus, we end this article

the way we began it, with the assertion that

research is worth it-that research should

be seen as an essential guide to policy

and practice. Recall our argument that our

experiences alone may misguide us, that

sometimes we do not know what we do

not know, that research allows us to take

a longer term view than our personal ex-

periences may allow, that research allows

us to pool our numbers and experiences,

and that research allows us into places and

situations that we may not be able to ob-

serve otherwise.

Recall that the goal of literacy and

school library researchers is much the same

as the goal of literacy and school library

educators: to improve teaching and learn-

ing. Literacy and school library educators

play a vital role in helping researchers

meet this goal. Careful reading, evaluation,

and interpretation by thoughtful and in-

formed educators offers our best chance at

realizing the full value of what this price-

less tool-research-has to offer.

REFERENCES

Anderson, G., with N. Arsenault. (1998).

Fundamentals of educational research (2nd

ed.). Bristol, PA: Falmer Press.

Asselin, M. (2001). Factors of effective

school libraries: A survey of perspectives

of teachers, teacher-librarians, and princi-

pals. Paper presented at the annual con-

ference of the International Association of

School Librarianship, Auckland, NZ.

August, D., a Shanahan, T. (2006). Devel-

oping reading and writing in Second-Lan-

guage Learners: Lessons from the Report of

the National Literacy Panel on Language-

Minority Children and Youth. Mahwah, NJ:

Lawrence Erlbaum.

Bos, C. S., ft Anders, P. L. (1990). Effects of

interactive vocabulary instruction on the

vocabulary learning and reading compre-

hension of junior-high learning disabled

students. Learning Disability Quarterly, 13,

13-42.

Buly, M. R., ft Valencia, S. W. (2002). Below

the bar: Profiles of students who fail state

reading assessments. Fducational Fvalua-

tion and Policy Analysis, 24, 219-239.

Clyde, L. A. (2003). Evidence-based prac-

tice in school librarianship: Evaluating the

research evidence. ACCFSS-CAULFIFLD

FAST THFN ALICF SPRINGS, J7(4), 26-

29.

Duke, N. K. (2000). 3.6 minutes per day:

The scarcity of informational text in first

grade. Reading Research Quarterly, 35,

202-224. doi: 10.159B/RRQ.35.2.1

Duke, N. K., Bennett-Armistead, V. S., ft

Roberts, E. M. (2003). Bridging the gap

between learning to read and reading to

learn. In D. M. Barone ft L. M. Morrow

(Eds.), Literacy and young children: Re-

seareh-based practices (pp. 226-242). New

York: Guilford Press.

Duke, N. K., ft Kays, J. (1998). "Can I say

'once upon a time'?": Kindergarten chil-

dren developing knowledge of information

book knowledge. Farly Childhood Research

Quarterly, 13, 295-318.

Duke, N. K., ft Mallette, M. H. (Eds.). (2004).

Literacy research methodologies. New York,

NY: Guilford.

Duke, N. K., ft Mallette, M. H. (Eds.). (2011).

Literacy research methodologies (2nd ed.).

New York, NY: Guilford.

Duke, N. K., Purcell-Gates, V., Hall, L. A., ft

Tower, C. (2006/2007). Authentic literacy

activities for developing comprehension

HI 1 6 T E A C H E R L I B R A R I A N 4 0 : 4

and writing. The Reading Teacher, 60, 344-

355.

Farstmp, A. E., ft Samuels, S. (Eds.). (2002).

What research has to say about reading in-

struction. Newark, DE: International Read-

ing Association.

Gall, M. D., Gall, J. P., ft Borg, W. R. (2007).

Educational research: An introduction (8th

ed.). New York, NY: Pearson Education.

Gaver, M. (1963). Effectiveness of central-

ized library service in elementary schools.

New Bmnswick, NJ: Rutgers.

Gavigan, K. (2012). Sequentially SmART-

Using Graphic Novels across the K-12 Cur-

riculum. Teacher Librarian, 35(5), 20-25.

Graham, S., ft Perin, D. (2006). Writing

next: Effective strategies to improve writing

of adolescents in middle and high schools.

A report to Carnegie Corporation of New

York. Washington, DC: Alliance for Excel-

lent Education.

Grover, R., ft Fowler, S. G. (1993). Recent

trends in school library media research.

School Library Media Quarterly, 21{4],

Retrieved from http://www.ala.org/aasl/

aaslpubsandjournals/slmrb/editorschoiceb/

infopower/selctgrover2

Guthrie, J. T., Wigfield, A., ft Perencev-

ich, K. C. (Eds.). (2004). Motivating reading

comprehension: Concept-Qriented Reading

Instruction. Mahwah, NJ: Lawrence Erl-

baum.

Harizell, G. (2004). Invisible to indispens-

able: Rethinking the library media spe-

cialist-school principal relationship. Santa

Barbara, CA: Libraries Unlimited.

Howe, K., ft Eisenhari, M. (1990). Standards

for qualitative (and quantitative) research:

A prolegomenon. Educational Researcher,

J9(4), 2-9.

Hughes-Hassell, S., Qverberg, E., ft Harris,

S. (2013). Lesbian, Gay, Bisexual, Trans-

gender, and Questioning (LGBTQ)-themed

literature for teens: Are school libraries

providing adequate collections? School Li-

brary Research, 16, Retrieved from http://

www.ala.org/aasl/slr/volumel6/hughes-

hassell-overberg-harris

Kachel, D. E., and the graduate students

of LSC 5530 School Library Advocacy.

(2011). School library research summa-

rized: A graduate class project. Mansfield,

PA: Mansfield University. Retrieved from

http://libweb.mansfield.edu/upload/ka-

chel/ImpactStudy.pdf

Kamil, M. L., Mosenthal, P. B., Pearson,

P. D., ft Barr, R. (Eds.). (2000). Handbook

of reading research (Vol. 3). Mahwah, NJ:

Lawrence Erlbaum.

Kamil, M. L., Pearson, P. D., Moje, E. B.,

ft Afflerbach, P. (Eds.). (2010). Handbook

of reading research (Vol. 4). Mahwah, NJ:

Lawrence Erlbaum.

Lance, K. C. (2001). Proof of the power:

Quality library media programs affect aca-

demic achievement. MultiMedia Schools,

S(4), 14-20.

Lance, K. C. (2002). Impact of school library

media programs on academic achievement

Teacher Librarian, 29(3), 29-34.

Lance, K. C, Wellborn, L., ft Hamilton-

Pennell, C. (1993). The impact of school

library media centers on academic achieve-

ment. Castle Rock, CO: Hi Willow Research

and Publishing.

Moss, B., ft Newton, E. (2002). An examina-

tion of the information text genre in basal

readers. Reading Psychology, 23, 1-13.

National Commission on Libraries and In-

formation Sciences. (2008). School libraries

work! Retrieved from http://www.scholas-

tic.com/content/collateral_resources/pdf/s/

slw3_2OO8.pdf

National Early Literacy Panel. (2008). De-

veloping early literacy: Report of the Na-

tional Early Literacy Panel Washington,

DC: National Institute for Literacy.

National Institute of Child Health and Hu-

man Development. (2000). Report of the

National Reading Panel. Teaching children

to read: An evidence-based assessment of

the scientific research literature on reading

and its implications for reading instruction.

(NIH Publication No. 00-4769). Washing-

ton, DC: U.S. Government Printing Office.

Pappas, C. C. (1993). Is narrative "primary"?

Some insights from kindergarierners' pre-

tend readings of stories and information

books. Journal of Reading Behavior, 25,

97-129.

Perry, K. H. (2009). Genres, contexts,

and literacy practices: Literacy brokering

among Sudanese refugee families. Reading

Research Quarterly, 44, 256-276.

Purcell-Gates, V., Duke, N., ft Mariineau,

J. A. (2007). Leaming to read and write

genre-specific text: Roles of authentic ex-

perience and explicit teaching. Reading Re-

search Quarterly, 42, 8-45.

Shanahan, T. (2002). What reading re-

search says: The promises and pitfalls of

applying research to reading education. In

A. E. Farstmp ft S. J. Samuels (Eds.), What

research has to say about reading instruc-

tion (pp. 8-24). Newark, DE: International

Reading Association.

Shavelson, R. J., ft Towne, L. (2004). What

drives scientific research in education?

Questions, not methods, should drive the

enterprise. American Psychological As-

sociation Qbserver, 17(4), Retrieved from

http://www.psychologicalscience.org/ob-

server/get\rticle.cfm?id= 1557

Shavelson, R. J., ft Towne, L. (Eds.). (2002).

Scientific research in education. Washing-

ton, DC: National Academy Press.

Snow, C. E., Burns, M. S., ft Griffin, P.

(Eds.). (1998). Preventing reading difficul-

ties in young children. Washington, DC:

National Academy Press.

Snowling, M. J. (2000). Dyslexia (2nd ed.).

Hoboken, NJ: Wiley-Blackwell.

APRIL 2 0 1 3 1 7

Todd, R. J. (2002, August). School librar-

ian as teachers: Learning outcomes and

evidence-based practice Libraries for Life:

Democracy, Diversity, Delivery. Paper pre-

sented at IFLA Council and General Con-

ference, Glasgow, Scotland. [ERIC Data-

base, ED472883].

Todd, R. J., a Kuhlthau, C. (2004). Stu- dent learning through Ohio sehool librar-

ies: Background, methodology and report of

findings. Columbus, OH: OELMA. Retrieved

from http://www.oelma.org/documents/

OBLMAReportofFindings.pdf

Valencia, S. W., Ö Buly, M. R. (2004). Be-

hind test scores: What struggling readers

really need. The Reading Teacher, 57, 520-

531.

Wheelock, A. (1999). Executive summary:

Findings from the evaluation of the Na-

tional Library Power Program. DeWitt

Wallace Reader's Digest Fund. Retrieved

from http://www.wallacefoundation.org/

knowledge-center/Libraries/Documents/

F i n d i n g s - F r o m - t h e - N a t i o n a l - L i b r a r y -

Power-Program.pdf

Wilkinson, I. A. G., Ö Bloome, D. (2008).

Research as principled, pluralistic argu-

ment. Reading Research Quarterly, 43, 6-8.

Wirkus, K. (2006). School library media

research: An analysis of articles found in

School Library Media Research and the

FRIC Database. School Library Media Re-

seareh, 9, Retrieved from http://www.ala.

org/aasl/aaslpubsandjournals/slmrb/slmr-

contents/volume09/best_of_eric

Nell K. Duke, University of Michigan

Nicole M. Martin University of North Car-

olina, Greensboro

Anne Trice T. Akers, University of North

Carolina, Greensboro

Complete reference for the study

Purpose and/or Questions

Methodology

Participants

Context

Data Collection Procedures

Measures (if applicable)

Materials (if applicable)

Intervention (if applicable)

Data Analysis Procedures

Findings

Limitations

Implications for policy, practice, research, and other

Other information for evaluation of study quality

Figure 1. Research study summary sheet.

1 8 T E A C H E R L I B R A R I A N 4 0 : 4

Table 1. Brief Descriptions and Standards of Qualify for Some Common Research Methodologies.

Methodology Description Five Qualify Indicators

Assessment

development

Case study

Content

analysis

These studies examine the reliability and

validify of assessments, attitude surveys,

and other research tools. Along with

other activities, researchers typically

give the assessment to a number of

pariicipants and then perform statistical

analyses to examine its validity and

reliabilify.

Case studies seek to describe naturally-

occurring phenomena. These studies

often focus on a single or small number

of cases, such as one classroom or three

reading groups at one grade level or

exemplary school library programs in

one state. Researchers fypically identify

themes or patterns, rather than making

claims about cause-effect relationships.

Content analysis is a methodology for examining the content of something,

such as instruction (e.g., how much

instructional time is devoted to

vocabulary instruction) or texts (e.g.,

what kinds of text, and in what

proporiions, are included in basal

readers). Content analysis is more about

the what in language, whereas discourse

analysis is more about the how with

language.

1. The targeted concepts have been carefully defined.

2. Evidence has been provided suggesting that current research/

theory has thoughtfully informed its creation and other people

have reviewed the assessment

3. Multiple fypes of reliability and validify have been discussed.

4. The assessment includes uncomplicated and unbiased questions.

Enough (but not too many) questions have been included to

understand the targeted concepts.

5. The assessment includes instructions that are easy to understand

and follow.

1. It is easy to envision the context in which the research has

occurred. Careful, vivid, and thorough descriptions have been

included.

2. Potential ethical issues (such as the researchers' incoming

biases or the consequences of the researchers' interactions with

pariicipants) have been acknowledged and, wherever possible,

sensitively addressed.

3. Direct quotations and excerpts from field notes are used to

suppori the study findings, and enough of them have been

included to inspire trust in the researchers' interpretations. 4. Multiple types of evidence have been used to inform conclusions.

Findings suggest that the different types of evidence have been

compared to check and confirm the identified patterns or themes. 5. It is easy to understand why and how the researchers did

what they did, as well as to see why they reached the specific

conclusions outlined in the research repori. Extensive

information about the study's design, preparation, data collection,

analysis, and results has been included.

1. The purpose of and theory and rationale for the content analysis

is clear.

2. The sample and context from which the content is drawn is

thoroughly described.

3. The variables for the content analysis are clearly defined and

operationalized, with reference to the theoretical framework and

rationale for the study.

4. There are at least two coders, who demonstrate high levels of

coding agreement (inter-rater reliabilify).

5. The procedures for going from codes to results are reporied

clearly.

A P R I L 2 0 1 3 19

1 Methodology Description Five Quality Indicators

Correlational data

Discourse analysis

Ethnography

Experimental a quasi- experimental research

This kind of research examines

relationships among variahles.

Researchers often conduct these studies

when they are interested in causes and

effects but are unable to control or alter

the variables. For example, correlational

research might examine the relationship

between exposure to lead paint and

reading difficulties or school library

funding and student success.

This methodology tries to gain insight

into the structures and meanings

underlying conversations and written

texts. Researchers examine previously or

newly recorded texts and develop systems

for uncovering patterns in the texts.

Ethnography is a specific type of

case study. Like case study research,

ethnographic research explores

phenomena by looking closely at specific

examples. This kind of research typically

involves extended, intense observations

and emphasizes cultural contexts.

Researchers often attempt to represent the

perspectives of insiders.

These designs investigate cause-effect

relationships. Researchers typically

identify a focus, such as the use of a

pariicular instructional approach, and

measure its outcomes. Researchers

attempt to eliminate alternative

explanations for outcomes by creating

groups who differ in only one way-for

example, in receiving or not receiving

a particular instructional approach.

In experiments, researchers typically

create groups hy randomly assigning

pariicipants. In contrast, researchers

use groups that already exist for quasi-

experiments.

1. Concepts are carefully defined.

2. Statistical techniques are appropriately and widely used in the

study.

3. Thoughtful attempts to identify and/or rule out alternative

explanations have been made.

4. The findings from other related studies and existing theory are

taken into account.

5. Any claims about cause-and-effect have been made with extreme

care.

1. Evidence that the data has been systematically collected and

carefully analyzed has been provided.

2. Researchers have provided plausible and thorough explanations

for why pariicular texts have been selected as focal texts and

how the coding system was developed to analyze them.

3. Researchers' interpretations have taken into account the original

contexts in which the texts were created, as well as existing

research and theory.

4. High scores from the inter-rater reliability process have been

reporied.

5. Many direct quotations have been used to suppori conclusions.

1. A large amount of time (often years) has been spent interacting

with the pariicipants.

2. Multiple types of evidence and evidence from different seftings

have been collected.

3. Researchers have carefully described the settings and

circumstances of pariicipants' lives, as well as their own

incoming perspectives and the roles they enacted during the

study.

4. Previous research and existing theory have been used to design

the study and interpret the findings.

5. The study goes heyond providing new information about the

focal topic by adding to and/or changing theory.

1. Groups are highly similar, ideally differing only in terms of the

variable of interest (e.g., receiving a treatment or not).

2. Researchers have tried to ensure that the research is valid and

reliable (e.g., scoring assessments without knowing whether they

are ftom the experimental or control group).

3. Appropriate and high-quality measures have been used to assess

outcomes.

4. The statistical analyses are appropriate and clearly linked

to the study design. Information that might affect readers'

interpretation of the analyses, such as missing data or violations

of the assumptions of pariicular statistical procedures, has been

discussed.

5. Enough details about each group's pariicipants and their study

experiences have heen included to compare pariicipants and tasks

to other studies and real-world situations.

2 0 T E A C H E R L I B R A R I A N 4 0 : 4

Methodology Description Five Quality Indicators

Formative ft design experiments

Historical research

Neuroimaging

Quantitative meta-analysis

In this methodology, data is collected

systematically for the purpose of

informing design or practice to reach

specified goals. Often, researchers and

teachers work together to implement

an instructional approach, investigate

factors that might influence its outcomes,

modify the approach to account for what

they have discovered, and implement

the revised instructional approach. This

implement-investigate-revise process

might continue for several rounds or

until the original goal is achieved.

In historical research, researchers attempt

to address a question about the past.

They examine artifacts fi-om or about the

time period, such as diaries, photographs,

court records, or legal documents.

Researchers may also interview people

associated with the event or topic. This

kind of research ofien searches for

patterns or themes that might inform

current issues. For example, a researcher

might examine past educational policies

for the purpose of revising or creating

present-day policy initiatives.

This kind of research tries to answer

questions about neurological structure

and/or function. Researchers examine

images of the brain and brain activity.

These studies are characterized by the

use of specialized medical equipment and

processes, such as electroencephalography

or functional magnetic resonance

imaging.

Researchers use this methodology

to synthesize the results of previous

research. Researchers systematically

collect studies that have addressed

the same or similar questions, then

conduct statistical analyses to identify

the trends across the collected studies.

Quantitative meta-analyses often focus

on the relative magnitudes of outcomes,

such as the average effect of a particular

instructional intervention.

1. The starting point (e.g., students' initial achievement) has been

carefully explained.

2. Previous research and existing theory have informed the

experiment.

3. Multiple rounds of data have been collected over time.

4. Revisions to the experimental design take into account the

outcomes and circumstances of previous rounds of data

collection.

5. The outcomes of each round are thoroughly discussed and

include unplanned results.

1. The settings and circumstances of the time period are carefully

described and are used to understand the study findings.

2. The research focus is precise and thoughtful.

3. Researchers describe what kind of and how much evidence is

available to study the topic. Throughout the research report, they

extensively use the available evidence and justify how and why

particular sources were selected.

4. Researchers employ widely accepted and systematic methods to

collect and analyze the data.

5. The limitations of the study are thoroughly discussed.

1. Research questions can be addressed appropriately by looking at

images of the brain and the brain's activity.

2. The tasks that participants are asked to complete seem suitable

and appear to be high in quality.

3. The methods used to collect evidence are appropriate and

straightforward.

4. Multiple approaches have been used to verify the findings.

5. Attempts have been made to compare and connect the findings

with other research.

1. The study focuses on an area of research that can support a

quantitative meta-analytic approach. Enough research reporis in

the focal area include effect sizes or information that could be

used to calculate effect sizes.

2. The search for previous research is systematic and thorough.

Researchers have used multiple sources to obtain studies, and

they have considered both published and unpublished works.

3. The criteria used in the quantitative meta-analysis seem

thorough, appropriate, and comprehensive enough to lead to the

identification of most or all relevant studies.

4. Appropriate statistical techniques are widely used in the study

design.

5. High levels of inter-rater reliability or agreement have been

reported.

APRIL 2 0 1 3 2 1

Methodology Description Five Quality Indicators

Single-subject experimental design

Survey research

Verbal Protocols

In this design, individuals are (or an

individual is) studied in such a way that

they each comprise their own comparison

group. For example, in an ABA

withdrawal design, repeated baseline

assessments are administered (A), then

an intervention is introduced and the

subject assessed repeatedly again (B), and

finally, the intervention is withdrawn and

the subject assessed additional times (A).

Differences in A and B suggest a possible

impact of the intervention.

Survey research usually elicits reporis

from pariicipants about themselves. The

purpose of this kind of research is usually

to understand something about the larger

group to which the pariicipants belong.

For example, researchers might survey

100 kindergarten teachers across a state

in order to leam about the beliefs of the

kindergarien teachers in that state or 100

principals to leam about their personal

experiences with school libraries. This

kind of research may involve different

kinds of interactions, such as face-to-face

or telephone interviews and computerized

or mail surveys.

Verbal protocols, also referred to as

think-aloud studies, typically gather

information about people's thought

processes. Researchers often ask

pariicipants to complete a specific task,

such as reading a book, and repori

what they are thinking. Pariicipants

from second grade to adulthood have

pariicipated in verbal protocol research.

1. The focus of the measures/assessment is believed to be

susceptible to intervention within a relatively shori time fróime.

2. Sensitive and reliable measures that can be administered

repeatedly are used.

3. Pariicipants and study procedures are described in great detail so

that others can replicate them.

4. The design is such that any effect observed can be reasonably

attributed to the intervention.

5. Any effects observed are evaluated for how much practice

difference they make for the individual.

1. The pariicipants have been carefully chosen. Individuals are

clearly representative of the larger group in which the researchers

are interested.

2. The survey questions are straightforward and unbiased.

3. The quality of the survey has been checked, and evidence of its

validity and reliability has been reporied.

4. Reasonable attempts have been made to get pariicipants

to respond, as well as to understand why nonresponding

pariicipants did not respond.

5. Information that might influence readers' interpretations of the

study, such as how the pariicipants differ from the targeted

population and what percentage of pariicipants responded, has

been included.

1. The texts and tasks used in the study seem to be good choices for

gathering information about the research questions.

2. The instmctions and prompts are carefully worded to avoid

influencing pariicipants' responses.

3. The procedures for collecting data, including the prompts used,

have been thoroughly described.

4. Plausible and logical reasons have been given for each research

design decision.

5. Clear and logical connections can be seen among the research

questions, research procedures, tasks, and materials.

Note. The information in this table was drawn in pari from "Standards for Reporiing on Empirical Social Science Research in AERA

Publications: American Educational Research Association," by P.A. Moss, J.W. Pelligrino, B.L. Schneider, R.P. Duran, M.A. Eisenhari, F.D.

Erickson, et al., 2006, Educational Researchers, 35(6), 33-40; "Qualitative Analysis on Stage: Making the Research Process More Public,"

by V.A. Anfara, Jr., K.M. Brown, and T.L Mangione, 2002, Educational Researcher, 31{7), 28-38; Literacy Research Methodologies [2"''

ed.), by N.K. Duke and M.H. Mallette (Eds.), 2011 ; New York: Guilford; and Educational Research: An Introduction (8"' ed.), by M.D. Gall,

J.P. Gall, and W.R. Borg, 2007, Boston: Allyn ft Bacon.

2 2 T E A C H E R L I B R A R I A N 4 0 : 4

Copyright of Teacher Librarian is the property of EL Kurdyla Publishing LLC and its content may not be

copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written

permission. However, users may print, download, or email articles for individual use.