SAM 3
qexadda5
ComparisonofQuantitativeandQualitativeResearchTraditionsEpistemologicalTheoreticalandMethodologicalDifferences.pdf
Comparison of Quantitative and Qualitative Research Traditions: epistemological, theoretical, and methodological differences
Kaya Yilmaz
Introduction Educational researchers in every discipline need to be cognisant of alternative research traditions to make decisions about which method to use when embarking on a research study. There are two major approaches to research that can be used in the study of the social and the individual world. These are quantitative and qualitative research. Although there are books on research methods that discuss the differences between alternative approaches, it is rare to find an article that exam- ines the design issues at the intersection of the quantitative and qualitative divide based on eminent research literature. The purpose of this article is to explain the major differences between the two research paradigms by comparing them in terms of their epistemological, theoretical, and methodological underpinnings. Since quantitative research has well-established strategies and methods but quali- tative research is still growing and becoming more differentiated in methodological approaches, greater consideration will be given to the latter.
Definition of the Terms What is quantitative research? It can be defined as research that explains phenom- ena according to numerical data which are analysed by means of mathematically- based methods, especially statistics. From a broader perspective, it can be defined as a type of empirical research into a social phenomenon or human problem, testing a theory consisting of variables which are measured with numbers and analysed with statistics in order to determine if the theory explains or predicts phenomena of interest (Creswell, 1994; Gay & Airasian, 2000).What is qualitative research? Although it is deemed ‘difficult to define’ because of its multifaceted nature underpinned by different paradigms (Hitchcock & Hughes, 1995, p. 26), a working definition has been provided by some researchers. Strauss and Corbin (1998) offer this definition: ‘By the term “qualitative research” we mean any type of research that produces findings not arrived at by statistical procedures or other means of quantification’ (pp. 10–11). But this definition is simplistic since it focuses on procedures and techniques used to collect and analyse data, ignoring other aspects of research design. It also tends to define the term from a quantitative perspective rather than focus on its characteristics. Gay and Airasian (2000, p. 627) define qualitative research as ‘the collection of extensive data on many variables over an extended period of time, in a naturalistic setting, in order to gain insights not possible using other types of research’. But this definition also suffers from an identical problem, since it uses a quantitative concept to define a qualitative term (Qualitative research is based on the epistemological assumption that social phe- nomena are so complex and interwoven that they cannot be reduced to isolated variables, so it is not appropriate to use the term variable when defining qualitative
bs_bs_banner
European Journal of Education, Vol. 48, No. 2, 2013
© 2013 John Wiley & Sons Ltd
research). Hence, qualitative research needs to be comprehensively defined to do justice to its key characteristics. Drawing on the research literature (Creswell, 2007, p. 37; Denzin & Lincoln, 1998, 2005, p. 3; Miles & Huberman, 1994, pp. 6–7; Patton, 2002, pp. 39–41), I define it as an emergent, inductive, interpretive and naturalistic approach to the study of people, cases, phenomena, social situa- tions and processes in their natural settings in order to reveal in descriptive terms the meanings that people attach to their experiences of the world.
It should be noted that qualitative research is not based on a single method- ology and does not belong to a single discipline (Denzin & Lincoln, 2005). It ‘draws on philosophical ideas in phenomenology, symbolic interactionism, herme- neutics and other traditions to support the attention on “quality” rather than “quantity”.’ (Brewer, 2003, p. 239).Therefore, the term is used as ‘an overarching category, covering a wide range of approaches and methods found within different research disciplines’ (Snape & Spencer, 2003, p. 3). There is a wide variety of theoretical paradigms, methodologies, research strategies and methods in qualita- tive research traditions, ranging from descriptive study, case study, field research, ethnography, participant observation, biographical method, life history, oral history, narrative inquiry to phenomenological research, ethno-methodology, sym- bolic interactionist study, grounded theory and action research.
Differences between the Two Research Paradigms When the characteristics of quantitative or qualitative research are discussed, the four essential elements of the research process must be addressed. They are epistemology, theoretical perspectives, methodology, and methods (Crotty, 1998). Denzin and Lincoln (1998) suggested that four basic issues structure the design of a research study: (a) Which paradigm or worldview will inform the study design? (b)Who or what will be studied? (c)Which research strategies will be used? and (d) Which research methods or tools will be used to collect and analyse data? This article takes into account these components when explaining the differences between quantitative and qualitative research approaches.
Quantitative and qualitative research designs differ in terms of their epistemo- logical, theoretical and methodological underpinnings. Quantitative research is informed by objectivist epistemology and thus seeks to develop explanatory univer- sal laws in social behaviours by statistically measuring what it assumes to be a static reality. It emphasises the measurement and analysis of causal relationships between isolated variables within a framework which is value-free, logical, reductionistic, and deterministic, based on a priori theories. A quantitative approach endorses the view that psychological and social phenomena have an objective reality that is independ- ent of the subjects being studied, i.e. the knower or the researcher and the known or subjects are viewed as relatively separate and independent. Hence, reality should be studied objectively by the researchers who should put a distance between themselves and what is being studied. On the other hand, qualitative research is based on a constructivist epistemology and explores what it assumes to be a socially con- structed dynamic reality through a framework which is value-laden,flexible,descrip- tive, holistic, and context sensitive; i.e. an in-depth description of the phenomenon from the perspectives of the people involved. It tries to understand how social experience is created and given meaning. From a qualitative perspective, reality or knowledge are socially and psychologically constructed. The qualitative paradigm views the relationship between the knower and the known as inextricably connected.
312 European Journal of Education, Part II
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
Therefore, the researcher is supposed to develop a close, empathic relationship with the subjects being studied (Bergman, 2008; Bryman, 1988; Cohen, Manion & Marrison, 2007; Creswell, 2007; Denzin & Lincoln, 1998; Gelo, Braakmann & Benetka, 2008; Patton, 2002).The epistemological, theoretical, and methodological differences between quantitative and qualitative research designs, together with their underlying assumptions, purposes, approaches and the researcher’s roles are summarised in Table I.
Quantitative methods require the researcher to use a pre-constructed stand- ardised instrument or pre-determined response categories into which the partici- pants’ varying perspectives and experiences are expected to fit. They generally demand randomly selected large representative samples in order for researchers to generalise their findings from the sample, i.e. from where the logic and power of probability sampling derive their purpose, generalisation. The major advantage of this method is that it allows one to measure the responses of a number of participants to a limited set of questions, thereby facilitating comparison and statistical aggregation of the data.The results of closed-ended questionnaires help the researchers to identify a general pattern of participants’ reactions to a treat- ment or programme. Quantitative methods and procedures allow the researchers to obtain a broad and generalisable set of findings and present them succinctly and parsimoniously. But, because they require a deductive approach and predeter- mined sets of standardised responses based on theory, they fail to provide insight into the participants’ individual or personal experiences. They do not let the respondents describe their feelings, thoughts, frames of reference, and experiences with their own words. Quantitative researchers are supposed to play a neutral role in the research process. Hence, the meaning participants ascribe to the phenom- enon studied is largely ignored in quantitative studies (Patton, 2002).
Unlike quantitative studies which are concerned with outcomes, generalisation, prediction, and cause-effect relationships through deductive reasoning, qualitative studies are concerned with process, context, interpretation, meaning or under- standing through inductive reasoning. The aim is to describe and understand the phenomenon studied by capturing and communicating participants’ experiences in their own words via observation and interview. What is emphasised is the examination of the context that influences people’s actions or interactions and the meaning that people ascribe to their experiences. People can elucidate how they make sense of the world around them and their experiences through interviews with open-ended questions. That is why qualitative research requires an in-depth study of people’s lives or the issues in their natural settings without resorting to standardised, pre-determined categories of analysis. Open-ended responses let the researcher understand and present the world as it is seen and experienced by the participants without predetermining those standpoints. Direct quotations docu- ment the participants’ depth of feelings, experiences, thoughts about what is happening, and meaning at a personal level. Hence, qualitative findings are far longer, more detailed and variable in content than quantitative ones. Purposeful sampling plays a key role. Irrespective of the kind of unit of analysis, the main aim of purposeful sampling in qualitative research is to select and study a small number of people or unique cases whose study produces a wealth of detailed information and an in-depth understanding of the people, programmes, cases, and situations studied. But, this sampling procedure limits the possibility of generalising research findings to other settings or situations, i.e. it does not provide parsimonious
Kaya Yilmaz 313
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
T A
B L
E I.
C om
pa ri
so n
of Q
ua nt
it at
iv e
(P os
it iv
is t)
an d
Q ua
lit at
iv e
(N at
ur al
is t)
M od
es of
In qu
ir y
Q ua
nt it
at iv
e M
od e
Q ua
lit at
iv e
m od
e
A ss
um pt
io ns
• R
ea lit
y is
si ng
le ,
ta ng
ib le
, an
d fr
ag m
en ta
bl e.
S oc
ia l
fa ct
s ha
ve an
ob je
ct iv
e re
al it
y. •
K no
w er
an d
kn ow
n ar
e in
de pe
nd en
t, a
du al
is m
. •
P ri
m ac
y of
m et
ho d
• V
ar ia
bl es
ca n
be id
en ti
fie d
an d
re la
ti on
sh ip
s m
ea su
re d
• In
qu ir
y is
ob je
ct iv
e, va
lu e-
fr ee
.
A ss
um pt
io ns
• R
ea lit
ie s
ar e
m ul
ti pl
e, co
ns tr
uc te
d, an
d ho
lis ti
c. R
ea lit
y is
so ci
al ly
co ns
tr uc
te d.
• K
no w
er an
d kn
ow n
ar e
in te
ra ct
iv e,
in se
pa ra
bl e.
• P
ri m
ac y
of su
bj ec
t m
at te
r •
V ar
ia bl
es ar
e co
m pl
ex ,
in te
rw ov
en ,
an d
di ffi
cu lt
to m
ea su
re .
• In
qu ir
y is
su bj
ec ti
ve ,
va lu
e- bo
un d.
P ur
po se
s •
G en
er al
is ab
ili ty
(T im
e an
d co
nt ex
t fr
ee ge
ne ra
lis at
io ns
th ro
ug h
no m
ot he
ti c
or ge
ne ra
lis ed
st at
em en
ts )
• P
re di
ct io
n •
C au
sa l
ex pl
an at
io ns
P ur
po se
s •
C on
te xt
ua lis
at io
n (O
nl y
ti m
e an
d co
nt ex
t bo
un d
w or
ki ng
hy po
th es
es th
ro ug
h id
io gr
ap hi
c st
at em
en ts
) •
In te
rp re
ta ti
on •
U nd
er st
an di
ng ac
to rs
’ pe
rs pe
ct iv
es
A pp
ro ac
h •
B eg
in s
w it
h hy
po th
es es
an d
th eo
ri es
• M
an ip
ul at
io n
an d
co nt
ro l
• U
se s
fo rm
al ,
st ru
ct ur
ed in
st ru
m en
ts •
E xp
er im
en ta
ti on
an d
in te
rv en
ti on
• D
ed uc
ti ve
• C
om po
ne nt
an al
ys is
• S
ee ks
co ns
en su
s, th
e no
rm •
R ed
uc es
da ta
to nu
m er
ic al
in di
ce s
• A
bs tr
ac t
la ng
ua ge
in w
ri te
-u p
A pp
ro ac
h •
E nd
s w
it h
hy po
th es
es or
gr ou
nd ed
th eo
ry •
E m
er ge
nc e
an d
po rt
ra ya
l •
R es
ea rc
he r
as th
e in
st ru
m en
t •
N at
ur al
is ti
c or
no ni
nt er
ve nt
io n
• In
du ct
iv e
• S
ea rc
he s
fo r
pa tt
er ns
• S
ee ks
pl ur
al is
m ,
co m
pl ex
it y
• M
ak es
m in
or us
e of
nu m
er ic
al in
di ce
s •
D es
cr ip
ti ve
w ri
te -u
p
R es
ea rc
he r
R ol
e •
D et
ac hm
en t
an d
im pa
rt ia
lit y
• O
bj ec
ti ve
po rt
ra ya
l •
E ti
c (o
ut si
de r’
s po
in t
of vi
ew )
R es
ea rc
he r
R ol
e •
P er
so na
l in
vo lv
em en
t an
d pa
rt ia
lit y
• E
m pa
th ic
un de
rs ta
nd in
g •
E m
ic (i
ns id
er ’s
po in
t of
vi ew
)
S ou
rc e:
A da
pt ed
fr om
G le
sn e
& P
es hk
in (1
99 2)
; L
in co
ln &
G ub
a (1
98 5)
.
314 European Journal of Education, Part II
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
information about the research topic studied (Denzin & Lincoln, 1998; Patton, 2002; Wolcott, 1994).
Methods of data collection and analysis are also different in the two approaches. Quantitative research uses questionnaires, surveys and systematic measurements involving numbers. Quantitative researchers use mathematical models and statistics to analyse the data and report their findings in impersonal, third-person prose by using numbers. In contrast, qualitative research uses par- ticipants’ observation, in-depth interviews, document analysis, and focus groups. The data are usually in textual, sometimes graphical or pictorial form. Qualitative researchers disseminate their findings in a first-person narrative with a combina- tion of etic (outsider or the researcher’s) and emic (insider or the participants’) perspectives (Denzin & Lincoln, 1998; Miles & Huberman 1994). Since qualita- tive findings are highly context and case dependent, researchers are expected to keep findings in context and report any personal and professional information that may have an impact on data collection, analysis, and interpretations. Bracketing their points of view and biases, the researchers must avoid making any judgement about whether the situation in which they are involved and participants are engaged is good or bad, appropriate or inappropriate. In addition, researchers should make their orientation, predispositions, and biases explicit. Lastly, qualita- tive research reports must provide the reader with sufficient quotations from participants (Patton, 2002).
There are some situations or questions that qualitative research methods illus- trate better than quantitative ones or vice versa. For instance, qualitative methods are especially effective to study a highly individualised programme in which learn- ers who have different abilities, needs, goals, and interests proceed at their own pace. In this case, quantitative methods can provide parsimonious statistical data through mean achievement scores and hours of instruction. But, in order to understand the meaning of the programme for individual participants, their points of view and experiences should be illustrated with their own words via such qualitative methods and techniques as case studies and interviews which provide the detailed, descriptive data needed to deepen our understanding of individual variation. On the other hand, some situations require quantitative research design to be effectively addressed. For example, quantitative methods are more helpful when conducting research on a broader scale or studying a large number of people, cases, and situations since they are cost-effective and statistical data can provide a succinct and parsimonious summary of major patterns (Patton, 2002).
More Elaboration on Qualitative Research The quantitative research paradigm has been practised for a long time, so its defining characteristics are well known. But that is not the case for qualitative research design. Hence, more elaboration is needed to illustrate its distinctive features. The post-positivist, post-structural, constructionist, and critical para- digms lay the bases for the ontological, epistemological, and methodological underpin- nings of qualitative research design. The qualitative inquiry is identified with a relativist ontology (the notion of multiple realities is accepted), a subjectivist epistemology (the idea that understandings are created through interaction between the knower and the unknown or subject), and a naturalistic (subjects are studied in their natural settings) set of methodological procedures (Denzin & Lincoln, 1998, p. 27). Creswell (2007) adds two more philosophical assumptions
Kaya Yilmaz 315
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
to qualitative research.These are axiological (the idea that no research endeavour is value-free in that researchers brings their values to what is researched) and rhe- torical (the language of research is subjective in the form of first person account) assumptions (p. 17). Philosophical assumptions of qualitative research are given in Table II.
Even though the qualitative research tradition is coloured by diversity of different and sometimes conflicting philosophical assumptions, theoretical lens, and prac- tical considerations, the essence of its design as commonly agreed by researchers can be summarised as follows: • Qualitative research design assumes that knowledge is not independent of the
knower, but socially constructed and that reality is neither static nor fixed. Since there are multiple realities that different cultural groups construct on the basis of their world views or value systems, there are multiple interpretations or perspectives on any event or situation. So, understanding the phenomenon
TABLE II. Basic Set of Beliefs or Philosophical Assumptions about Characteristics of Qualitative Research
Assumptions Questions Characteristics Implications for Practice (Examples)
Ontological What is the nature of reality?
Reality is subjective and multiple, as seen by participants in the study
Researcher uses quotes and themes in words of participants and provides evidence of different perspectives
Epistemological What is the relationship between the researcher and that being researched?
Researcher attempts to lessen distance between himself or herself and that being researched
Researcher collaborates, spends time in field with participants, and becomes an ‘insider’
Axiological What is the role of values?
Researcher acknowledges that research is value laden and that biases are present
Researcher openly discusses values that shape the narrative and includes his or her own interpretation in conjunction with the interpretations of participants
Rhetorical What is the language of research?
Researcher writes in a literary, informal style using the personal voice and uses qualitative terms and limited definitions
Researcher uses an engaging style of narrative, may use first-person pronoun, and employs the language of qualitative research
Methodological What is the process of research?
Researcher uses inductive logic, studies the topic within its context, and uses an emerging design
Researcher works with particulars (details) before generalisations, describes in detail the context of the study, and continually revises questions from experiences in the field
Source: Adapted from Creswell (2007).
316 European Journal of Education, Part II
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
under investigation from the perspectives of the participants involved is essential.
• It is holistic, flexible or emergent. It looks at the larger picture or process, searching for an understanding of the whole over time.
• It seeks answers to ‘what’, ‘how’ and ‘why’ questions in terms of quality rather than quantity, amount, intensity, or frequency.
• It postulates that events, cases, processes, situations, individuals and their behaviours are unique, context-dependent and largely non-generalisable. Hence, what is needed is not reductionism but ‘thick description’ of purpose- fully selected small samples or cases.
• It stipulates that people are intentional and creative in their actions, actively construct their social world, and make meanings in and through their activities. People interpret events, contexts and situations, and act on the basis of those events.
• It looks at relationships within a system or culture or face-to-face interactions among people in a given social setting. It emphasises the importance of under- standing a given social setting rather than making predictions about that setting.
• It assumes that events, processes, situations and behaviours change over time and are affected by context.
• It requires researchers to stay in the setting over a substantial period and urges them to develop a model of what occurred in the social setting.
• It requires the researcher to become the research instrument. Hence, the researcher must be able to observe behaviour and interview people face-to- face. The researcher should establish close contact with the research partici- pants when collecting data which need to be detailed, rich, complex, and extensive.
• It demands time in analysis that is equal to the time in the field, calling for ongoing analyses of the data. The bottom-up approach to data analysis with open coding strategies should be practised to allow themes and patterns to emerge from data.
• It involves informed consent decisions and is responsive to ethical concerns. • It incorporates room for description of the role of the researcher as well as
description of the researcher’s own biases and ideological preference (Cohen, Manion, & Marrison, 2007, p. 21; Creswell, 2009, pp. 175–176, 195; Denzin & Lincoln, 1998, p. 42; Snape & Spencer, 2003, p. 5).
Issues Related to Criteria for Evaluating Research: Reliability and Validity Because quantitative and qualitative researchers differ in terms of their approach to defining the concepts of reliability and validity (or determining the criteria for evaluating the quality of a research study), the definition and meaning of these terms for both research designs are explained in separate sections.
Reliability andValidity in Quantitative Research
Reliability means consistency or the degree to which a research instrument meas- ures a given variable consistently every time it is used under the same condition with the same subjects. It is important to note that reliability applies to data not to measurement instruments. From different perspectives or approaches, researchers can evaluate the extent to which their instruments provide reliable data. Types of
Kaya Yilmaz 317
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
reliability can be briefly explained as follows (Huck, 2000; Keppel, 1991;Trochim, 2005):
Test-retest reliability refers to the extent that the same test administered by the researcher to a single group of subjects on two different occasions gives highly positively correlated results. Two sets of scores from the same test should be correlated for the researcher to claim they are consistent (i.e. assessment of the stability of the instrument over time). Parallel forms reliability demonstrates whether two forms of the same instrument administered to the same group of people to measure the same characteristic such as intelligence give highly posi- tively correlated results. The researcher examines whether there is a consistency between the scores obtained for any examinee across the two settings (i.e. the degree of equivalence across forms). Internal consistency reliability indicates whether measuring instruments possess internal consistency or the results of the instrument administered to a group of people on one occasion correlate very positively. There should be consistency across the parts of a measuring instru- ment or subsets of questions. To judge that the full instrument possesses high internal consistency reliability, the researcher determines the extent to which parts of a test hang together and measure the same thing. Inter-rater (Inter- observer) reliability refers to the process whereby the researcher gathers data by asking raters to evaluate a set of objects, pictures etc., and then quantifies the degree of consistency among the raters. To that end, the researcher computes an index of interrater reliability.
Validity refers to the accuracy of research data. A researcher’s data can be said to be valid if the results of the study measurement process are accurate. That is, a measurement instrument is valid to the degree that it measures what it is supposed to measure. There are different types of validity. Internal validity refers to whether there is a causal relationship between the treatment and the outcome. External validity reflects the degree to which one can generalise research results or the effects of the treatment beyond the present conditions of testing; that is, other settings, programmes, persons, places, times, cases, or approaches. Construct valid- ity refers to the degree to which conclusions can be made from the operationali- sations of a study to the theoretical constructs on which operationalisations are based. In other words, the treatment or the programme should reflect the construct on which they are based. For instance, if the study examined the effects of simulation or role playing on students’ ability to empaphise with historical agents, the treatment (simulation) should accurately reflect the construct of simulation, and the measured outcome (historical outcome) should reflect the construct of historical empathy. Conclusion validity indicates whether there is a relationship between the independent variable and the dependent variable or outcome (Huck, 2000; Keppel, 1991).
Reliability andValidity in Qualitative Research
Rather than using quantitative researchers’ jargons, qualitative researchers prefer to use their own terms to communicate what is meant by reliability, validity, and objectivity in quantitative language. Some researchers have even argued that deter- mining the quality of qualitative studies via quantitative concepts or measures such as reliability and validity is not only irrelevant but also misleading (Creswell, 2009, p. 190; Davies & Dodd, 2002, p. 280; Steinke, 2004, p. 186; Stenbacka, 2001, p. 551). Since qualitative research is focused on meaning and interpretation in
318 European Journal of Education, Part II
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
cases which are unique and context-bound, ‘traditional thinking about generaliz- ability falls short . . . and reliability in the traditional sense of replicability is pointless’ (Denzin & Lincoln, 1998, p. 51). It is believed that because ontological, epistemological, and theoretical assumptions of qualitative research are so funda- mentally different from those of quantitative research, it should be judged on its own terms. Hence, it is proposed that rather than the concepts of validity and reliability, an alternative set of criteria based on qualitative concepts need to be used to judge the trustworthiness of a qualitative research which needs its own criteria for evaluation (Gibbs, 2007; Lincoln & Guba, 1985; Wolcott, 1994).
The concept of validity in quantitative study corresponds to the concept of credibility, trustworthiness, and authenticity in qualitative study which means that the study findings are accurate or true not only from the standpoint of the researcher but also from that of the participants and the readers of the study (Creswell & Miller, 2000). The concept of reliability in quantitative study is comparable, but not identical, with the concept of dependability and auditability in qualitative study, which means that the process of the study is consistent over time and across different researchers and different methods or projects (Gibbs, 2007; Miles & Huberman, 1994). To judge the quality or (a) credibility and (b) dependability of a qualitative study, the following questions compiled from various studies can be asked (Miles & Huberman, 1994, pp. 278–279): Credibility (instead of validity) questions: • How context-rich and detailed are the basic descriptions? • Does the account ‘ring true’, make sense, seem convincing or plausible, enable
a ‘vicarious presence’ for the reader? • Is the account rendered comprehensive, respecting the configuration and tem-
poral arrangement of elements in the local context? • Did triangulation among complementary methods and data sources generally
lead to converging conclusions? If not, is there a coherent explanation for this? • Are the presented data linked to the categories of prior or emergent theory if
used? • Are the findings internally coherent and concepts systematically related? • Were guiding principles used for confirmation of propositions made explicit? • Are areas of uncertainty identified? • Was negative case or evidence sought for? Found? What happened then? • Have rival explanations been actively considered? What happened to them? • Were the conclusions considered to be accurate by the participants involved in
the study? If not, is there a coherent explanation for this?
Dependability (instead of reliability) questions: • Are research questions clearly defined and the features of the study design
congruent with them? • Are basic paradigms and analytic constructs clearly specified? • Are the researcher’s role and status within the site explicitly described? • If multiple field-researchers are involved, do they have comparable data collec-
tion protocols? • Do multiple observers’ accounts converge, in instances, settings, or times when
they might be expected to? • Were data connected across the full range of appropriate settings, times,
respondents suggested by the research questions?
Kaya Yilmaz 319
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
• Were coding checks made and did they show adequate agreements? • Were data quality checks for bias, deceit, informant knowledgeability etc.
made? • Do findings show meaningful parallelism across data sources (informants,
contexts, and times)? • Were any forms of peer or colleague review employed?
Lincoln and Guba are commonly acknowledged to have made a great contribution to the criteria debate in qualitative research by developing parallel criteria to the concepts of validity and reliability (Spencer et al., 2003). Their alternative criteria are constantly cited in the research literature and still considered to be the yard- stick or the ‘gold standard’ (Spencer et al., 2003; Whittemore, Chase & Mandle, 2001).To assess the rigour of qualitative research, Lincoln and Guba (1985) resort to the concepts of credibility, transferability, dependabilty, and confirmability to express the quantitative concepts of internal validity, external validity (generalis- ability), reliability, and objectivity respectively. Credibility means that the partici- pants involved in the study find the results of the study true or credible. Transferability is achieved if the findings of a qualitative study are transferable to other similar settings.Thick description of the setting, context, people, actions, and events studied is needed to ensure transferability or external validity in quantitative terms.The study has dependability (reliability) if the process of selecting, justifying and applying research strategies, procedures and methods is clearly explained and its effectiveness evaluated by the researcher and confirmed by an auditor, which is called ‘audit trail’. The study enjoys confirmability when its findings are based on the analysis of the collected data and examined via an auditing process, i.e. the auditor confirms that the study findings are grounded in the data and inferences based on the data are logical and have clarity, high utility or explanatory power (Table III).
Incorporated to these concepts by Guba and Lincoln (1989) in their discussion of the issues related to validity and reliability are fairness (the various perspectives of all participants should be given equal consideration), educative authenticity (the study enables the participants to educate themselves by raising their conscious- ness), catalytic authenticity (the study encourages activity and decision making), and empowerment (the study promotes the participants’ ability to make choices about their professional activity).
In short, terms such as credibility, trustworthiness, authenticity, neutrality or confirmability, dependability, applicability or transferability and the like are those that qualitative researchers use most in their discussion of the concepts of
TABLE III. Criteria for Judging the Quality of a Research Study: Quantitative vs. Qualitative Terms
Aspect Quantitative terms Qualitative terms
Truth value Internal validity Credibility Applicability External validity or generalizability Transferability Consistency Reliability Dependability Neutrality Objectivity Confirmability
Source: Adapted from Lincoln and Guba (1985).
320 European Journal of Education, Part II
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
reliability and validity1. Patton (2002) argues that judging ‘quality’ constitutes the foundation for perceptions of credibility. Issues related to quality and credibility correspond to the audience and intended inquiry purposes.Therefore, the criteria for judging quality and credibility depend on the philosophical underpinnings, theoretical orientations, and purposes of a particular qualitative research. Taking this crucial point into account, Patton (2002, pp. 542–544) suggests alternative sets of five criteria for judging the quality or credibility of a qualitative inquiry: (1) traditional scientific research criteria, (2) social construction and constructivist criteria, (3) artistic and evocative criteria, (4) critical change criteria, and (5) evaluation standards and principles (Table IV).
The credibility of a qualitative study is affected by the extent to which system- atic data collection procedures, multiple data sources, triangulation, thick and rich description, external reviews or member checking, external audits, and other techniques for producing trustworthy data are used. According to Patton (2002), three distinct but related inquiry elements determine the credibility of a qualitative research:
(1) rigourous methods to do fieldwork that yield high-quality data that are systematically analysed with attention to issues of credibility; (2) the credibility of the researcher, which is dependent on training, experience, track, record, status, representation of self; and (3) philosophical beliefs in the value of qualitative inquiry, i.e. an appreciation of naturalistic inquiry, quali- tative methods, inductive analysis, purposeful sampling, and holistic thinking (pp. 552–553).
For a qualitative study to be credible and trustworthy, the data must first and foremost be sufficiently descriptive and include a great deal of pure description of people, activities, interactions, and settings so that the reader or reviewer can understand what occurred and how it occurred. The basic criterion to judge the credibility of data is the extent to which they allow the reader to enter the situation or setting under study. In other words, rich and detailed or thick description of the setting and participants is a must.The researcher must provide an accurate picture of the empirical social world as it exits to those under investigation, rather than as he or she imagines it to be. The descriptions must be factual, accurate, detailed but without being overburdened with irrelevant information or trivia. In addition, researchers should overtly reveal the biases they bring to the study and discuss how their background such as gender, ethnicity, disciplinary orientation and ideo- logical viewpoint affected the interpretation of the findings. Since the nature of qualitative inquiry is fundamentally people-oriented, qualitative researchers must get close enough to the people and situation being studied in order to capture what actually takes place and what people actually say; i.e. develop an in-depth understanding of the phenomenon under investigation. To that end, they should spend prolonged time in the setting with the participants without dismissing the negative or discrepant cases observed in the setting. Member checking (the partici- pants check and evaluate the final research report to determine if its descriptions and themes accurately reflect their viewpoints), peer debriefing (involving another researcher in reviewing the study report to see if it fits or resonates with the experience of both the participants and the audience rather than the researcher), and external auditor (an independent researcher who, unlike the peer debriefer, is not familiar with the researcher, reviews the study project to evaluate its accuracy)
Kaya Yilmaz 321
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
TABLE IV. Criteria for Judging the Quality and Credibility of Qualitative Inquiry
Traditional Scientific Research Criteria � Objectivity of the inquirer [attempts to minimise bias] � Validity of the data � Systematic rigour of fieldwork procedures � Triangulation [consistency of findings across methods and data sources] � Reliability of coding and pattern analysis � Correspondence of findings to reality � Generalisability [external validity] � Strength of evidence supporting causal hypothesis � Contributions to theory
Construction and Constructivist Criteria � Subjectivity acknowledged [discusses and takes into account biases] � Trustworthiness and authenticity � Triangulation [capturing and respecting multiple perspectives] � Reflexivity and praxis � Particularity [doing justice to the integrity of unique cases] � Enhanced and deepened understanding [Verstehen] � Contributions to dialogue
Artistic and Evocative Criteria � Opens the world to us in some way � Creativity � Aesthetic quality � Interpretive vitality � Flows from self; embedded in lived experience � Stimulating � Provocative � Connects and moves the audience � Voice distinct and expressive � Feels true, authentic or real
Critical Change Criteria � Critical perspective: Increases consciousness about injustice � Identifies nature and sources of inequalities and injustice � Represents the perspective of the less powerful � Makes visible the ways in which those with more power exercise and benefit from this power � Engages those with less power respectfully and collaboratively � Builds the capacity of those involved to take action � Identifies change-making strategies � Clear historical and values context
Evaluation Standards and Principles � Utility � Feasibility � Propriety � Accuracy [balance] � Systematic inquiry � Evaluator competence � Integrity/honesty � Respect for people [fairness] � Responsibility to the general public welfare [taking into account diversity of interests and
values]
Source: Adapted from Patton (2002).
322 European Journal of Education, Part II
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
should be employed as verification strategies to ensure the accuracy of the account. Because only one research method is unlikely to adequately address the problem of rival explanations, employing multiple data collection methods to study the same setting, issue, or programme increases the credibility of the find- ings by eliminating or reducing errors linked to a particular method. Thus, trian- gulation or combination of interviewing, observation, and document analysis contributes to a rigorous qualitative research study. Five types of triangulation enhance verification of qualitative analysis, adding depth and breadth to under- standing of the issue under investigation.These are methods triangulation, sources triangulation, analyst triangulation, theory/perspective triangulation, and meth- odological triangulation, all of which together enable the researcher to gain a broader and deeper understanding of the research issue (Creswell, 2009; Denzin & Lincoln, 1994; Patton, 2002).
Corbin and Strauss (2008) identify certain research situations and conditions that affect the quality of a qualitative study.These are methodological consistency, methodological awareness, clarity about the purpose of the study, having training in qualitative research tradition, self-awareness about one’s biases and assump- tions, creativity and openness to new ideas, sensitivity for the research including the research data and the participants, and a willingness to do research for its own sake and to work hard (pp. 302–304).
Conclusion Educational researchers, especially graduate students, need to acquaint them- selves with different research approaches. Quantitative and qualitative research approaches represent the two ends of the research continuum. They differ in terms of their epistemological assumptions, theoretical frameworks, methodo- logical procedures and research methods. Whereas the former is based on posi- tivism or objective epistemology, relies on quantitative measures for collecting and analysing data, and aims to make predictions and generalisations, the latter is based on constructivism, draws on naturalistic methods for data collection and analysis, and aims to provide an in-depth understanding of people’s experiences and the meanings attached to them. Having been viewed not only as competitive but also incompatible research paradigms for some decades, they are now con- sidered as alternative strategies for research. Both approaches have their own strengths and weaknesses in their design and application. Which approach should be used when planning a research depends on several factors such as the type of questions asked, the researcher’s training or experinces, and the audience.
Kaya Yilmaz, Marmara University, Ataturk Faculty of Education, Department of Elementary Education, Kadikoy, Istanbul,Turkey, [email protected]
NOTE
1. However, the term validity is used by some researchers in relation to qualita- tive research. Schwandt (1997) defines validity as the extent to which the qualitative account accurately represents the research participants’ views of social phenomena and is credible to them. Likewise, reliability is defined as the extent to which the qualitative study provides an ‘understanding’ of a situation, setting, case, programme, or event that otherwise would be confusing
Kaya Yilmaz 323
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
and enigmatic (Eisner, 1991, p. 58). Lincoln and Guba (1985) use the concept of dependability to refer to reliability in quantitative studies (p. 300).
REFERENCES
BERGMAN, M. M. (2008) Advances in Mixed Methods Research (Los Angeles, Sage).
BREWER, J. D. (2003) Qualitative research, in: R. L. MILLER & J. D. BREWER (Eds) The A-Z of Social Research: a dictionary of key social science research concepts (pp. 238–241) (Thousand Oaks, Sage).
BRYMAN, A. (1988) Quantity and Quality in Social Research (London, Routledge). COHEN, L., MANION, L. & MARRISON, K. (2007) Research Methods in Education
(6th edition) (New York, Routledge). CORBIN, J. & STRAUSS, A. (2008). Basics of Qualitative Research: techniques and
procedures for developing grounded theory (3rd edition) (Thousand Oaks, Sage). CRESWELL, J. W. (1994) Research Design: qualitative & quantitative approaches
(Thousand Oaks, Sage). CRESWELL, J.W. (2007) Qualitative Inquiry and Research Design: choosing among five
traditions (2nd edition) (Thousand Oaks, Sage). CRESWELL, J. W. (2009) Research Design: qualitative, quantitative and mixed
approaches (3rd edition) (Thousand Oaks, Sage). CRESWELL, J.W. & MILLER, D. (2000) Determining validity in qualitative inquiry,
Theory into Practice, 39, pp. 124–130. CROTTY, M. (1998) The Foundations of Social Research: meaning and perspective in
the research process (Thousand Oaks, Sage). DAVIES, D. & DODD, J. (2002) Qualitative research and the question of rigor,
Qualitative Health Research, 12, pp. 279–289. DENZIN, N. K. & LINCOLN, Y. S. (1994) Introduction: entering the field of
qualitative research, in: N. K. DENZIN & Y. S. LINCOLN (Eds) Handbook of Qualitative Research (pp. 1–17) (London, Sage).
DENZIN, N. K. & LINCOLN, Y. S. (1998). Strategies of Qualitative Inquiry (Thousand Oaks, Sage).
DENZIN, N. K. & LINCOLN, Y. S. (2005) Handbook of Qualitative Research (3rd edition) (Thousand Oaks, Sage).
EISNER, E.W. (1991) The Enlightened Eye: qualitative inquiry and the enhancement of educational practice (New York, NY, Macmillan Publishing Company).
GAY, L. R. & AIRASIAN, P. (2000) Educational Research: competencies for analysis and application (6th edition) (Upper Saddle River, Prentice Hall).
GELO, O., BRAAKMANN, D. & BENETKA, G. (2008) Quantitative and qualitative research: beyond the debate, Integrative Psychological Behavioral Science, 42, pp. 266–290.
GIBBS, G. R. (2007) Analyzing qualitative data, in: U. FLICK (Ed) The Sage Qualitative Research Kit (London, Sage).
GLESNE, C. & PESHKIN, A. (1992) Becoming Qualitative Researchers: an introduction (New York, Longman).
GUBA, E. & LINCOLN, Y. (1989) Fourth Generation Evaluation (Newbury Park, Sage).
HITCHCOCK, G. & HUGHES, D. (1995) Research and the Teacher: a qualitative introduction to school-based research (2nd edition) (London, Taylor & Francis).
324 European Journal of Education, Part II
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
HUCK, S. W. (2000) Reading Statistics and Research (3rd edition) (New York, Longman).
KEPPEL, G. (1991) Design and Analysis: a researchers’ handbook (Upper Saddle River, Prentice Hall).
LINCOLN, Y. S. & GUBA, E. G. (1985) Naturalistic Inquiry (Newbury Park, Sage).
MILES, M. & HUBERMAN, M. (1994) Qualitative Data Analysis (2nd ed) (Thou- sand Oaks, Sage).
PATTON, M. Q. (2002) Qualitative Research & Evaluation Methods (3rd Edition) (Thousands Oaks, Sage).
SCHWANDT, T. A. (1997) Qualitative Inquiry: A dictionary of terms (Thousand Oaks, Sage).
SNAPE, D. & SPENCER, L. (2003) The foundations of qualitative research, in: J. RITCHIE & J. LEWIS (Eds) Qualitative Research Practice: a guide for social science students and researchers (pp. 1–23) (Thousand Oaks, Sage).
SPENCER, L., RITCHIE, J., LEWIS, J. & DILLON, L. (2003) Quality in Qualitative Evaluation:a framework for assessing research evidence (London, National Centre for Social Research, Government Chief Social Researcher’s Office).
STEINKE, I. (2004) Quality criteria in qualitative research, in: U. FLICK, E. V. KARDORFF & I. STEINKE (Eds) A Companion to Qualitative Research (pp. 184–190) (Thousand Oaks, Sage).
STENBACKA, C. (2001) Qualitative research requires quality concepts of its own, Management Decision, 39, pp. 551–555.
STRAUSS, A. L. & CORBIN, J. (1998) Basics of Qualitative Research: techniques and procedures for developing grounded theory (2nd edition) (Thousand Oaks, Sage).
TROCHIM, W. M. (2005) Research Methods Knowledge Base. www.socialresearch methods.net/kb
WHITTEMORE, R., CHASE, S. K. & MANDLE, C. L. (2001) Validity in qualitative research, Qualitative Health Research, 11, pp. 522–537.
WOLCOTT, H. F. (1994) Transforming Qualitative Data: description, analysis, interpretation (Thousand Oaks, Sage).
Kaya Yilmaz 325
© 2013 John Wiley & Sons Ltd
14653435, 2013, 2, D ow
nloaded from https://onlinelibrary.w
iley.com /doi/10.1111/ejed.12014 by A
ugusta U niversity, W
iley O nline L
ibrary on [02/10/2023]. See the T erm
s and C onditions (https://onlinelibrary.w
iley.com /term
s-and-conditions) on W iley O
nline L ibrary for rules of use; O
A articles are governed by the applicable C
reative C om
m ons L
icense
