Evaluate a Qualitative Study

3.3 Cri�quing a Qualita�ve Study

How does one assess the overall worth and credibility of a qualita�ve research study or proposal? What characteris�cs cons�tute strong research studies, and what characteris�cs cons�tute weak ones? The methods and guidelines for evalua�ng research studies are fairly detailed and tedious. Not all studies are worthy or rigorous, and it is important that you, as a consumer of psychological research, are able to iden�fy poten�al problems. Just because a study claims to have valid and reliable results does not necessarily mean that it does. For example, some studies may u�lize incorrect sta�s�cal or data analysis procedures, so the results generated may not be correct or complete. Addi�onally, some studies may use inappropriate sampling techniques for the type of research design being conducted. As professionals in the field, you will need to be able to iden�fy which parts of a study are valid, which parts can be considered acceptable with cau�on, and which parts have significant limita�ons or are downright misleading.

Leedy and Ormrod (2010) reviewed standards from experienced qualita�ve researchers and compiled a list of general criteria to apply when evalua�ng a qualita�ve study:

1. Purposefulness: Does the research ques�on(s) drive the research process and the methods to collect and analyze the data?

2. Explicitness of Assump�ons and Biases: Does the researcher describe any assump�ons, expecta�ons, or biases that might influence how the data are collected, analyzed, and interpreted?

3. Rigor: Does the researcher use rigorous, precise, and thorough methods to collect, record, and analyze the data? Does the researcher take steps to remain objec�ve throughout the study?

4. Open-mindedness: Is the researcher willing to modify interpreta�ons when newly collected data do not support previously collected data?

5. Completeness: Does the researcher describe the phenomenon in all its complexity? Does the researcher spend sufficient �me in the field examining the phenomenon, detail all aspects of the phenomenon (e.g., se�ng, behaviors, percep�ons), and provide a holis�c picture of the phenomenon?

6. Coherence: Do the data show consistent findings with the measurement used and across mul�ple measurement methods used?

7. Persuasiveness: Does the researcher provide logical arguments, and does the evidence support one interpreta�on of the data?

8. Consensus: Do other studies and researchers in the field agree with the interpreta�ons and explana�ons? 9. Usefulness: Does the study provide useful implica�ons for future research, a more thorough

understanding of the phenomenon, or lead to interven�ons that could enhance the quality of life? (p. 187)

In addi�on to these criteria, there are several factors to consider when evalua�ng the various sec�ons of a research study or proposal. The next seven sec�ons will discuss how to cri�cally evaluate the literature review, the purpose statement, the sampling methods, the procedures, the instruments, the results, and the discussion sec�on of a qualita�ve study.

Evaluating the Literature Review Section

As men�oned in Chapter 1, the purpose of the literature review is to support the need for research to be conducted. Literature reviews should be thorough and comprehensive and contain all relevant research on the specific topic being studied. Literature reviews should also be objec�ve, showing no biases toward the selec�on of ar�cles being reviewed, and include previous research that relates to the current study. The following ques�ons, adapted from Houser (2009), will assist in the evalua�on of the literature review:

Do the researchers present an adequate ra�onale for conduc�ng the study? What is the significance of the study? What difference will it make to the field? Is the literature review thorough and comprehensive? Do the researchers demonstrate any poten�al biases in the literature review? Are all important concepts clearly defined by the researchers? Do the researchers clearly describe previous methods that are relevant to understanding the purpose for conduc�ng this study?

Evaluating the Purpose Statement

The purpose statement provides the aim or intent of the study. It is generally found in the Introduc�on sec�on, as the last paragraph before the Literature Review sec�on. Purpose statements can be wri�en as a declara�ve statement or in the form of a ques�on or ques�ons. They should include the type of research methods and design used and describe the variables and popula�on studied. When evalua�ng the purpose statement, it is also important to examine whether the purpose and research problem are in fact researchable. Purpose statements and research problems are researchable only when the variables of interest can be opera�onalized—that is, defined in a measurable and objec�ve way. Considering these requirements, the following ques�ons, adapted from Houser (2009), can assist in the evalua�on of the purpose statement:

Does the ar�cle clearly present the purpose statement? Is the purpose statement clearly based on the argument developed in the literature review? Are the variables of interest (i.e., independent and dependent) clearly iden�fied in the purpose statement?

Evaluating the Methods Section—Sampling

The Sampling sec�on includes thorough and detailed informa�on on the sample used and the techniques or methods used to select the sample. Descrip�ons of the sample should include all relevant demographic characteris�cs (e.g., age, ethnicity, sex) as well as size. Unlike quan�ta�ve approaches, which usually require large samples, qualita�ve techniques do not have any restric�ons on sample size. Thus, sample size depends on what the research wants to know, the purpose of the inquiry, what it will be useful for, how credible it will be, and what can be done with available �me and resources. Qualita�ve research can be very costly and �me-consuming, so choosing informa�on-rich cases will be most valuable. As noted by Pa�on (2002), "The validity, meaningfulness, and insights generated from qualita�ve inquiry have more to do with the

informa�on-richness of the cases selected and the observa�onal/analy�cal capabili�es of the researcher than with sample size" (p. 245).

The sampling techniques employed should also be discussed, including detailed informa�on about how the sample was selected and what sampling methods were used (e.g., purposive sampling, snowball sampling). In contrast to quan�ta�ve research, which strives for generalizable representa�ve sampling, qualita�ve research typically focuses on rela�vely smaller samples, and even some�mes only single cases, that are selected purposefully. "Purposeful sampling refers to selec�ng informa�on-rich cases for study in depth" (Pa�on, 2002, p. 230). When evalua�ng the sampling methods sec�on, it is important to examine whether appropriate sampling techniques were used for the type of research design that was employed and the research ques�ons proposed. The following list covers a few of the most common sampling procedures used in qualita�ve studies (these are discussed in more detail in Chapter 4):

Purposive sampling (or judgment sampling): The researcher selects a sample that will yield the most informa�on to answer the research ques�ons. Quota sampling (a type of purposive sampling): The researcher determines the number of par�cipants and what characteris�cs will be needed, and then selects a sample based on those. Theore�cal sampling: The researcher selects a sample that will assist him or her in developing a theory. Convenience sampling: The researcher selects anyone who shows up for the study, regardless of individual demographics. Snowball sampling (a type of purposive sampling): The researcher collects data on a few par�cipants that he or she has access to and then asks those par�cipants for referrals to other individuals who are within the same popula�on.

All these approaches serve a somewhat different purpose. However, the underlying principle common to all of these techniques is selec�ng informa�on-rich cases.

The following ques�ons, adapted from Houser (2009), are provided to assist in the evalua�on of the sampling methods sec�on:

What type of sampling method is used? Are the sampling procedures consistent with the purpose and research ques�ons? Are relevant demographic characteris�cs of the sample clearly iden�fied? Do the methods of sample selec�on provide a good representa�ve sample, based on the popula�on? Are there any apparent biases in the selec�on of the sample? Is the sample size large enough for the study proposed?

Evaluating the Methods Section—Procedures

The procedures sec�on provides a detailed descrip�on of everything that was conducted in the study. For qualita�ve studies, this involves primarily the type of research design that was employed. When evalua�ng the procedures sec�on, it is important to examine whether the research design is appropriate for the study,

as well as whether it is consistent with the purpose and research ques�ons. The following ques�ons, adapted from Houser (2009), are provided to assist in the evalua�on of the procedures sec�on:

What type of research design is used (e.g., case study, phenomenological study)? Is the research design consistent with the purpose and research ques�ons? Did the researcher provide a detailed descrip�on of what was conducted? Did the researcher introduce any bias in the procedures used?

Evaluating the Methods Section—Instruments

The instruments sec�on provides a detailed descrip�on regarding the types of instruments and measures that were used to collect the data. In qualita�ve research, common instrumenta�on includes interviews, observa�ons, and journals. When evalua�ng the instruments sec�on, it is important to consider whether the instruments were appropriate for the study and the sample and whether there were any limita�ons in the types of instruments u�lized. The following ques�ons, adapted from Houser (2009), are provided to assist in the evalua�on of the instruments sec�on:

Is there a clear and adequate descrip�on of the instruments (e.g., data collec�on measures) used? What types of measures were used in the study (observa�ons, interviews, etc.)? What are some poten�al problems or limita�ons of the types of measures used? Does the instrument appear to be appropriate for the sample?

Evaluating the Results Section

The results sec�on describes findings from the study. Unlike quan�ta�ve studies, which focus on sta�s�cal analyses and results, qualita�ve studies include descrip�ons about the findings. When evalua�ng the Results sec�on, it is important to examine how the data were analyzed (using themes, pa�erns, codes, etc.), whether concrete examples supported the themes or concepts, and how adequate the descrip�ons were to the findings. The following ques�ons, adapted from Houser (2009), are provided to assist in the evalua�on of the results sec�on:

What strategies were used for coding and interpre�ng the data? Were they clearly described? Are concrete examples provided that link to iden�fied themes or concepts? Are the examples adequate?

Evaluating the Discussion Section

The discussion sec�on summarizes the purpose of the research and what the findings imply for future research and actual prac�ce. Addi�onally, the discussion sec�on includes alterna�ve explana�ons and poten�al limita�ons of the findings. The following ques�ons, adapted from Houser (2009), are provided to assist in the evalua�on of the discussion sec�on:

Do the researchers clearly restate the purpose and research ques�ons?

Do the researchers clearly discuss the implica�ons of the findings and how they relate to theories, other findings, and actual prac�ce? Do the researchers provide alterna�ve explana�ons of the findings? Do the researchers iden�fy poten�al limita�ons of the study and the results? Do the researchers iden�fy possible direc�ons for future research?