quantitative study
5.7 Critiquing a Quantitative Study
As discussed in Chapter 1 (Section 1.5, Searching the Literature) and Chapter 3 (Section 3.3, Critiquing a Qualitative Study), the methods andguidelines for evaluating research studies are fairly detailed for both quantitative and qualitative studies. For any type of study, it is important toread the entire article so that you can determine which parts of the study are valid, which parts can be considered acceptable but with caution,and which parts have significant limitations. As mentioned in Chapter 3, a study does not necessarily have valid and reliable results even if itclaims that it does. Not all studies are worthy of being taken seriously, and it is important that you, as a consumer of psychological and othertypes of research, are able to identify potential problems.
There are several factors to consider when evaluating the sections of a quantitative research study or proposal. The following sections discusshow to critically evaluate the literature review, the purpose statement, the sampling methods, the procedures, the instruments, the results, andthe discussion.
Evaluating the Literature Review Section
Similar to a qualitative study, the literature review section of a quantitative study is meant to provide an adequate rationale for conducting thestudy and to show what difference the study hopes to make in the field. Literature reviews should be thorough and comprehensive and containrelevant research on the topic being studied. They should also be objective, unbiased in the selection of articles being reviewed, and includeprevious research that relates to the current study. Considering these daunting requirements, the following questions, adapted from Houser(2009), can help you to evaluate the literature review:
· Do the researchers present an adequate rationale for conducting the study?
· What is the significance of the study? What difference will it make to the field?
· Is the literature review thorough and comprehensive?
· Do the researchers demonstrate any potential biases in the literature review?
· Are all important concepts clearly defined by the researchers?
· Do the researchers clearly describe previous methods that are relevant to understanding the purpose for conducting this study?
Evaluating the Purpose Statement and Hypotheses
As discussed in Chapter 3, the purpose statement does the following: provides the aim or intent of the study; the focus, methodology,population, and geographical location of the study; the specific area of the research and the central concepts or ideas of the study; justificationfor the method of inquiry (qualitative, quantitative, or mixed); and a description of possible conclusions and their implications. It is generallyfound in the Introduction, as the last paragraph before the Literature Review section. Purpose statements can be written in declarative form orin the form of a question or questions. They should include the type of research methodology and design used, as well as a description of thevariables and population studied. When evaluating the purpose statement, it is important to examine whether the purpose and researchproblem are researchable. Purpose statements and research problems are researchable when the variables of interest can be operationalized, ordefined in a measurable and objective way.
It is important to note that not all researchers use purpose statements in quantitative studies, but instead generally provide only hypotheses. Itis not typical for a researcher to include the null hypothesis, as it can be inferred that every researcher has started out with one. Rather,alternative types of research hypotheses are given. For example, some research will include alternative nondirectional hypotheses. Theseindicate that there is enough evidence from the literature to suggest a difference, but not enough to predict the direction of that difference.Another hypothesis is the alternative directional hypothesis, which suggests a difference and the direction of that difference.
Hypotheses guide researchers in selecting appropriate statistical analyses, so there should be consistency between the hypotheses and statisticaltests chosen. The following questions, adapted from Houser (2009), are provided to help you to evaluate the purpose statement:
· Does the article clearly state the purpose statement?
· Is the purpose statement clearly based on the argument developed in the literature review?
· Are the variables of interest (i.e., independent and dependent) clearly identified in the problem statement?
· What type of hypothesis is presented?
· Is there a clear rationale for the direction of the hypothesis, based on the literature review?
· Is the hypothesis testable and falsifiable?
· Is the hypothesis consistent with current theories or research?
Evaluating the Methods Section: Sampling
The Sampling section includes thorough, detailed information about the sample that was chosen and the sampling techniques or methods thatwere used to select it. Descriptions of the sample should include all relevant demographic characteristics (e.g., age, ethnicity, gender) as well asthe size. Unlike qualitative approaches, which do not have hard-and-fast rules regarding sample size, quantitative approaches often require muchlarger samples. As noted in Chapter 4 (Section 4.3), Gay, Mills, and Airasian (2009) provided the following guidelines for determining a sufficientsample size based on the size of the population:
· For populations that include 100 or fewer individuals, the entire population should be sampled.
· For populations that include 400–600 individuals, 50% of the population should be sampled.
· For populations that include 1,500 individuals, 20% of the population should be sampled.
· For populations that are 5,000 or larger, about 8% of the population should be sampled. (Note, however, that it is possible to have a samplethat is so large that it does not justify the increased costs, and may produce significant differences that are too small to be meaningful).
The sampling techniques employed should be discussed, including detailed information about how the sample was selected and what methodswere used (e.g., simple random sampling, stratified sampling, cluster sampling). The following are a few of the most common samplingprocedures used in quantitative studies. These are discussed in more detail in Chapter 4, Section 4.3, Sampling From the Population.
· Simple Random Sampling : The researcher randomly picks study participants from a list of everyone in the population.
· Stratified Random Sampling : The researcher breaks the sampling frame into subgroups and then samples a random number from eachsubgroup.
· Cluster Sampling : The researcher randomly selects intact groups.
· Systematic Random Sampling : The researcher selects every nth participant from a finite list of those in the population.
Sampling errors and sampling biases should also be considered. Sampling errors can occur when the sample selected is not representative ofthe population. This generally occurs when smaller sample sizes are used. Sampling bias occurs when the researcher purposely selectsparticipants in a particular way that introduces bias into the sample. Convenience samples typically contain sampling bias because the samplesare not usually representative of the entire population.
The following questions, adapted from Houser (2009), can help you evaluate the sampling methods section:
· What type of sampling method is used?
· Are the sampling procedures consistent with the purpose and hypotheses?
· Are relevant demographic characteristics of the sample clearly identified?
· Do the methods of sample selection provide a good representative sample, based on the population?
· Are there any apparent biases in the selection of the sample?
· Is the sample large enough for the study proposed? How has size been justified?
· To whom can study results be generalized?
Evaluating the Methods Section: Procedures
The Procedures section describes in detail everything that was conducted in the study. For quantitative studies, it discusses the study variables,how informed consent was obtained, how the participants were assigned to groups, and how the researcher controlled for extraneous orconfounding variables. This section should clearly describe the independent variable as well as give details about whether a control groupexisted, and if so, how it was treated. Additionally, it should cover how the participants were randomly assigned to the experimental group andcontrol group. When evaluating the procedures section, one should assess whether there are any threats to the internal or external validity ofthe study. Furthermore, it is important to examine whether the research design was appropriate for the study and whether it was consistentwith the purpose and research questions. The following questions, adapted from Houser (2009), can assist in the evaluation of the Proceduressection:
· What type of research design was used?
· Was the research design consistent with the purpose and hypotheses?
· Did the researcher provide a detailed description of what was conducted?
· Did the researcher introduce any bias in the procedures used?
· Was the independent variable clearly defined?
· Were there any threats to the internal validity of the study?
· Were there any threats to the external validity of the study?
Evaluating the Methods Section: Instruments
The Instruments section describes in detail the types of instruments and measures that were used to collect the data. In quantitative research, avariety of common instruments are used: direct observation and behavioral measures, self-report inventories, ratings of peoples’ behaviors,questionnaires, physiological approaches, interviews, and standardized assessments.
When evaluating the Instruments section, consider whether the instruments were appropriate for the study and the sample and whether therewere any limitations in the types of instruments selected. As noted by Houser (2009), “An important consideration in assessing the objectivityand quality of a dependent measure is the extent to which the procedures and scoring methods are standardized” (p. 167). A test is not highlyreliable or valid simply because it is standardized. Whenever evaluating instruments, it is important to assess the test’s reliability and validity aswell as its use in various populations. It is also important to assess whether the tests selected were consistent with the sample selected, theresearch problem, and the hypotheses. Studies that employ standardized assessments or standardized administration and scoring proceduresgenerally obtain more reliable and valid data collection and results. The following questions, adapted from Houser (2009), can help guide theevaluation of the Instruments section:
· Is there a clear and adequate description of the instrument (data collection measures) used?
· What types of instruments were used in the study (self-report inventories, ratings, etc.)?
· Are the instruments standardized?
· What are some of the potential problems or limitations with the types of instruments used?
· Does the instrument appear to be appropriate for the sample?
· Is the instrument consistent with the hypotheses?
· Is there a discussion about the reliability and validity of the instruments used?
Evaluating the Results Section
The Results section describes the findings of the study. Unlike qualitative studies, which focus more on descriptions about the findings,quantitative studies are usually presented mathematically through statistical analyses.
Two forms of statistics are presented in the Results section: descriptive and inferential. Descriptive statistics summarize the data collected fromthe sample, such as demographic information. Inferential statistics use statistical methods or analyses to test hypotheses. When evaluating theResults section, consider whether appropriate statistics were used and whether those statistics were consistent with the purpose of the studyand hypotheses. The following questions, adapted from Houser (2009), will help:
· What descriptive statistics were used?
· Do the descriptive statistics adequately describe the sample?
· What types of inferential statistics were used?
· Are the inferential statistics used adequate to answer the hypotheses?
· Is the research design consistent with the statistical methods used?
Evaluating the Discussion Section
Whether evaluating a quantitative or qualitative study, the Discussion section summarizes the purpose of the research, as well as the findings’implications for future research and actual practice. Additionally, the Discussion section includes alternative explanations and potentiallimitations of the findings. The following questions, adapted from Houser (2009), can shed light on the Discussion section:
· Did the researchers clearly restate the purpose and hypotheses?
· Did the researchers clearly discuss the implications of the findings and how they relate to theories, other findings, and actual practice?
· Did the researchers provide alternative explanations of the findings?
· Has there been appropriate generalization of significant findings beyond the sample to the population?
· Did the researchers identify potential limitations of the study and the results?
· Did the researchers identify possible directions for future research?