Unit VIII Essay

RCH 8301, Quantitative Research Methods 1

Course Learning Outcomes for Unit VIII Upon completion of this unit, students should be able to:

3. Explain the dimensions of research validity. 3.1 Examine the differences between internal and external validity.

4. Discriminate between components of internal and external validity.

4.1 Describe the threats to internal and external validity.

Course/Unit Learning Outcomes

Learning Activity

3.1, 4.1

Unit Lesson Chapter 23, pp. 417–431 Chapter 24, pp. 433–442 Unit VIII Essay

Required Unit Resources Chapter 23: Evaluating Research Validity: Part I, pp. 417–431 Chapter 24: Evaluating Research Validity: Part II, pp. 433–442

Unit Lesson

Evaluating Research Validity

In this final unit, we will be reviewing many of the terms and concepts from previous chapters since our goal will be to learn how to evaluate the quality of the design and analysis in a quantitative research study. The quality of a research project may vary considerably. Since all research connections in any research or related documents are disseminated in the field, there is a need to ensure that the design, methodology, findings, and quality of the general content are standardized. It implies that the variations in the research should be minimized through providing valid answers to the questions developed in the study. The validity in this context includes construct, internal, and external validity. The validity of the research is important, but the aspect of reliability bears equal measure. For the case of reliability, it is imperative that the constituency of the measure is achieved over time, across items, and across various researchers. However, for this unit, the focus is on the validity of the research and, specifically, the framework for the evaluation of the research (e.g., the Cook and Campbell framework) (Gliner et al., 2017). In the evaluation of the research validity, the variables and their measurement levels must be taken into consideration. Some of the questions that must be asked and answered involve appreciating the key independent, antecedent, or predictor variables and the key dependents or outcome variables as well as their level of measurement. Using the Cook and Campbell framework, the evaluation of external research validity considers the following argument: A charge of neglecting external validity can be made against a researcher who has invented construct validity. Based on the aforementioned argument, it is evident that the threats to the external validity, also known as the interaction effects, involve the variables in the research (both X and the other variables). The Cook and Campbell framework has been widely used in the research design for quasi-experimental research where concerns about internal validity and the treatment of the samples or groups are difficult to compare at the

UNIT VIII STUDY GUIDE

Evaluating Research Validity

RCH 8301, Quantitative Research Methods 2

UNIT x STUDY GUIDE

Title

baseline. The complications in the validity of the quasi-experimental research are compounded by the random assignment, which is where each participant in the study has an equal chance to be assigned to the comparison or intervention group (Campbell & Stanley, 1963). As a result, Cook and Campbell published their work on quasi-experimental research; the primary messages in the text are to ensure that any form of bias is minimized, generalizability is maximized, and the use of randomized designs consider the use of clusters instead of the individual. The Cook and Campbell framework, therefore, call for the estimation of validity in the research when a deductive process must be used. The deductive approach requires that the researcher systematically think through all of the internal and external validity threats and thereby minimize their influence on the data. In the cases where these threats can be eliminated, it is important that the confident conclusions on any possible relationship be said to be causal. Otherwise, in the case of quasi-experimental research, where there are high chances of greater threats to the internal validity of the research, the validity can be achieved through conducting randomized trials. Furthermore, in the deduction process, the aspects that must be considered include the main research questions (hypotheses), the research approach, general classification of the approach (randomized or quasi-experimental), and the specific experimental design name for the case of quasi- or randomized experimental research. Overall, the evaluation of the research validity considers various dimensions. They include the overall measurement of reliability and statistics, which is where the overall rating of the measurement reliability and that of the statistics used in the study are established. Also, for the internal validity, the researcher must evaluate and establish the equivalence of the groups based on the characteristics of each participant as well as establish the control of extraneous experience and those of the environment variables. Thus, upon selecting the independent and dependent variables in the research, stating the research question, and identifying the research approach, it is easier to rate the threats to the internal and external validity and thereby rate the measurement of the validity and reliability of the research. Moreover, the other major steps that must be appreciated in the measurement of the research validity include the systematic measurement of the validity and reliability of each variable and the measurement of the validity of the constructs. Researchers must also consider the reliability of the statics (i.e., instruments and measures, statistical techniques, statistical results, and interpretation of the analysis) used in the study and thereby ascertain the validity of the whole study. The rating for the validity answers questions on whether the group participants were randomly assigned, whether there were attempts made to check the similarity of key variables, and the level of retention during the study and across the groups. Moreover, the coefficients below provide the guideline on the evaluation of measurement of validity.

The overall measurement of the validity of constructs is also important in the research, which is where the evaluation of the validity of the intervention is considered. Cronbach and Meehl (1955) stated, “construct validation was introduced in order to specify types of research required in developing tests for which the conventional views on validation are inappropriate” (p. 299). Thus, in the measuring of the construct validity, the objective is to evaluate the degree to which the research and its claims are appropriate while subsuming the other types of evidence in the validity. The

researcher must appreciate the threats to the construct validity such as guessing the hypothesis, having biases in the experimental design, having confounding variables, and having a narrow definition of the predicted outcome; the expectations of the researcher can also be a threat. As a result, the evaluation of the construct validity must be done so that there is an examination of the correlation of measures regarding known variables or those variables that are related to the construct. Otherwise, the construct validity in a research study should be treated as a judgment based on the accumulation of correlations from various studies that have used the instrument under investigation. In this regard, pilot studies usually help in the testing of construct validity prior to the main research.

(Gliner et al., 2017, p.425)

RCH 8301, Quantitative Research Methods 3

UNIT x STUDY GUIDE

Title

Peer Review

Credible research includes peer-reviewed journal articles. Peer review means that the work was reviewed for accuracy by experts in the respective field of study. Peer review is an important part of the quality control of scholarly writing and ensures that there are no flaws in the design or methodology of the research. A peer reviewer also makes certain that the work has been reported correctly and that the results have been interpreted and analyzed correctly. A peer reviewer makes certain that the work is readable and written at a high academic level, that the work is truthful and accurate, and that all citations and references are relevant. A journal article that has been peer reviewed and published is considered credible and, therefore, improves the reliability of the research.

Conclusively, the evaluation of research validity and reliability is not an individual researcher’s task but, rather, is a joint of effort. In the peer-review process, the author’s work is subjected to scrutiny by other experts in the same field to help evaluate and check the validity and reliability of the research findings and its suitability for publication. Based on the approval of the peer reviewers, the researcher makes a judgment on whether the findings and the overall content is suitable to contribute to his or her research topic.

References

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Houghton-Mifflin.

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings.

Houghton-Mifflin. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4),

281–302. https://psycnet.apa.org/record/1956-03730-001 Gliner, J. A., Morgan, G. A., & Leech, N. L. (2017). Research methods in applied settings: An integrated

approach to design and analysis (3rd ed.). Routledge. Vaeenma. (n.d.). Standard model of the research process (ID 85620966) [Photograph]. Dreamstime.

https://www.dreamstime.com/stock-photo-standard-model-research-process-presenting-diagram- image85620966

Learning Activities (Nongraded) Nongraded Learning Activities are provided to aid students in their course of study. You do not have to submit them. If you have questions, contact your instructor for further guidance and information. Review the “Interpretation Questions” and “Application Problems” at the end of Chapters 23 and 24.

Vaeenma. (n.d.)