VI Powerpoint

RCH 5301, Research Design and Methods 1

Course Learning Outcomes for Unit VI At the end of this unit, you should be able to:

1. Integrate philosophical assumptions and worldviews held by researchers. 1.1 Appraise philosophical assumptions, research worldviews, research methodologies,

research traditions, and research designs.

2. Discriminate among the characteristics of qualitative and quantitative research methods. 2.1 Deduce differences between a quantitative methodological research strategy and

qualitative research strategy. 2.3 Compare and contrast descriptive statistics and inferential statistics.

3. Evaluate research designs.

3.1 Differentiate between descriptive, causal, and explanatory research designs based on their unique characteristics.

4. Apply research methods within a research design.

4.1 the relationship of population, sampling frame, sampling design, and generalization. 4.3 Illustrate the use of hypothesis testing for determining statistical significance.

Required Unit Resources Chapter 7: Research Questions and Hypotheses (ULO 4.3) Chapter 8: Quantitative Methods (ULO 4.3) Read the following sections from this chapter:

• Interpreting Results and Writing a Discussion Section • Components of an Experimental Study Method Plan • Participants • Variables • Instrumentation and Materials • Experimental Procedures • Threats to Validity • The Procedure • Data Analysis • Preregistering the Study Plan • Interpreting Results and Writing a Discussion Section

Unit Lesson Lesson: Using Statistical Significance to Evaluate Data Analysis Results (ULO 4.3)

Statistical Significance and Hypothesis Testing Hypotheses are a best guess about the researcher’s phenomenon of interest, and they serve as a starting point for testing the guess. The null hypothesis is the assumption that no significant relationship or difference exists between the variables being tested. The null hypothesis is designated by Ho. The alternative hypothesis is the assumption that a significant relationship or difference exists between variables being tested. The

UNIT VI STUDY GUIDE Quantitative Data Analysis and Inferential Statistics

RCH 5301, Research Design and Methods 2

UNIT x STUDY GUIDE Title

alternative hypothesis is designated by Ha. The null and alternative hypotheses are mutually exclusive, meaning that after statistical results are produced, one will be accepted and the other rejected. It is useful to think of hypothesis testing as serving one of two purposes: a) testing for statistically significant relationships between variables, or b) testing for statistically significant differences between group means. Correlational research methods, such as correlation and regression, look for statistically significant relationships between variables. Causal-comparative methods, such as t-test and ANOVA, look for statistically significant differences between group means. To prevent confusion when conducting hypothesis testing, it is advisable to state hypotheses in a specific and standard format. Specificity relates to precisely defining the variables to be measured and tested. Standardization pertains to using the same verbiage when stating the null and alternative hypotheses. In the examples below, the highlighted verbiage is standard language that can be used for most hypotheses. Relationship Between Variables Example One – Statistical test looking for a relationship between variables. Ho1 There is no statistically significant relationship between x and y. Ha1 There is a statistically significant relationship between x and y. Differences Between Groups Example Two – Statistical test looking for a difference between group means. Ho2: There is no statistically significant difference in means between Group A and Group B. Ha2: There is a statistically significant difference in means between Group A and Group B.

Hypothesis Testing Steps Much of the content in this course was created to help students understand the practical use of research methods. Nowhere is this more obvious than the use of statistical analysis. It is more important to understand how to interpret the statistics to aid decision-making than it is to be able to explain all the math behind the results. That is the domain of academics and statisticians. The following steps will highlight the relative simplicity of hypothesis testing. Hypothesis testing follows a logical progression regardless of whether the researcher is testing for relationships (correlation) or testing for differences (causal-comparative). According to Cooper and Schindler (2014) that progression includes the following steps:

1. State the null and alternative hypotheses. 2. Choose a statistical procedure that will be used to test the hypotheses (e.g., correlation, regression, t-

test, ANOVA, etc.) 3. Select the desired alpha (α), or significance level, that represents the risk of a Type I error. A .05 α is

customarily used in research. 4. Use the chosen statistical procedure to test the data, which produces a p value. This is accomplished

using a statistical application. 5. Interpret the test results. If the p value < .05 α, reject the null hypothesis and accept the alternative

hypothesis. If the p value ≥ .05 α, accept the null hypothesis and reject the alternative hypothesis.

Statistical Significance and Importance Are significant results important? Are non-significant results useless? Many results can be statistically significant but offer little practical importance. Bell et al. (2022) provide a clear example of how statistically significant results can have no practical significance. As they explained, a significant r of .10 equating to R squared of 1% is essentially useless for decision-making.

RCH 5301, Research Design and Methods 3

UNIT x STUDY GUIDE Title

Conversely, non-significant results can offer the practitioner valuable insights about the domain of interest. Assume a global company’s safety director wants to assess the effectiveness of a new safety training program. Two different groups of employees; those having taken the training and those who have not, are being tested for significant differences in recordable work-related injuries. The hypotheses are:

• Ho: There is no statistically significant difference in recordable work-related injuries between Group A and Group B.

• Ha: There is a statistically significant difference in recordable work-related injuries between Group A and Group B.

After an independent samples t-test is conducted, the results indicate a p value 17 > .05 α, therefore, the result is non-significant, and the null hypothesis is accepted (and the alternative hypothesis is rejected) that there is no statistically significant difference in recordable work-related injuries between Group A and Group B. Although the conclusion to be drawn from the results of the hypothesis test is that the training is not effective, this finding has immense practical value. The company may save thousands, hundreds of thousands, or millions of dollars by not implementing the new, but ineffective, safety training program. So, even though the results are not statistically significant, they offer great practical significance by providing invaluable insights for decision-making.

Statistical Significance and Hypothesis Testing Summary

• Hypotheses either test for relationships between variables or differences between group means. • Hypotheses should be specific and use standardized verbiage. • Hypothesis testing follows a logical progression of steps. • Significant results do not necessarily mean they offer practical importance. • Non-significant results do not necessarily mean they lack practical importance. • Significant results do not provide definitive proof the alternative hypothesis is true since a Type I error

is always possible. • Non-significant results do not provide definitive proof the null hypothesis is true since a Type II error is

always possible.

The Lady Tasting Tea Experiment and the Origin of .05 α As mentioned previously, an alpha of .05 is customarily used in research as the cut-off for statistical significance, and many other applications of hypothesis testing. There is an interesting story of the origin of the .05 α. It goes like this. Once upon a time in Cambridge, England, on a lovely summer day during the 1920s, a group of academics and their friends were enjoying afternoon tea. One of the women present made a claim to the astonished group that tea tasted differently depending on whether the tea was poured first, or the milk was poured first, and that she could blindly differentiate cups by taste alone. To say the academics were skeptical would be an understatement. One of the academics in attendance, Dr. Ronald Fisher, decided to give her the benefit of the doubt by constructing an ad-hoc, blind taste test, an experiment to test the hypothesis that the lady could differentiate cups by taste (Salsburg, 2001). The hypotheses were:

• Ho: There is no statistically significant difference in the lady’s ability to distinguish cups by taste vs. random selection.

• Ha: There is a statistically significant difference in the lady’s ability to distinguish cups by taste vs. random selection.

For the purposes of revealing the nature of statistical significance and its relation to probability for hypothesis testing, assume the following three hypothesis tests were administered to the lady. Test 1 In the first test, two choices were available: a) cup 1, tea poured first, and b) cup 2, milk poured first. If the lady was able to identify either cup 1 or cup 2 by taste alone, nobody would be impressed since she had a 1

RCH 5301, Research Design and Methods 4

UNIT x STUDY GUIDE Title

out of 2 chance of success, or 50% probability (see graphic), of making a correct selection by simply guessing. Based on the results of this test, Ha would have been accepted that the lady could distinguish cups; however, this test would have had a very low threshold for statistical significance and a very high chance for a Type I error with what is effectively a .50 α. Test 2 A second test involved four cups: a) cup 1, tea poured first, b) cup 2, tea poured first, c) cup 3, milk poured first, and d) cup 4 milk poured first. A combination calculator can be used to determine that these four cups result in 6 2-cup combinations from which the lady could have chosen. Combination 1 = Cup 1, Cup 2 Combination 2 = Cup 3, Cup 4 Combination 3 = Cup 1, Cup 3 Combination 4 = Cup 1, Cup 4 Combination 5 = Cup 2, Cup 3 Combination 6 = Cup 2, Cup 4 If the lady had successfully identified the 2 cups with tea added first (Combination 1) or the 2 cups with milk added first (Combination 2), it might have generated some applause, but it would be unlikely most observers would have been overly impressed since she had a 1 out of 6 chance, or 16.7% probability (see graphic), of selecting the correct combination by simply guessing. Once again, based on the results of this test, Ha would have been accepted that the lady could distinguish cups based on taste alone. This test would have had a moderate threshold for statistical significance and moderate chance of a Type I error with what is effectively a .167α. Test 3 This final test scenario uses 6 cups, with 3 cups having tea added first, and 3 cups having milk added first. Again, using a combination calculator, these 6 cups result in 20 3-cup combinations from which the lady could have chosen. If the lady had successfully identified the 3 cups with tea added first, or the 3 cups with milk added first, that would have certainly elicited hoots and hollers. Observers would truly be impressed since she had only a 1 out of 20 chance, or 5% probability, of selecting the correct permutation by simply guessing. Based on the results of this test, Ha would have been accepted that the lady could distinguish cups based on taste alone. This is a high threshold for statistical significance and a very low chance of a Type I error with what is effectively a .05 α.

(Sustainability Methods, 2021)

Several years after the tea-tasting experiment, in 1929, Fisher wrote The Statistical Method in Psychical Research in which he recommended p value < .05 α as an arbitrary, but convenient, cut-off for evaluating statistical significance. He described a statistically significant event as one that would rarely be observed in the absence of an independent explanatory variable(s). Rarity, from Fisher’s perspective, would be an event

RCH 5301, Research Design and Methods 5

UNIT x STUDY GUIDE Title

that occurs randomly not more often than 1 in 20 tries, which translates into rejecting the null hypothesis and accepting the alternative hypothesis when a statistical test returns a p value < .05 α. Legend holds that the lady tasting tea was indeed able to successfully select three like cups. The question remains as to whether this was one of those rare statistically significant events that was observed in the absence of an explanatory variable, or if the test measured a true effect (Fisher, 1929; Lewis & Sauro, 2021). Fisher was a mathematical genius who dedicated his life to science and math and was famous for his contribution to the field of statistics and the many statistical procedures he created. Although Fisher’s writings have been pored over, there is no indication if the Lady Tasting Tea experiment was the creative spark for codifying .05 α as the arbitrary and convenient threshold for statistical significance, or if the Lady Tasting Tea experiment was simply an opportunity to promote Fisher’s preconceived desire to see the scientific community begin to adopt .05 α as the threshold for statistical significance. Regardless, .05 α has become the standard, and the Lady Tasting Tea experiment has become legendary in statistical history.

Additional Inferential Tests of Differences Sometimes, a researcher will find that One-Way ANOVA is limited in the number and types of variables that can be included in the analysis. There are, fortunately, additional causal-comparative procedures that provide flexibility to meet the needs of any combination or quantity of variables. The following will compare these additional procedures with One-Way ANOVA. They include Two-Way ANOVA, ANCOVA, One-Way MANOVA, Two-Way MANOVA, and MANCOVA.

Terminology Terminology is once again important to understand the content in this section. The following will describe the three main variable types used in the additional causal-comparative tests to be discussed below. These variables are named response variable, factor, and covariate. Response Variable When using ANOVA, ANCOVA, One-Way MANOVA, Two-Way MANOVA, and MANCOVA, the dependent variable (DV) is referred to as a response variable. A response variable is a continuous variable, which means it is measured on at least an interval or ratio scale level. Examples of response variables include income, time, height, length, depth, weight, growth, temperature, moisture, humidity, speed, distance, frequency, antibodies, grades, etc. Factor A factor is an independent variable (IV) that is sometimes referred to as the grouping variable. The factor is a categorical variable, which means it is measured on a nominal scale and specifies group membership. Examples of factors include socio-economic status, race, ethnicity, religion, education level, industry, business type, department, work shift, work team, batch, lot, production line, athletic team, gate, turnstile, country, state, county, city, neighborhood, society, culture, treatment, eye color, hair color, etc. Covariate A covariate is a continuous independent variable, which means it is measured on at least an interval or ratio scale level.

One-Way ANOVA (Analysis of Variance) Like the t-test, ANOVA is a powerful causal-comparative inferential statistical procedure for analyzing differences between means; however, it has an advantage over the bivariate t-test in that it can test for differences in more than two means. For example, a researcher may be interested in determining if statistically significant differences exist between average income (response variable) depending on degree held (IV factor). More specifically, average income could be compared among the holders of the following degrees: high school diploma, associate’s degree, bachelor’s degree, master’s degree, and terminal degree. In this example, five income means would be compared for differences.

RCH 5301, Research Design and Methods 6

UNIT x STUDY GUIDE Title

This example used one response variable and one IV factor, which is the reason for the name One-Way ANOVA. Now, suppose the researcher is interested in analyzing the mean differences for a single response variable, but for two IV factors. Adding two or more factors requires the use of Two-Way ANOVA.

Two-Way ANOVA (Analysis of Variance) Two-Way ANOVA is used when a researcher is interested in determining if statistically significant differences exist in means for a single response variable, but for two factors (Field, 2005). Expanding on the above example, the researcher could compare average income (response variable) depending on degree held (Factor1) and region (Factor2). At a granular level, the Two-Way ANOVA would compare average income for a) the holders of the following degrees: high school diploma, associate’s degree, bachelor’s degree, master’s degree, and terminal degree, and b) region, including Northeast, Southeast, Mid-Atlantic, Midwest, Southwest, Northwest, and West. It should be noted that there can be more than two factors used in this procedure. Whenever more than two factors are used in ANOVA, with one response variable, it is referred to as factorial ANOVA.

ANCOVA (Analysis of Covariance) ANCOVA is similar to ANOVA in that there is only a single response variable, which is continuous. However, ANCOVA uses two IVs: a) one that is a categorical factor, and b) one that is a continuous variable. Therefore, the C in ANCOVA represents the test of covariance between the response variable and the continuous variable. The continuous IV is called a covariate (Field, 2005). In the original example, a researcher was interested in determining if statistically significant differences exist between average income (response variable) depending on degree held (Factor1), such as high school diploma, associate degree, bachelor’s degree, master’s degree, and terminal degree. Using ANCOVA, the researcher now includes the covariate GPA (Covariate1).

RCH 5301, Research Design and Methods 7

UNIT x STUDY GUIDE Title

One-Way MANOVA (Multivariate Analysis of Variance) Until this point, only one DV continuous response variable was used in the ANOVA and ANCOVA examples. This changes with MANOVA and the addition of the M, which represents multivariate. One-way MANOVA involves two continuous response variables and one categorical factor (Field, 2005). For example, a researcher is interested in determining if statistically significant differences exist between: a) average income (Response1), and b) average hours worked each week (Response2) depending on degree held (Factor1), such as high school diploma, associate’s degree, bachelor’s degree, master’s degree, and terminal degree.

Two-Way MANOVA (Multivariate Analysis of Variance) Like ANOVA, MANOVA can take a two-way approach, which includes two continuous response variables and two IV categorical factors (Field, 2005). For example, the researcher in this example is interested in determining if statistically significant differences exist between: a) average income (Response1), and b) average hours worked each week (Response2) depending on: c) degree (Factor1), such as high school diploma, and AA Degree, Bachelor’s Degree, Master’s Degree, and Terminal Degree, and d) region (Factor2), including Northeast, Southeast, Mid-Atlantic, Midwest, Southwest, Northwest, and West. It should be noted that there can be more than two categorical factors used in this procedure, and more than two continuous response variables. Whenever more than two factors or more than two response variables are used in MANOVA, it is referred to as factorial MANOVA.

RCH 5301, Research Design and Methods 8

UNIT x STUDY GUIDE Title

MANCOVA (Multivariate Analysis of Covariance) MANCOVA is similar to MANOVA with the addition of the C, representing covariance. MANCOVA and MANOVA both include two or more continuous response variables, but MANCOVA includes at least one categorical factor and at least one continuous variable, which is the covariate (Field, 2005). In this example of MANCOVA, the researcher is interested in determining if statistically significant differences exist between: a) average income (Response1), and b) average hours worked each week (Response2) depending on: c) degree (Factor1 categorical variable) such as high school diploma, associate’s degree, bachelor’s degree, master’s degree, and terminal degree, and d) hours of sleep each week (Covariate1 continuous variable).

Conclusion Many students will never apply the statistical tests discussed in this course, but most will be exposed to information that includes the results of these types of procedures, like regression and ANOVA, throughout their careers. That is why it is important to have a good foundation for interpreting basic statistical results. Statistical results are also often published in daily news reports on a variety of topics, so knowledge of interpretation is helpful even outside of one’s interests. While it is not necessary for most students to memorize the purpose and particulars of procedures like ANCOVA, MANOVA, and MANCOVA, it is important that students are aware of the robustness of causal-comparative tests. If a need arises in this area, a good research website or textbook can help anyone determine the appropriate test for their needs.

RCH 5301, Research Design and Methods 9

UNIT x STUDY GUIDE Title

References Bell, E., Bryman, A., & Harley, B. (2022). Business research methods (6th ed.). Oxford University Press.

https://online.vitalsource.com/#/books/9780192640505 Cooper, D. R., & Schindler, P. S. (2014). Business research methods (12th ed.). McGraw-Hill. Creswell, J. W., & Creswell, J. D. (2022). Research design: Qualitative, quantitative, and mixed methods

approaches (6th ed.). SAGE. https://online.vitalsource.com/#/books/9781071817964 Field, A. (2005). Discovering stats using SPSS (2nd ed.). SAGE. Fisher, R. A. (1929). The statistical method in psychical research. Proceedings of the Society for Psychical

Research, 39, 189–192 (1929). Lewis, J., & Sauro, J. (2021). For statistical significance, must p be < .05? Measuring U.

https://measuringu.com/setting-alpha/ Salsburg, D. (2001). The lady tasting tea: How statistics revolutionized science in the twentieth century. Henry

Holt and Company. Sustainability Methods. (2021, September 20). Designing studies [Image].

https://sustainabilitymethods.org/index.php/Designing_studies