Discussion: Statistical Significance and Meaningfulness 8210

Student 1

      I am going to break down the sentence “given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level.” In order to critically analyze its meaning, the operative phrases to be analyzed include exploratory, traditional levels of significance, null hypothesis, and .10 level.  Now, the first concept mentioned is ‘exploratory in nature’. Exploratory is typically a purpose of qualitative research and if this is the case, the data obtained from a qualitative study should not be quantified. However, in this case, I believe the exploratory in nature means that the research is still exploring possibilities and is exploring options for a more formal hypothesis. 

In many cases, the best way to test a hypothesis is to state the null hypothesis which means there is no relationship or no difference (Frankfort-Nachmias, Leo-Guerrero, & Davis, 2020). So, what the statement is saying is that they used a traditional level of significance to confirm the hypothesis and reject the null hypothesis. Relaxing traditional levels of significance to the .10 level means lowering the likelihood of rejecting the null hypothesis (Warner, 2020). The .10 is an alpha level use to test the level of significance of the data. What I would tell the authors about the footnote is that although most statistics apply a .5 level which represents a low risk of a type 1 error (incorrectly rejecting the null), it is common to see researchers use the .10 in exploratory research (Warner, 2020).

References:

Frankfort-Nachmias, C., Leon-Guerrero, A., & Davis, G. (2020). Social statistics for a diverse society (9th ed.). Thousand Oaks, CA: Sage Publications.

Warner, R. M. (2012). Applied statistics from bivariate through multivariate techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.

Student 2

Scenario: A research paper claims a meaningful contribution to the literature based on finding statistically significant relationships between predictor and response variables. In the footnotes, you see the following statement, "given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level."

The scenario states, "given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level". Exploratory research is implemented during an investigation when defining an existing problem without a provided conclusion/results. This statement alone speaks to the type of research conducted and how it is being carried out thus far. When referring to the traditional level of significance, it means that a hypothesis has been formulated and the "p-value" is less than or equal to a considered significance. The significance level is crucial because it shows the probability in which the null hypothesis could be rejected when it is true. According to Frankforth-Nachmias & Leon-Guerrero (2020), a null hypothesis is a statement that reflects "no difference which contradicts the research hypothesis and is always expressed in terms of population parameters; When the null hypothesis is rejected, it strengthens the belief in the research hypothesis. It increases the researcher's confidence. When testing a null hypothesis, first, the researcher makes a "guess" about a specific value, then there is a random selection. The researcher then compares the mean of the observed sample (Warner, 2012). 

As a reader/reviewer, the response I would provide to the authors about this footnote is that within the exploratory research, the traditional level of significance is the probability in which the research guess may be true despite the possibility of being rejected. If the null hypothesis has become relaxed to a .10 level, that means that the confidence of the null hypothesis being rejected has decreased. This reflects that the research is progressing in confidence, and there is a conclusion of the correct results.

Resources

Frankfort-Nachmias, C., Leon-Guerrero, A., & Davis, G. (2020). Social statistics for a diverse society (9th ed.). Thousand Oaks, CA: Sage Publications.

Chapter 8, "Testing Hypothesis: Assumptions of Statistical Hypothesis Testing" (pp. 241-242)

Wartner, R.M. (2012). Applied Statistics from bivariate through multivariate techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.

Chapter 3, "Statistical Significance Testing" (pp. 81-124)

Student 3

The statistical significance and meaningfulness are essential and vital to evaluating statistics because the statistical significance focuses on a statistic's value. In contrast, the meaningfulness factors applicability in the real world (Laureate Education (Producer), 2016f). A researcher needs to align the statistics with the claim and justifying using the data (Laureate Education (Producer), 2016f).  

With the statement "traditional levels of significance to reject the null hypotheses were relaxed to the .10 level," it would signify that the predictability or P-value is less strict.  Dr. Matt Jones stated in the video Meaningfulness vs. Statistical Significance that just because you have a P-value below 0.01, it doesn't mean the effect is significant, the difference may be small between two groups (Laureate Education (Producer), 2016f). Therefore, a level of .10 level may allow for additional probabilities but it may not be a very large significance.   

In this specific case, because they indicate the research is exploratory, by relaxing the level to .10 they allow additional probabilities, analysis, or data with continued literature. By reporting this information, the authors are also allowing the reader to determine if data is usable or applicable for their research or study. It is an ethical thing to report.

 Laureate Education (Producer). (2016). Meaningfulness vs. statistical significance [Video file]. Baltimore, MD: Author.

Student 4

Scenario:  A research paper claims a meaningful contribution to the literature based on finding statistically significant relationships between predictor and response variables. In the footnotes, you see the following statement, "given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level."

         In VandenBos , 2007, defines exploratory research a study that is conducted when not much is known about a particular phenomenon. In exploratory research, one typically seeks to identify multiple possible links between variables.  The null hypothesis contradicts the research hypothesis and states that there is no difference between the population mean and some specified value. It is also referred to as the hypothesis of "no difference,"  In hypothesis testing, we hope to reject the null hypothesis to provide indirect support for the research hypothesis. Rejection of the null hypothesis will strengthen our belief in the research hypothesis and increase our confidence in the importance and utility of the broader theory from the research hypothesis was derived. (Frankfort-Nachmias, Leon-Guerrero & Daivs 2020). 

         Sharma, 2017, states that most often level of significance of 5% is chosen as a standard practice. However, levels like 1% and 10% can also be chosen e.g. if our p-value is 0.7, we say that our results are insignificant at 5% level (and we should accept our null hypothesis at this level) and are significant at 10% level (and we should reject our null hypothesis at this level). Following the example set by Sir Ronald Fisher, most users of statistics assume than a value of .05 represents an acceptably small risk of Type I error in most situations. However, in exploratory research, investigators are sometimes willing to use alpha levels (such as  a = .10) that correspond to a higher risk of Type I error. (Warner, 2012)  I would say to the authors, that there is a 10% chance of finding relationships between predictor and response given that the null hypothesis is true.

Frankfort-Nachmias, c,. Leon-Guerrero, A., & Davis, G. (2020). Social statistics for a diverse society (9th ed.) Thousand Oaks, CA: Sage Publications

Sharma, S. (2017). p-value and level of significance explained. Data Science Central, [blog] Retrieved from https://www.datasciencecentral.cpm/profiles/blogs/p-value-and-level-of-significance-explained

VandenBos, G. R. (Ed.) (2007). APA dictionary of psychology

Warner, R. M. (2012). Applied statistics from bivariate through multivariate techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.