Week 5 Discussion 3

BUS 308 Week 5 Lecture 3

A Different View: Effect Sizes

Expected Outcomes

After reading this lecture, the student should be familiar with:

1. What effect size measures exist for different statistical tests. 2. How to interpret an effect size measure. 3. How to calculate an effect size measure for different tests.

Overview

While confidence intervals can give us a sense of how much variation is in our decisions, effect size measures help us understand the practical significance of our decision to reject the null hypothesis. Not all statistically significant results are of the same importance in decision making. A difference in means of 25 cents is more important with means around a dollar than with means in the millions of dollars, yet with the right sample size both groups can have this difference be statistically significant.

Effect size measures help us understand the practice importance of our decision to reject the null hypothesis.

Excel has limited functions available for us to use on Effect Size measures. We generally need to take the output from the other functions and generate our Effect Size values.

Effect Sizes

One issue many have with statistical significance is the influence of sample size on the decision to reject the null hypothesis. If the average difference in preference for a soft drink was found to be ½ of 1%; most of us would not expect this to be statistically significant. And, indeed, with typical sample sizes (even up to 100), a statistical test is unlikely to find any significant difference. However, if the sample size were much larger; for example, 100,000; we would suddenly find this miniscule difference to be significant!

Statistical significance is not the same as practical significance. If for example, our sample of 100,000 was 1% more in favor of an expensive product change, would it really be worthwhile making the change? Regardless of how large the sample was, it does not seem reasonable to base a business decision on such a small difference.

Enter the idea of Effect Size. The name is descriptive but at the same time not very illuminating on what this measure does. We will get to specific measures shortly, but for now, let’s look at how an Effect Size measure can help us understand our findings. First, the name: Effect Size. What effect? What size? In very general terms, the effect we are monitoring is the effect that occurs when we change one of the variables. For example, is there an effect on the average compa-ratio when we change from male to female. Certainly, but not all that much, as we found no significant difference between the average male and female compa-ratios. Is there an effect when we change from male to female on the average salary? Definitely. And it is

much larger than what we observed on the compa-ratio means. We found a significant difference in the average salary for males than females – around $14,000.

The Effect Size measures looks at the impact of the variables on our outcomes; large impacts suggest that variables are important, while small impacts might suggest that the variable is not particularly important in determining changes in outcomes. We could, for example, argue that both male and females in the population had the same compa-ratio mean and what we observed in the sample was simply the result of sample error. Certainly, our test results and confidence intervals could support this.

Now, when do we look at an Effect Size; that is, when should we go to the effort of calculating one. The general consensus is that the Effect Size measure only adds value to our analysis if we have already rejected the null hypothesis. This makes sense, if we found no difference between the variables we were looking at, why try to see what effect changing from one to the other would do. We already know, not much.

When we reject a null hypothesis due to a significant test statistic (one having a p-value less than our chosen alpha level), we can ask a question: was this rejection due to the variable interactions or was it due to the sample size? If due to a large sample size, the practical significance of the outcome is very low. It would often not be “smart business” to make a decision based on those kinds of results. If, however, we have evidence that the null was rejected due to a significant interaction by the variables, then it makes more sense to use this information in making decisions.

Therefore, when looking at Effect Sizes, we tend to classify them as large, moderate, or small. Large effects mean that the variable interactions caused the rejection of the null, and our results have practical significance. If we have small effect size measures, it indicates that the rejection of the null was more likely to have been caused by the sample size, and thus the rejection has very little practical significance on daily activities and decisions.

OK, so far:

• Effect sizes are examined only after we reject the null hypothesis, they are meaningless when we do not reject a claim of no difference.

• Large effect size values indicate that variable interactions caused the rejection of the null hypothesis, and indicate a strong practical significance to the rejection decision.

• Small effect size values indicate that the sample size was the most likely cause of rejecting the null, and that the outcome is of very limited practical significance.

• Moderate effect sizes are more difficult to interpret. It is not clear what had more influence on the rejection decision and suggests only moderate practical significance. These results might suggest a new sample and analysis.

Different statistical tests have different effect size measures and interpretations of their values. Here are some that relate to the work we have done in this course.

• T-test for independent samples. Cohen’s D is found by the absolute difference between the means divided by the pooled standard deviation of the entire data set. A large effect is .8 or above, a moderate effect is around .5 to .7, and a small effect is .4 or lower. Interpretation of values between these levels is up to the researcher and/or decision maker.

• One-sample T-test. Cohen’s D is found by the absolute difference between the means divided by the standard deviation of the tested variable data set. A large effect is .8 or above, a moderate effect is around .5 to .7, and a small effect is .4 or lower. Interpretation of values between these levels is up to the researcher and/or decision maker.

• Paired T-test. Effect size r = square root of (t^2/(t^2 + df)). A large effect is .4 or above, a moderate effect is around .25 to .4, and a small effect is .25 or lower.

• ANOVA. Eta squared equals the SS between/SS total. A large effect is .4 or above, a moderate effect is .25 to .40, and a small effect is .25 or lower.

• Chi Square Goodness of Fit tests (1-row actual tables). It is, also called Effect size r = square root (Chi Square statistic/(N * (c -1)), where c equals the number of columns in the table. A large effect is .3 or above, a moderate effect is .3 to .5, and a small effect is .3 or lower.

• Chi Square Contingency Table tests. For a 2x2 table, use phi = square root of (chi square value/N). A large effect is .5 or above, a moderate effect is .3 to .5, and a small effect is .3 or lower.

• Chi Square Contingency Table tests. For larger than a 2x2 table, use Cramer’s V = square root (chi square value/((smaller of R or C)-1)). A large effect is .5 or above, a moderate effect is .3 to .5, and a small effect is .3 or lower.

• Correlation. Use the absolute value of the correlation, A large effect is .4 or above, a moderate effect is .25 to .4, and a small effect is .25 or lower.

Would using these measures change any of our test interpretations?

Summary

Effect size measures change our focus from merely finding differences or associations to interpreting how important these might be in “real world” applications and decisions. While the different tests have different ways of calculating and interpreting their Effect Size value, all share a common theme. “Large” values mean that the differences or associations are caused by the variables and have strong practical importance, while “small” values mean that the findings were more likely caused by large samples and have very little practical significance or importance in “real world” decision-making or actions.

Please ask your instructor if you have any questions about this material.

When you have finished with this lecture, please respond to Discussion Thread 3 for this week with your initial response and responses to others over a couple of days.