Effect Size and Power
Limitless84
EffectSizeandPower_TPE.pptxNONPARAMETRICS, Effect size, and Power
1
Statistical significance...so what?
The limits of hypothesis testing
2
Statistical Significance
Remember this does not mean the same thing as practical significance!
So, how do we know the practical significance?
3
We’ve actually already done this….
Remember “variance” from correlation and regression??
This was a measure of “effect size”
4
How are effect sizes used?
To report practical significance of results
To compare findings within a study
To compare several studies (meta-analysis)
5
Small Effect Size, Important Result
Even small effects can be important for serious problems (rare diseases, cancer treatments, reducing fatalities in surgeries)
The cost of implementing the changes is minimal, so it is worth making practical changes based on a small effect
6
Large Effect Size, Unimportant Result
Costly and impractical changes need to be made
Ethical concerns
7
Interpreting and reporting effect size
Cohen’s d (can be used in different types of tests)
Correlation: r2
Independent and dependent t-tests: eta squared
One-way ANOVA: eta squared
Chi-square: phi and cramer’s v
8
Statistical Power
Power is the probability a study will produce a statistically significant result IF the research hypothesis is true (or the null hypothesis is false).
9
Statistical Power
Can help you determine how large a sample size you need
Another tool for interpreting results that are not significant or those that are significant with little practical importance (small effect size)
A power analysis is done BEFORE you conduct your study
10
Power: It’s Relative!
11
What Determines Power
Predicted effect size
Sample size
Significance level
Type of statistical test conducted
12
Effect Size
The larger the predicted effect size the more power your study has.
Or the higher the probability of rejecting the null hypothesis (if it is in fact false).
Why? Because there is less overlap between the distributions (either due to a large difference in means or b/c each distribution has a small standard deviation)
13
Sample Size
Remember the larger the sample size the smaller the standard deviation of the distribution of means becomes...
So there is less overlap in the populations...
This is not the same as effect size!
14
How to Increase Power
Increase effect size by increasing predicted difference
Increase effect size by decreasing population standard deviation
Increase sample size
Use less extreme level of significance
Use a one-tailed test
15
Copyright © 2011 by Pearson Education, Inc. All rights reserved
Role of Power When a Result is Not Statistically Significant
A nonsignificant result from a study with low power is truly inconclusive.
A nonsignificant result from a study with high power suggests that:
the research hypothesis is false or
there is less of an effect than was predicted when calculating power
16
Copyright © 2011 by Pearson Education, Inc. All rights reserved
Effect Size and Power in Research Articles
Articles often mention effect size.
Effect size is a crucial factor in meta- analyses, and thus is almost always reported in meta-analyses.
Power is sometimes discussed when evaluating nonsignificant results.
17
Effect Size
Young adults who were alcohol dependent at age 21 scored lower at age 18 than those who were not alcohol dependent at age 21 on Traditionalism (d=.49), Harm Avoidance (d=.44), Control (d=.64), and Social Closeness (d=.40), and higher on Aggression (d=.86), Alienation (d=.66) and Stress Reaction (d=.50).
18
Power
Three of the six studies included in the meta-analysis reported nonstatistically significant differences in serious adverse event rates, and concluded that there was no risk, despite having power of less than .37 to detect the reported differences. A high probability of Type II error may lead to erroneous clinical inference resulting in harm. The statistical power for nonsignificant tests should be considered in the interpretation of results.
19
