POLI 205

Boomer18

Tokunaga_Ch10LectureNotes.pdf

Home >Political Science homework help >POLI 205

4/5/2018

Chapter 10: Errors in Hypothesis

Testing, Statistical Power, and Effect

Size

Decision‐Making Errors • A review

– Within hypothesis testing, we rely on probability to decide whether or not to reject the null hypothesis

– Because the decision is based on probability, not certainty, it’s possible that the decision about the null hypothesis may be correct or incorrect

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

4/5/2018

Decision‐Making Errors

• A research scenario – Polygraph tests attempt to measure psychophysiological responses to relevant and irrelevant questions about a crime

– 68 suspects interrogated with polygraph, • 30 later confessed to the crime

• How accurate was the polygraph?

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Decision‐Making Errors

Polygraph decision

Not guilty Guilty

Suspect actuality

Innocent

Correct decision

Incorrect decision

Guilty

Incorrect decision

Correct decision

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

4/5/2018

Decision‐Making Errors: Elaad et al. (1992)

Polygraph decision

Not guilty Guilty

Suspect actuality

Innocent

Guilty

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Two Errors in Hypothesis Testing: Type I and Type II Error

• The previous example illustrates the errors that can be made in hypothesis testing

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Decision about H0 Do not reject Reject

Reality in Population

H0 is true

Correct decision

Type I error

H1 is true

Type II error

Correct decision

4/5/2018

Two Errors in Hypothesis Testing: Type I and Type II Error

• Type I error – The null hypothesis is true, but we reject it and conclude that an effect exists.

– Probability of making a Type I error • p(Type I error) =  •  = .05, so a 5% chance of concluding an effect exists when it doesn’t

• p(not making Type I error) = 1 ‐  • 95% chance of not making a Type I error

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Two Errors in Hypothesis Testing: Type I and Type II Error

• Why is Type I error a concern? – Saying an effect exists when it doesn’t communicates false information

• Imagine a drug‐trial in which there was no improvement in depression, but the researchers claimed there was.

• What would the consequences be?

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

4/5/2018

Two Errors in Hypothesis Testing: Type I and Type II Error

• Why does Type I error occur? – These errors occur due to chance factors and fluctuations in sampling

– The only way to completely avoid Type I errors is to never reject the null hypothesis regardless of the evidence against it

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Two Errors in Hypothesis Testing: Type I and Type II Error

• Type II error – The null hypothesis is false, but we fail to reject it and conclude that no effect exists

– Probability of making a Type II error • p(Type II error) =  •  is difficult to determine, be cause we frequently don’t know true population values

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

4/5/2018

Two Errors in Hypothesis Testing: Type I and Type II Error

– Probability of not making a Type II error

• p(not making a Type II error) = 1 ‐  • Statistical power = 1 ‐  • Power is the ability to detect an effect when it does exist in the population

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Two Errors in Hypothesis Testing: Type I and Type II Error

• Why is Type II error a concern? – It may result in a non‐communication of correct information

• If a researcher believes no effect exists, she may be less likely to attempt to publish the research

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

4/5/2018

Two Errors in Hypothesis Testing: Type I and Type II Error

• Why does Type II error occur? – Chance fluctuations

– Small effects that are difficult to detect

– Research design issues

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Type I and Type II Error: A Summary

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Decision about H0

Do not reject Reject

Reality in Population

H0 is true

Correct decision p = 1 ‐ 

Type I error p = 

H1 is true

Type II error p = 

Correct decision p = 1 - 

Statistical power

4/5/2018

Controlling Type I and Type II Error

• Controlling Type I error: Make it more difficult to reject H0 – Lower  (i.e., .05 to .01)

– However, a more stringent  also increases the probability of making a Type II error

• The more difficult we make it to reject H0, the more likely we are to miss effects that actually exist.

Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

Controlling Type I and Type II Error

• Controlling Type II error: Make it more likely to reject H0 – Increase sample size

• Larger sample size yields lower critical values

– Raise  (i.e., .05 to .10) • However, this increases p(Type I error)

– Use directional alternative hypothesis (H1)

– Increase between‐group variability • Maximize differences between groups by using effective experimental manipulations

– Decrease within‐group variability Howard T. Tokunaga, Fundamental Statistics for the Social and Behavioral Sciences © SAGE Publications, 2016

4/5/2018

Measures of Effect Size

• Influence of sample size on inferential statistics – Sample size influences the decision to reject H0, and affects the conclusions we draw

– The larger the sample size, the more likely we are to reject H0, even if the means of the samples are the same

– We want large sample sizes to be confident in our conclusions, but it would be helpful to have a gauge not reliant on sample size

Measures of Effect Size

• Effect size as variance accounted for – Variance is the amount of variability in a set of scores for a variable

– Researchers attempt to understand, describe, predict, and explain variance (differences)

4/5/2018

Measures of Effect Size: Difference Between Two Sample Means

• r squared (r2) : percentage of variance in one variable accounted for by another variable

Measures of Effect Size: Difference Between Two Sample Means

4/5/2018

Measures of Effect Size: Difference Between Two Sample Means

• Interpreting measures of effect size – Cohen (1988)

• “Small" effect produces r2 of .01

• “Medium" effect produces r2 of .06

• “Large" effect produces r2 of .15 or greater

• Presented with results of inferential statistics as a supplemental piece of information

Measures of Effect Size: Difference Between Two Sample Means

• Why calculate measure of effect size? – Unaffected by sample size

– Range from 0.00 to 1.00, and allow for comparisons across samples and studies

– Aids in interpretation of results: • A significant result may be of small ‘practical significance’

• Non‐significant results can be better understood with effect size comparisons

4/5/2018

• Cohen’s d: magnitude of difference between means of two groups measured in standard deviation units

− How different are two group means from one another?

Measures of Effect Size: Difference Between Two Sample Means

4/5/2018

Measures of Effect Size: Difference Between Two Sample Means

• Interpreting Cohen’s d – “Small" effect produces Cohen’s dof .20

– “Medium" effect produces Cohen’s dof .50

– “Large" effect produces Cohen’s dof .80 or greater.

Looking Ahead • This chapter discussed decision‐making errors and how to minimize them. Although it is impossible to completely eliminate them, it is important to understand the consequences of making these errors.

• The next chapter discusses a statistical procedure used to compare differences between the means of three or more groups that comprise one independent variable.