Research Method

11/23/2020

1

CHAPTER 11

More on Experiments:

Confounding and Obscuring Variables

Chapter Overview

▪ Threats to internal validity: Did the independent variable really cause the difference?

▪ Interrogating null effects: What if the independent variable does not make a difference?

Review: Experiments

▪ Independent variable (IV): manipulated variable (the “cause”)

▪ Conditions: levels of an independent variable

▪ Dependent variable (DV): measured, outcome variable (the “effect”)

▪ Control variable: any variable that an experimenter holds constant

▪ Different than a comparison group!

11/23/2020

2

The Four Big Validities

Why Experiments Support Causal Claims

▪ Experiments establish covariance

▪ Experiments establish temporal precedence

▪ Well-designed experiments establish internal validity

Well-Designed Experiments Establish Internal Validity

▪ Internal validity (third-variable problem) → Is there a third variable (C) that is associated with variables A and B independently?

▪ Confounds → alternative explanations (third variable (C)) that threaten internal validity

11/23/2020

3

Threats to Internal Validity: Did the Independent Variable Really Cause the Difference?

▪ The really bad experiment → a cautionary tale!

▪ Six potential internal validity threats in one-group, pretest/posttest designs

▪ Three potential internal validity threats in any experiment

▪ With so many threats, are experiments still useful?

The Really Bad Experiment

• Known as one-group pretest/posttest design → there

is one group of participants who are measured on a

pretest, exposed to a treatment/intervention/change and

then measured on a posttest

The Really Bad Experiment: Graphic

11/23/2020

4

Potential Internal Validity Threats

1. Design Confounds

2. Selection Effects

3. Order Effects

4. Maturation threats

5. History threats

6. Regression threats

7. Attrition threats

8. Testing threats

9. Instrumentation threats

10. Observer bias

11. Demand Characteristics

12. Placebo Effect

Maturation Threats

▪ What are maturation threats? → a change in behavior that emerges over time

▪ How are maturation threats prevented?

▪ Have a comparison group!

History Threats

▪ What is a history threat? → results when some external or

“historical” event affects most members of the treatment group

at the SAME time as the treatment

▪ How are history threats prevented?

▪ Look at the change in comparison group!

11/23/2020

5

Regression Threats

▪ What is a regression threat? → a statistical concept in which extremely

high or low performance at Time 1 is

less extreme at Time 2 (i.e., closer to

the average)

▪ How can regression threats be

prevented?

▪ Use comparison groups

▪ Inspect the results

Attrition Threats

▪ What is an attrition threat?

→ reduction in participant

numbers from pretest to

posttest

▪ How can attrition threats be

prevented?

▪ Remove the dropped participant

scores from the pretest (or look

at the pretest scores of the

dropouts)

Testing Threats

• What is a testing threat? → a type of order effect in which

there is a change in participants as a results of experiencing the

DV more than once!

• How can testing threats be prevented?

– 1) using a posttest only design or 2) using alternative forms of the test at

pretest or posttest or 3) using a comparison group

11/23/2020

6

Instrumentation Threats

▪ What is an instrumentation threat? → occurs when a measuring instrument changes over time

▪ How can instrumentation threats be prevented? ▪ 1) use a posttest only design OR

▪ 2) use the same observers for pretest/posttest (or retrain them between) OR

▪ 3) counterbalance the order of the pretest and posttest measurements

Three Potential Internal Validity Threats in Any Study

▪ Observer bias → bias caused by researcher’s expectations influencing how they interpret the results

▪ Demand characteristics → bias that occurs when participants figure out what the research study is about and change their behavior

▪ Controlling for observer bias and demand characteristics (double-blind study, masked study)

▪ Placebo effects → effect is present when people receive a treatment and improve, but only because they believe they are receiving a valid or effective treatment

Placebo Effects

▪ Designing studies to

rule out the placebo

effect

▪ Double-blind placebo

control study

▪ Is that really a placebo

effect?

11/23/2020

7

With So Many Threats, Are Experiments Still Useful?

▪ Valid → reasonable, accurate and justifiable

▪ Validity → the appropriateness of a conclusion or decision

CHAPTER 11

More on Experiments:

Confounding and Obscuring Variables

Chapter Overview

▪ Threats to internal validity: Did the independent variable really cause the difference?

▪ Interrogating null effects: What if the independent variable does not make a difference?

11/23/2020

8

Review: Experiments

▪ Independent variable (IV): manipulated variable (the “cause”)

▪ Conditions: levels of an independent variable

▪ Dependent variable (DV): measured, outcome variable (the “effect”)

▪ Control variable: any variable that an experimenter holds constant

▪ Different than a comparison group!

Review: Experiments with One Independent Variable

Analyzing Causal Claims

▪ t-test → a statistic to test the difference between two group averages ▪ Different types of t-tests depending on the type of claim (association

versus causal) and the experimental design (between groups versus within groups)

▪ Example: Paired t-tests, independent sample t-tests

▪ One-way ANOVA (analysis of variance) → statistic that tests the differences between 3 or more group averages

Look at the difference in means

(averages) between your groups

(the levels of your

independent/manipulated

variable)

11/23/2020

9

W h ic

h s

t a t is

t ic

a l t e s t ?

Interrogating Null Effects: What If the Independent Variable Does Not Make a Difference?

▪ Null effect (null result) → when the independent variable did not affect the dependent variable (there is no covariance between the IV and DV)

▪ Perhaps there is not enough between-groups difference.

▪ Perhaps within-groups variability obscured the group differences.

▪ Perhaps there really is no difference.

Perhaps There Is Not Enough Between-Groups Difference

▪ Ceiling & floor effects

▪ Ceiling effect → participants’ scores on the DV are clustered at the high end

▪ Floor effect → participants’ scores on the DV are clustered at the low end

▪ How do you prevent these? → Manipulation checks!

▪ Manipulation check = a second DV included in a study to make sure the IV manipulation worked

▪ Check to see if you have a design confound

▪ Design confounds can act in reverse

▪ Weak manipulations → maybe your manipulation wasn’t strong enough?

▪ How was the independent variable operationalized? (ask about construct validity!)

▪ Insensitive measures → maybe your dependent variable wasn’t sensitive enough?

▪ How was the dependent variable operationalized? (ask about construct validity!)

11/23/2020

10

Perhaps Within-Groups Variability Obscured the Group Differences

▪ Measurement error → any factor that can inflate or

deflate a person’s true score on the DV

▪ Solution 1: Use reliable, precise measurements

▪ Solution 2: Measure more instances

▪ Individual differences → individual differences spread

out scores within each group

▪ Solution 1: change the design to a within-groups or

matched groups design

▪ Solution 2: add more participants

▪ Situation noise → any kind of external distraction that

could cause variability within-groups that obscure

between-groups differences

▪ Solution: controlling the surroundings of an

experiment

▪ Another name for these solutions: power

▪ Power → the likelihood that a study will yield a

statistically significant result when the IV really has

an effect

Perhaps There Really Is No Difference:

Not Enough Variability Between Levels

Null Effects May Be Published Less Often

▪ Null results can be just as interesting as experiments that show group differences.

▪ Null effects are rarely reported in the popular media.