Exam 3
Monet013
Chapter10LT1.pptxResearch Methods in Psychology
Quasi-Experimental Designs
1
Characteristics of True Experiments
Manipulate Independent Variable (IV)
Treatment, comparison conditions
High degree of control
Choice of the DVs
Random assignment to conditions
Unambiguous outcome regarding effect of IV on DV
Internal validity
2
Applied Research
Goals
Test external validity of lab findings
Improve conditions in which people live and work (natural settings)
Quasi-experiments
Procedures that approximate the conditions of highly controlled laboratory experiments
3
Permission
Difficult to gain permission to conduct true experiments in natural settings
Difficult to gain access to participants
Random assignment perceived as unfair
People want a “treatment”
Random assignment is best way to determine whether a treatment is effective
Use “waiting-list” control group or alternate treatments
Tablets in English and science classes example
Obstacles to Conducting True Experiments in Natural Settings
4
Advantage of True Experiments
Threats to internal validity are controlled
8 general threats to internal validity
| history | regression |
| maturation | selection |
| testing | subject attrition |
| instrumentation | additive effects with selection |
5
Threats to Internal Validity
History
When an event occurs at the same time as the treatment and changes participants’ behavior
Participants’ “history” includes events other than treatment
Difficult to infer treatment has an effect
6
History Threat, continued
Does a campus recycling awareness campaign influence the amount of paper, plastic, and cans in campus bins?
History threat: Suppose at week 4 (X = treatment) a popular celebrity also starts to promote recycling in the media.
Can you conclude the campus campaign was effective?
7
Week
Recycling (Kg)
Threats to Internal Validity, continued
Maturation
Participants naturally change over time.
These maturational changes, not treatment, may explain any changes in participants during an experiment.
8
Maturation Threat, continued
Does a new reading program improve 2nd graders’ reading comprehension?
Reading comprehension improves naturally as children mature over the year.
Can you conclude the reading program was effective?
9
Reading Comprehension
Threats to Internal Validity, continued
Testing
Taking a test generally affects subsequent testing.
Participants’ performance on a measure at the end of a study may differ from an initial testing because of their familiarity with the measure.
10
Testing Threat, continued
Does teaching a new problem solving strategy influence people’s ability to solve problems quickly?
If similar problems are used in the pretest, faster problem solving at post-test may be due to familiarity with the test.
Can we conclude the new strategy improves problem-solving ability?
11
Minutes (Mean)
Threats to Internal Validity, continued
Instrumentation
Instruments used to measure participants’ performance may change over time
Example: observers may become bored or tired
Changes in participants’ performance may be due to changes in instruments used to measure performance, not to a treatment.
12
Instrumentation, continued
Suppose a police protection program is implemented to decrease incidence of assault.
At the same time the program is implemented (X), reporting laws change such that what constitutes assault is broadened.
Can we conclude the program was effective (or ineffective)?
13
Week
Assaults
Threats to Internal Validity, continued
Regression
Individuals sometimes perform very well or very poorly because of chance (e.g., luck).
Chance factors are not likely present during 2nd testing, so scores will not be as extreme.
Scores will “regress” (go toward) the mean.
Regression effects, not treatment, may account for changes in participants’ performance over time.
14
Regression, continued
Suppose students are selected for an enrichment program because of their very high scores on a brief test.
Regression: to the extent the test is an unreliable measure of ability, we can expect their scores to regress to the mean at the 2nd testing.
Can we conclude the enrichment program was effective (or ineffective)?
15
Test Score (Mean)
Threats to Internal Validity, continued
Subject attrition
When participants are lost from the study (attrition), the group equivalence formed at the start of the study may be destroyed.
Differences between treatment and control groups at the end of the study may be due to natural differences in those who remain in each group.
16
Subject Attrition, continued
Suppose an exercise program is offered to employees who would like to lose weight.
At Time 1, N = 50
M weight = 225 pounds
At Time 2, N = 25 (25 drop out of study)
Suppose the 25 who stayed in program weighed, on average, 150 pounds at Time 1
Did the exercise program help people to lose weight?
17
MeanWeight
Threats to Internal Validity, continued
Selection
Occurs when differences exist between individuals in treatment and control groups at the start of a study
These differences become alternative explanations for any differences observed at the end of the study
Random assignment controls the selection threat
18
Selection, continued
Suppose a community recycling program is tested. Individuals who are interested in recycling are encouraged to participate.
Evaluation: Compare the weight of garbage (i.e., not recycled) from participants in the program with weight of garbage from those not in the new program.
Can we tell if the new recycling program is effective?
19
Recycle Program
mean lbs/week
Threats to Internal Validity, continued
Additive effects with selection
When one group of participants in an experiment
Responds differently to an external event (history)
Matures at a different rate
Is measured more sensitively by a test (instrumentation)
These threats (rather than treatment) may account for any group differences at the end of a study.
20
Additive effects with selection, continued
Suppose School A starts a program (X) to prevent alcohol abuse on campus (Week 4). The DV is number of alcohol-related infractions in student residences.
School B is a comparison.
During Week 4 the newspaper at School A reports a student death due to intoxication (“local history effect”).
Is the program effective?
Week
# Infractions
Threats to Internal Validity, continued
With no comparison group, must rule out:
history, maturation, testing, instrumentation, regression, subject attrition, selection
When there is a comparison group, you must rule out these threats:
selection, additive effects with selection
Adding a comparison group helps rule out many threats to internal validity.
22
Stretching Exercise, page 323
Threats to Internal Validity, continued
Threats even true experiments may not eliminate
Contamination
resentment, rivalry, diffusion of treatments
Experimenter expectancy effects
Novelty effects (including Hawthorne effect)
Threats to external validity
Treatment effects may not generalize
Best way to assess external validity: replication
24
Quasi-Experiments
“Quasi-” (resembling) experiments
Important alternative when true experiments are not possible
Lack the high degree of control found in true experiments
Often no random assignment
Researchers must seek additional evidence to eliminate threats to internal validity
25
The One-Group Pretest-Posttest Design
“Bad experiment” or “pre-experimental design”
Intact group is selected to receive a treatment
e.g., a classroom of children, a group of employees
Pretest is 1st Observation (O1)
Treatment is implemented (X)
Posttest is 2nd Observation (O2)
O1 X O2
26
One-Group Pretest-Posttest Design, cont.
O1 X O2
None of the threats to internal validity are controlled.
Any change between pretest (O1) and posttest (O2) may be due to treatment (X) or
History
Maturation
Testing
Or instrumentation, regression, subject attrition, selection
27
Quasi-Experimental Designs
Nonequivalent Control Group Design
A group similar to the treatment group serves as a comparison group
Obtain pretest and posttest measures for individuals in both groups
Random assignment to groups is not used
Pretest scores are used to determine whether the groups are equivalent
Equivalent only on this dimension
28
Nonequivalent Control Group Design, continued
| Treatment | ||||
| ↓ | ||||
| O1 | X | O2 | ← treatment group | |
| -------------------------------------------------- | ||||
| O1 | O2 | ← nonequivalent control group | ||
| pretest | posttest |
Nonequivalent Control Group Design, continued
Example: Does taking a research methods course improve reasoning ability?
Compare students in research methods and developmental psychology courses
DV: 7-item test of methodological and statistical reasoning ability
Suppose group differences are observed at the posttest
30
Nonequivalent Control Group Design, continued
By adding a comparison group, rule out these threats to internal validity:
history
maturation
testing
instrumentation
regression
Assume that these threats happen the same to both groups, therefore, can’t be used to explain posttest differences.
31
Mean Reasoning Score
Nonequivalent Control Group Design, continued
What threats are not ruled out?
Selection
Without random assignment to conditions, the two groups are probably not equivalent on many dimensions.
These preexisting differences may account for group differences at the posttest.
32
Nonequivalent Control Group Design, continued
Additive effects with selection
The two groups
May have different experiences (selection X history or “local history effect”)
May mature at different rates (selection X maturation)
May be measured more or less sensitively by the instrument (selection X instrumentation)
May drop out of the study (courses) at different rates (differential subject attrition)
May differ in terms of regression to the mean (differential regression)
33
Quasi-Experiments, continued
Simple Interrupted Time Series Design
Observe a DV for some time before and after a treatment is introduced.
Archival data are often used.
Look for clear discontinuity in the time-series data for evidence of treatment effectiveness.
O1 O2 O3 O4 X O5 O6 O7 O8
34
Simple Interrupted Time-Series Design, cont.
Example: Study habits
Intervention: An instructional course to change students’ study habits
Implemented during summer following the sophomore year (after semester 4)
DV: semester GPA
Suppose a discontinuity is observed when the treatment (X) is introduced
35
Simple Interrupted Time-Series Design, cont.
What threats can be ruled out?
Maturation: assume maturational changes are gradual, not abrupt
Testing (GPA): if testing influences performance, these effects are likely to show up in initial observations (before X)
Testing effects less likely with archival data
Regression: if scores regress to the mean, they will do so in initial observations
discontinuity
36
Semester
Mean GPA
Quasi-Experiments, continued
Time Series with Nonequivalent Control Group Design
Add a comparison group to the simple time-series design
O1 O2 O3 O4 X O5 O6 O7 O8
--------------------------------------------------------------
O1 O2 O3 O4 O5 O6 O7 O8
37
Time Series with Nonequivalent Control Group Design, continued
Example: Study habits
Suppose a nonequivalent control group is added—these students don’t participate in the study habits course
Who could be in the comparison group?
What threats would you be able to rule out?
38
Semester
Mean GPA
An Example
Study to determine if well being is increased if nursing home residents are given the opportunity to make daily personal decisions (how their room is arranged, visits, movie choices)
Two groups: choice group and no-choice group
Assignment to groups was done by floor in a nursing home
These floors were chosen due to similarity in the residents’ physical and psychological health and prior SES
Questionnaires administered 1 week before and three weeks into the study
Staff members rated residents before and after treatment (alertness, sociability, and activity)
Contest—guess the number of jelly beans in a jar.
What is the independent variable?
What is/are the dependent variable(s)?
What type of quasi-experimental design?
Which threats to internal validity are controlled?
Which threats are not controlled?
