db week3 cj methods
Experimental and Quasi-Experimental Designs
Chapter 5
*
Introduction
Experiments are best suited for explanation and evaluation research
Experiments involve:
Taking action
Observing the consequences of that action
Especially suited for hypothesis testing
Often occur in the field
The Classical Experiment
- Classical experiment: a specific way of structuring research
- Involves three major components:
Independent variable and dependent variable
Pretesting and posttesting
Experimental group and control group
Independent and Dependent Variables
The independent variable takes the form of a dichotomous stimulus that is either present or absent
It varies (i.e., is independent) in our experimental process
The dependent variable is the outcome, the effect we expect to see
Might be physical conditions, social behavior, attitudes, feelings, or beliefs
Pretesting and Posttesting
Subjects are initially measured in terms of the DV prior to association with the IV (pretested)
Then, they are exposed to the IV
Then, they are remeasured in terms of the DV (posttested)
Differences noted between the measurements on the DV are attributed to influence of IV
Experimental and Control Groups
Experimental group: exposed to whatever treatment, policy, initiative we are testing
Control group: very similar to experimental group, except that they are NOT exposed
Can involve more than one experimental or control group
If we see a difference, we want to make sure it is due to the IV, and not to a difference between the two groups
Placebo
We often don’t want people to know if they are receiving treatment or not
We expose our control group to a “dummy” independent variable just so we are treating everyone the same
Medical research: participants don’t know what they are taking
Ensures that changes in DV actually result from IV and are not psychologically based
Double-Blind Experiment
Experimenters may be more likely to “observe” improvements among those who received drug
In a double-blind experiment, neither the subjects nor the experimenters know which is the experimental group and which is the control group
Selecting Subjects
First, must decide on target population – the group to which the results of your experiment will apply
Second, must decide how to select particular members from that group for your experiment
Cardinal rule – ensure that experimental and control groups are as similar as possible
Randomization
- Randomization: produces an experimental and control group that are statistically equivalent
- Essential feature of experiments
- Eliminates systematic bias
Experiments and Causal Inference
Experimental design ensures:
Cause precedes effect via taking posttest
Empirical correlation exists via comparing pretest to posttest
No spurious 3rd variable influencing correlation via posttest comparison between experimental and control groups, and via randomization
Example of Research Using an Experimental Design
Researchers at the University of Maryland conducted an evaluation of the Baltimore Drug Court using an experimental design. For their research, eligible offenders were randomly assigned to either the drug court or to ‘”treatment as usual”. The results of the randomization process were given to the judges as a recommendation. In most cases, the judges, who agreed to participate in the study beforehand, sentenced offenders in accordance with the randomization. The results showed that participants in the drug court were less likely to recidivate than those in the control group.
For more information see Gottfredson, D.C., Najaka, S.S. & Kearley, B. (2003). Effectiveness of drug treatment courts: Evidence from a randomized trial. Criminology, 2(2), 171-196.
Internal Validity Threats
Internal Validity: refers to the possibility that conclusions drawn from experimental results may not reflect what went on in experiment
History: external events may occur during the course of the experiment
Maturation: people constantly are growing
Testing: the process of testing and retesting
Instrumentation: Changes in the measurement process
Internal Validity Threats: Continued
Statistical regression: extreme scores regress to the mean
Selection bias: the way in which subjects are chosen
Experimental mortality: subjects may drop out prior to completion of experiment
Ambiguous Casual Time Order: the dependent variable actually caused the change in the stimulus
Generalizability and Threats to Validity
- Researchers also face problems with generalizing results from experiments
- Generalizability: do the results of an experiment really tell us what would happen in the real world?
Construct Validity Threats
Construct validity: the correspondence between the empirical test of a hypothesis and the underlying causal process that the experiment represents
Link construct and measures to theory
Clearly indicate what constructs are represented by what measures
Decide how much treatment is required to produce change in DV
External Validity Threats
External validity: whether the results from experiments in one setting will be obtained in other settings
Significant for experiments conducted under carefully controlled conditions rather than more natural conditions
But, this reduces internal validity threats!
A conundrum!
Internal validity must be established before external validity is an issue
Statistical Conclusion Validity Threats
Statistical conclusion validity: whether we are able to determine if two variables are related
Becomes an issue when findings are based on small samples
More cases allows you to reliably detect small differences; less cases result in detection of only large differences
Variations in the Classical Experimental Design
- Post-test Only design
No pretest measure is used
Used when pretest might bias results
- Factorial Design
Two experimental groups are used
Used to determine necessary amount of treatment
Quasi-Experimental Designs
When randomization is not possible
quasi = “to a certain degree”
Quasi-Experiment: an experiment to a certain degree
Do not have as stringent of a control over internal validity threats as true experiments
Two categories: non-equivalent-groups designs and time series designs
Nonequivalent-Groups Designs
When we cannot randomize, we cannot assume equivalency; hence the name
We take steps to make groups as comparable as possible
Match subjects in E and C groups using important variables likely related to DV under study
Aggregate matching – comparable average characteristics
Cohort Designs
Cohort – group of subjects who enter or leave an institution at the same time
Necessary to ensure that two cohorts being examined against one another are actually comparable
Time-Series Designs
Examine a series of observations over time
Interrupted – observations compared before and after some intervention
Instrumentation threat to internal validity is likely because changes in measurements may occur over a long period of time
Often use measures produced by CJ organizations
Variations in Time-Series Designs
- Interrupted Time Series Design with a Non-Equivalent Comparison Group
- Time-Series Design with Switching Replications
Variable-Oriented Research, Case Studies and Scientific Realism
- Case-oriented research: many cases are examined to understand a small number of variables
- Variable-oriented research: a large number of variables are studied for a small number of cases
Case studies: researcher centers on an in-depth examination of one or a few cases on many dimensions
In-depth examinations of a few cases
*