Need 4 pages done

carren45

quasi-experimentalmethod-encyclopedia-of-measurement-and-statistics1.pdf

Home >Business & Finance homework help >Management homework help >Need 4 pages done

Quasi-Experimental Method

In: Encyclopedia of Measurement and Statistics

By: M. H. Clark & William R. Shadish

Edited by: Neil J. Salkind

Book Title: Encyclopedia of Measurement and Statistics

Chapter Title: "Quasi-Experimental Method"

Pub. Date: 2011

Access Date: August 17, 2018

Publishing Company: Sage Publications, Inc.

City: Thousand Oaks

Print ISBN: 9781412916110

Online ISBN: 9781412952644

DOI: http://dx.doi.org/10.4135/9781412952644

Print pages: 806-808

This PDF has been generated from SAGE Research Methods. Please note that the

pagination of the online version will vary from the pagination of the print book.

http://dx.doi.org/10.4135/9781412952644

Quasi-experiments, like all experiments, manipulate treatments to determine causal effects

(quasi-experiments are sometimes referred to as nonrandomized experiments or observational

studies). However, quasi-experiments differ from randomized experiments in that units are not

randomly assigned to conditions. Quasi-experiments are often used when it is not possible to

randomize ethically or feasibly. Therefore, units may be assigned to conditions using a variety

of nonrandomized techniques, such as permitting units to self-select into conditions or

assigning them based on need or some other criterion. Unfortunately, quasi-experiments may

not yield unbiased estimates as randomized experiments do, because they cannot reliably rule

out alternative explanations for the effects. To improve causal inferences in quasi-experiments,

however, researchers can use a combination of design features, practical logic, and statistical

analysis. Although researchers had been using quasi-experiments designs long before 1963,

that was the year Donald T. Campbell and Julian C. Stanley coined the term quasi-experiment.

The theories, practices, and assumptions about these designs were further developed over the

next 40 years by Campbell and his colleagues.

Threats to Validity

In 1963, Campbell and Stanley created a validity typology, including threats to validity, t o

provide a logical and objective way to evaluate the quality of causal inferences made using

quasi-experimental designs. The threats are common reasons that explain why researchers

may be incorrect about the causal inferences they draw from both randomized and quasi-

experiments. Originally, Campbell and Stanley described only two types of validity, internal

validity and external validity. Thomas D. Cook and Campbell later added statistical conclusion

validity and construct validity.

Of the four types of validity, threats to internal validity are the most crucial to the ability to make

causal claims from quasi-experiments, since the act of randomization helps to reduce the

plausibility of many internal validity threats. Internal validity addresses whether the observed

covariation between two variables is a result of the presumed cause influencing the effect.

These internal validity threats include the following:

Ambiguous temporal precedence: the inability to determine which variable occurred first,

thereby preventing the researcher to know which variable is the cause and which is the

effect.

Selection: systematic differences between unit characteristics in each condition that could

affect the outcome.

History: events that occur simultaneously with the treatment that could affect the outcome.

Encyclopedia of Measurement and StatisticsPage 2 of 7

Maturation: natural development over time that could affect the outcome.

Regression: occurs when units selected from their extreme scores have less extreme scores

on other measures, giving the impression that an effect had occurred.

Attrition: occurs when those who drop out of the experiment are systematically different in

their responses from those who remain.

Testing: Repeatedly exposing units to a test may permit them to learn the test, giving the

impression that a treatment effect had occurred.

Instrumentation: Changes in the instrument used to measure responses over time or

conditions may give the impression that an effect had occurred.

Additive and interactive threats to internal validity: T h e i m p a c t o f a t h r e a t c a n b e

compounded by, or may depend on the level of, another threat.

The other three types of validity also affect the ability to make causal conclusions between the

treatment and outcome but do not necessarily affect quasi-experiments more than any other

type of experiment. Statistical conclusion validity addresses inferences about the how well the

presumed cause and effect covary. These threats, such as low statistical power and violation of

statistical assumptions, are essentially concerned with the statistical relationship between the

presumed cause and effect. Construct validity addresses inferences about higher-order

constructs that research operations represent. These threats, such as reactivity to the

experimental situation (units respond as they want to be perceived rather than to the intended

treatment) and treatment diffusion (the control group learns about and uses the treatment),

question whether the researchers are actually measuring or manipulating what they intended.

External validity addresses inferences about whether the relationship holds over variation in

persons, settings, treatment variables, and measurement variables. These threats, such as

interactions of the causal treatment with units or setting, determine how well the results of the

study can be generalized from the sample to other samples or populations.

Basic Types of Quasi-Experiments

While there are many variations of quasi-experimental designs, basic designs include, but are

not limited to, (a) one-group posttest-only designs, in which only one group is given a treatment

and observed for effects using one posttest observation; (b) nonequivalent control group

designs, in which the outcomes of two or more treatment or comparison conditions are studied,

but the experimenter does not control assignment to conditions; (c) regression discontinuity

designs, in which the experimenter uses a cutoff score from a continuous variable to determine

assignment to treatment and comparison conditions, and an effect is observed if the regression

line of the assignment variable on outcome for the treatment group is discontinuous from that

of the comparison group at the point of the cutoff; and (d) interrupted time series designs, in

Encyclopedia of Measurement and StatisticsPage 3 of 7

which many (100 or more) consecutive observations on an outcome over time are available, and

treatment is introduced in the midst of those observations to determine its impact on the

outcome as evidenced by a disruption in the time series after treatment; and (e) single-group or

single-case designs, in which one group or unit is repeatedly observed over time (more than

twice, but fewer than in a time series), while the scheduling and dose of treatment are

manipulated to demonstrate that treatment affects outcome.

The causal logic of threats to validity can also be applied to two other classes of designs that

are not quasi-experiments because the cause is not manipulated, as it is in the previous five

designs. These are case-control designs, in which a group with an outcome of interest is

compared with a group without that outcome to see how they differ retrospectively in exposure

to possible causes, and correlational designs, in which observations on possible treatments

and outcomes are observed simultaneously to see if they are related. These designs often

cannot ensure that the cause precedes the effect, making it more difficult to make causal

inferences than in quasi-experiments.

Design Features

To prevent a threat from occurring or to diagnose its presence and impact on study results,

researchers can manipulate certain features within a design, thereby improving the validity of

casual inferences made using quasi-experiments. These design features include (a) adding

observations over time before (pretests) or after (posttests) treatment to examine trends over

time; (b) adding more than one treatment or comparison group to serve as a source of

inference about the counterfactual (what would have occurred to the treatment group if they

had not received the treatment); (c) varying the type of treatment, such as removing or varying

a treatment; and (d) using nonrandomized assignment methods that the researcher can control

or adjust, such as using a regression discontinuity design or matching. All quasi-experiments

are combinations of these design features, chosen to diagnose or minimize the plausibility of

threats to validity in a particular context.

New designs are added to the basic repertoire of designs using these elements. For example,

by adding pretest observations to a posttest-only, nonequivalent control group design, existing

pretest differences between the treatment and control groups can better be measured and

accounted for, which helps reduce effects of selection. Likewise, adding a comparison group to

a time series analysis can assess threats such as history. If the outcome for the comparison

group varies over time in the same pattern as the treatment outcome, history is a likely threat.

Examples

Encyclopedia of Measurement and StatisticsPage 4 of 7

In 1983 and 1984, the Arizona State Lottery implemented a campaign aimed at increasing the

sale of state lottery tickets by offering free tickets to retail customers, who were not asked if

they wanted to buy tickets by store clerks. While the researchers examined the effects of the

campaign by using various designs, one of the methods used a nonequivalent control group

design with pretests and posttests. The 44 retail stores that implemented the campaign were

matched with 22 control stores on market shares (the proportion of total state ticket sales made

by each retail store for a single game). All stores were measured on their market shares before

the campaign intervention (pretest) and after the program intervention (posttest). Results

indicated that there were no differences between the treatment and control groups in market

shares at pretest. However, stores that participated in the campaign profited significantly more

in market shares than did the control group at posttest.

In 1974, Cincinnati Bell began charging 20¢ per call to local directory assistance and found an

immediate and large drop in local directory-assisted calls once this charge was imposed. One

hundred eighty monthly observations were collected from 1962 through 1976, which assessed

the number of local and long-distance directory-assisted calls. Results of this study found that

the number of local and long-distance directory-assisted local calls steadily increased from

1962 (approximately 35,000 calls per day for local calls and 10,000 calls per day for long-

distance calls) until 1973 (approximately 80,000 calls per day for local calls and 40,000 calls

call per day for long-distance calls). However, once the charge for the local calls was imposed,

the number of directory-assisted local calls decreased to approximately 20,000 calls per day in

1974. However, the directory-assisted long-distance calls, which did not have a fee imposed,

continued to slowly increase over time. This study was an interrupted time series quasi-

experiment using a nonequivalent control group (directory-assisted long-distance calls).

Statistical Adjustments

While Campbell emphasized the importance of good design in quasi-experiments, many other

researchers sought to resolve problems in making causal inferences from quasi-experiments

through statistical adjustments. One such method uses propensity scores, the conditional

probability that a unit will be in a treatment condition given a set of observed covariates. These

scores can then be used to balance treatment and control units on predictor variables through

matching, stratifying, covariate adjustment, or weighting. Another method, selection bias

modeling, attempts to remove hidden bias that occurs when unobserved covariates influence

treatment effects by modeling the selection process. A third method uses structural equation

modeling (SEM) to study causal relationships in quasi-experiments by modeling latent variables

to reduce bias caused by unreliable measures. While these statistical adjustments have been

Encyclopedia of Measurement and StatisticsPage 5 of 7

shown to reduce some of the bias present in quasi-experiments, each of these methods has

limitations that prevent it from accounting for all of the sources of biased estimates. Therefore, it

is often more effective to obtain less biased estimates through good designs than through

elaborate statistics.

Conclusion

Quasi-experiments may never rule out threats to internal validity as well as randomized

experiments; however, improving the designs can reduce or control for those threats, making

causal conclusions more valid for quasi-experiments than they would otherwise be. This can

most easily be done by using designs that are most appropriate for the research question and

by adding design features to address particular plausible threats to validity that may exist.

While certain conditions within field studies may hinder the feasibility o f u s i n g m o r e

sophisticated quasi-experimental designs, it is important to recognize the limitations of designs

that are used. In some cases, statistical adjustments can be used to improve treatment

estimates; however, even then, causal inferences should be made with caution.

treatment

M. H.Clark and William R.Shadish

http://dx.doi.org/10.4135/9781412952644.n369