Lecture: Quasi-experimental Designs and More

QUASI-EXPERIMENTAL DESIGNS AND MORE Quasi-experimental, Single Case, and Between Subjects Research Designs

Slide 1 Transcript As mentioned before, quasi-experimental research differs from true experiments mainly because random assignment is not required, either for individuals or groups. Also, Quasiexperimental designs may not include a comparison group. The variable for a Quasiexperimental study is one that already exists and cannot be manipulated or applied differentially among participants. Quasi-experimental studies are not able to control for all confounding variables, so they are not able to completely rule out some of the alternate explanations of the results. Researchers must take all the variables into consideration as they decipher results from quasi-experimental studies. So far, we have visited non- experimental and quasi-experimental research designs. In the second part of this module, we move into the experimental designs. A particular kind of experimental design is the single subject, or single case, research study. The single subject design can be used with an individual or a group, which serve as their own control and experimental subjects. The third component of this module will move into experimental research designs that examine levels of a factor applied once to individuals who have been assigned to groups at each level. This design is one that can be used to establish cause and effect.

Quasi-experimental Group Designs

One-group designs Posttest only

Lacks a comparison/control group to establish cause Single measure of DV, but many threats to internal validity

Pretest-posttest Pretest is the control; no separate control group Compares scores with same participants before and after Many threats to internal validity

Nonequivalent control group designs (not random)

Nonequivalent posttest Cause cannot be determined Susceptible to the threat of selection differences

Nonequivalent pretest-posttest Compare scores before and after intervention for both groups Susceptible to threat of selection differences

Slide 3 Transcript Among the four categories of quasi-experimental research designs that will be presented in this module, the first two relate to groups. First, there are the one-group research designs that use a posttest only or pretest-posttest design. Sometimes, the research requires observation of a single group. When this happens, there is no comparison group to demonstrate cause. The one-group posttest only design, also known as the one-shot case study, measures a dependent variable for one group of participants after the independent variable is introduced. With no control group used for comparison, this design has several threats to internal validity, like history and maturation effects, and is generally considered a poor research design. The second type is the one-group pretest-posttest design where the same results, or dependent variable, is measured with the same participants before and after the independent variable is administered. While this does allow for comparison of the results, still there is not a control group that did not experience the independent variable, so this design is weakened for internal validity as well. Even though there is a comparison, since random selection was not used, researchers are not able to control the extent to which other factors may have influenced the results. The second category of group designs relates to having a control group, except that the participants are not assigned randomly. This means the pre-existing factors cannot be controlled by the researchers, but they can make every effort to match participants with certain pre-existing characteristics. The same two versions described for one-group designs, posttest only or pretest-posttest, can be applied here. Again, with no random selection of participants, cause cannot be determined, although differences in results can be evaluated and associated with influences from the applied independent variable. With the pretest-posttest design, there is an added advantage where values can be compared before and after for differences between the group receiving the independent variable and the group that does not. As before, though, selection differences might be contributing to the differences.

Quasi-experimental Time Series & Developmental Designs

Time Series Basic design

Multiple measure of DV to establish baseline Measuring DV multiple times after intervention Weakness if intervening variable is present

Interrupted time series design Measure DV multiple times before/after naturally occurring event

Control time series design Basic or interrupted design includes non equivalent control group Improved internal validity

Developmental Study Designs Longitudinal

No IV, extended, multiple, same participants Cross-Sectional

Different participants, cohorts same Cohort-Sequential

Cohorts measured at different times Note: Cross-sectional and cohort- sequential have higher internal validity

Slide 5 Transcript A third category for quasi-experimental designs is the time series. The basic, sometimes called simple, design consists of making a series of observations, then introducing an intervention or new dynamic, and then making additional observations. So, in this type of design a baseline is established and if there is a substantial change observed in the second series of observations we could conclude that our intervention was the cause. The major weakness is that some unrecognized event may have occurred at about the same time as our intervention, or independent variable, and may have introduced an intervening history factor. As a footnote, Alexander Fleming used this design in his discovery of penicillin.

In the basic series we manipulate the intervention, however, in the interrupted series design the intervention occurs naturally. The rest is also similar to the basic design in that the postintervention measures determine if there has been an effect. With the control time series design, we now add a nonequivalent control group with parallel observations, the difference being that the control group does not experience the intervention. This design has greater internal validity than the simple version since an outside event would likely alter both groups in a similar way and further isolate changes due to the intervention.

The fourth category of quasi-experimental designs are those studying developmental changes over the human lifespan, for instance. There are three recognized version of developmental studies. In a previous module, these were mentioned, but they will be included here because they are part of the quasi-experimental methodologies. The first is the longitudinal design which observes the same participants, measured by changes in a dependent variable, over an extended period of time – often decades. There is no independent variable introduced by researchers and just the one group is being followed. The disadvantage is that there are several threats to internal validity like attrition, motivation, and testing effect from repeated exposures to the measurement. An advantage is that individual differences among the participants can be studied separately from the others. The cross-sectional design, on the other hand, does not follow the same participants. Each age group is called a cohort and is formed by sharing common characteristics, like the year of birth, for example. The advantage of using this design is that individuals in each cohort are observed just once, which avoids the threats to internal validity.

A cohort-sequential design is a comprise between longitudinal and cross-sectional research, with the intention to strengthen the research plan. Several observations are taken over time, although not as long as some longitudinal studies, and cohort groups are selected. The variation occurs as the cohorts are measured at different times during the course of the study. This approach reduces threats to internal validity since part of the sample is a cross-section of characteristics and the same participants are the ones observed over the extended time period.

Elements of Single Subject Research Design

Sample size is N=1 Studies changes in only one individual or group

Participant serves as own control, establishes baseline

DV measured for each individual participant, not averaged across groups or participants

To be an experiment, must meet three elements of control:

Randomization Manipulation Comparison/control group

Advantage: Allows for critical analysis of each individual measure

Slide 7 Transcript The single-subject or single-case design is applied when the sample size is one. This can be one individual person, or one group. In these designs, the individual or group serves as their own control in much the same way as a time series study. Basically, there is a nonintervention phase where a baseline is measured, then a post-intervention measure. Sometimes these are reported as an A -B design. Naturally, repeated measures become an ABABAB design and so on. As an experiment, the single subject design must meet the required criteria – first, randomization (of the interventions being studied, e.g.), second, manipulation of the independent variables. This is known as the single-variable rule, which states that only one variable at a time should be manipulated. And third, making the comparisons as the study progresses. Single subject studies can be done in three ways which will be explained next.

Elements of Between-Subjects Experimental Design

Two basic experimental research strategies

Between-subjects and Within-subjects

Compares separate groups Includes control group Allows one value per participant

Compares between two or more samples

Advantages Each value independent of others Researcher controls assignment to groups

Disadvantages Individual differences influence values and variability Assignment bias threatens internal validity Imitation, compensatory rivalry, resentful demoralization

Slide 9 Transcript Single-subject designs can be classified into three major categories: (1) A-B-A withdrawal, also known as reversal designs, (2) multiple-baseline designs, and (3) alternating interventions, also called changing-criterion designs. Single-subject designs alternate baseline and intervention phases over the course of several trials or observations. For the first one, the A-B-A reversal, as you would expect there is a baseline measure which might require several measurements taken. Then the treatment or intervention is performed and may also involve repeated measures. The treatment or intervention is withdrawn or stopped, and another series of measures is taken until baseline is restored. It is important to note that the return to the original baseline would be expected once the independent variable is removed. The number of measures taken in each phase will vary in experiments, but the longer it goes the greater risk there is for confounding variables to materialize. Measures typically are made for changes in levels or trends of the dependent variable as compared with baseline measures. Variations of this design, as mentioned before, might repeat the sequence so you wind up with something like A-B-A-B-A-B. The second category, multiple-baseline designs, would be used when there cannot be a return to the original baseline measure, or it is not possible to withdraw or reverse the intervention or independent variable. Also, this design might be used when the intervention can be withdrawn but the effect will linger or influence what comes next. Multiple, in the title, refers to multiple independent or dependent variables. With multiple baseline studies, data are collected on several measures or behaviors for one subject, or one behavior for several subjects, or one behavior and on subject in several settings. Eventually, all behaviors will be evaluated in all settings. When applying interventions across behaviors, for instance, they must be independent of each other. So far, only one intervention at a time has been introduced. The third category allows for successive interventions without a return to baseline. You might see this expressed as an A-B1-B2-B3 and is called the changing, or alternating, -criterion design. This would obviously be measured with a trend analysis in mind to see. The attraction of this design is that it can be performed without a baseline on some cases, since the goal is not to assess intervention versus no intervention. Rather, it is to assess the extent of change from intervention to intervention. Also, this design does not require withdrawal of treatments and allows for several interventions to be introduced quickly. It almost goes without saying that the dependent variable being measured must be pre-existing with the participant, since we are only calculating changes in that behavior or dimension an are not attempting to create it.

Types of Single Subject Designs

A-B-A Withdrawal design Multiple-baseline design Can be used when return to baseline not expected Treatment successively administered over time to different participants, behaviors, or settings

Changing-criterion design

Baseline phase followed by successive treatment phases

Criterion measure changed between phases Must meet criterion of one treatment phase before next

Does not require reversal to baseline

Enables analysis of gradually improving behavior

DV must pre-exist with participant

Slide 11 Transcript Remembering that the greater the control by the researcher, the higher the internal validity, one way to assess how much influence the intervention has is in watching the pattern of change across the phases of interventions. If the pattern is stable, the internal validity increases. Another indicator is in how much change has occurred between phases, presuming there is stability within each phase. The larger the change, the higher the internal validity. What is important here is the criterion used to measure the magnitude of change, which is to say at what point is the magnitude of change meaningful. Since there are no practice, carryover, or contrast effects from multiple exposures, fewer intervening variables are likely to be involved, which enhances internal validity. However, in single subject designs more participants are needed to offset the situation where each participant is only exposed to one level of the independent variable. As for external validity, it is generally pretty low in single subject research because of the small sample sizes. Still, if there are consistent patterns or findings, it may be possible to generalize more broadly.

Validity in Single Subjects Designs

Internal validity increases when:

Stable pattern across phases of intervention Magnitude of change increases Criterion measure if magnitude is accurate Fewer intervening variables involved

External validity low since sample sizes are small

External validity increases when:

Consistent patterns are observed Can generalize across behaviors, participants, or settings

Slide 13 Transcript There are two basic research designs associated with the experimental research strategy – between subject’s design and within-subjects design. The next module takes on the within subject’s approach, so first we will begin with the between-subjects version. The defining characteristic of a between-subjects design is that it compares separate groups of individuals. Since this is an experimental approach, a control group is established. Another feature is that it allows only one value per participant. In this way, every score represents a separate, unique participant. You might also hear this described as an independent measures design. The general goal of a between-subjects experiment is to determine whether differences exist between two or more levels or conditions to compare something, all still associated with one independent variable. Among the advantages for this approach is that each individual value generated is independent of the other values from the participant. Also, there is no practice effect or fatigue issue intervening from other treatments. A further advantage is that the researcher does have control over assignment of individuals to groups. Disadvantages can occur with individual differences influencing score values and variability within interventions. Internal validity can be threatened through assignment bias when the researcher selects groups, or by differential attrition as participants withdraw before the study is completed. With regard to the control group, there might be unintended diffusion or imitation of treatment effects noted in the experimental group. Compensatory rivalry might exist when the control group competes to equal performance seen in the experimental group, or resentful demoralization when the untreated group learns about the other group receiving the intervention and subsequently becomes less motivated.

Between-subjects Research and the Independent Variable

Researcher controls IV and known intervening factors

To minimize influence, researcher may:

Intentionally match participants against DV Limit participants to those with preferred characteristics

Setting may be: Staged: deliberately preconceived Natural: no control of conditions by researcher

Greater variation & less error variance

Significant differences Error variance in scores is caused by individual differences

More the values overlap, larger the error variance

Suggests individual differences were cause, not the IV

Slide 15 Transcript As we know, the researcher selects the participants, assigns them randomly to groups, and then manipulates the independent variable for the experimental group. Another aspect of control is that the researcher makes every effort to remove the influence of any other variable. In controlling these influencing factors, the researcher will address the differences among participants in the study, and anything in the environment for the study that might cause unwanted differences between the groups. Sometimes a researcher might intentionally select participants by matching them against the dependent variable or by limiting participants to those having particular characteristics identified by the researcher. With these issues in mind, the researcher has a choice to make about how to control these various potentially influential factors, so the experiment is performed in staged, i.e., deliberately pre-conceived, conditions as opposed to a more natural environment. One of the requirements for an experimental design is to exercise somewhat rigid controls over participants and conditions in a laboratory-like situation, which means that generalizing to something beyond those conditions is difficult to justify. In the natural settings, though, having less control over factors means it can be more generalizable but becomes less valid internally. The more the between group varies and the less variances overlap between groups, the more likely there will be significant differences. The converse is also true – when the values or results overlap among participants or in the group, the error variance will increase and there will likely be fewer significant differences found.

Between-subjects Research with Multiple Independent Samples

Multiple groups selected and treated independently

Compare results for each level of the IV

If different populations, use quasi-experimental Independent sample t-test for two groups (interval or ratio) One-way ANOVA for three+ groups same population

Goal of multiple samples approach

Minimize individual differences Reduce researcher selection influences Use random assignment into groups

Slide 17 Transcript Acknowledging the similarities with quasi-experimental designs, the between-subjects experimental design allows for cause and effect determination. Cause and effect can be determined with the between-subjects design, and sometimes involves multiple groups selected and treated independently. A researcher might select two or more groups from the same population and compare results for each level of the independent variable applied. If the participants were selected from different populations, a quasi-experimental approach would be more appropriate. The statistical application for a between subjects two samples design would be an independent sample t-test. With more than two groups, the researcher might choose to conduct a one-way (meaning one independent variable) analysis of variance. The goal for two or more independent samples is to minimize the possibility of individual differences or something associated with the intervention process causing differences among the groups. One of the requirements for an experiment is randomness, so once the participants are selected from the population the requirement can be achieved through random assignment of the participants into different groups.

Between-subjects Research and the Dependent Variable

Self-report Easy, but not always accurate Economical Large volume of data possible

Behavioral May be direct or unobtrusive Can track many items using structured rating Potential rating bias by experimenter

Physiological Unbiased for most part Extensive training and greater expense May introduce ethical concerns

Artifact may be a factor

Slide 19 Transcript Dependent variables are measured to determine how much influence the independent variable has produced. The types of measures often are by self-report instruments, various behavioral measures, or more objective physiological measures. Self-report measures are relatively easy to administer and are economical by comparison to other measures. Quite a bit of data can be accumulated using these reports. However, the items reported may be inaccurate because participants are not truthful, get confused, or just guess when they do not truly know what response to make. Observing behaviors can be done openly or unobtrusively. This means the participant sees or is aware of the researcher recording observations, or they do not see them and cannot tell what is being recorded and when. An advantage is that the observer recorder can track many items and use a structured rating device, although this also introduces potential rating bias. The use of physiological measures permits unbiased measures for the most part but is subject to how setting and thresholds are determined. Physiological measures can be expensive and require highly trained technicians, which might introduce ethical and confidentiality concerns. Also, there are various artifacts that can influence recording devices and yield inaccurate data. So, we have covered the quasi-experimental designs and started on the experimental ones focusing on administering one independent variable. This will continue in the next module, and until then, have a productive and healthy week.