Need 4 pages done

CHAPTER6-designandanalysisofexperimentalandquasi-experimentalinvestigations.docx

Home >Business & Finance homework help >Management homework help >Need 4 pages done

CHAPTER 6 - Design and Analysis of Experimental and Quasi-Experimental Investigations Andrea Follmer Greenhoot

from Blackwell Handbook of Research Methods in Clinical Psychology

This chapter is concerned with experimental and quasi-experimental designs and the statistical approaches most commonly employed in conjunction with them. The section on design describes basic configurations of true experiments and quasi-experiments, explains the advantages and disadvantages of each of these two families of design, and discusses some general considerations for either type of design. The section on analysis focuses on the analysis of experimental and quasi-experimental data with continuous response variables ( chapter 9 , this volume, for information about the analysis of qualitative dependent variables). This section begins with a brief discussion of preliminary, descriptive analyses, followed by a description of methods for inferential analyses. Because experimental and quasi-experimental studies are designed to address questions about group differences in average performance, data from these studies are typically analyzed using some form of an analysis of variance (ANOVA) or extensions of that technique. However, it is important to note that the statistical techniques discussed here may be appropriate for the analysis of nonexperimental data; indeed, these procedures are appropriate for the analysis of data from any investigation in which the primary research questions involve an assessment of differences in group performance.

Design Experimental designs

True experiments involve the manipulation of one or more independent variables by the investigator and random assignment of subjects to experimental groups or treatments.

An experiment with two or more independent variables (i.e., factors) that are completely crossed has a factorial design. Combinations of levels of the independent variables form the groups or cells of the experimental design. For example, an experimenter might measure depressive symptoms of patients who took one of three drugs (drug A, drug B, or a placebo) and participated in one of two types of therapy. By combining the three levels of the drug factor with the two levels of therapy, six cells or groups are formed. With random assignment of subjects to groups, each individual has an equal chance of being assigned to any experimental group, reducing the likelihood of differences between treatment groups being due to initial differences in the group samples. As such, true experiments permit the investigator to make strong cause and effect conclusions about the relations between the independent and dependent variables.

Random assignment procedures

Two general procedures can be used to randomly assign subjects to experimental conditions. With the simpler procedure, free random assignment, subjects are randomly distributed among experimental groups; this procedure leads to a completely randomized experimental design. The alternative procedure, matched random assignment, is often used when an investigator suspects that certain subject characteristics (e.g., age, IQ, or prior hospitalization) are related to the dependent variable. This method is particularly desirable when the influence of a confounding variable is likely to be strong enough that it will mask experimental effects. Pretest scores or existing records are used to create blocks of subjects who are similar on the confounding variable(s), and treatment levels are assigned at random to subjects within each block. By controlling for initial differences among subjects on confounding factors, the blocking procedure facilitates detection of experimental effects. When this randomization procedure is used, the experiment has a randomized block design.

Posttest-only versus pretest-posttest designs

In the simplest experimental design one or more experimental groups are compared to a control group in a “posttest.” In other words, subjects are randomly assigned to groups, and responses are measured once after the introduction of the experimental manipulation. Group differences in mean scores on the response variable are assumed to reflect the effect of the independent variable because the randomization process reduces the probability of initial group differences in the outcome variable. Posttest-only designs work particularly well with large samples, because larger samples increase the likelihood that random assignment will actually lead to equality between groups. To strengthen the claim that the randomization process resulted in initially equivalent groups, also it is common practice to measure other characteristics of the sample that might be related to the outcome of interest.

Whereas posttest-only designs only permit the evaluation of between-subject effects, pretest-posttest designs allow for the assessment of within-group changes in response to an experimental manipulation. In some pretest-posttest designs subjects are first randomly assigned to groups and then pretested before the experimental manipulation. Pretest scores allow the investigator to verify the success of random assignment in creating equivalent groups. Other pretest-posttest designs involve pretesting subjects first, matching them on the basis of pretest scores, and randomly assigning members of matched pairs to groups. This procedure helps to ensure that equivalent groups are created in the first place. A pretest-posttest design is especially preferable when the number of subjects is small or when the subjects are likely to differ substantially on a characteristic related to the outcome of interest. The design is also recommended if a researcher suspects that subjects’ responses to a manipulation depend on their initial scores on the dependent measure; with the pretest-posttest design the investigator can statistically control for initial levels, making it easier to detect experimental effects. Further, when the examination of within-subject changes in the outcome is an explicit goal of the research, a pretest-posttest design is clearly the more appropriate one. One disadvantage of pretest-posttest designs is that repeated testing sometimes introduces the problem of differential carryover effects ( Maxwell and Delaney, 1990 ). When carryover effects are likely, a posttest-only design is preferred.

Quasi-experimental designs

Quasi-experiments have some of the characteristics of true experiments, including direct manipulation of one or more variables of interest. As in basic experiments, combinations of the different levels of the independent variables form the experimental groups. What distinguishes quasi-experiments from true experiments is that they do not involve random assignment of subjects to conditions. Quasi-experimental designs can be categorized into two subtypes: nonequivalent control group designs and interrupted time series designs. Nonequivalent control group designs

In research with human subjects it is sometimes impossible for an investigator to randomly assign subjects to treatment groups. In many institutions individuals are assigned to groups for educational or social purposes. For example, children are aggregated into schools, grades, and classrooms, and patients are aggregated into hospitals and wards. In these situations researchers are rarely permitted to reassign individuals to test a hypothesis, and instead must compare preexisting groups that receive different treatments. Intact groups, however, might differ on characteristics related to the outcome variable. For example, patients assigned to different wards might vary in symptom level or demographic characteristics such as age. Similarly, children attending different schools might come from different socioeconomic and educational backgrounds. Within schools, children may be aggregated into classrooms by school staff on the basis of academic or behavioral characteristics, or by parents who request particular teachers for their children. Because of the potential for initial group differences, the ability to draw causal inferences from nonequivalent control group investigations depends on other features of the study design, such as whether the design includes a pretest.

Posttest-only versus pretest-posttest designs

Posttest-only designs for nonequivalent control group designs are not ordinarily recommended (e.g., Morgan, Gliner, and Harmon, 2000 ). The researcher is unable to rule out the threat of selection bias because baseline information about the equivalence of groups is not collected. That is, group differences in the outcome could be attributable to either the experimental manipulation or initial group differences. Although measurements of sample characteristics that might be related to the variables of interest could be used to strengthen the claim of group equivalence, the results of a posttest-only quasi-experiment are generally more difficult to interpret than those of the pretest-posttest alternative. Quasi-experiments that employ some sort of pretest provide clearer information about the relationships between the independent and dependent variables, because the use of a pretest permits an assessment of the initial equivalence of the experimental groups. Pretest-posttest designs, however, do not eliminate the threat of “local history,” or an experience occurring between pretest and posttest for one group that might influence the outcome of interest. Thus, the likelihood of such experiences for the different groups should be assessed.

Interrupted time series designs

In contrast to nonequivalent control group designs, which involve between-subjects comparisons to test treatment effects, interrupted time series designs assess treatment effects through within-subjects comparisons. These designs, including ABA, ABAB, and multiple baseline designs, can be applied to either groups of subjects or single subjects. The logic of the designs is generally the same regardless of whether one or several subjects are involved: subjects are exposed to a series of changes in the independent variable, and their responses are measured in each phase. To illustrate, in ABA designs, baseline measurements on the outcome variable are taken repeatedly for a given time period, followed by the introduction of an experimental manipulation and the measurement of responses to the manipulation. Finally, the manipulation is removed and baseline conditions are reinstated. Changes in subjects’ responses during the experimental manipulation phase combined with reversal of the effects in the final phase provide strong evidence for a causal link between the independent and dependent variable ( Gelfand and Hartmann, 1984 ; and chapter 5 , this volume, for a more detailed description of these procedures).

Inclusion of additional explanatory variables

Both true experiments and quasi-experiments frequently include measurements of non-manipulated classificatory or continuous variables. For instance, an investigator might be interested in whether subjects from two or more populations (e.g., males and females) respond similarly to an experimental manipulation (e.g., drug A and a placebo). Alternatively, a researcher might want to control for continuous extraneous variables, such as age or prior hospitalizations, when assessing experimental effects. The inclusion of such variables permits the investigator to demonstrate the generalizability of treatment effects, and increases the sensitivity of tests of treatment effects by accounting for some within-group variability. In data analysis, classificatory variables can simply be treated as additional factors. Methods for dealing with continuous covariates (i.e., analysis of covariance) will be discussed at some length in the analysis section of this chapter.

Advantages of factorial designs

How many independent variables should be examined in an experiment or quasi-experiment? Methodologists generally agree that factorial designs have several advantages over a series of single-factor studies (e.g., Maxwell and Delaney, 1990 ; Kirk, 1982 ). One major benefit of factorial experiments is their ability to detect the presence of interactions between factors. That is, factorial studies provide information about whether each variable operates in the same way across levels of the other variables. For example, a researcher interested in the effects of a new drug on depressive symptoms could use a factorial design to determine whether the drug affects subjects receiving therapy differently from those not receiving therapy. Even if no interactions are expected, a factorial design enhances generalizability of the findings because effects can be generalized across levels of the other factor. Finally, factorial experiments employ subjects more economically than a series of experiments focusing on one factor at a time. These advantages do not imply that investigators should try to design studies manipulating all factors that could possibly affect the outcome of interest. The high-order interactions created by four- or five-factor designs are usually too complex to be informative. Thus, designs with several manipulated factors (i.e., more than three) are uncommon in the behavioral sciences ( Maxwell and Delaney, 1990 ).

Between-subjects versus within-subjects designs

Most of the discussion of design thus far has centered on between-subjects factors, but very often experiments and quasi-experiments include factors that vary within subjects. For example, rather than exposing different groups to different treatment conditions, a researcher could apply two or more treatment conditions to the same subjects, counterbalancing the order of treatments between subjects (note that this type of design is desirable only when treatment effects are expected to be temporary). Other within-subjects designs involve comparing subjects’ scores on several different outcome variables, as in the case of profile analysis, or comparing subjects’ scores on the same variables measured repeatedly, as in pretest-posttest designs or longitudinal studies. The within-subjects approach may be adopted for several reasons. First and foremost, within-subjects designs are often used to explore research questions that between-subjects designs cannot address (e.g., questions about changes in individuals over time). In addition, participants are used more efficiently in within-subjects designs than in between-subjects designs; more information is gathered from each subject and the number of subjects required for a particular level of power is lower ( Maxwell and Delaney, 1990 ). Finally, within-subjects factors remove some variability due to individual differences among subjects from the error term, reducing the error variance and increasing power to detect experimental effects ( Vonesh, 1983 ; Winer, 1971 ). The most serious disadvantage of within-subject designs is the potential for differential carryover from repeated measures, which may bias estimates of treatment effects (e.g., Maxwell and Delaney, 1990 ). Thus, the effects of some factors are better assessed in between-subjects designs.

Analysis

Research questions about experimental and quasi-experimental data involve comparisons between experimental conditions on one or more outcome variables. Measures of individual performance are combined into measures of typical performance for each condition, and differences between conditions provide information about whether there is evidence for a cause and effect relationship between the independent and dependent variables. More generally, examination of the data involves measuring one or more dependent variables under conditions that are identified by one or more categorical variables. Thus, as mentioned earlier, the techniques for analyzing experimental and quasi-experimental data are also appropriate for examining nonexperimental data in which subjects are aggregated into groups according to one or more classification variables, such as age, gender, or disease diagnosis.

Data analysis can be partitioned into two phases, each of which is critical to the investigator’s scientific conclusions. The first phase is a preliminary descriptive phase in which the investigator becomes familiar with the data. Once the preliminary phase has been applied, formal inferential analyses are used to determine the likelihood that patterns identified in the descriptive analyses could have occurred by chance (i.e., the statistical significance of patterns). Each of these two phases will be discussed in turn.

Descriptive analyses

Preliminary descriptive analyses are essential to understanding the meaning of data. In this phase of analysis, the investigator constructs graphs or plots and calculates descriptive statistics to become familiar with data, looking for important or unexpected patterns. All too often, after completing data collection, investigators are eager to look for statistically significant effects and conduct formal inferential tests prematurely, before taking the time to familiarize themselves with their data. Yet the descriptive phase of analysis is as critical for drawing scientific conclusions as the formal inferential phase; indeed, it is difficult to interpret the results of inferential analyses without actually understanding what the data look like.

To become familiar with experimental and quasi-experimental data, investigators often construct histograms or stem and leaf plots to illustrate the frequency distribution of scores (see figures 6.1 and 6.2). These graphic displays provide information about the shape of the distribution, so that the investigator can examine the symmetry, modality, and peakedness of the distribution for departures from normality. For example, the frequency distribution in figure 6.1 shows a positively skewed distribution, suggesting that there might be a floor effect on the assessment instrument. In contrast, figure 6.2 is an illustration of an approximately normal distribution. As we will see below, the assumption that scores are normally distributed is required for the most common statistical procedures used to analyze continuous response variables. Frequency distributions should also be examined for deviations or outliers that fall well outside the overall pattern, as illustrated in figure 6 .1. In this case the researcher should search for explanations for the atypical response, and decide how to handle it ( Stevens, 1984 or Tukey, 1977 , for detailed discussions of the detection and treatment of outliers).

Figure 6.1 Illustration of a frequency distribution that is positively skewed and contains an outlier.

Figure 6.2Illustration of an approximately normal frequency distribution.

When research questions focus on differences between groups or experimental conditions the most commonly used descriptive statistics are the mean, which measures the central tendency of the scores, and the standard deviation, which measures the spread or variability of the scores. Means and standard deviations are ordinarily calculated and compared for each cell of the experimental design, as well as for the total sample. When more than one independent variable (or classification variable) is being investigated, marginal means may also be calculated. A marginal mean is the mean score for a particular level of one independent variable, averaged across levels of the other factor. Comparisons of marginal means for an independent variable provide information about the main effect of that variable. Comparisons of individual cell means provide information about interactions between independent variables. To illustrate, table 6.1 shows cell, marginal, and overall means for an artificial example of a study examining the effects of an exercise program on resting heart rate in male and female subjects, assuming equal cell sample sizes. Thus, two explanatory variables, gender and program, are considered here, each of which has two levels. Comparison of the marginal means for males and females suggests no differences according to gender. The marginal mean for the control program is somewhat higher than that for the intervention program. Comparison of the cell means suggests the presence of an interaction between gender and program; among females, average heart rate was lower in the intervention program than in the control program, whereas there appear to be no differences between programs among the males. Of course, it is not possible at this point to determine whether these differences are simply the result of chance variation. Inferential statistics must be used to determine whether the observed mean differences are statistically significant.

Table 6.1 Sample cell means for a 2-factor study
*Program*
*Gender*	*Control*	*Intervention*	*Marginal mean (gender)*
Females	Y11 = 90.3	Y21 = 79.9	Y.1 = 85.1
Males	Y12 = 83.8	Y22 = 84.4	Y.2 = 84.1
Marginal mean (program)	Y1. = 87.1	Y2. = 82.2	Y.. = 84.6

Inferential analyses

Once an investigator has explored patterns in the data, formal inferential tests are used to determine the extent to which these patterns could have arisen by chance. Experimental and quasi-experimental studies are designed to address questions about mean differences between treatment conditions in one or more response variables. Thus, the analysis of experimental or quasi-experimental data typically involves relating one or more categorical independent variables (e.g., an experimental treatment) to one or more continuous outcome variables. When all factors are manipulated between subjects, the most appropriate analysis strategy is an analysis of variance (ANOVA). The following sections begin with a discussion of the basic assumptions and principles of ANOVA, followed by an overview and illustration of the use of several types of ANOVA models and extensions of ANOVA, for dealing with continuous covariates or within-subjects factors.

Assumptions of ANOVA

ANOVA requires several assumptions that are associated with either the F statistic or linear models. First, it is assumed that all observations are random samples from the k populations or groups. Second, random errors of the observations are assumed to be independent and normally distributed within each population, with a mean of 0. Thus, the observations are also independent and normally distributed within each population. Finally, the variance of the random errors (and therefore the variance of the observations) is assumed to be the same in all populations.

Computational details

ANOVA partitions the variation in a dependent variable into different parts that are attributed to one of the explanatory variables or to random error. The variation attributable to random error is represented by the within-group variation, calculated as the sum of the squared deviations of the individual scores from the group means, or the sum of squares due to error (SSE). The SSE, divided by its corresponding degrees of freedom (the error mean square, or MSE), is an estimate of the random error or common population variance, s2. The variation associated with each explanatory variable is measured as the variability between treatment groups, calculated as the sum of squared deviations of treatment group means from the overall sample mean, or the sum of squares due to treatments (SST). When SST is large compared to SSE, we know that most of the variability in the dependent variable is due to differences between groups rather than differences within groups. The SST divided by its degrees of freedom is called the treatment mean square (MST), and under the null hypothesis of no group differences MST is also an estimate of the common population variance s2. When MST is much larger than MSE, the null hypothesis can be rejected. Thus, tests of the null hypothesis are constructed as a ratio of these two mean squares, MST/MSE, which has an F distribution under the null hypothesis. The probability of an F value at least as large as the one observed (the p value) can be calculated, providing a test of significance of group differences.

Table 6.2 presents the formulas for the calculation of variance components for a simple one-way ANOVA, in which a single classification variable is related to one dependent variable. Two subscripts are used in these formulas. The first subscript indicates the treatment group (i = 1st to kth group), whereas the second subscript indicates the individual observation within each group (j = 1stto observation). The calculations are only slightly more complicated when more than one explanatory variable is involved, provided there are equal numbers of observations in each cell. For example, the SST for each independent variable is indicated by the deviations of the marginal means, rather than the cell means, from the overall mean, and the sums of squares for interaction effects are indicated by deviations of the cell means from the overall mean, less the sums of squares components for the main effects.

Table 6.2 General one-way ANOVA table
*Source*	Df	SS	MS	F
Between	k-1		MST = SST/(k-1)	MST/MSE
Within	n-k		MSE = SSE/(n-k)
Total	n-1

ANOVA as a general linear model

The computational details described above correspond to the traditional sum of squares approach to analysis of variance. This approach to ANOVA was developed in the day of desk calculators and uses a formula simple enough for hand calculation. This approach is limited, however, because it cannot be used in two or more factor studies with unbalanced data 1 (i.e., data with unequal cell sizes). A more general analytic approach called a general linear model (GLM) can be used with balanced or unbalanced data, but requires computer software packages that can manipulate matrices. Although ANOVA and regression are traditionally viewed as distinct approaches to data analysis, both can actually be written as specific cases of the GLM. In the GLM framework a subject’s score on a dependent variable is defined as a linear combination of main effects, interaction effects, and an error. In the GLM approach to ANOVA, group means are estimated as linear combinations of regression parameters. Groups are defined by combinations of indicator variables that take the value of 0 or 1. For instance, a one-way ANOVA can be written by the use of a single indicator variable for each level of the categorical variable. To illustrate, if variable A has three levels, indicator variables are created as follows:

X1 = 1 for A = 1 = 0 for A = 2 or 3 X2 = 1 for A = 2 = 0 for A = 1 or 3 X3 = 1 for A = 3 = 0 for A = 1 or 2

The linear model for this example is:

yi = ß0 + ß1X1i + ß2X2i + ß3X3i

Variation is partitioned into a portion attributable to the regression model (or explanatory variables) and a portion due to random error. These variance components are a function of predicted values, calculated using least squares estimates of the regression parameters, and observed values. For example, in a model with one explanatory variable, the regression sum of squares (SSR) is calculated as the sum of squared deviations of the predicted values from the overall observed mean. Variation due to random error (SSE) is calculated as the sum of squared deviations of individual observed values from the predicted values. As in ANOVA, the error mean square in GLM is an estimate of the population variance, s2. When the null hypothesis is true, the regression mean square is also an estimate of s2; thus F-tests for the effects of explanatory variables are constructed as ratios of the regression and error mean squares.

When the data are balanced (i.e., when cell sizes are equal) the GLM approach and the traditional approach give identical solutions. The GLM approach is more flexible than the traditional approach because it can handle both balanced and unbalanced data and it can also be more readily extended to incorporate continuous explanatory variables and repeated measures. These characteristics, as well as the prevalence of computers for data analysis, make GLM the current preferred approach ( Kirk, 1982 ; Kleinbaum, Kupper, and Muller, 1988 ). It is often argued, however, that students should continue to learn about the traditional sum of squares approach to ANOVA because it elucidates many of the basic concepts of experimental design (e.g., Kirk, 1982 ; Collier and Hummel, 1977 ). Moreover, even when GLM machinery is used to predict a continuous outcome variable from one or more group variables, the term ANOVA continues to be used simply because it is a convenient way of conveying these characteristics of the analysis strategy. Indeed, in the remainder of this chapter, the analysis examples are conducted using GLM machinery, but the more traditional terms will be used to describe them.

One-way ANOVA

One-way ANOVA deals with the effect of a single between-subjects factor or variable on a single outcome variable. The independent variable can have two or more levels; thus a one-way ANOVA assesses differences in two or more population means. In the simplest case, a one-way ANOVA that compares two population means is equivalent to a two-sample t-test. If there are kpopulation means, the null hypothesis is that all k means are equal, whereas the alternative hypothesis is that all means are not equal.

To illustrate a one-way ANOVA, consider a recent study by Donnelly et al. (1996) , which examined the effectiveness of an intervention program in promoting physical fitness and preventing obesity among elementary school children. The investigation had a quasi-experimental design with a pretest, a posttest, and two intermediate observations. For two years, three cohorts of children (in the 3rd through 5th grades at the beginning of the study) at the intervention school participated in a program with enhanced physical activity, nutrition education, and a modified school lunch (low fat and sodium), whereas children in the control school continued with their regular school lunch and physical activity programs. Several measures of physical fitness, such as aerobic capacity, resting heart rate, lat pulls, bench presses, and knee extensions, were measured at the beginning and end of each school year.

Suppose Donnelly and his colleagues were interested in age differences in physical fitness as measured by lat pull scores at the initial assessment. To examine this issue, mean bench press scores would first be calculated for each grade. These data are shown in table 6.3, and as can be seen, lat pull scores seemed to increase with age. A one-way ANOVA could then be used to determine whether grade differences were statistically significant. Table 6.4 summarizes the results of a one-way ANOVA with grade as the classification variable and lat pulls as the dependent measure. As can be seen, the mean square for grade is larger than the error mean square. The resulting F ratio of 5.41 is significant at p= .006, indicating that lat pull scores do differ between the three age cohorts.

Table 6.3 Mean scores and standard deviations for lat pull scores (lbs.), by age
*Grade (at pretest)*	*Mean*	SD
3rd graders (n = 29)	52.7	8.2
4th graders (n = 47)	55.7	11.2
5th graders (n = 26)	61.5	10.2

Table 6.4 Results of one-way ANOVA examining age differences in lat pull scores
*Source*	Df	SS	MS	F
Grade	2	1114.3	557.1	5.41
Error	99	10201.8	103.0
Total	101	11316.0

The F ratio associated with the grade effect does not provide information about which grades performed differently from each other. When a factor has more than two levels, the overall F test is only a first step in analyzing the data. If the omnibus F test is significant, more specific mean comparisons or contrasts are then considered to determine which means are significantly different from each other. A contrast is defined by a set of known, ordered coefficients applied to the means such that at least two coefficients are nonzero, and all coefficients sum to zero . Some contrasts involve pairwise comparisons between means. For example, in the Donnelly et al. (1996) example, one might wish to determine whether the 3rd graders’ lat pull scores differ from those of the 4th graders. The null hypothesis for this contrast is Ho: u3rd = u4th, which can be rewritten as Ho: u3rd - u4th = 0. The contrast for testing this hypothesis is identified by the coefficients (-1 1 0). Other contrasts involve more complex comparisons. For instance, the coefficients (-1 ½ ½) could be used to determine whether 3rd graders’ lat pull scores differ from the average of 4th and 5th graders’ scores. If multiple follow-up tests are conducted, the researcher runs the risk of inflating the Type I error rate. In addition, some contrasts may have been hypothesized before the data were collected (a priori or planned comparisons), whereas others might arise after the data are examined (a posteriori or post hoc comparisons). When decisions about mean comparisons are made after examining the data, the testing procedure is further biased in favor of rejecting the null hypothesis because only comparisons that appear to be significant would be tested. A variety of procedures have been developed for controlling the Type I error rate when carrying out multiple tests of planned and post hoc contrasts.

Multiple comparison procedures

Each of the multiple comparison methods discussed here involves calculating an F statistic for each contrast and comparing it to a critical value. When all comparisons are planned in advance, the Type I error rate can be adjusted by reducing the significance level (e.g., α = .05) for each contrast by dividing it by the number of planned contrasts. This procedure is called the Bonferroni or Dunn procedure and can be used with simple pairwise comparisons or more complex contrasts. For example, if an investigator plans to test two contrasts, each would be tested at a significance level of α = .025. Tukey’s HSD (honestly significant difference) is designed specifically to test all pairwise comparisons, either planned or unplanned. The logic of this method is that each observed F is compared to a critical value (q2/2) selected such that the largest pairwise difference will exceed that value only 5 percent of the time when the null hypothesis is true. Although Tukey’s HSD is limited to situations in which sample sizes are equal across treatment levels, Tukey (1953) and Kramer (1956) proposed a modification to Tukey’s HSD for unequal cell sizes (the Tukey-Kramer method). This method has been shown to be more powerful than the Bonferroni method for testing pairwise comparisons. When more complex, post hoc comparisons are of interest, Scheffe’s method is recommended. Following a similar logic to Tukey’s HSD, Scheffe’s approach involves comparing observed F values for each contrast to a critical value of (k-1)F.05; k-1, n-k (where k is the number of groups), which ensures that the largest F value for any possible contrast in the data (pairwise or complex) will exceed that value only 5 percent of the time when the null hypothesis is true. Scheffe’s method should be used only when the investigator is interested in complex comparisons and when these contrasts are unplanned (e.g., Maxwell and Delaney, 1990 ). When only paired comparisons are of interest, Tukey’s HSD or the Tukey-Kramer extension are more powerful than Scheffe’s method. In the case of planned complex comparisons, Bonferroni’s procedure is typically more powerful than Scheffe’s method. Although the Bonferroni, Tukey, and Scheffe methods are the most commonly used multiple comparison procedures, several additional approaches have been suggested in the statistical literature. More comprehensive reviews of these procedures can be found elsewhere (e.g., Miller, 1981 ; Kirk, 1982 ).

To test all pairwise grade comparisons in lat pull scores for the Donnelly et al. (1996) example, the most appropriate procedure is the Tukey-Kramer adjustment. Table 6.5 shows the F values for each contrast and the associated p values according to the Tukey-Kramer criteria. As can be seen, 3rd graders differ from the 5th graders, whereas the 4th graders do not significantly differ from either the 3rd or 5th graders.

Two-way (and higher) ANOVA

Two-way ANOVAs are designed to assess the relations between two classification variables, each with two or more levels, and an outcome variable. Thus, factorial designs are typically analyzed with two-way or higher-way ANOVAs. This strategy can also be used with randomized block designs, with block considered an additional factor, although this method is less powerful than analysis of covariance ( Maxwell and Delaney, 1990 ; Kirk, 1982) . For the sake of simplicity, the discussion here will focus on two-factor studies, but these methods are easily generalized to multifactor studies.

Table 6.5 F and p values for pairwise grade comparisons
*Contrast*	*F(1,99)*	*Tukey-Kramer p value*
3rd vs. 4th graders	1.60	.419
4th vs. 5th graders	5.55	.053
3rd vs. 5th graders	10.48	.005

Hypothesis testing in two-way ANOVAs involves an assessment of the main effects of each factor as well as the interaction between the factors. Most statisticians recommend that the interaction effects be considered before main effects (e.g., Applebaum and Cramer, 1974 ). If the interaction is significant, simple effect tests should be carried out to assess the effect of each independent variable at each level of the other independent variable (more on simple effects below). If the interaction is not significant, the investigator should then test main effects. There are several methods for calculating the sums of squares for main effects in two-way ANOVAs. These methods yield equivalent results when cell sizes are equal. When cell sizes are unequal, the different methods can lead to dramatically different conclusions about the presence of main effects. Type I sums of squares, or “added in order” tests, test the main effect of the first factor entered in the model ignoring the effects of the other factor and the interaction. The test of the second factor allows for main effects of the first factor, but ignores the interaction. Type II sums of squares test the main effects of each factor allowing for the effect of the other factor, but ignoring the interaction effect. Finally, Type III sums of squares test the main effects of each factor allowing for all other specified effects, including the interaction. The Type I approach is not usually recommended because ignoring the effects of one factor defeats the purpose of a factorial design ( Maxwell and Delaney, 1990 ). The choice between Type II and Type III sums of squares is somewhat controversial (e.g., Cramer and Applebaum, 1980 ). The Type II method is more powerful than the Type III approach when the population interaction is zero. However, the statistical test of the interaction could fail to detect a true interaction in the population. As a result, many statisticians recommend using the Type III approach, unless there are strong theoretical arguments for a nonzero population interaction (e.g., Maxwell and Delaney, 1990 ; Kleinbaum, Kupper, and Muller, 1988 ). Most statistics programs (e.g., SAS and SPSS) automatically provide the results for Type III sums of squares.

In two-way and higher-way ANOVAs, main effects for multilevel factors may be investigated in the same manner as in one-way ANOVAs, using contrasts and the multiple comparison procedures described earlier. To further explore interactions, tests of simple effects are used to examine the effects of each factor at each level of the other factor. If a simple effect test is significant and the factor has more than two levels, one usually compares individual cell means. To control the Type I error rate, the Bonferroni adjustment is usually recommended ( Maxwell and Delaney, 1990 ; Kirk, 1982 ). For example, tests of factor A at b different levels of factor B are conducted using α = .05/b. Likewise, tests of factor B at a different levels of factor A are conducted using α = .05/a. In the physical fitness example, a two-way ANOVA could be used to examine the effects of both grade level and gender on children’s physical fitness measured by their initial lat pull scores. The mean scores shown in table 6.6 suggest that lat pulls increase with age, and that gender differences might vary across grades. Specifically, females’ scores were higher than those of males in the 3rd and 4th grades, but male scores were higher than those of females in the 5th grade, suggesting that there may be an interaction between gender and grade. Table 6.7 summarizes the results of the 2 × 3 ANOVA analyzing these data, using the Type III sums of squares. As can be seen, the F test for the interaction does not reach significance, making tests of simple effects unnecessary. As in the one-way ANOVA, a significant main effect of grade was observed. To further investigate this main effect, pairwise grade comparisons were conducted using the Tukey-Kramer method. These tests indicated that both 3rd and 4th graders scored significantly lower than 5th graders, Fs(1, 96) > 8.35, ps < .013.

Table 6.6 Mean scores (and standard deviations) for lat pull scores (lbs.), by age and gender
*Grade (at pretest)*	*Females*	*Males*
3rd graders	54.9 (8.3)	50.0 (7.4)
4th graders	57.2 (9.3)	53.3 (13.6)
5th graders	59.1 (8.5)	66.1 (11.9)

Table 6.7 Results of a two-way ANOVA examining grade and gender differences in lat pull scores
*Source*	df	*Type III SS*	MS	F	p
Grade	2	1421.05	710.53	7.12	.0013
Gender	1	7.46	7.46	0.07	.7851
Grade × gender	2	567.19	283.59	2.84	.0632
Error	96	9579.68	99.79
Total	101	11316.05

Analysis of covariance (ANCOVA)

Sometimes an investigator wishes to control for extraneous variables when assessing the relation between an independent variable and a response. One way to control for such a variable is to include it in the analysis model, so that the effects of the independent variable are adjusted for the presence of the control variable, or covariate, in the model. The usual statistical technique for carrying out this adjustment process is called an analysis of covariance (ANCOVA). In the typical analysis of covariance, one of the predictors is continuous and the other is categorical. Separate parallel regression lines are estimated for each level of the classification variable. Thus, ANCOVA assumes equivalent slopes for all groups, or that there is no interaction between the classification variable and the continuous covariate. Main effects of the classification variable are represented as intercept differences, or differences in the relative levels of the regression lines. More complex ANCOVA models can include more than one factor and more than one covariate ( Maxwell, Delaney, and O’Callaghan, 1993 , for a discussion of these more complex models).

ANCOVA is also the preferred method for analyzing data from randomized block designs ( Feldt, 1958 ; Maxwell and Delaney, 1990 ). In contrast to the ANOVA approach in which block is treated as a factor, the ANCOVA approach uses all the quantitative information in the covariate or blocking variable, and consumes fewer degrees of freedom. ANCOVA can also be used to analyze pretest-posttest data, as long as the analysis of change from pretest to posttest is not of interest to the investigator. In the ANCOVA approach, group differences in posttest scores are assessed with the pretest score included as a covariate. This method is highly recommended for the analysis of pretest-posttest data because it is usually more powerful than the alternative method of analyzing change scores ( Huck and McLean, 1975 ; Maxwell, Delaney, and Dill, 1984 ).

Most methodologists recommend that investigators test the assumption of parallelism of regression lines before proceeding with the standard ANCOVA strategy (e.g., Kleinbaum, Kupper, and Muller, 1988 ; Maxwell and Delaney, 1990 ). The test of parallelism, or homogeneity of slopes, is essentially a test of the interaction between the covariate and the classification variable to determine whether the relation between the covariate and the dependent variable differs at different levels of the classification variable. The model used to test the parallelism hypothesis is referred to by a variety of names, including ANCOVA with heterogeneous slopes or the heterogeneous regressions model. Maxwell and Delaney (1990) suggest that the heterogeneous regressions model be used if the test of the interaction term approaches significance (p < .20) or if there are theoretical reasons to suspect heterogeneity of regressions. If neither of these two conditions applies, the investigator should proceed with the standard ANCOVA assuming parallel slopes.

When the heterogeneous regressions model applies, the interpretation of the main effect of the classification variable is complicated, especially if the covariate is considered a nuisance variable by the investigator. In ANCOVA the main effect of the classification variable is represented by the distance between the regression lines; if the regression lines are not parallel, however, this distance varies continuously as a function of the covariate. How should this distance be estimated? There are two ways to address this problem. One approach, elaborated by Rogosa (1980) , involves selecting a single point along the covariate dimension at which to test the main effect of the classification variable. The estimate will be most accurate when the selected point is at the center of the covariate’s distribution. The alternative approach involves testing for main effects at multiple points along the covariate’s distribution to identify regions in which treatment effects are significant ( Potthoff, 1964 , developed a method for simultaneously conducting these tests so as to control the Type I error rate). This procedure is recommended when a researcher is interested in the covariate itself, or when the regression lines intersect. The approach is especially useful when a researcher is using the covariate to identify ranges within which particular treatments will be effective. The reader is advised to consult Neter, Wasserman, and Kutner (1985) , Maxwell, Delaney, and O’Callaghan (1993) , or Maxwell and Delaney (1990) for further details about these procedures.

To demonstrate these procedures, ANCOVA was used with the Donnelly et al. (1996) physical fitness data to assess the effects of the intervention program controlling for pretest fitness levels. Specifically, group (intervention versus control) and pretest bench press scores were used to predict bench press scores at the final assessment. An initial test of the interaction between group and pretest bench press scores was not significant, F (1, 98) = 0.05, p= .82, thus a standard ANCOVA model was examined.

Figure 6.3 Estimated posttest bench press scores as a function of group, adjusting for pretest bench press scores (data from Donnelly et al., 1996).

The results of this model are shown in table 6.8. As can be seen, both group and pretest scores were related to posttest bench press scores. To further illustrate the nature of the effects, the slope and intercept estimates from this model were used to plot the regression lines for each group (see figure 6.3). The distance between the two lines indicates that children in the intervention group had higher bench press scores at the end of the program than children in the control group. The slope of the lines indicates that in both groups, individuals with higher baseline bench press scores had higher posttest scores as well.

Multivariate analysis of variance (MANOVA)

When more than one measurement is taken on the same subjects, the measurements are often correlated with each other. To account for these correlations it often makes sense to use a multivariate analysis strategy. MANOVA is used when the investigator measures more than one dependent variable, each representing qualitatively different constructs that share some conceptual meaning. For example, MANOVA is appropriate for analyzing several different measures of physical fitness, such as lat pull scores, aerobic capacity, and bench press scores. In the standard MANOVA the investigator is interested in jointly testing the effects of one or more between-subjects classification variables on the set of outcome measures. The procedure is a generalization of ANOVA in which linear combinations of the dependent variables are formed to maximize the differences between the levels of the independent variable. The resulting composite scores, rather than scores on the separate dependent measures, are tested for significance. There are several reasons for considering a MANOVA rather than a series of separate univariate ANOVAs ( Stevens, 1992 ; Harris, 1993 ). When there is a weak treatment effect across several outcome variables, MANOVA has greater power for detecting the effect than single ANOVAs. In addition, the MANOVA approach reduces the probability of a Type I error when the dependent variables are correlated.

Table 6.8 Results of ANCOVA examining group differences in posttest bench press scores, controlling for pretest scores
*Source*	df	*Type III SS*	MS	F	p
Group	1	1315.98	1315.98	16.14	.0001
Pretest bench press	1	3748.60	3748.60	45.97	.0001
Error	99	8072.78	81.54
Total	101	14044.10

When the multivariate test of a between-subjects effect is significant, several follow-up tests may be of interest. For example, when the factor has more than two levels, contrasts can be used to determine which groups differ from one another at the multivariate level. In addition, an investigator will often want to determine which dependent variables contribute to multivariate significance, most often by examining univariate tests of each dependent variable. Some methodologists argue that multiple comparison procedures for controlling Type I error are not necessary in multivariate analyses because the overall multivariate test provides adequate protection (e.g., Hummel and Sligo, 1971 ). Others suggest the Bonferroni adjustment for multiple planned comparisons and pairwise post hoc comparisons (e.g., Stevens, 1992 ; Maxwell and Delaney, 1990 ; Timm, 1975 ). For complex post hoc comparisons a multivariate extension of Scheffe’s method developed by Roy and Bose (1953) is sometimes recommended for the selection of the appropriate critical value (e.g., Morrison, 1976 ; Harris, 1993 ). This method controls Type I error rate for all possible pairwise and complex group comparisons for each dependent variable, as well as for different linear combinations of the variables. As such, this procedure allows for unrestricted analysis, but is highly conservative ( Hummel and Sligo, 1971 ; Stevens, 1992 ). For additional detail about multiple comparison procedures in MANOVA, the reader should refer to Stevens (1992) or Harris (1993) .

Suppose Donnelly et al. (1996) were interested in assessing grade and gender differences in baseline scores on several indicators of physical fitness. To address this question, a MANOVA was used to examine the joint effects of grade and gender on a set of four outcome measures: aerobic capacity, knee extensions, lat pulls, and bench presses. The multivariate test statistics for this analysis are summarized in table 6.9. As shown in the table, the MANOVA revealed significant multivariate main effects of both grade and gender. To further investigate the grade effect, pairwise multivariate tests were conducted. Even with the significance level adjusted according to the Bonferroni procedure (a = .05/3 = .017), all three contrasts were significant, Fs(4, 93) > 4.95, ps < .0012. Univariate tests of these differences were then examined to determine which dependent measures contributed to the pairwise multivariate significance, adjusting the significance level for the number of dependent variables (a = .05/4 = .0125). These tests indicated that the difference between 3rd graders and 4th graders was limited to knee extensions, F (1, 96) = 18.06, p= .0001. Fifth graders outperformed 3rd graders on both knee extensions and lat pulls, Fs (1, 96) > 13.49, ps < .0004. Fifth graders also outperformed 4th graders on knee extensions and lat pulls, Fs(1, 96) > 8.38, ps < .0047. The contributions of the four dependent measures to the significant multivariate gender effect were also examined through univariate tests, which indicated that males outperformed females in aerobic capacity and bench press scores, Fs (1, 96) > 14.53, ps < .0002.

Table 6.9 Results of MANOVA examining grade and gender differences in four physical fitness measures (aerobic capacity, knee extensions, lat pulls, and bench presses)
*Source*	*Num df*	*Den df*	F	p
Grade	8	186	6.95	.0001
Gender	4	93	8.99	.0001
Grade × gender	8	186	1.05	.40

Repeated measures ANOVA

Like MANOVA, a repeated measures ANOVA is used to analyze multiple measurements on the same subjects. The repeated measures approach is typically used to examine multiple measurements on the same outcome variable. In the simplest repeated measures ANOVAs, the researcher is interested in within-subjects differences in the repeated measure. In more complex repeated measures ANOVAs, between-subjects effects of one or more classification variables, as well as interactions between the within-subjects and between-subjects variables, are also of interest. In contrast, in MANOVA the investigator is primarily interested in between-subjects effects, tested jointly across a set of dependent measures. Repeated measures ANOVAs are recommended for investigations in which change over time is of particular interest, such as longitudinal or developmental studies, studies of learning and memory, and some pretest-posttest studies.

There are two approaches to repeated measures ANOVAs. The univariate approach involves blocking the data by subject, and treating block as a factor in a univariate ANOVA to account for individual differences in subjects’ responses to the repeated measure. Thus, the univariate repeated measures ANOVA will always have at least two factors, one representing subjects and the other representing the within-subjects or repeated measures variable (e.g., time). The univariate approach requires an assumption of sphericity, or homogeneity of the treatment difference variances. In other words, it is assumed that all pairs of levels of the within-subjects variable have equal correlations. This assumption, however, is frequently violated; measurements taken close in time are often more highly correlated than measurements taken farther apart in time. Several adjusted univariate tests have been developed to handle violations of the sphericity assumption, including the Geisser-Greenhouse lower bound correction, the Box adjustment, and the Huynh-Feldt adjustment ( Stevens, 1992 , for greater detail). Alternatively, an investigator could choose to use the multivariate approach to repeated measures ANOVAs, which require no sphericity assumption. The multivariate strategy involves transforming the dependent variables into a set of difference scores and testing the hypothesis that these scores equal zero. Between-subjects effects are measured by averaging across levels of the within-subjects variable. When the within-subjects variable has only two levels, the univariate and multivariate approaches provide identical results. When sphericity holds, the univariate approach is the more powerful strategy; when sphericity is violated, the choice between the adjusted univariate procedures and the multivariate approach is complicated . Some methodologists (e.g., Cole and Grizzle, 1966 ; Maxwell and Delaney, 1990 ) recommend routine use of the multivariate approach for theoretical reasons (e.g., because repeated measures data are essentially multivariate in nature).

In repeated measures ANOVAs, significant overall tests for within-subjects effects, between-subjects effects, and their interactions can be followed up with a variety of more specific comparisons, depending on the hypotheses of interest. Comparisons involving between-subjects effects with more than two levels can be tested with contrasts that average over scores on the within-subject factor. The multiple comparison procedures recommended for univariate ANOVAs are also appropriate for between-subjects comparisons in the repeated measures case ( Maxwell and Delaney, 1990 ). Contrasts may also be used to determine the nature of within-subject effects. For example, the investigator might want to test for linear, quadratic, and cubic trends in the repeated measure ( Morrison, 1976 , for more details about trend analysis). One way to explore a significant interaction between within- and between-subjects variables is to test the simple effect of the between-subjects variable at each level of the within-subjects variable. For instance, the researcher could examine the univariate one-way ANOVAs of the between-subjects variable at each level of the within-subjects variable. The Bonferroni adjustment is recommended to control the Type I error rate when testing multiple planned contrasts or unplanned pairwise comparisons involving within-subjects effects. For post hoc complex comparisons, the Roy and Bose method is preferred (e.g., Morrison, 1976 ; Harris, 1993 ).

In the physical fitness example, a multivariate repeated measures ANOVA was used to assess change over time in physical fitness scores as a function of group (intervention versus control). The model included the four-level within-subjects variable of time, the between-subjects variable group, and the interaction between time and group. As shown in table 6.10, all three effects were significant, indicating that the effect of group varied across time. To explore the nature of this interaction, univariate tests of the group effect were conducted at each level of time. These analyses revealed that the intervention group had higher bench press scores than the control group at the final assessment, F (1, 97) = 20.78, p < .0001, but not at the earlier three time points. Thus, these findings suggest that only after two academic years did the intervention program lead to significant strength gains relative to the control group.

Table 6.10 Results of a repeated measures ANOVA examining group differences in bench press scores over time
*Source*	*Num df*	*Den df*	F	p
Group	1	97	10.92	.0013
Time	3	95	18.83	.0001
Group × time	3	95	4.29	.007

Conclusions

The goal of this chapter is to provide an overview of common experimental and quasi-experimental designs and the statistical analysis strategies most often associated with them. The information presented here is by no means exhaustive; a multitude of more complex extensions of these methods are possible. Kirk (1982) provides in-depth coverage of a variety of experimental designs for the behavioral sciences. More complex designs than those described here can usually be analyzed using some form of the general linear model. These strategies are likely to be generalizations of those presented here, which are also based on the general linear model. The interested reader is referred to Graybill (1961) or Kirk (1982) for further information about the general linear model.

Notes

1 The method of unweighted means is an adaptation of the traditional sum of squares approach for unbalanced data, but this technique provides test statistics that only approximate F statistics under the null hypothesis.

References

· APA

· Chicago

· Harvard

· MLA

CHAPTER 6 - Design and analysis of experimental and quasi-experimental investigations. (2005). In M. C. Roberts, & S. S. Ilardi (Eds.), Blackwell handbook of research methods in clinical psychology. Oxford, UK: Blackwell Publishers. Retrieved from https://login.proxy1.ncu.edu/login?url=https://search.credoreference.com/content/entry/bkhrmcp/chapter_6_design_and_analysis_of_experimental_and_quasi_experimental_investigations/0?institutionId=1633