The instruction to follow is in the browse files. 0 plagiarism. Please zero plagiarism

Cosmo Condina/Stone/Getty Images

chapter 5

Experimental Designs— Determining Cause-and-Effect

Relationships

Chapter Contents

• Experiment Terminology • Key Features of Experiments • Experimental Validity • Experimental Designs • Analyzing Experiments • Wrap-Up: Avoiding Error • Critiquing a Quantitative Study • Mixed Methods Research Designs

CO_

CO_

new85743_05_c05_213-280.indd 213 6/18/13 12:53 PM

214

CHAPTER 5Introduction

One of the oldest debates within psychology concerns the relative contributions that biology and the environment make in shaping our thoughts, feelings, and behaviors. Do we become who we are because it is hard-wired into our DNA or in response to early experiences? Do people take on their parents’ personality quirks because they carry their parents’ genes or because they grew up in their parents’ homes? There are, in fact, several ways to address these types of questions. In fact, a consortium of researchers at the University of Minnesota has spent the past 2 decades comparing pairs of identical and fraternal twins to tease apart the contributions of genes and environment. You can read more at the research group’s website, Minnesota Center for Twin and Family Research, http://mctfr.psych.umn.edu/.

An alternative to using twin pairs to separate genetic and environmental influence is through the use of exper- imental designs, which have the pri- mary goal of explaining the causes of behavior. Recall from Chapter 2 (Section 2.1, Overview of Research Designs) that experiments can speak to cause and effect because the exper- imenter has control over the envi- ronment and is able to manipulate variables. One particularly ingenious example comes from the laboratory of Michael Meaney, a professor of psychiatry and neurology at McGill University, using female rats as experimental subjects (Francis, Dioro, Liu, & Meaney, 1999). Meaney’s research revealed that the parenting ability of female rats can be reliably classified based on how attentive

they are to their rat pups, as well as how much time they spend grooming the pups. The question tackled in this study was whether these behaviors were learned from the rats’ own mothers or transmitted genetically. To answer this question experimentally, Meaney and colleagues had to think very carefully about the comparisons they wanted to make. It would have been insufficient to simply compare the offspring of good and bad moth- ers—this approach could not distinguish between genetic and environmental pathways.

Instead, Meaney decided to use a technique called cross-fostering, or switching rat pups from one mother to another as soon as they were born. This resulted in four combinations of rats: (1) those born to inattentive mothers but raised by attentive ones, (2) those born to attentive mothers but raised by inattentive ones, (3) those born and raised by attentive mothers, and (4) those born and raised by inattentive mothers. Meaney then tested the rat pups several months later and observed the way they behaved with their own offspring. The setup of this experiment allowed Meaney to make clear comparisons between the influence of birth mothers and the rearing process. At the end of the study, the conclusion was crystal clear: Maternal behavior is all about the environment. Those rat pups that ultimately grew up to be inattentive mothers were those who had been raised by inatten- tive mothers.

Creatas Images/Thinkstock

Researchers at the University of Minnesota work with twins in order to study the impact of genetics versus upbringing on personality traits.

TX_

TX

new85743_05_c05_213-280.indd 214 6/18/13 12:53 PM

215

CHAPTER 5Section 5.1 Experiment Terminology

This final chapter is dedicated to experimental designs, in which the primary goal is to explain behavior. Experimental designs rank highest on the continuum of control (see Figure 5.1) because the experimenter can manipulate variables, minimize extraneous vari- ables, and assign participants to conditions. The chapter begins with an overview of the key features of experiments and then covers the importance of both internal and external validity of experiments. From there, the discussion moves to the process of designing and analyzing experiments and a summary of strategies for minimizing error in experiments. It concludes with guidelines for critiquing a quantitative study.

Figure 5.1: Experimental designs on the continuum of control

5.1 Experiment Terminology

Before we dive into the details, it is important to cover the terminology that we will use to describe different aspects of experimental designs. Much of this will be famil-iar from previous chapters, with a few new additions. First, let’s review the basics.

Recall that a variable is any factor that has more than one value. For example, height is a variable because people can be short, tall, or anywhere in between. Depression is a variable because people can experience a wide range of symptoms, from mild to severe. The independent variable (IV) is the variable that is manipu- lated by the experimenter in order to test hypotheses about cause. The dependent variable (DV) is the variable that is measured by the experimenter in order to assess the effects of the independent variable. For example,

Increasing Control . . .Increasing Control . . .

• Ethnographic Study • Phenomenological Study • Grounded Theory Study • Case Study • Archival Research • Observational Research

Qualitative and Descriptive Methods

• Survey Research

Predictive Methods

• Pre-experiments • Quasi-experiments • “True” Experiments

Experimental Methods

Flying Colours Ltd/Photodisc/Thinkstock

A variable is any factor that has more than one value, such as height.

new85743_05_c05_213-280.indd 215 6/18/13 12:53 PM

216

CHAPTER 5Section 5.2 Key Features of Experiments

in an experiment testing the hypothesis that fear causes prejudice, fear would be the inde- pendent variable and prejudice would be the dependent variable. To keep these terms straight, it is helpful to think of the main goal of experimental designs. That is, we test hypotheses about cause by manipulating an independent variable and then looking for changes in a dependent variable. Thus, our independent variable causes changes in the dependent variable; for example, fear is hypothesized to cause changes in prejudice.

Any manipulation of independent variables results in two or more versions of the vari- able. One common way to describe the versions of the independent variable is in terms of different groups, or conditions. The most basic experiments have two conditions: The experimental condition receives treatment designed to test the hypothesis, while the control condition does not receive this treatment. In our fear and prejudice example, the participants who make up the experimental condition would be made to feel afraid, while the participants who make up the control condition would not. This setup allows us to test whether introducing fear to one group of participants leads them to express more prejudice than the other group of participants, who are not made fearful.

Another common way to describe these versions is in terms of levels of the indepen- dent variable. Levels describe the specific set of circumstances created by manipulat- ing a variable. For example, in the fear and prejudice experiment, the variable of fear would have two levels—afraid and not afraid. There are countless ways to introduce fear into the experiment. One option would be to adopt the technique used by the Stan- ford social psychologist Stanley Schachter (1959), who led participants to believe they would be exposed to a series of painful electric shocks. In Schachter ’s study, the painful shocks never happened, but they did induce a fearful state as people anticipated them. So those at the “afraid” level of the independent variable might be told to expect these shocks, while those at the “not afraid” level of the independent variable would not be given this expectation.

At this stage, it may seem odd to have two sets of vocabulary terms—“levels” and “condi- tions”—for the same concept. However, there is a subtle difference in how these terms are used once we get into advanced experimental designs. As the designs become more com- plex, it is often necessary to expand independent variables to include several groups and multiple variables. Once this happens, we will need different terminology to distinguish between the versions of one variable and the combinations of multiple variables. We will return to this complexity later in the chapter, in Section 5.4, Experimental Designs.

5.2 Key Features of Experiments

The overview of research designs (Chapter 2, Section 2.1) described the overall pro-cess of experiments in the following way: A researcher controls the environment as much as possible so that all participants have the same experience. He or she then manipulates, or changes, one key variable, and then measures the outcomes in another key variable. In this section, we will examine this process of control in more detail. Experi- ments can be distinguished from all other designs by three key features: manipulating variables, controlling the environment, and assigning people to groups comprising exper- imental and control conditions.

new85743_05_c05_213-280.indd 216 6/18/13 12:54 PM

217

CHAPTER 5Section 5.2 Key Features of Experiments

Manipulating Variables

The most crucial element of an experiment is that the researcher must manipulate, or change, some key variable. To study the effects of hunger, for example, a researcher could manipulate the amount of food given to the participants. Or, to study the effects of tem- perature, the experimenter could raise and lower the temperature of the thermostat in the laboratory. Because these factors are under the researchers’ direct control, they can feel more confident that changing them contributes to changes in the dependent variables.

In Chapter 2 we discussed the main shortcoming of correlational research: These designs do not allow us to make causal statements. As you’ll recall from that chapter (as well as from Chapter 4), correlational research is designed to predict one variable from another.

One of the examples in Chapter 2 concerned the correlation between income levels and happiness, with the goal of trying to predict happiness levels based on knowing people’s income level. If we measure these as they occur in the real world, we cannot say for sure which variable causes the other. However, we could settle this question relatively quickly with the right experiment. Let’s say we bring two groups into the laboratory, give one group $100 and a second group nothing. If the first group were happier at the end of the study, this would support the idea that money really does buy happiness. Of course, this is a rather simplistic look at the connection between money and happiness, but because it manipulated levels of money, this study would bring us closer to making causal state- ments about the effects of money.

To manipulate variables, it is necessary to have at least two versions of the variable. That is, to study the effects of money, we need a comparison group that does not receive money. To study the effects of hunger, we would need both a hungry and a not-hungry group. Hav- ing two versions of the variable distinguishes experimental designs from the structured observations discussed in Chapter 3 (Observational Research), in which all participants received the same set of conditions in the laboratory. Even the most basic experiment must have two sets of conditions, which are often an experimental group and a control group. But, as we will see later in this chapter, experiments can become much more complex. You might have one experimental group and two control groups, or five degrees of food deprivation, ranging from 0 to 12 hours without food. Your decisions about the number and nature of these groups will depend on consideration of both your hypotheses and previous literature.

When it comes to the manipulation of variables, there are three options available. First, environmental manipulations involve changing some aspect of the setting. Environ- mental manipulations are perhaps the most common in psychology studies, and they include everything from varying the temperature to varying the amount of money people receive. The key is to change the way different groups of people experience their time in the laboratory—it is either hot or cold, and they either receive or don’t receive $100. Sec- ond, instructional manipulations involve changing the way a task is described in order to change participants’ mind-sets. For example, you could give all participants the same math test but describe it as an intelligence test for one group and a problem-solving task for another. Because an intelligence test is thought to have implications for life success, you might expect participants in this group to be more nervous about their scores. Finally, an invasive manipulation involves taking measures to change internal, physiological

new85743_05_c05_213-280.indd 217 6/18/13 12:54 PM

218

CHAPTER 5Section 5.2 Key Features of Experiments

processes and is usually conducted in medical settings. For example, studies of new drugs involve administering the drug to volunteers to determine whether it has an effect on some physical or psychological symptom. Or, for example, studies of cardiovascular health often involve having participants run on a treadmill to measure how the heart functions under stress.

Finally, there is one qualification to the rule that we must manipulate a variable. In many experi- ments, researchers divide up participants based on an inherent difference (e.g., gender) or person- ality measures (e.g., self-esteem or neuroticism) that capture stable individual characteristics among people. The idea behind these personal- ity measures is that someone scoring high on a measure of neuroticism (for example) would be expected to be more neurotic across situations than someone scoring lower on the measure. Using this technique allows us to compare how, for example, men and women, or people with high and low self-esteem, respond to manipula- tions. When existent differences are used in an experimental context, they are referred to as quasi- independent variables—“quasi,” or “nearly,” because they are being measured, not manipu- lated, by the experimenter, and thus do not meet the criteria for a regular independent variable. Because these variables are not manipulated, an experimenter cannot make causal statements about them. In order for a study to count as an experiment, these quasi-independent variables would have to be combined with a true indepen- dent variable. This could be as simple as com-

paring how men and women respond to a new antidepressant drug—gender would be quasi-independent while drug type would be a true independent variable.

Controlling the Environment

The second important element of experimental designs is that the researcher has a high degree of control over the environment. In addition to manipulating variables, a researcher conducting an experiment ensures that the other aspects of the environment are the same for all participants. For instance, if you were interested in the effects of temperature on people’s mood, you could manipulate temperature levels in the laboratory so that some people experienced warmer temperatures and other people cooler temperatures. But it would be equally important to make sure that other potential influences on mood were the same for both groups. That is, you would want to make sure that the “warm” and “cool” groups were tested in the same room, around the same time of day, and by the same experimenter.

iStockphoto/Thinkstock

Medical studies often use invasive manipulation to change internal, physiological processes.

new85743_05_c05_213-280.indd 218 6/18/13 12:54 PM

219

CHAPTER 5Section 5.2 Key Features of Experiments

The overall goal, then, is to control extraneous variables, or variables that add noise to your hypothesis test. In essence, the more you are able to control extraneous variables, the more confidence you can have in the results of your hypothesis test. As we will discuss in the section “Experimental Validity,” the impact of extraneous variables can vary in a study. Let’s say we conduct the study on temperature and mood, and all of our partici- pants are in a windowless room with a flickering fluorescent light. This would likely have an influence on mood—making everyone a little bit grumpy—but cause few problems for our hypothesis test because it would affect everyone equally. Table 5.1 shows hypothetical data from two variations of this study, using a 10-point scale to measure mood ratings. In the top row, participants were in a well-lit room; we can see that participants in the cooler room reported being in a better mood (i.e., an 8 versus a 5). In the bottom row, all participants were in the windowless room with flickering lights. These numbers suggest that people were still in a better mood in the cooler room (5) than a warm room (2), but the flickering fluorescent light had a constant dampening effect on everyone’s mood.

Table 5.1: Influence of an extraneous variable

Cool Room Warm Room

Variation 1: Well-Lit 8 5

Variation 2: Flickering Fluorescent 5 2

Assigning People to Conditions

The third key feature of experimental designs is that the researcher can assign people to receive different conditions, or versions, of the independent variable. This is an important piece of the experimental process: The experimenter not only controls the options—warm versus cool room; $100 versus no money, etc.—he or she also gets to control which par- ticipants get each option. Whereas a correlational design might assess the relationship between current mood and choosing the warm room, an experimental design will have some participants assigned to the warm room and then measure the effects on their mood. In other words, an experimenter is able to make causal statements because that person causes things to happen.

The most common, and most preferable, way to assign people to conditions is through a process called random assignment. An experimenter who uses random assignment makes a separate decision for each participant as to which group he or she will be assigned to before the participant arrives. As the term implies, this decision is made randomly—by flipping a coin, using a random number table (for an example, see http://stattrek.com /tables/random.aspx), drawing numbers out of an envelope, or some other random pro- cess. The overall goal is to try to balance out existent differences among people, as illus- trated in Figure 5.2. So, for example, some people might generally be more comfortable in warm rooms, while others might be more comfortable in cold rooms. If each person who shows up for the study has an equal chance of being in either group, then the groups in the sample should reflect the same distribution of differences as the population.

new85743_05_c05_213-280.indd 219 6/18/13 12:54 PM

220

CHAPTER 5Section 5.2 Key Features of Experiments

Figure 5.2: Random assignment

The 25 participants in our sample consist of a mix of happy and sad people. The goal of random assignment is to have these differences distributed equally across the experimental conditions. Thus, the two groups on the right each consist of six happy and six sad people, and our random assignment was successful.

Another significant advantage of forming groups through random assignment is that it helps to avoid bias in the selection and assignment of subjects. For example, it would be a bad idea to assign people to groups based on a first impression of them because partici- pants might be placed in the cold room if they arrived at the laboratory dressed in warm clothing. Experimenters who make decisions about condition assignments ahead of time can be more confident that the independent variable is responsible for changes in the dependent variable.

It is worth highlighting the difference here between random selection and random assign- ment (discussed in Chapter 4, Section 4.3, Sampling From the Population). Random selec- tion means that the sample of participants is chosen at random from the population, as with the probability sampling methods discussed in Chapter 4. However, most psychol- ogy experiments use a convenience sample of individuals who volunteer to complete the study. This means that the sample is often far from fully random. However, a researcher can still make sure that the group assignments are random so that each condition contains an equal representation of the sample.

In some cases—most notably, when samples are small—random assignment may not be sufficient to balance an important characteristic that might affect the results of a particular study. Imagine conducting a study that compared two strategies for teaching students complex math skills. In this example, it would be especially important to make sure that both groups contained a mix of individuals with, say, average and above-average intelli- gence. For this reason, it would be necessary to take extra steps to ensure that intelligence was equally distributed among the groups, which can be accomplished with a variation on random assignment called matched random assignment. This requires the experi- menter to obtain scores on an important matching variable (in this case, intelligence), rank participants based on the matching variable, and then randomly assign people to condi- tions. Figure 5.3 shows how this process would unfold in our math skills study. First, par- ticipants are given an IQ test to measure existing differences in intelligence. Second, the experimenter ranks participants based on these scores, from highest to lowest. Third, the experimenter would move down this list in order and randomly assign each participant

new85743_05_c05_213-280.indd 220 6/18/13 12:54 PM

221

CHAPTER 5Section 5.2 Key Features of Experiments

to one of the conditions. This process still contains an element of random assignment, but adding the extra step of rank ordering ensures a more balanced distribution of intelligence test scores across the conditions.

Figure 5.3

The 20 participants in our sample represent a mix of very high, average, and very low intelligence test scores (measured 1–100). The goal of matched random assignment is to ensure that this variation is distributed equally across the two conditions. The experimenter would first rank participants by intelligence test scores (top box) and then distribute these participants alternately between the conditions. The result is that both groups (lower boxes) contain a good mix of high, average, and low scores.

9 9 9 8 9 7 9 6 9 6 9 4

9 3 8 0 7 9 7 8 7 7 7 6

7 5 74 7 3 5 0 4 9 4 8

4 7 4 6

9 9 9 7

9 5 9 3

7 9 7 7

7 5 7 3

4 9 4 7

9 8 9 6

9 4 8 0

7 8 7 6

74 7 5

4 8 4 6

Group A Group B

Research: Making an Impact

The Stanford Prison Experiment

The landmark 1971 Stanford Prison Experiment had an extremely widespread impact on multiple fields and real-life settings such as prison reform, the ethics of human research, and terrorism and torture tactics (Clements, 1999; Haney & Zimbardo, 1998). The study, conducted by Phillip Zimbardo and his colleagues at Stanford University, placed volunteer participants in a simulated prison envi- ronment and randomly assigned them to play the roles of “guards” and “prisoners.” These 24 par- ticipants had been selected based on their personality traits that marked them as “good apples” and who had no previous evidence of antisocial behavior (Haney, Banks, & Zimbardo, 1973). Neverthe- less, the simulated prison quickly took on the characteristics of a real prison, with simulated situa- tions of dominance, dehumanization, severe psychological distress, and the unexpected phenomena of social control, obedience, and effects of power (Haney, 2002; Banuazizi & Movahedi, 1975). The prisoners were subjected to humiliation similar to what has been seen in prison scandals such as Abu Ghraib: nakedness, sexual humiliation, verbal torment, chains, and bags over prisoners’ heads. Although planned to run for 2 weeks, the experiment was stopped after only 6 days due to the severe psychological harm it was causing the prisoners and the unexpected behavior of the prison guards.

(continued)

new85743_05_c05_213-280.indd 221 6/18/13 12:54 PM

222

CHAPTER 5Section 5.3 Experimental Validity

Research: Making an Impact (continued) The study led to major reform in the ethical guidelines for psychological research and humane treat- ment, and in fact has never been replicated in a scientific setting. Following publication of this experi- ment, the Supreme Court saw an influx of cases regarding prisoner treatment and the structure of the prison system. In an interesting twist, one of the original researchers, Craig Haney, was inspired to pursue a career in prison reform based on what he learned from this study. Zimbardo himself went on to testify on behalf of the soldiers accused of abuse at Abu Ghraib, highlighting the power- ful influence of social roles (e.g., prison guard) on our behavior. An overwhelming majority of social and psychological research has found the punitive punishment system to not only be ineffective but actually deleterious to prisoner behavior and recidivism. The suggestions from current research include calls to restructure prison “power” dynamics in an effort to increase prisoner safety and reduce guard brutality.

5.3 Experimental Validity

Chapter 2 (Section 2.2) discussed the concept of validity, or the degree to which mea-sures capture the constructs that they were designed to capture. For example, a measure of happiness needs to actually capture differences in people’s levels of happiness. In this section, we return to the subject of validity in an experimental context. Similar to our earlier discussion, validity refers here to whether the experimental results are demonstrating what we think they are demonstrating. We will cover two types of validity that are relevant to experimental designs. The first is internal validity, which assesses the degree to which results can be attributed to independent variables. The second is external validity, which assesses how well the results generalize to situations beyond the spe- cific conditions laid out in the experiment. Taken together, internal and external validity provide a way to assess the merits of an experiment. However, each of these has its own threats and remedies, as discussed in the following sections.

Internal Validity

In order to have a high degree of internal validity, experimenters strive for maximum control over extraneous variables. That is, they try to design experiments so that the inde- pendent variable is the only cause of differences between groups. But, of course, no study is ever perfect, and there will always be some degree of error. In many cases, errors are the result of unavoidable causes, such as the health or mood of the participants on the day of the experiment. In other cases, errors are caused by factors that are, in fact, under the experimenter’s control. In this section, we will focus on several of these more manageable threats to internal validity and discuss strategies for reducing their influence.

new85743_05_c05_213-280.indd 222 6/18/13 12:54 PM

223

CHAPTER 5Section 5.3 Experimental Validity

Experimental Confounds To avoid threats to the internal validity of an experiment, it is important to control and minimize the influence of extraneous variables that might add noise to a hypothesis test. In many cases, extraneous variables can be considered relatively minor nuisances, as when our mood experiment was accidentally run in a depressing room. But now, let’s say we ran our study on temperature and mood, and owing to a lack of careful planning, we accidentally placed all of the warm-room participants in a sunny room, and the cool-room participants in a windowless room. We might very well find that the warm-room partici- pants were in a much better mood. But would this be the result of warm temperatures or the result of exposure to sunshine? Unfortunately, we would be unable to tell the differ- ence because of a confounding variable, or confound (in the case of correlation studies, third variable). The confounding variable changes systematically with the independent variable. In this example, room lighting would be confounded with room temperature because all of the warm-room participants were also exposed to sunshine, and all of the cool-room participants to artificial lighting. This combination of variables would leave us unable to determine which variable actually had the effect on mood. The result would be that our groups differed in more than one way, which would seriously hinder our ability to say that the independent variable (the room) caused the dependent variable (mood) to change.

It may sound like an oversimplifica- tion, but the way to avoid confounds is to be very careful in designing experiments. By ensuring that groups are alike in every way but the experi- mental condition, one can generally prevent confounds. This is somewhat easier said than done because con- founds can come from unexpected places. For example, most studies involve the use of multiple research assistants who manage data collec- tion and interact with participants. Some of these assistants might be more or less friendly than others, so it is important to make sure each of them interacts with participants in all conditions. If your friendli- est assistant works with everyone in the warm-room group, for exam- ple, it would result in a confounding variable (friendly versus unfriendly assistants) between room and research assistant. Consequently, you would be unable to separate the influence of your independent variable (the room) from that of the confound (your research assistant).

Digital Vision/Thinkstock

The demeanor of the person running a study may be a confounding variable.

new85743_05_c05_213-280.indd 223 6/18/13 12:54 PM

224

CHAPTER 5Section 5.3 Experimental Validity

Selection Bias Internal validity can also be threatened when groups are different before the manipulation, which is known as selection bias. Selection bias causes problems because these inherent differences might be the driving factor behind the results. Imagine you are testing a new program that will help people stop smoking. You might decide to ask for volunteers who are ready to quit smoking and put them through a 6-week program. But by asking for volunteers—a remarkably common error—you gather a group of people who are already somewhat motivated to stop smoking. Thus, it is difficult to separate the effects of your new program from the effects of this a priori motivation.

One easy way to avoid this problem is through either random or matched-random assign- ment. In the stop-smoking example, you could still ask for volunteers, but then randomly assign these volunteers to one of the two programs. Because both groups would consist of people motivated to quit smoking, this would help to cancel out the effects of motivation. Another way to minimize selection bias is to use the same people in both conditions so that they serve as their own control. In the stop-smoking example, you could assign vol- unteers first to one program and then to the other. However, you might run into a prob- lem with this approach—participants who successfully quit smoking in the first program would not benefit from the second program. This technique is known as a within-subject design, and we will discuss its advantages and disadvantages in the subsection “Within- Subject Designs” in Section 5.4, Experimental Designs.

Differential Attrition Despite your best efforts at random assignment, you could still have a biased sample at the end of a study as a result of differential attrition. The problem of differential attri- tion (sometimes called the mortality threat) occurs when subjects drop out of experimental groups for different reasons. Let’s say you’re conducting a study of the effects of exercise on depression levels. You manage to randomly assign people to either 1 week of regular exercise or 1 week of regular therapy. At first glance, it appears that the exercise group shows a dramatic drop in depression symptoms. But then you notice that approximately one third of the people in this group dropped out before completing the study. Chances are you are left with those who are most motivated to exercise, to overcome their depres- sion, or both. Thus, you are unable to isolate the effects of your independent variable on depression symptoms. While you cannot prevent people from dropping out of your study, you can look carefully at those who do. In many cases, you can spot a pattern and use it to guide future research. For example, it may be possible to discover a profile of people who dropped out of the exercise study and use this knowledge to increase retention for the next attempt.

Outside Events As much as we strive to control the laboratory environment, participants are often influ- enced by events in the outside world. These events—sometimes called history effects— are often large-scale events such as political upheavals and natural disasters. The threat to research is that it becomes difficult to tell whether participants’ responses are the result of the independent variable or the historical event(s). One great example of this comes from a paper published by social psychologist Ryan Brown, now a professor at the University

new85743_05_c05_213-280.indd 224 6/18/13 12:54 PM

225

CHAPTER 5Section 5.3 Experimental Validity

of Oklahoma, on the effects of receiving different types of affirmative action as people were selected for a leadership position. The goal was to determine the best way to frame affirmative action in order to avoid undermining the recipient’s confidence (Brown, Charnsangavej, Keough, Newman, & Rentfrow, 2000). For about a week during the data collection process, students at the University of Texas, where the study was being con- ducted, were protesting on the main lawn about a controversial lawsuit regarding affir- mative action policies. The result was that participants arriving for this laboratory study had to pass through a swarm of people holding signs that either denounced or supported affirmative action. These types of outside events are difficult, if not impossible, to control. But, because these researchers were aware of the protests, they made a decision to exclude from the study data gathered from participants during the week of the protests, thus mini- mizing the effects of these outside events.

Expectancy Effects One final set of threats to internal validity results from the influence of expectancies on people’s behavior. This can cause trouble for experimental designs in three related ways. First, experimenter expectancies can cause researchers to see what they expect to see, leading to subtle bias in favor of their hypotheses. In a clever demonstration of this phe- nomenon, the psychologist Robert Rosenthal asked his graduate students at Harvard University to train groups of rats to run a maze (Rosenthal & Fode, 1963). He also told them that based on a pretest, the rats had been classified as either bright or dull. As you might have guessed, these labels were pure fiction, but they still influenced the way that the students treated the rats. Rats labeled “bright” were given more encouragement and learned the maze much more quickly than rats labeled “dull.” Rosenthal later extended this line of work to teachers’ expectations of their students (Rosenthal & Jacobson, 1968) and found support for the same conclusion: People often bring about the results they expect by behaving in a particular way.

One common way to avoid experimenter expectancies is to have participants interact with a researcher who is “blind” (i.e., unaware) to the condition that each participant is in. The researcher may be fully aware of the research hypothesis, but his or her behav- ior is unlikely to affect the results. In the Rosenthal and Fode (1963) study, the graduate students’ behavior influenced the rats’ learning speed only because they were aware of the labels “bright” and “dull.” If these had not been assigned, the rats would have been treated fairly equally across the conditions.

Second, participants in a research study often behave differently based on their own expectancies about the goals of the study. These expectancies often develop in response to demand characteristics, or cues in the study that lead participants to guess the hypothe- sis. In a well-known study conducted at the University of Wisconsin, psychologists Leon- ard Berkowitz and Anthony LePage found that participants would behave more aggres- sively—by delivering electric shocks to another participant—if a gun was in the room than if there were no gun present (Berkowitz & LePage, 1967). This finding has some clear implications for gun control policies, suggesting that the mere presence of guns increases the likelihood of violence. However, a common critique of this study is that participants may have quickly clued in to its purpose and figured out how they were “supposed” to behave. That is, the gun served as a demand characteristic, possibly making participants act more aggressively because they thought it was expected of them.

new85743_05_c05_213-280.indd 225 6/18/13 12:54 PM

226

CHAPTER 5Section 5.3 Experimental Validity

To minimize demand characteristics, researchers use a variety of techniques, all of which attempt to hide the true purpose of the study from participants. One common strategy is to use a cover story, or a misleading statement about what is being studied. In Chapter 1 (Section 1.4, Hypotheses and Theories, and Section 1.7, Ethics in Research), we discussed Milgram’s famous obedience studies, which discovered that people were willing to obey orders to deliver dangerous levels of electric shocks to other people. In order to disguise the purpose of the study, Milgram described it to people as a study of punishment and learning. And the affirmative action study by Ryan Brown and colleagues (Brown et al., 2000) was presented as a study of leadership styles. The goal in using these cover stories is to give participants a compelling explanation for what they experience during the study and to direct their attention away from the research hypothesis.

Another strategy is to use the unrelated-experiments technique, which leads participants to believe that they are completing two different experiments during one laboratory ses- sion. The experimenter can use this bit of deception to present the independent variable during the first experiment and then measure the dependent variable during the second experiment. For example, a study by Harvard psychologist Margaret Shih and colleagues (Shih, Pittinsky, & Ambady, 1999) recruited Asian American females and asked them to complete two supposedly unrelated studies. In the first, they were asked to read and form impressions of one of two magazine articles; these articles were designed to make them focus on either their Asian American identity or their female identity. In the second experiment, they were asked to complete a math test as quickly as possible. The goal of this study was to examine the effects on math performance of priming different aspects of identity. Based on previous research, these authors predicted that priming an Asian American identity would remind participants of positive stereotypes regarding Asians and math performance, whereas priming a female identity would remind participants of negative stereotypes regarding women and math performance. As expected, priming an Asian American identity led this group of participants to do better on a math test than did priming a female identity. The unrelated-experiments technique was especially useful for this study because it kept participants from connecting the independent variable (maga- zine article prime) with the dependent variable (math test).

A final way in which expectancies shape behavior is the placebo effect, meaning that change can result from the mere expectation that change will occur. Imagine you wanted to test the hypothesis that alcohol causes people to become aggressive. One relatively easy way to do this would be to give alcohol to a group of volunteers (aged 21 and older) and then measure how aggressive they became in response to being pro- voked. The problem with this approach is that people also expect alcohol to change their behavior, so you might see changes in aggression simply because of these expecta- tions. Fortunately, there is an easy solution: Add a placebo control group to your study that mimics the experimental condition in every way but one. In this case, you might tell all participants that they will be drinking a mix of vodka and orange juice but only add vodka to half of the participants’ drinks. The orange-juice-only group serves as the placebo control, so any differences between this group and the alcohol group can be attributed to the alcohol itself.

new85743_05_c05_213-280.indd 226 6/18/13 12:54 PM

227

CHAPTER 5Section 5.3 Experimental Validity

External Validity

In order to attain a high degree of external validity in their experiments, researchers strive for maximum realism in the laboratory environment. External validity means that the results extend beyond the particular set of circumstances created in a single study. Recall that science is a cumulative discipline and that knowledge grows one study at a time. Thus, each study is more meaningful to the extent that it sheds light on a real phenom- enon and to the extent that the results generalize to other studies. Let’s examine each of these criteria separately.

Mundane Realism The first component of external validity is the extent to which an experiment captures the real-world phenomenon under study. One popular question in the area of aggression research is whether rejection by a peer group leads to aggression. That is, when people are rejected from a group, do they lash out and behave aggressively toward the members of that group? Researchers must find realistic ways to manipulate rejection and measure aggression without infringing on participants’ welfare. Given the need to strike this bal- ance, how real can things get in the laboratory? How do we study real-world phenomena without sacrificing internal validity?

The answer is to strive for mundane realism, meaning that the research replicates the psy- chological conditions of the real-world phenomenon (sometimes referred to as ecological validity). In other words, we need not re-create the phenomenon down to the last detail; instead, we aim to make the laboratory setting feel like the real world. Researchers study- ing aggressive behavior and rejection have developed some rather clever ways of doing this, including allowing participants to administer loud noise blasts or serve large quanti- ties of hot sauce to those who rejected them. Psychologically, these acts feel like aggressive revenge because participants are able to lash out against those who rejected them, with the intent of causing harm, even though the behaviors themselves may differ from the ways people exact revenge in the real world.

In a 1996 study, Tara MacDonald and her colleagues at Queen’s University in Ontario, Canada, examined the relationship between alcohol and condom use (MacDonald, Zanna, & Fong, 1996). The authors pointed out a puzzling set of real-world data: Most people reported that they would use condoms when engaging in casual sex, but the rates of unprotected sex (i.e., having sexual intercourse without a condom) were also remarkably high. In this study, the authors found that alcohol was a key factor in causing “common sense to go out the window” (p. 763), resulting in a decreased likelihood of condom use. But how on earth might they study this phenomenon in the laboratory? In the authors’ words, “even the most ambitious of scientists would have to conclude that it is impos- sible to observe the effects of intoxication on actual condom use in a controlled laboratory setting” (p. 765).

new85743_05_c05_213-280.indd 227 6/18/13 12:54 PM

228

CHAPTER 5Section 5.3 Experimental Validity

To solve this dilemma, MacDonald and colleagues developed a clever technique for study- ing people’s intentions to use condoms. Participants were randomly assigned to either an alcohol or placebo condition, and then they viewed a video depicting a young couple that was faced with the dilemma of whether to have unprotected sex. At the key decision point in the video, the tape was stopped and participants were asked what they would do in the situation. As predicted, participants who were randomly assigned to consume alcohol said they would be more willing to proceed with unprotected sex. While this laboratory study does not capture the full experience of making decisions about casual sex, it does a nice job of capturing the psychological conditions involved.

Generalizing Results The second component of external validity is the extent to which research findings gener- alize to other studies. Generalizability refers to the extent to which the results extend to other studies, using a wide variety of populations and a wide variety of operational defi- nitions (sometimes referred to as population validity). If we conclude that rejection causes people to become more aggressive, for example, this conclusion should ideally carry over to other studies of the same phenomenon, using different ways of manipulating rejection and different ways of measuring aggression. If we want to conclude that alcohol reduces intentions to use condoms, we would need to test this relationship in a variety of set- tings—from laboratories to nightclubs—using different measures of intentions.

Thus, each study that we conduct is limited in its conclusions. In order for your par- ticular idea to take hold in the scientific literature, it must be replicated, or repeated in different contexts. These replications can take one of four forms. First, exact replication involves trying to re-create the original experiment as closely as possible in order to verify the findings. This type of replication is often the first step following a surprising result, and it helps researchers to gain more confidence in the patterns. The second and much more common method, conceptual replication, involves testing the relationship between conceptual variables using new operational definitions. Conceptual replications would include testing our aggression hypotheses using new measures or examining the link between alcohol and condom use in different settings. For example, rejection might be operationalized in one study by having participants be chosen last for a group project. A conceptual replication might take a different approach: operationalizing rejection by having participants be ignored during a group conversation or voted out of the group. Likewise, a conceptual replication might change the operationalization of aggression by having one study measure the delivery of loud blasts of noise and another measure the amount of hot sauce that people give to their rejecters. Each variation studies the same concept (aggression or rejection) but uses slightly different operationalizations. If all of these variations yielded similar results, this would provide further evidence of the under- lying ideas—in this case, that rejection causes people to be more aggressive.

The third method, participant replication, involves repeating the study with a new popu- lation of participants. These types of replication are usually driven by a compelling theory as to why the two populations differ. For example, you might reasonably hypothesize that the decision to use condoms is guided by a different set of considerations among college students than among older, single adults. Finally, constructive replication re- creates the original experiment but adds elements to the design. These additions are typically designed to either rule out alternative explanations or extend knowledge about

new85743_05_c05_213-280.indd 228 6/18/13 12:54 PM

229

CHAPTER 5Section 5.3 Experimental Validity

the variables under study. In our rejection and aggression example, you might test whether males and females respond the same way or perhaps compare the impact of being rejected by a group versus an individual.

Internal Versus External Validity

We have focused on two ways to assess validity in the context of experimental designs. Internal validity assesses the degree to which results can be attributed to independent vari- ables; external validity assesses how well results generalize beyond the specific conditions of the experiment. In an ideal world, studies would have a high degree of both of these. That is, we would feel completely confident that our independent variable was the only cause of differences in our dependent variable, and our experimental paradigm would perfectly capture the real-world phenomenon under study.

In reality, though, there is often a trade-off between internal and external validity. In Mac- Donald and colleagues’ study on condom use (MacDonald et al., 1996), the researchers sacrificed some realism in order to conduct a tightly controlled study of participants’ intentions. In Berkowitz and LePage’s (1967) study on the effect of weapons, the research- ers risked the presence of a demand characteristic in order to study reactions to actual weapons. These types of trade-offs are always made based on the goals of the experiment. To give you a better sense of how researchers make these compromises, let’s evaluate three fictional examples.

Scenario 1: Time Pressure and Stereotyping Dr. Bob is interested in whether people are more likely to rely on stereotypes when they are in a hurry. In a well-controlled laboratory experiment, participants are asked to categorize ambiguous shapes as either squares or circles, and half of these partici- pants are given a short time limit to accomplish the task. The independent variable is the presence or absence of time pressure, and the dependent variable is the extent to which people use stereotypes in their classification of ambiguous shapes. Dr. Bob hypothesizes that people will be more likely to use stereotypes when they are in a hurry because they will have fewer cognitive resources to carefully consider all aspects of the situation. Dr. Bob takes great care to have all participants meet in the same room. He uses the same research assistant every time, and the study is always conducted in the morning. Consistent with his hypothesis, Dr. Bob finds that people seem to use shape stereotypes more under time pressure.

The internal validity of this study appears high—Dr. Bob has controlled for other influ- ences on participants’ attention span by collecting all of his data in the morning. He has also minimized error variance by using the same room and the same research assistant. In addition, Dr. Bob has created a tightly controlled study of stereotyping through the use of circles and squares. Had he used photographs of people (rather than shapes), the attractiveness of these people might have influenced participants’ judgments. But here’s the trade-off: By studying the social phenomenon of stereotyping using geometric shapes, Bob has removed the social element of the study, thereby posing a threat to mundane real- ism. The psychological meaning of stereotyping shapes is rather different from the mean- ing of stereotyping people, which makes this study relatively low in external validity.

new85743_05_c05_213-280.indd 229 6/18/13 12:54 PM

230

CHAPTER 5Section 5.3 Experimental Validity

Scenario 2: Hunger and Mood Dr. Jen is interested in the effects of hunger on mood; not surprisingly, she predicts that people will be happier when they are well fed. She tests this hypothesis with a lengthy laboratory experiment, requiring participants to be confined to a laboratory room for 12 hours with very few distractions. Participants have access to a small pile of magazines to help pass the time. Half of the participants are allowed to eat during this time, and the other half is deprived of food for the full 12 hours. Dr. Jen—a naturally friendly person— collects data from the food-deprivation groups on a Saturday afternoon, while her grumpy research assistant, Mike, collects data from the well-fed group on a Monday morning. Her independent variable is food deprivation, with participants either not deprived of food or deprived for 12 hours. Her dependent variable consists of participants’ self-reported mood ratings. When Dr. Jen analyzes the data, she is shocked to discover that participants in the food-deprivation group were much happier than those in the well-fed group.

Compared with our first scenario, this study seems high on external validity. To test her predictions about food deprivation, Dr. Jen actually deprives her participants of food. One possible problem with external validity is that participants are confined to a laboratory setting during the deprivation period with only a small pile of magazines to read. That is, participants may be more affected by hunger when they do not have other things to dis- tract them. In the real world, people are often hungry but distracted by paying attention to work, family, or leisure activities. But Dr. Jen has sacrificed some external validity for the sake of controlling how participants spend their time during the deprivation period. The larger problem with her study has to do with internal validity. Dr. Jen has accidentally confounded two additional variables with her independent variable: Participants in the deprivation group have a different experimenter and data are collected at a different time of day. Thus, Dr. Jen’s surprising results most likely reflect the fact that everyone is in a better mood on Saturday than on Monday and that Dr. Jen is more pleasant to spend 12 hours with than Mike is.

Scenario 3: Math Tutoring and Graduation Rates Dr. Liz is interested in whether specialized math tutoring can help increase graduation rates among female math majors. To test her hypothesis, she solicits female volunteers for a math skills workshop by placing flyers around campus, as well as by sending email announcements to all math majors. The independent variable is whether participants are in the math skills workshop, and the dependent variable is whether participants gradu- ate with a math degree. Those who volunteer for the workshop are given weekly skills tutoring, along with informal discussion groups designed to provide encouragement and increase motivation. At the end of the study, Dr. Liz is pleased to see that participants in the workshops are twice as likely as nonparticipants to stick with the major and graduate.

The obvious strength of this study is its external validity. Dr. Liz has provided math tutor- ing to math majors, and she has observed a difference in graduation rates. Thus, this study is very much embedded in the real world. But, as you might expect, this external validity comes at a cost to internal validity. The biggest flaw is that Dr. Liz has recruited volunteers for her workshops, resulting in selection bias for her sample. People who volunteer for extra math tutoring are likely to be more invested in completing their degree and might also have more time available to dedicate to their education. Dr. Liz would also need to be

new85743_05_c05_213-280.indd 230 6/18/13 12:54 PM

231

CHAPTER 5Section 5.3 Experimental Validity

mindful of how many people drop out of her study. If significant numbers of participants withdrew, she could have a problem with differential attrition, so that the most motivated people stayed with the workshops. One relatively easy fix for this study would have been to ask for volunteers more generally, and then randomly assign these volunteers to take part in either the math tutoring workshops or a different type of workshop. While the sample might still have been less than random, Dr. Liz would at least have had the power to assign participants to different groups.

A Note on Qualitative Research Validity and Reliability

As discussed in Chapter 3, the validity of a quantitative study hinges on whether the experimental results demonstrate what we think they are demonstrating; reliability refers to whether the experimental results will yield the same or similar results in other experi- ments. The concepts of validity and reliability in qualitative research do not carry the same meanings as they do in quantitative research, nor is the goal to generalize the results to individuals, sites, or places outside of those under study. As Creswell (2009) notes, quali- tative validity “means the researcher checks for the accuracy of the findings by employing certain procedures, while qualitative reliability indicates that the researcher’s approach is consistent across different researchers and different projects” (p. 190). Because qualita- tive research does not include experimental results, numerical output, and data analyses, many qualitative researchers argue that they must evaluate the quality of their results dif- ferently and focus more on the trustworthiness or overall worth of their data. Thus, they ask, are the findings worthy of attention? And, how do you evaluate them?

To evaluate the trustworthiness or the validity and reliability of qualitative studies, Guba and Lincoln (1994) proposed the following alternative criteria outlined in Table 5.2 that are utilized by many qualitative researchers.

Table 5.2: Criteria for evaluating quantitative research and qualitative research

Criteria for evaluating quantitative research Alternative criteria for evaluating qualitative research

Internal validity • Assesses whether the independent variable

is the only possible explanation for the dependent variable

Credibility • Used to assess “the accuracy of the

identification and description of the subject of the study” (Smith & Davis, 2010, p. 51)

• Examines whether the research is credible or believable from the perspective of the participant

External validity • Evaluates whether the results can be applied

to different populations and settings

Transferability • Focuses on the transferability of findings to

other settings and groups

• Transferability is enhanced by providing thorough and clear reports so that the results can be transferred to a different context

(continued)

new85743_05_c05_213-280.indd 231 6/18/13 12:54 PM

232

CHAPTER 5Section 5.3 Experimental Validity

Table 5.2: Criteria for evaluating quantitative research and qualitative research (continued)

Criteria for evaluating quantitative research Alternative criteria for evaluating qualitative research

Objectivity • Empirical findings that can be confirmed by

others and corrected through subsequent research

Confirmability • The extent to which the qualitative report “is

accurate, unbiased, and can be confirmed by others” (Smith & Davis, 2010, p. 51)

• Confirmability is enhanced by having other researchers review drafts and point out inconsistencies, contradictions, and biases

• Confirmability is also enhanced by providing thorough reports regarding the procedures that were used to check and recheck the data

Reliability • Assesses whether the methods will yield

similar results in other studies

Dependability • “The extent to which the researcher believes

the same results would be produced if the study were replicated” (Smith & Davis, 2010, p. 51)

• Emphasizes the need for the researcher to account for and describe the changes that occur in the setting and how these changes affected the way the researcher approached the study

There have been lengthy debates about the value of including alternative sets of criteria for judging qualitative research. Although the criteria for evaluating validity and reli- ability in quantitative and qualitative research may seem similar and appear to be mere relabeling of concepts, it should be noted that the procedures utilized to assess them are not. For example, to ensure validity in qualitative studies, researchers employ one or more of the following strategies:

• Triangulation: Using multiple sources of data collection to build justification for themes. If multiple sources of data confirm themes, this process can be consid- ered to add to the validity of the study.

• Member checking: Asking participants to review the final report and confirm whether the descriptions or themes are accurate.

• Providing rich, thick descriptions: Describing in detail the setting, participants, and procedures. This process can add to the validity of the findings.

• Clarifying researcher bias: Self-reflections on any bias the researcher brings to the study helps create an open and honest report.

• Presenting negative or discrepant information: The researcher discussing any infor- mation that runs counter to the themes.

• Peer debriefing: Utilizing peer debriefers that review and ask questions about the study.

new85743_05_c05_213-280.indd 232 6/18/13 12:54 PM

233

CHAPTER 5Section 5.4 Experimental Designs

• Spending prolonged time in the field: Spending long periods of time in the field allows the researcher to develop in-depth understandings of the phenomenon of interest. The more experience a researcher has with participants in their natu- ral setting, the more valid the findings will be.

• External auditors: Employing an independent reviewer who is not familiar with the research or project who can provide an objective assessment of the study.

To determine whether qualitative research approaches are consistent or reliable, research- ers must first ensure that all steps of the procedures are documented thoroughly. Gibbs (2007) suggests the following strategies:

• Checking transcripts: The researcher checks written transcripts against tape-recorded information to ensure that mistakes were not made during transcription.

• Ensuring codes are stable: Verifying that a shift in the meaning of codes did not occur during the process of coding. This is accomplished by constantly compar- ing data with codes and providing detailed descriptions of the codes.

• Coordinating communication: The researcher communicates the analyses to coders through regular documented meetings.

• Cross-checking codes: The researcher cross-checks codes developed by other researchers and compares the results with his or her own.

5.4 Experimental Designs

There are three types of experimental designs that can be used in research stud-ies: pre-experimental designs, quasi-experimental designs, and true experimental designs. Each design differs in the degree to which it controls for confounding or hidden variables; in turn, the degree of control affects the internal validity of the study. Pre-experimental designs have little to almost no control; instead, they involve studying a single group or unbalanced groups that are not randomly assigned and then introduc- ing an intervention or treatment during the study. Quasi-experimental designs offer some control and may or may not include a control group, but the participants are not randomly assigned to groups. True experimental designs give the researcher maximum control and involve randomly assigning participants and manipulating the independent variable. As we will see later in this discussion, true experiments include randomized group selection and assignments, a control group, and a large degree of control over confounding variables.

The best types of designs are those that offer control over confounding variables and include random assignment of the participants. Although true experiments provide a much stronger and more valid design, sometimes they cannot be used for a study for particular reasons. The following sections will discuss examples of the various types of designs. This is not an exhaustive list, and researchers can modify designs or combine them in various ways.

new85743_05_c05_213-280.indd 233 6/18/13 12:54 PM

234

CHAPTER 5Section 5.4 Experimental Designs

Pre-Experimental Designs

Pre-experimental designs follow basic experimental procedures but cannot show cause- and-effect relationships. Some researchers argue that pre-experimental designs confer so little control that they have minimal scientific value. As a result, these designs should be used only when making tentative hypotheses and should be followed up with more con- trolled research (Leedy & Ormrod, 2010). As we will see in the following three sections, pre-experimental designs generally enroll a single group, although some designs include experimental and control groups that are not randomly selected.

The three pre-experimental designs we describe next are very limited in terms of the conclusions that can be drawn from them. In comparison, quasi-experimental designs (described later in this section) confer more control over confounding variables.

One-Shot Case Study One-shot case studies are the most basic type of experimental designs. Because they do not include a control or comparison group, it is impossible to know whether the outcome results would have been better if the intervention had not been provided. This type of pre-experimental design involves using one group (Group A), introducing an interven- tion (X), and then administering a posttest observation (O) to determine the effects of the intervention, as shown below:

Group A: X O

Additionally, since this type of design does not use a pretest, it is impossible to determine whether changes within the group have actually taken place. As a result, one-shot case studies have low internal validity.

In addition, because one-shot case studies confer little to no control over confounding variables, variables such as maturation or other environmental conditions could have influenced the participant’s performance. For instance, let’s say a researcher wants to measure whether praising first-grade students makes a difference in their reading scores. The researcher selects a group of students and increasingly praises those students. At the end of the study, the researcher finds that reading scores have improved. However, the researcher cannot be sure that the scores improved because of the praise, as it is possible that the students selected were already good in reading or that their parents were helping them to learn reading strategies at home. While this type of design is easy to carry out, its results are nearly meaningless.

One-Group Pretest–Posttest Design The one-group pretest–posttest design involves one group but includes both a pretest (O1) before the intervention (X) is given and a posttest (O2) after the intervention is pro- vided, as shown below:

Group A: O1 X O2

new85743_05_c05_213-280.indd 234 6/18/13 12:54 PM

235

CHAPTER 5Section 5.4 Experimental Designs

This design gives the researcher a little more control over the study than a one-shot case study, as it includes a pretest to measure performance before the intervention is provided. Thus, after the posttest is administered, one can know that some sort of change has taken place. However, even though it includes a pretest and a posttest, the researcher still cannot be sure whether some sort of confounding variable contributed to the results.

Suppose, for example, that a researcher wanted to know whether having a group of students take an 8-week course on GRE preparation would improve their scores. The researcher would give the students a form of the GRE test, have all of them participate in the 8-week preparation course, and then give the students an alternate form of the GRE. If the scores improved after the 8-week course, the researcher might conclude that the course produced an increase in GRE scores. However, this conclusion may or may not be accurate, and here is why. Let’s say some of the students studied GRE material at home or participated in another GRE study group at the same time. How could we say that the results were the result of only the 8-week GRE preparation course? The answer is, we cannot.

Static Group Comparison or Posttest-Only With Nonequivalent Groups Static group comparison (or posttest only, with nonequivalent groups’ designs) involves both an experimental group and a control group, although participants are not randomly assigned to either group. This type of design is similar to the one-shot case study design, in that an intervention (X) is provided to one group (Group A) and then a posttest is used to measure the effects of the intervention (O). However, this design includes a second control group (Group B) that does not receive the intervention. The idea here is to include the second group to see whether receiving the intervention really affects the participants’ performance (O). Using our reading score example from the one-shot case study section, the researcher would include a group of students (Group A) who are praised and a group of students (Group B) who are not praised. Since the participants were not randomly assigned to the groups, there would be no way to determine whether the groups were equivalent or whether the results could be attributed to praising or some other factor, such as motivation or IQ. Thus, this design still offers limited control over confounding variables and shares similar problems with one-shot case studies and one-group pretest– posttest designs.

The posttest-only with nonequivalent groups’ design takes the following form:

Group A: X O

Group B: O

Quasi-Experimental Designs

Quasi-experimental designs fare better than pre-experimental designs in terms of both level of control and internal validity. They employ both experimental and control groups with which to make comparisons. However, similar to pre-experimental designs, quasi- experimental designs do not include randomization, either in the selection of group participants or in the presentation of different interventions within the experimental

new85743_05_c05_213-280.indd 235 6/18/13 12:54 PM

236

CHAPTER 5Section 5.4 Experimental Designs

group (Leedy & Ormrod, 2010). Although quasi-experimental designs control more for confounding variables than do pre-experimental designs, they do not control for them completely. These variables must be considered and acknowledged when analyzing and interpreting the data, as alternative explanations for the results are possible. The follow- ing sections discuss the four most common types of quasi-experiments.

Nonequivalent Pretest–Posttest Control-Group Design The nonequivalent pretest–posttest control-group design is probably the most popular quasi-experimental design. In this design, both the experimental group (Group A) and the control group (Group B) receive a pretest observation (O) and a posttest observa- tion (O), although only the experimental group receives the intervention (X). Neither the experimental group nor the control group is randomly assigned. This design takes the following form:

Group A: O X O

Group B: O O

The advantage of this design is that its pretest and posttest allow for comparisons. For example, researchers are able to see whether receiving or not receiving the intervention had an effect on the participants. The disadvantage of this design is that it does not include random assignment to groups. Random assignment allows us to confirm that the two groups are similar in terms of the dependent variable. However, in this design, we cannot be sure that the experimental group and control group are equivalent with respect to age, gender, and other demographic variables. While this design can rule out some alternative explanations for the study results, it cannot control for all confounding variables. As a consequence, the results may not be valid.

Single-Group Interrupted Time Series Design Time series designs consist of making a series of observations over time, introducing an intervention at some point during the study, and then making additional observations (Leedy & Armrod, 2010). Single-group time series designs contain only one experimen- tal group, which is observed over time to evaluate predictable patterns of events. For example, suppose you wanted to examine the effects that a 3-week Republican presiden- tial campaign (X) would have on the pro-choice beliefs (favoring legalized abortion) of a group of independent voters in a particular area (Group A). The campaign might include commercial ads, phone calls, or presentations and speeches. Assuming that you had a reli- able measure of pro-choice beliefs in these voters prior to the campaign, you could start observations (O) a few months before the campaign and then continue observations for a few months after the campaign had ended. The design would take the following form:

Group A: O O O O O X O O O O O

So did the campaign affect Group A’s pro-choice beliefs? Researchers could determine whether an increase in beliefs actually occurred. However, they could not determine

new85743_05_c05_213-280.indd 236 6/18/13 12:54 PM

237

CHAPTER 5Section 5.4 Experimental Designs

whether the change was solely due to the campaign without including an equivalent con- trol group, a pretest, and a posttest. Thus, the major disadvantage of this design is that it cannot allow researchers to conclude that the results were caused by the intervention and not by some other real-world event or confounding variable.

Control-Group Interrupted Time Series Design Another variation of the time series design is the control-group interrupted time series design. This design is similar to the single-group time series design; however, it includes a control group that does not receive the intervention. Thus, the design takes the follow- ing form:

Group A: O O O O O X O O O O O

Group B: O O O O O O O O O O

Using the campaign example discussed in the previous design, in this design, the con- trol group would not be exposed to the campaign. Instead, they would simply receive observations throughout the study at the same time as the experimental group. Providing a control group ensures greater internal validity because it allows researchers to exam- ine whether the campaign had a direct effect on participants’ beliefs. Assuming that the groups were fairly equivalent demographically and that there were no other confound- ing variables that could have influenced their beliefs, we should see a change only in Group A.

Reversal Time Series Design Reversal time series designs use a variable intervention approach in order to minimize the possibility that outside influences might affect the results. This type of design includes one experimental group (Group A), presence or absence of the intervention over time (X and No X, respectively) and observations of the dependent variable at regular inter- vals throughout the study (O). Thus, each participant is either exposed or not exposed to the intervention at various points during the study. After each exposure or nonexposure, observations are made. The design takes the following form:

Group A: X O No X O X O No X O

For example, let’s say you were interested in studying whether including audiovisual pre- sentations in an online courseroom would help students learn statistics. On some weeks, you might want to include video presentations in addition to PowerPoint slide shows, while on other weeks, you might omit the audiovisual material altogether and include just the PowerPoint slides. This approach would let you examine how effectively students learn under both conditions and how effective adding audiovisual materials can be for student learning. If no confounding variables influenced student performance, and audio- visual materials did, in fact, promote student learning, you should see positive changes in student performance on the weeks that included audiovisual materials. If confounding variables were present, you might see very different results.

new85743_05_c05_213-280.indd 237 6/18/13 12:54 PM

238

CHAPTER 5Section 5.4 Experimental Designs

True Experimental Designs

In contrast to pre-experimental and quasi-experimental designs, true experimental designs offer a greater degree of control, randomly select participants and assign them to groups, and randomly assign treatments or interventions to these groups. These char- acteristics all increase the internal validity of a study, which, as we learned earlier in this chapter, reflects the degree to which results can be attributed to the independent variables. Thus, a higher degree of internal validity ensures that the results are related to the inde- pendent variable rather than to other, extraneous variables. And, unlike pre-experimental and quasi-experimental designs, true experiments can determine a clear cause-and-effect relationship between independent and dependent variable. With true experiments, there is a limited chance that the results are not directly related to the independent variable.

The following three designs are the most highly recommended true experimental designs in the literature because they have been found to best control for the threats to internal validity (Campbell & Stanley, 1966). These designs also demonstrate the impact that ran- dom selection and assignment have on the internal validity of a study.

Pretest–Posttest Control Group Design The pretest–posttest control group is a classic true experimental design. It begins with randomly assigning (R) participants into both an experimental and a control group. Both groups are administered a pretest or observation (O) at the beginning of the study and a posttest or observation (O) at the end of the study; only the experimental group is exposed to the intervention (X). Thus, the control group is isolated from the intervention and is subject only to observations. The design takes the following form:

Group A: R O X O

Group B: R O O

Such a design controls for two major threats to internal validity. First, randomly assign- ing participants to groups allows researchers to assume that the two groups were equal in terms of demographic characteristics before the study began. Second, including a pre- test and a posttest for both groups allows researchers to control for confounding factors such as history, maturation, and testing, since both groups should be affected equally by those. Thus, in this design, one should see a change in the experimental group only, which would indicate a cause-and-effect relationship. If a change occurs in the control group, then the researcher would need to consider whether some other confounding variable, such as the presence of a human observer, may have influenced the study.

Solomon Four-Group Design One weakness of the pretest–posttest control group design is that the process of observ- ing participants at the beginning of the study may influence how the experimental group responds to the intervention (Leedy & Omrod, 2010). Thus, the pretest observation may motivate participants to respond to the intervention in a more positive and beneficial manner. To address this weakness, Solomon (1949) proposed an extension of the pretest– posttest control group design.

new85743_05_c05_213-280.indd 238 6/18/13 12:54 PM

239

CHAPTER 5Section 5.4 Experimental Designs

As we can see in the following depiction, the Solomon four-group design enrolls the experimental and control groups, included in the pretest–posttest control group design, along with two additional groups. The two additional groups are also randomly assigned (R); however, they do not receive a pretest observation. Instead, one group receives the intervention (X) and posttest observation (O), and the other group receives only a posttest observation (O). This design allows the researcher to control for the influences of a pretest observation.

Group A: R O X O

Group B: R O O

Group C: R X O

Group D: R O

By adding two groups, the Solomon four-group design also increases the external validity of the study. For instance, if Groups C and D differed in the same way that Groups A and B did, one could assume that a pretest is not required and could generalize the findings to other situations where a pretest had not been provided.

Although this design was developed to eliminate the possibility of pretest influences, it is not without limitations. One disadvantage is the large sample size needed to ensure randomly assigned groups. Another is the time demanded for carrying out a more com- plex design. Finally, Smith and Davis (2007) explained that no available statistical test can treat all four data sets at the same time; therefore, conducting statistical analyses may be challenging.

Posttest-Only Control Group Design The posttest-only control group design resembles the pretest–posttest control group design, except that the pretest is removed from both the experimental and control groups. It is also identical to the additional two groups’ protocol in the Solomon four-group design.

Posttest-only group designs randomly assigns participants to an experimental group and a control group. Only the experimental group receives the intervention (X), whereas both the experimental and control groups receive a posttest observation (O). There are no pre- test observations. The design takes the following form:

Group A: R X O

Group B: R O

Does the lack of a pretest make this design less desirable than the other two true experi- mental designs? The answer is no. Rather, randomly assigning participants to groups is critical in this design because it ensures that the groups are demographically equivalent at the beginning of the study. Thus, any changes during the posttest observation in the experimental group are likely the result of the intervention and not differences between

new85743_05_c05_213-280.indd 239 6/18/13 12:54 PM

240

CHAPTER 5Section 5.4 Experimental Designs

the two groups. So random assignment and including a control group both control for threats to internal validity, making this a very powerful design.

As we have observed, true experiments exhibit more control over a study and provide for greater internal validity. They are often preferred over pre-experimental and quasi- experimental designs because their results can be generalized to a larger population. Table 5.3 summarizes the characteristics of each type of experimental design.

Table 5.3: Characteristics of pre-experimental, quasi-experimental, and true experimental designs

Design Characteristics Pre-Experimental Design

Quasi- Experimental Design

True Experimental Design

Inclusion of a control group In some cases, but not always

Often Yes, always

Random selection of participants from a population

No No Yes, always

Random assignment of participants to groups

No No Yes, always

Random assignment of treatments or interventions to groups

No No Yes, always

Control over extraneous or confounding variables

None Some Yes

Designing True Experiments

The process of designing true experiments boils down to deciding what variables to manipulate and how to do so. In this section, we will cover two broad issues related to experimental design: deciding how to structure the levels (different versions) of an inde- pendent variable and deciding on the number of independent variables necessary to test the hypotheses. While these decisions may seem tedious, they are at the crux of design- ing successful experiments, and, therefore, are the key to performing successful tests of research hypotheses.

Levels of the Independent Variable

In designing internally valid experiments, the primary goal is to ensure that the levels of independent variables are equivalent in every way but one. This allows researchers to make causal statements about the effects of that single change. There are two primary ways in which researchers can control what levels of the independent variables the groups will receive: using between-subject designs and within-subject designs.

new85743_05_c05_213-280.indd 240 6/18/13 12:54 PM

241

CHAPTER 5Section 5.4 Experimental Designs

Between-Subject Designs In some of the research designs discussed so far (primarily quasi-experimental and true experimental designs), the levels of our independent variables have represented two dis- tinct groups—participants are in either the control group or the experimental group. This type of design is referred to as a between-subject design because the levels differ between one subject and the next. Each participant who enrolls in the experiment is exposed to only one level of the independent variable. That is, an individual participant might be in either the experimental or the control group.

Most of the examples so far illustrated between-subject designs: Participants receive either alcohol or placebo; students read an article designed to prime either their Asian or their female identity; and graduate students train rats that are falsely labeled either bright or dull. This approach is very common and has the advantage of using distinct groups to represent each level of the independent variable. In other words, participants who are asked to consume alcohol are completely distinct from those asked to consume the pla- cebo drink. However, this is only one option for structuring the levels of the independent variable. In this section, we will examine two additional ways to structure these levels.

Within-Subject Designs In some cases, the levels of the independent variable represent the same participants but at different time periods. This type of design is referred to as a within-subject design because the levels differ within people. Each participant who enrolls in the experiment would be exposed to both or all levels of the independent variable. That is, every par- ticipant would be in both the experimental and the control group. Within-subject designs are often used to compare changes over time in response to various stimuli. For example, you might measure anxiety symptoms before and after people are locked in a room with a spider or measure depression symptoms before and after people undergo drug treatment.

Within-subject designs have two main advantages over between-subject designs. First, because the same people determine both levels of the independent variable, these designs require fewer participants. Let’s say you decided to collect data from 20 participants at each level of your independent variable. In a between-subject design with three levels, you would need 60 people. However, if you ran the same experiment as a within-subject design—exposing the same group of people to three different sets of circumstances—you would need only 20 people. Thus, within-subject designs are often a good way to conserve resources.

Second, participants serve as their own control group, allowing the researcher to minimize a big source of error variance. Remember that one key feature of experimental design is the researcher’s power to assign people to groups; this is done to randomly distribute subject differences across the levels of the independent variable. Using a within-subject design solves the problem of subject differences in another way—by examining changes within people. For instance, in the example regarding spiders and anxiety, some participants would likely have higher baseline anxiety than others. By measuring changes in anxiety in the same group of people before and after spider exposure, the researcher would be able to minimize the effects of individual differences.

new85743_05_c05_213-280.indd 241 6/18/13 12:54 PM

242

CHAPTER 5Section 5.4 Experimental Designs

Problems With Within-Subject Designs Within-subject designs also have two clear disadvantages compared with between-subject designs. First, there is the risk of carryover effects, in which the effects of one level are still present when another level is introduced. Because the same people are exposed to all levels of the independent variable, it can be difficult to separate the effects of one level from the effects of the others. One common paradigm in research on emotions is to show participants several film clips that elicit different types of emotion. People might view one clip showing a puppy playing with a blanket, another showing a child crying, and another showing a surgical amputation. Even without seeing these in full color, you can imagine that it would be hard to shake off the disgust triggered by the amputation in order to expe- rience the joy triggered by the puppy.

When researchers use a within-subject design, they take steps to minimize carryover effects. In studies of emotion, for example, researchers typically show a brief, neutral clip, such as waves rolling onto a beach, between emotional clips so that participants experi- ence each emotion after viewing a benign image. Another simple technique is to collect data from the control condition first whenever possible. In the example study of spiders and anxiety, it would be important to measure baseline anxiety at the start of the experi- ment before exposing people to spiders. Once people had been surprised by a spider, it would be hard to get them to relax enough to collect control ratings of anxiety.

Second, there is a risk of order effects, meaning that the order in which levels are pre- sented can moderate their effects. Order effects fall into two categories. The practice effect happens when participants’ performance improves over time simply due to making repeated attempts. This is a particular problem in studies that examine learning. Let’s say you use a within-subject design to compare two techniques for teaching people to solve

logic problems. Participants would learn technique A, then take a logic test, then learn technique B, and then take a second logic test. The possible problem that arises is that participants will have had more opportunities to practice logic problems by the time they took the second test. This would make it difficult to separate the effects of practice from the effects of different teaching techniques.

The flipside of practice effects is the phenomenon of the fatigue effect, which happens when participants’ performance decreases over time owing to repeated testing. Let’s say you ran a variation of the experiment just described, attempting to teach people to improve their reaction time. Participants might learn each tech- nique and have their reaction time

Ryan McVay/Photodisc/Thinkstock

Carryover effects can be understood through the example of monitoring people’s reactions to different film clips. How they feel about one image may influence how they react to the next image.

new85743_05_c05_213-280.indd 242 6/18/13 12:54 PM

243

CHAPTER 5Section 5.4 Experimental Designs

tested several times after each one. The problem would be that people gradually started to become tired, and their reaction times slowed down due to fatigue. Thus, it would be dif- ficult to separate the effects of fatigue from the effects of the different teaching techniques.

The result of both of these problems is that the order of presentation becomes confounded with the level of the independent variable. Fortunately, there is a relatively easy way to avoid both carryover and fatigue effects by using a process called counterbalancing. Counterbalancing involves varying the order of presentation to groups of participants. The simplest approach is to divide participants into as many groups as there are combina- tions of levels in the experiment. That is, the researchers create a group for each possible order, allowing them to identify the effects of encountering the conditions in different orders. In the preceding examples, the learning experiments involved two techniques: A and B. To counterbalance these techniques across the study, it would be necessary to divide the participants into two groups. One group would be exposed to A and then B; the other group would be exposed to B and then A. When it came time to analyze the data, researchers would be able to examine the effects of both presentation order and teaching technique. If the order of presentation made a difference, then one could conclude that the A/B group differed from the B/A group in some way.

Mixed Designs The third common way to structure the levels of an independent variable is in a mixed design (discussed in more detail in Section 5.8), which contains at least one between- subject variable and at least one within-subject variable. So, in the example just discussed, participants would be exposed to both teaching techniques (A and B), but in only one of two possible orders of presentation. In this case, teaching technique is a within-subject variable because participants experience both levels. And presentation order is a between- subject variable because participants experience only one level. When there is one of each in the overall experiment, this is a mixed design.

One common use of mixed designs is in studies that compare the effects of different drugs. Imagine you wanted to compare three new drugs—Drug X, Drug Y, and a placebo con- trol—to determine which had the strongest effects on reducing depression symptoms. To do this study, you would want to measure depression symptoms on at least three occa- sions: before starting drug treatment, after a few months of taking the drug (or placebo), and then again after a few months of stopping the drug (to assess relapse rates). So the participants would be given one of three possible drugs and then measured at each of three time periods. In this mixed design, measurement time is a within-subject variable because participants experience all three levels, whereas the drug is a between-subject vari- able because participants experience only one of three possible levels.

The hypothetical results of this study are shown in Figure 5.4. You can see that the placebo pill has no effect on depression symptoms; depression scores in this group are the same at all three measurements. Drug X appears to significantly improve depression symptoms; depression scores drop steadily across measurements in this group. And, strangely, Drug Y seems to make depression worse; depression scores increase steadily across measure- ments in this group. The mixed design allows researchers both to track people over time and to compare different drugs in one study.

new85743_05_c05_213-280.indd 243 6/18/13 12:54 PM

244

CHAPTER 5Section 5.4 Experimental Designs

Research: Thinking Critically

Outwalking Depression

By Michael Otto

There is a great deal of evidence that exercise not only improves mood and enhances well-being but also is an effective intervention for depression. That is, well-conducted clinical trials have repeatedly shown mood benefit from exercise in adults with clinical depression. Indeed, there is evidence that exercise provides benefits at levels similar to that found for antidepressant medication.

Now, there is new evidence for the power of exercise when other treatments for depression have not provided adequate help. A recent study, published in the August 2011 issue of the Journal of Psy- chiatric Research, provides data that exercise can provide benefits when medication alone does not (Moto-Pereira et al, 2011). Specifically, researchers in Portugal examined the effects of exercise—in this case programmed episodes of walking for 30–45 minutes five times per week—on depressed adults who failed to respond to two previous trials of antidepressant medication. In the study, all patients remained on their antidepressant medication, and two thirds of the sample received the program of regular (walking) exercise.

The results were dramatic. Of those patients who received only medication, no average changes in depression mood ratings were seen over the next 12 weeks. In contrast, clear improvement was seen in those who exercised, with 10 of the 19 patients who exercised showing a response or full remis- sion in symptoms during this time.

(continued)

Figure 5.4: Example of a mixed-subjects design

D e

p re

s s

io n

S c

o re

30

35

Baseline Time 1 Time 2

Time of Measurement (Within–Subject)

Drug Y

Placebo

Drug X

25

20

15

10

5

0

new85743_05_c05_213-280.indd 244 6/18/13 12:54 PM

245

CHAPTER 5Section 5.4 Experimental Designs

Research: Thinking Critically (continued) Although this was a small study, the results were consistent with previous exercise studies. These studies underscore the importance of considering combined treatment strategies for patients who do not respond to one type of intervention—if one treatment does not work, consider using other resources with evidence for success. In the case of treatment of depression, these resources promi- nently include psychotherapy (with a wealth of studies indicating that specific forms of psychother- apy like Cognitive Behavior Therapy or Interpersonal Psychotherapy can bring about timely relief of depression), a range of medication treatments, or, with increasing evidence, regular exercise. Within these choices there is not a clear cure-all option, but, importantly, there is a range of options to be pursued to try to find the right fit for any particular person suffering from depression.

A number of important features of the Portuguese study deserve additional note. First, the study provided additional evidence that high intensities of exercise are not required to bring about mood changes. Moderate exercise is adequate for mood benefit. Second, the study used an exercise pro- gram that relied on the individual efforts of the patient. Exercise was performed on a treadmill at the treatment center for one out of the five weekly sessions; the rest of the sessions were up to the individual. This individual program of exercise helped show that the mood effects of exercise were not simply due to the social contact from a group of patients exercising together. It also showed that moderate exercise is an accessible option even for chronically depressed individuals. Third, the exer- cise prescription included the use of regular reminders for exercise: keeping walking shoes in a visible location, having support for the walking program from family, or arranging for written or cell phone reminders for walking sessions. Simple reminders like these can have a powerful effect on helping people keep up with any new habit.

Otto, M., & Smits, J. (2011, July 26). Outwalking depression: Consider exercise when the antidepressants have not helped. Psychology Today. Retrieved from http://www.psychologytoday.com/blog/exercise-and-mood/201107/outwalking-depression

Think about it:

1. Identify the following essential aspects of this experimental design:

a. What are the independent and dependent variables in this study? b. How many levels does the independent variable have? c. Is this a between-subject, within-subject, or mixed design? d. Draw a simple table labeling each condition.

2. What existent differences between groups should the researchers be sure to take into account? Name as many as you can.

3 How should the researchers assign participants to the conditions in order to ensure that exis- tent differences cannot account for the results?

4. How might expectancy effects influence the results of this study? Can you think of any ways to control for this?

5. Briefly state how you would replicate this study in each of the following ways:

a. exact replication b. conceptual replication c. participant replication d. constructive replication

new85743_05_c05_213-280.indd 245 6/18/13 12:54 PM

246

CHAPTER 5Section 5.4 Experimental Designs

One-Way Versus Factorial Designs

The second big issue in creating experimental designs is to decide on the number of inde- pendent variables to manipulate. In some cases, we can test hypotheses by manipulating a single independent variable and measuring the outcome, for example, giving people either alcohol or a placebo drink and measuring the intention to use condoms. In other cases, hypotheses involve more complex combinations of variables. Earlier in the chap- ter, we discussed research findings that people tend to act more aggressively after a peer group has rejected them. But this hypothesis could be extended by asking what happens when people are rejected by members of the same sex versus members of the opposite sex. And one could go a step further and test whether the attractiveness of the rejecters matters, for a total of three independent variables. These examples illustrate two broad categories of experimental design, known as one-way and factorial designs.

One-Way Designs If a study involves giving people either alcohol or a placebo and then measuring outcomes, it has a one-way design, which has only one independent variable with two or more levels. These tend to be the simplest experiments and have the advantage of testing manipulations in isolation. One-way designs are used in the majority of drug studies. These types of study compare the effects on medical outcomes for people randomly assigned, for instance, to take the antidepressant drug Prozac or a placebo. Note that a one-way design can still have multiple levels—in many cases, it is preferable to test several different doses of a drug. So, for example, researchers might test the effects of Prozac by assigning people to take doses of either 5 mg, 10 mg, 20 mg once daily, or a placebo control. If so, the independent vari- able would be the drug dose, and the dependent variable would be a change in depression symptoms. This design would allow researchers to compare all three drug doses to a pla- cebo control, as well as to compare the effects of each dose. Figure 5.5 shows hypothetical results from this study. Notice that even those receiving the placebo showed a drop in depression symptoms, with the maximum benefit caused by the 10-mg dose of Prozac.

Figure 5.5: Comparing drug doses in a one-way design

Decrease in Depression Symptoms

Placebo 5mg 10mg 20mg 0

-5

-10

-15

-20

-25

new85743_05_c05_213-280.indd 246 6/18/13 12:54 PM

247

CHAPTER 5Section 5.4 Experimental Designs

Factorial Designs Despite the appealing simplicity of one-way designs, experiments conducted in the field of psychology with only one independent variable are relatively rare. The real world is much more complicated, which means that studies that focus on people’s thoughts, feel- ings, and behaviors must try to capture the associated complexity. Thus, the rejection and aggression example discussed previously is not that far-fetched. If a researcher wanted to manipulate the presence of rejection, the sex of the rejecters, and the attractiveness of the rejecters in a single study, the experiment would need a factorial design. Factorial designs are those that have two or more independent variables, each of which has two or more levels. When using a factorial design, the purpose is to observe both the effects of indi- vidual variables and the combined effects of multiple variables.

Factorial designs have their own terminology to reflect the fact that they comprise both individual variables and combinations of variables. At the beginning of this chapter, we learned that the versions of an independent variable are referred to as both levels and conditions, and that there is a subtle difference between the two. This difference comes into play when discussing factorial designs. Specifically, levels refer to the versions of each independent variable, while conditions refer to the groups formed by combinations of independent variables. Let’s walk through one variation of our rejection and aggression example from this perspective: The first independent variable has two levels because par- ticipants are either rejected or not rejected. And the second independent variable has two levels because members of the same sex or the opposite sex do the rejecting. To determine the number of conditions in this study, calculate the number of different experiences that participants can have in the study. This is a simple matter of multiplying the levels of separate variables, so 2 is multiplied by 2, for a total of four conditions.

Researchers also have a way to quickly describe the number of variables in their design: A two-way design has two independent variables; a three-way design has three indepen- dent variables; an eight-way design has eight independent variables, and so on. Even more useful, the system of factorial notation offers a simple way to describe both the number of variables and the number of levels in experimental designs. For instance, you might describe your design as a 2 3 2 (pronounced “two by two”), which instantly com- municates two things: (1) you have two independent variables, indicated by the presence of two separate numbers, and (2) each independent variable has two levels, indicated by the number 2 listed for each one.

The 2 3 2 Design One of the most common factorial designs also happens to be the simplest one: the 2 3 2 design. These designs have two independent variables, with two levels each, for a total of four experimental conditions. The simplicity of these designs makes them a useful way to become more comfortable with some of the basic concepts of experiments. This section will walk you through an example of a 2 3 2 design and analyze it in detail.

Beginning in the late 1960s, social psychologists developed a keen interest in understand- ing the predictors of helping behavior. This was inspired, in large part, by the tragedy of Kitty Genovese, who was killed outside her apartment building while none of her neigh- bors called the police (Gansberg, 1964). (See Research: Making an Impact in Chapter 2,

new85743_05_c05_213-280.indd 247 6/18/13 12:54 PM

248

CHAPTER 5Section 5.4 Experimental Designs

Section 2.1.) In one representative study, Princeton psychologists John Darley and Bibb Latané examined people’s likelihood of responding to a staged emergency. Participants were led to believe that they were taking part in a group discussion over an intercom sys- tem, but in reality, all the other participants’ voices were prerecorded. The key indepen- dent variable was the number of other people supposedly present, ranging from two to six. A few minutes into the conversation, one participant appeared to have a seizure. The recording went like this (actual transcript; Darley & Latané, 1968):

I could really-er-use some help so if someone would-er-give me a little h-hel- puh-er-er-er c-could somebody er-er-hel-er-uh-uh-uh [choking sounds] . . . I’m gonna die-er-er-I’m . . . gonna die-er-hel-er-er-seizure-er [chokes, then quiet].

What do people do in this situation? Do they help? How long does it take? Darley and Latané discovered that two things happen as the group became larger: People were less likely to help at all, and those who did help took considerably longer to do so. One of the primary conclusions to come out of this and other studies is that people are less likely to help when other people are present because the responsibility for helping is diffused among the members of the crowd (Darley & Latané, 1968).

Building on this conclusion, the sociologist Jane Piliavin and her colleagues (Piliavin, Rodin, & Piliavin, 1969; Piliavin, Piliavin, & Rodin, 1975) explored the influence of two additional variables on helping behaviors by staging an emergency on a New York City subway train in which a person who was in on the study appeared to collapse in pain. Specifically, the researchers manipulated two variables in their staged emergency. The first independent variable was whether there was a medical intern nearby, who could be easily identified by wearing blue scrubs. The second independent variable was whether the victim had a large disfiguring scar on his face. The combination of these variables resulted in four conditions, as shown in Table 5.4. The dependent variable in this study was the percentage of people taking action to help the person appearing to collapse in pain.

Table 5.4: 2 3 2 Design of the Piliavin et al. study

No intern Intern

No scar 1 2

Scar 3 4

The authors predicted that bystanders would be less likely to help if there was a perceived medical professional nearby since he or she was considered more qualified to help the victim. They also predicted that people would be less likely to help when the confeder- ate had a large scar because previous work had demonstrated convincingly that people avoid contact with those who are disfigured or have other stigmatizing conditions (e.g., Goffman, 1963). As seen in Figure 5.6, these hypotheses were supported by the results. Both the presence of a scar and the presence of a perceived medical professional reduced the percentage of people who came to help. But there’s something else going on in these

new85743_05_c05_213-280.indd 248 6/18/13 12:54 PM

249

CHAPTER 5Section 5.4 Experimental Designs

results: When the confederate is not scarred, having an intern nearby leads to a small decrease in helping (from 88% to 84%). However, when the confederate had a large facial scar, having an intern nearby decreased helping from 72% to 48%! In other words, it seems these variables had a combined effect on helping behavior. We will examine these com- bined effects more closely in the next section.

Figure 5.6: Sample 2 3 2 Design: Results from Piliavin et al. (1975)

Other Factorial Designs Experimental designs can often be more complex than a simple 2 3 2 format. In another variation of the rejection and aggression study, participants would be categorized in one of eight cells—they are either rejected or not, by a person of either the same sex or the opposite sex, who happens to be either attractive or unattractive. This design would have eight conditions (2 3 2 3 2), as shown in Table 5.5a, with the separate conditions num- bered for illustration purposes.

Table 5.5a: Levels versus conditions in a 2 3 2 3 2 design

Rejected Not Rejected

Rejecter Attractive Unattractive Attractive Unattractive

Same sex 1 2 5 6

Opposite sex 3 4 7 8

This is a 2 3 2 3 2 design for a total of eight conditions (numbered 1–8). “Rejection” has two levels: rejected and not rejected. “Attractiveness” has two levels: attractive and unattractive. “Sex” has two levels: the same sex and the opposite sex rejecter.

% H

e lp

in g

100

No Scar Scar

90

80

70

60

50

40

30

10

20

0

No Intern Intern

new85743_05_c05_213-280.indd 249 6/18/13 12:54 PM

250

CHAPTER 5Section 5.4 Experimental Designs

A different study might describe a design as 2 3 3, which would tell people two things: There are two independent variables, and one variable has two levels (the 2) while the other variable has three levels (the 3). One possibility for this 2 3 3 design is illustrated in Table 5.5b. For this hypothetical study, the researcher was interested in the effects of dis- comfort on mood. She manipulated both hunger, with three levels, and pain, with two lev- els. Participants were asked to abstain from eating for 4 hours (“mild hunger”) or 24 hours (“extreme hunger”), or they were allowed to eat before the study (“no hunger”). During the laboratory session, participants were asked either to hold their hand in a bucket of ice water (“pain”) or not (“no pain”). As you can see in Table 5.5b, this resulted in six possible conditions for her participants to experience.

Table 5.5b: Levels versus conditions in a 2 3 3 design

No Hunger Mild Hunger Extreme Hunger

Pain 1 2 3

No Pain 4 5 6

This is a 2 3 3 design for a total of six conditions (numbered 1–6). “Pain” has two levels: presence and absence of pain. “Hunger” has three levels: no hunger, mild hunger, and extreme hunger.

Main Effects and Interactions

When experiments involve only one independent variable (IV), the analyses can be as simple as comparing two group means—as with the example in Chapter 2 (Section 2.4, Hypothesis Testing) comparing the happiness levels of couples with and without chil- dren. But what about cases where the design has more than one independent variable?

In a factorial design, there are two types of effect: A main effect refers to the effect of each independent variable on the dependent variable, averaging values across the lev- els of other variables. A 2 3 2 design has two main effects; a 2 3 2 3 2 design has three main effects because there are three independent variables. An interaction occurs when the variables have a combined effect; that is, the effects of one independent variable are different depending on the levels of the other independent variable. So applying this new terminology to the Piliavin et al. (1969, 1975) “subway emergency” study yields three possible results (possible because researchers eventually use statistical analyses to verify them):

1. The main effect of scar: Does the presence of a scar affect helping behavior? Yes. More people help in absence of a facial scar. In Figure 5.6, you can see that

the bars on the left (no scar) are, on average, higher than those on the right (scar). 2. The main effect of intern: Does the presence of an intern affect helping behavior? Yes. More people help when there is no medical intern on hand. In Figure 5.6,

notice that the red bars (no intern) are, on average, higher than the orange bars (intern).

new85743_05_c05_213-280.indd 250 6/18/13 12:54 PM

251

CHAPTER 5Section 5.4 Experimental Designs

3. The interaction between scar and intern: Does the effect of one variable depend on the effect of another variable?

Yes. If you refer to Figure 5.6, you can see that the presence of a medical intern matters more when the victim has a facial scar. In visual terms, the gap between red and orange bars is much larger in the bars on the right. This indicates an inter- action between scar and intern.

Let’s walk through a fictional example. Imagine you were interested in people’s percep- tions of actors in different types of movies. You might predict that some actors are better suited to comedy and others are better suited to action movies. One simple way to test this hypothesis would be an experiment that showed four movies in a 2 3 2 design, using the same two actors in two movies (for a total of four conditions). Our first independent vari- able would be the movie type, with two levels: action and comedy. Our second indepen- dent variable would be the actor, with two levels: Will Smith and Arnold Schwarzenegger. Our dependent variable would be the ratings of each movie on a 10-point scale. This gives us three possible results:

1. The main effect of actor: Do people generally prefer Will Smith or Arnold Schwar- zenegger, regardless of the movie?

2. The main effect of movie type: Do people generally prefer action or comedy movies, regardless of the actor?

3. The interaction between actor and movie type: Do people prefer each actor in a dif- ferent kind of movie? (i.e., are ratings affected by the combination of actor and movie type?)

After collecting data from a sample of participants, we end up with the following average ratings for each movie, shown in Table 5.6.

Table 5.6: Main effects and marginal means: The actor study

Arnold Will Marginal Mean

Action 6 1.5 3.75

Comedy 5 8 6.5

Marginal Mean 5.5 4.75

These two means let us compare movie ratings, ignoring the actor. People have a slight preference for Comedy (6.5) over Action (3.75). These two means let us compare actor ratings, ignoring the movie type. People have a slight preference for Arnold (5.5) over Will (4.75).

Remember that main effects represent the effects of one independent variable, averaging across the levels of the other independent variable. To average across levels, we calculate the marginal mean, or the combined mean across levels of another factor. In other words, the marginal mean for action movies is calculated by averaging together the ratings of both Arnold Schwarzenegger and Will Smith in action movies. The marginal mean for Arnold Schwarzenegger is calculated by averaging ratings of Arnold Schwarzenegger in

new85743_05_c05_213-280.indd 251 6/18/13 12:54 PM

252

CHAPTER 5Section 5.4 Experimental Designs

both action and comedy movies. Doing this for our 2 3 2 design results in four mar- ginal means, which are presented alongside the participant ratings in Table 5.6. We would need to verify these patterns with statistical analyses, but it appears we have two main effects, based on these marginal means: People have a slight preference for comedy over action movies, as well as a slight preference for Arnold Schwarzenegger’s acting over Will Smith’s acting, regardless of the movie.

What about the interaction? Our main hypothesis here is that some actors perform best in some genres of movies (e.g., action or comedy) than they do in other genres. This suggests that the actor and the movie type combine to influence people’s ratings of the movies. We can get a sense of this from examining the means in Table 5.6, but it is even easier to appre- ciate in a graph. Figure 5.7 shows the mean of participants’ ratings across the four condi- tions. Focusing first on the ratings of Arnold Schwarzenegger, notice that participants did have a slight preference for him in action (6) versus comedy (5) roles. Then, examining ratings of Will Smith, you can see that participants had a strong preference for him in comedy (8) versus action (1.5) roles. Together, this set of means indicates an interaction between actor and movie type because the effects of one variable depend on another. Or, in plain English: People’s perceptions of the actor depend on the type of movie he is in. This pattern of results is a nice fit for our hypothesis that certain actors are better suited to certain types of movie: Arnold should probably stick to action movies, and Will should definitely stick to comedies.

Figure 5.7: Interaction in the actor study

Before we move on to analyses, let’s look at one more example from a published experi- ment. A large body of research in social psychology suggests that stereotypes can neg- atively impact performance on cognitive tasks (e.g., tests of math and verbal skills). According to Stanford social psychologist Claude Steele and his colleagues, the fear of confirming negative stereotypes about one’s group acts as a distraction. This distraction— which they term stereotype threat—makes it hard to concentrate and perform well, and thus leads to lower scores on a cognitive test (Steele, 1997). One of the primary implica- tions of this research is that ethnic differences in standardized test scores can be viewed as a situational phenomenon, so if we change the situation, the differences should go

F re

q u

e n

c y

Arnold Will

9

8

7

6

5

4

3

1

2

0

Action

Comedy

new85743_05_c05_213-280.indd 252 6/18/13 12:54 PM

253

CHAPTER 5Section 5.5 Analyzing Experiments

away. In the first published study of stereotype threat, Claude Steele and Josh Aronson (1995) found that when African American students at Stanford were asked to indicate their race before taking a standardized test, this was enough to remind them of negative stereotypes, and they performed poorly. But when the testing situation was changed, and participants were no longer asked their race, these students performed at the same level as Caucasian students. It is worth emphasizing that these were Stanford students and had therefore met admissions standards for one of the best universities in the nation. Even this group of elite students was susceptible to situational pressure but performed at their best when the pressure was eliminated.

In a great application of stereotype threat, social psychologist Jeff Stone at the Univer- sity of Arizona asked both African American and Caucasian college students to try their hand at putting on a golf course (Stone, Lynch, Sjomeling, & Darley, 1999). Putting was described as a test of natural athletic ability to half of the participants and as a test of sports intelligence to the other half. Thus, there were two independent variables: the race of the participants (African American or Caucasian) and the description of the task (“ath- letic ability” or “sports intelligence”). Note that “race” in this study was technically a quasi-independent variable because it was not manipulated. This resulted in a total of four conditions, and the dependent variable was the number of putts that participants managed to make. Stone and colleagues hypothesized that describing the task as a test of athletic ability would lead Caucasian participants to worry about the stereotypes regard- ing their poor athletic ability. In contrast, describing the task as a test of sports intelligence would lead African American participants to worry about the stereotypes regarding their lower intelligence.

Consistent with their hypotheses, Stone and colleagues found an interaction between race and task description but no main effects. That is, neither race was better at the putting task overall, and neither task description had an overall effect on putting performance. But the combination of these variables proved fascinating. When the task was described as measuring sports intelligence, the African American participants did poorly for fear of confirming negative stereotypes about their overall intelligence. But when the task was described as measuring natural athletic ability, the Caucasian participants did poorly for fear of confirming negative stereotypes about their athleticism. This is a beautiful illustra- tion of an interaction; the effects of one variable (task description) depend on the effects of another (race of participants). And this is further confirmation of the power of the situation: Neither group did better or worse overall, but both were responsive to a situationally induced fear of confirming negative stereotypes.

5.5 Analyzing Experiments

So far, we have been drawing conclusions about our experimental findings in concep-tual terms. But naturally, before we actually make a decision about the status of our hypotheses, we have to conduct statistical analyses. In this section, we will cover the most common statistical techniques for analyzing experimental data.

new85743_05_c05_213-280.indd 253 6/18/13 12:54 PM

254

CHAPTER 5Section 5.5 Analyzing Experiments

Dealing With Multiple Groups

You may find yourself wondering why we need a special technique for experimental designs. After all, we learned in Chapter 2 (Section 2.4, Hypothesis Testing) that we can compare two pairs of means using a t-test. So why not just use several t-tests to analyze our experimental designs? In our movie ratings study, we could analyze the data using a total of six t-tests to capture every possible pair of means: Arnold Schwarzenegger in a comedy versus Will Smith in a comedy; Arnold Schwarzenegger in an action movie ver- sus Will Smith in an action movie; Arnold Schwarzenegger in a comedy versus an action movie; Will Smith in a comedy versus an action movie; Will Smith in a comedy versus Arnold Schwarzenegger in an action movie; and finally Will Smith in an action movie versus Arnold Schwarzenegger in a comedy.

But there is a problem with this approach: The odds of developing a Type I error (a false positive) are increased with every statistical test. We typically set our alpha level at .05 for a t-test, meaning that we’re comfortable with a 5% chance of a Type I error. Unfortunately, if we conduct six t-tests, each one has a 5% chance of a Type I error, meaning that we’re now rather likely to draw conclusions about a false positive. The moral of this story is that we need a statistical approach that reduces the number of comparisons we perform. Fortunately, this can be accomplished using a statistical technique called the analysis of variance (ANOVA), which tests for differences by comparing the amount of variance explained by the independent variables to the amount of variance explained by error. The following sections explore ANOVA in more detail.

The Logic of ANOVA

The logic behind the analysis of variance is rather straightforward. As we have discussed throughout this course, the variability in data can be divided into systematic and error variance. That is, we can attribute some of the variability to the factors we are studying, but there will always be a degree of random error. In our movie ratings study, some of the variability in these ratings can be attributed to our independent variables (differences in actors and movie types), while some of the variability is the result of other factors— perhaps some people simply like movies more than other people.

The ANOVA works by comparing the influence of these different sources of variance. We always want to explain as much of the variance as possible through the independent variables. If the independent variables have more influence than random error does, this is good news. But if error variance has more influence than the independent variables, this is bad news for the hypotheses. You can get a sense of this by comparing the three pie charts in Figure 5.8. The proportion of variance explained by our independent variables is shaded in yellow, while the proportion explained by error is shaded in red. In the top graph, approximately 80% of the variance can be explained by our independent variables, which can be viewed as a good result. But in the middle graphs, variance is explained equally by the independent variables and by error, and in the bottom graph, only 30% of the variance is explained by the independent variables. Thus, in the latter two graphs, our independent variables do no better than random error at explaining our results.

new85743_05_c05_213-280.indd 254 6/18/13 12:54 PM

255

CHAPTER 5Section 5.5 Analyzing Experiments

One more analogy may be helpful. In the field of engineering, the term signal-to-noise ratio is used to describe the amount of light, sound, energy, and so forth that is detectable above and beyond background noise. This ratio is high when the signal comes through clearly and low when it is mixed in with static or other interference. Like- wise, when you try to tune in your favorite radio station, the goal is to find a clear signal that is not covered up by static. Believe it or not, the ANOVA statistic is doing the same thing. That is, the anal- ysis tells us whether differences in experimen- tal conditions (signal) are detectable above and beyond error variance (noise). In the next section, we’ll take a closer look at how ANOVA makes these distinctions.

Calculating ANOVA

The ANOVA statistic—abbreviated with the capi- tal letter F—works by comparing the variance that

is explained by group differences with the variance explained by error. This is really just a minor adaptation, with new terminology, of the formula we have encountered before, describing the components of total variance in the dataset:

Total Variance 5 Systematic Variance 1 Error Variance

In conducting an ANOVA, these two components will have special names, but the idea is the same. Specifically, we refer to systematic variance as between-group variance, or the variance explained by differences in our independent variables. In the movie ratings study, between-groups variance represents the variability in ratings that can be attributed to different actors, different movies, and the combinations of these independent variables. In addition, we refer to error variance as within-group variance, or the total variance that occurs across each condition. In the movie ratings study, within-groups variance would represent the sum total of all other influences on movie ratings (i.e., the error).

In this day and age, ANOVA (F) is calculated using one of the numerous statistical soft- ware programs; these programs take columns of data and churn out significance tests to test hypotheses. But in order to understand what these significance tests mean, it is worth walking through a conceptual overview of the elements in the ANOVA formula. The start- ing point for the F statistic is the calculation of several sum of squares, or the sum of the squared deviations between individual scores and the overall sample mean. We encoun- tered these sums of squares (abbreviated SS) in Chapter 3 (Section 3.5), in our discussion of calculating the standard deviation.

Explained by IVs

Explained by Error

Explained by IVs

Explained by Error

Explained by IVs

Explained by Error

Figure 5.8: Comparing sources of variance

new85743_05_c05_213-280.indd 255 6/18/13 12:54 PM

256

CHAPTER 5Section 5.5 Analyzing Experiments

Using this notation, our components of the variance formula can be rewritten as follows:

SS T 5 SS

BG 1 SS

WG SS

T 5 sum of squares total

SS BG

5 sum of squares between groups SS

WG 5 sum of squares within groups

In conceptual terms, SS BG

represents the deviation between each condition’s mean and the overall mean of the sample. If you refer to the data from our movie ratings study in Table 5.6, you’ll see the four condition means of 6, 5, 1.5, and 8. The overall mean of movie ratings—across conditions—is 5.13 (i.e., [6 1 5 1 1.5 1 8] 4 4). To compute the SS

BG , we

subtract each of the four condition means from 5.13, square the difference, and add up these squared deviation scores.

In conceptual terms, SS WG

represents the deviation between each individual score and the condition mean for that individual. To compute SS

WG in our movie ratings study, we sub-

tract each individual score from the relevant condition mean, square the difference, and add up the squared deviation scores.

That’s a lot of math, so let’s take a step back. In essence, these terms are calculating and comparing two sources of variation around the overall mean. When we combine all the participants’ movie ratings, and ignore everything about what they were rating, we get an overall mean of 5.13. But, naturally, individual participants deviate from this mean— some people gave higher ratings and some gave lower ratings. The point of calculating SS terms is to understand these deviations. Our between-group SS gives us an index of how much each condition deviates from the overall mean, while our within-group SS gives us an index of how much individuals differ from the mean of their condition. If the former is higher, it means that the groups’ results matter; if the latter is higher, it means that indi- vidual quirks mean more than our independent variables.

These sums of squares are the first step toward calculating our F statistic, but they have a flaw that has to be addressed. Namely, the sum of squares is an imprecise measure because it does not take sample size into account. You’ll notice that the SS terms are calcu- lated by adding up the squared deviation scores for each participant (or each condition). If our experiment happened to have a large sample, the SS

WG would automatically be higher

because there would be a greater number of squared deviation scores. Likewise, if our experiment happened to have several conditions, the SS

BG would be inflated. Fortunately,

this flaw is easily corrected by calculating a value called the mean square, which corrects the sum of squares for sample size by dividing by the appropriate degrees of freedom. The resulting values represent the average deviation from the sample mean, corrected for the number of scores that went into the calculation—much like the final step in our calcu- lation of standard deviations in Chapter 3 (Section 3.5). So, our components of variance formula is altered one more time:

MS T 5 MS

BG 1 MS

WG MS

T 5 mean square total

MS BG

5 mean square between groups MS

WG 5 mean square within groups

new85743_05_c05_213-280.indd 256 6/18/13 12:54 PM

257

CHAPTER 5Section 5.5 Analyzing Experiments

We now have all the information we need to calculate the F statistic (ANOVA), using the following formula:

F 5 between-groups variability within-groups variability

5 MSBG MSWG

This is the formula statistical software programs use to calculate an F value. F is expressed as a ratio of variances, expressing group differences relative to error. Thus, the bigger the F value, the higher the ratio of group differences to error, and the bigger the influence of independent variables. In other words, we want F values to be as large as possible because this indicates that our experimental manipulations make a difference on the out- come variable.

We can also frame the components of this ratio in terms of the null hypothesis versus the experimental hypothesis. If our experimental hypothesis is true, MS

WG represents

random variation and MS BG

represents meaningful differences between groups. Thus, MS

BG . MS

WG , and F will be a large value. In contrast, if the null hypothesis is true, both

MS WG

and MS BG

represent random variation because our experimental manipulations do not have a meaningful effect. Thus, MS

BG 5 MS

WG , and the value of F is 1. One final pos-

sibility here is that our value of F can occasionally be less than 1. This occurs when results are more influenced by random variation than they are by the independent variables; in other words, MS

WG . MS

BG .

One-Way Versus Factorial ANOVA

Before we move on from the steps in our ANOVA, there is one more level of complexity to explore. The formula we have been discussing so far describes an ANOVA only for a one- way design. When there is one independent variable, MS

T 5 MS

BG 1 MS

WG . However,

a factorial design has multiple independent variables, and the between-group variance term (MS

BG ) is divided into several elements. As we discussed, a 2 3 2 design has three

possible results: a main effect of variable A; the main effect of B; and the interaction of A and B. To account for these effects, we split our between-group variance into three new components:

MS BG

5 MS A 1 MS

B 1 MS

AB MS

A 5 MS

B 1 MS

AB MS

A 5 mean square for the main effect of A

MS B

5 mean square for the main effect of B MS

AB 5 mean square for the interaction of A and B

This results in another update to our components of variance formula, replacing the MS BG

term with these three components:

MS T 5 (MS

A 1 MS

B 1 MS

AB ) 1 MS

WG

This also results in three F values, expressing each mean square relative to within-group variance. These are illustrated next using our movie ratings study. Our 2 3 2 design in that study yields a main effect of actor, a main effect of movie type, and the actor * movie interaction:

new85743_05_c05_213-280.indd 257 6/18/13 12:54 PM

258

CHAPTER 5Section 5.5 Analyzing Experiments

Research: Thinking Critically

Love Ballad Leaves Women More Open to a Date

Medical News Today

If you’re having trouble getting a date, French researchers suggest that picking the right soundtrack could improve the odds. Women were more prepared to give their number to an “average” young man after listening to romantic background music, according to research that appears in the journal Psychology of Music, published by SAGE.

There’s plenty of research indicating that the media affects our behaviour. Violent video games or music with aggressive lyrics increases the likelihood of aggressive behaviour, thoughts, and feel- ings—but do romantic songs have any effect? This question prompted researchers Nicolas Guéguen and Céline Jacob from the Université de Bretagne-Sud along with Lubomir Lamy (2009) from Univer- sité de Paris-Sud to test the power of romantic lyrics on 18–20 year old single females. And it turns out that at least one romantic love song did make a difference.

Guéguen and Jacob were part of a research team that had already shown how romantic music played in a flower shop led to male customers spending more money. This time the researchers used questionnaires to pinpoint agreed-upon neutral and romantic songs. They chose ‘Je l’aime à mourir,’ a well-known love song by French songwriter Francis Cabrel, and the neutral song ‘L’heure du thé,’ by Vincent Delerm. A group of young women separate from the main study rated 12 young male volunteers for attractiveness, and the researchers picked the one rated closest to “average” to help with the experiment.

(continued)

FACTOR 5 MSACTOR

MSWG

FMOVIE 5 MSMOVIE

MSWG

FACTOR 3 MOVIE 5 MSACTOR 3 MOVIE

MSWG For illustration purposes, consider the following formula, representing the components of variance in a three-way design (which, incidentally, would yield seven F values!). As our designs become more complex, it becomes easier to appreciate the benefits of statistical software packages.

MS T 5 (MS

A 1 MS

B 1 MS

C 1 MS

AB 1 MS

BC 1 MS

AC 1 MS

ABC ) 1 MS

WG MS

A 5 mean square for the main effect of A

MS B 5 mean square for the main effect of B

MS AB

5 mean square for the interaction of A and B MS

BC 5 mean square for the interaction of B and C

MS AC

5 mean square for the interaction of A and C MS

ABC 5 mean square for the three-way interaction of A, B, and C

new85743_05_c05_213-280.indd 258 6/18/13 12:54 PM

259

CHAPTER 5Section 5.5 Analyzing Experiments

Research: Thinking Critically (continued) The researchers then set up a scenario where the 87 females each spent time in a waiting room with background music playing, before moving to a different room where the experimenter instructed her to discuss the difference between two food products with the young man. Once the experimenter returned, she asked them to wait for a few moments alone, and this gave the “average” male a chance to use his standard chat-up line: “My name is Antoine, as you know, I think you are very nice and I was wondering if you would give me your phone number. I’ll phone you later and we can have a drink together somewhere next week.”

The love song in the waiting room almost doubled Antoine’s chances of getting a woman’s num- ber—52% of participants responded to his advances under the influence of Francis Cabrel, whereas only 28% of those who had heard the “neutral” song by Vincent Delerm offered their details.

“Our results confirm that the effect of exposure to media content is not limited to violence and could have the potential to influence a high spectrum of behaviour,” says Guéguen. “The results are inter- esting for scientists who work on the effect of background music on individuals’ behaviour.”

The results also add weight to a general learning model proposed by Buckley and Anderson (2006) to explain the effect of media exposure. Their model states that media exposure in general, and not only aggressive or violent media, affects individuals’ internal states, which explains why prosocial media fosters prosocial outcomes.

Why did the music have this effect? It may be that, as shown in earlier research, the music induced positive affect (in psychological terms, affect is the experience of feeling or emotion). Positive affect is associated with being more receptive to courtship requests. Alternatively, the romantic content of the song may have acted as a prime that then led to displays of behaviour associated with that prime. In either case, further research is needed before the researchers will commit to wider gen- eralisations on the targeted use of love songs. But if you’re a hopeful single, awareness of the back- ground music certainly won’t do any harm.

Epping, J. (2010, June 21). Love ballad leaves women more open to a date. Medical News Today. Retrieved from http://www .medicalnewstoday.com/articles/192324.php

Think about it:

1. In this experiment, the type of song (love song or neutral song) is confounded with at least one other variable. Try to identify one. Do you think that this confounded variable would make a difference? How would you design a study that would overcome this?

2. Describe how demand characteristics might compromise the internal validity of this study. Can you think of any ways around this?

3. Toward the end of the article, the authors suggest that one explanation for these results is that the romantic music put the women into a more positive mood, and that this in turn made them more receptive to the man. How could you design a study that tested this hypothesis?

4. Given the nature of the dependent variable in this study, would an ANOVA test be appropriate? What would be the more appropriate statistical test, and why?

new85743_05_c05_213-280.indd 259 6/18/13 12:54 PM

260

CHAPTER 5Section 5.5 Analyzing Experiments

Exploring Your Data

Now that we have F values, what do we do with them? At the end of conducting an ANOVA, we have a yes-or-no answer to the following question: Do our experimental groups have a systematic effect on the dependent variable? The answer lets us decide whether to reject the null hypothesis, but it does not tell us everything we want to know about the data. In essence, a significant F value tells us that there is a significant difference between the groups, but it does not tell us what the difference is. If we conducted an ANOVA on our movie ratings study, we would see a significant interaction between actor and movie, but we would need to take additional steps to determine the meaning of this interaction.

In this section, we will walk through the process of exploring and interpreting ANOVA results in order to make sense of the data. Our example is drawn from a published study by Newman, Sellers, and Josephs (2005), which was designed to explore the effects of tes- tosterone on cognitive performance. Previous research had suggested that testosterone is involved in two types of complex human behavior. On one hand, people with higher tes- tosterone tend to perform better on tests of spatial skills, such as having to rotate objects mentally, and perform worse on tests of verbal skills, such as listing all the synonyms for a particular word. These patterns are thought to reflect the influence of testosterone on developing brain structures. On the other hand, people with higher testosterone are also more concerned with gaining and maintaining high status relative to other people. Tes- tosterone is correlated with a person’s position in the hierarchy and tends to rise and fall when people win and lose competitions, respectively. In one study by the sociologist Alan Mazur and his colleagues, testosterone levels were measured before, during, and after a series of professional chess matches. They found that testosterone rose in both players in anticipation of the competition, then rose even further in the winners, but plummeted in the losers (Mazur, Booth, & Dabbs, 1992).

Newman and colleagues (2005) set out to test the combination of these variables. Based on previous research, they hypothesized that people with higher testosterone would be uncomfortable when they were placed in a low-status position, leading them to perform worse on cognitive tasks. The researchers tested this hypothesis by randomly assigning people to a high-status, low-status, or control condition and then administering a spatial and a verbal test. This resulted in a 2 (testosterone: high or low) 3 3 (condition: high status, low status, control) between-subject design, for a total of six groups. Note that “testosterone” in this study is a quasi-independent variable because it is measured rather than manipulated by the experimenters.

Once the results were in, the ANOVA revealed an interaction between testosterone and status but no main effects. The results of the study are shown in Figure 5.9. These bars rep- resent z-scores that combine the spatial and verbal tests into one number. So what do these numbers mean? How do we make sense out of the patterns? This involves a combination of comparing means and calculating effect sizes, as discussed next.

new85743_05_c05_213-280.indd 260 6/18/13 12:54 PM

261

CHAPTER 5Section 5.5 Analyzing Experiments

Figure 5.9: Exploring your data: Results from Newman et al. (2005)

Comparing Means The first step in interpreting our results is to compare the various pairs of means within the design. This might seem counterintuitive, since the whole point of the ANOVA is to test for effects without comparing individual means. Our goal, therefore, is to somehow explore differences in conditions without inflating Type I error rates. There are two strate- gies for achieving this balance.

Planned comparisons (also called a priori comparisons) involve comparing only the means for which differences were predicted by the hypothesis. In the experiment by New- man et al. (2005), the hypothesis explicitly stated that high-testosterone people should perform better in a high-status position than a low-status position. So a planned compari- son for this prediction would involve comparing two means with a t-test: high T, high status (the highest red bar); and high T, low status (the lowest yellow bar). Consistent with the researchers’ hypothesis, high-testosterone people did perform higher on both tests, t(27) 5 2.35, p 5 0.01, but only in a high-status position. Type I errors are of less concern with planned comparisons because only a small number of theoretically driven compari- sons are being conducted.

If you refer to the graph of these results in Figure 5.9 and compare high- with low- testosterone people, you’ll notice another interesting pattern: In a high-status position, high-testosterone people do better than low-testosterone people. But in a low-status position, this pattern is reversed, and high-testosterone people do worse. However, the researchers did not predict these mean comparisons, so it would be cheating to do planned contrasts. Instead, they would use a second strategy called a post hoc comparison, which controls the overall alpha by taking into account the fact that multiple comparisons are being performed. In most cases, post hoc tests are allowed only if the overall F test result is significant.

C o

m b

in e d

C o

g n

it iv

e T

e s t

S c o

re

High Testosterone Low Testosterone

T Level

Leader

Control

Follower

–0.6

–0.4

–0.2

0

0.2

0.4

0.6

new85743_05_c05_213-280.indd 261 6/18/13 12:54 PM

262

CHAPTER 5Section 5.5 Analyzing Experiments

One popular way to conduct post hoc tests while minimizing the error rate is to use a tech- nique called a Bonferroni correction. This technique, named after the Italian mathemati- cian who developed it (Carlo Emilio Bonferroni, 1892–1960), involves simply adjusting the alpha level by the number of comparisons that are performed. For example, imagine you want to conduct 10 follow-up post hoc tests to explore your data. The Bonferroni cor- rection would involve dividing the alpha level (.05) by the number of comparisons (10), for a corrected alpha level of .005. Then, rather than using a cutoff of .05 for each test, you use this more conservative Bonferroni-corrected value. Translation: Rather than accepting a Type I error rate of 5%, we are moving to a more conservative 0.5% cutoff to correct for the number of comparisons that we are performing.

Another popular alternative to the Bonferroni correction is called Tukey’s HSD (for Honestly Significant Difference). This test works by calculating a critical value for mean comparisons (the HSD) and then using this critical value to evaluate whether mean comparisons are significantly different. This test manages to avoid inflating Type I error because the HSD is calculated based on the sample size, the number of experimental conditions, and the MS

WG , which essentially tests all the comparisons at once. (Note that

ANOVA will tell us only that the means are different but not which of them, so for the pur- pose of identifying the means that are different from others, one can apply Tukey’s test.) In the study by Newman et al. (2005), both of these post hoc tests were significant: Compared with those low in testosterone, high-testosterone people did better in a high-status posi- tion but worse in a low-status position. This suggests that high testosterone magnifies the effect of testing situations on cognitive performance.

Calculating Effect Size As you’ll remember from our discussion in Chapter 2 (Section 2.4), statistical significance is only part of the story; researchers also want to know how big the effects of their inde- pendent variables are. There are several ways to calculate effect size, but in general, bigger values mean a stronger effect. One of these statistics, Cohen’s d (named for Jacob Cohen, 1923–1998), is calculated as the difference between two means divided by their pooled standard deviation. The resulting values can therefore be expressed in terms of standard deviations; a d of 1 means that the means are one standard deviation apart. How big should we expect our effects to be? Based on his analyses of typical effect sizes in the social sciences, Cohen suggested the following benchmarks: d 5 .20 is a small effect; d 5 .40 is a moderate effect; and d 5 .60 is a large effect. In addition to these qualitative categories, effect size values can be interpreted in terms of standard deviation units. So, a d of 1 is equivalent to a standard deviation of 1. In other words, a large effect in social and behav- ioral sciences accounts for a little more than half of a standard deviation.

In interpreting the results of their testosterone experiment, Newman and colleagues (2005) computed effect size measurements for two of the key mean comparisons. First, they compared high-testosterone people in the high- and low-status conditions; the size of this effect was d 5 .78. Second, they compared the high- and low-testosterone people in the low-status condition; the size of this effect was d 5 .61. Both of these effects fall into the “large” range based on Cohen’s benchmarks. More important, taken together with the mean comparisons, they help us to understand the way testosterone affects behavior. The authors concluded that cognitive performance stems from an interaction between biology (testosterone) and environment (assigned status) such that high-testosterone people are

new85743_05_c05_213-280.indd 262 6/18/13 12:54 PM

263

CHAPTER 5Section 5.6 Wrap-Up: Avoiding Error

more responsive to their status in a given situation. When they are placed in a high-status position, they relax and perform well. But when placed in a low-status position, they become distracted and perform poorly. This nuanced conclusion is possible only through an exploration of the data, using mean comparisons and effect size measures.

5.6 Wrap-Up: Avoiding Error

As we wrap up our final chapter, it is worth thinking back to one of the key concepts in Chapter 2: Type I and Type II errors. Regardless of the research question, the hypothesis, or the particulars of the research design, all studies have the goal of making accurate decisions about the hypotheses. That is, we need to be able to correctly reject the null hypothesis when it is false and to fail to reject the null when it is true. But from time to time, and despite our best efforts, we make mistakes when we draw conclu- sions about our hypotheses, as summarized in Table 5.7. A Type I error, or “false posi- tive,” involves falsely rejecting a null hypothesis and getting excited about an effect that is the result of chance. A Type II error, or “false negative,” involves failing to reject the null hypothesis and missing an effect that is real and interesting. (If you need more of a refresher on these terms, refer back to Section 2.4, Hypothesis Testing.)

Table 5.7: Review of Type I and Type II errors

Researcher’s Decision

Reject Null Fail to Reject Null

Null is FALSE Correct Decision Type II Error

Null is TRUE Type I Error Correct Decision

In this section, we take a problem-solving approach to minimizing both of these errors in an experimental context. It turns out that each error is primarily under the researcher’s control at different stages in the research process, which means there are different strate- gies for reducing each error.

Avoiding Type I Error

Type I errors occur when results are caused by chance but are mistakenly interpreted as significant. We can generally reduce the odds of this happening by setting our alpha level at p , .05, meaning that we will get excited only about results that have less than a 5% chance of Type I error. However, Type I errors can still occur as a result of using either extremely large samples or large numbers of statistical comparisons. Large samples can make small effects seem highly significant, so it is important to set a more conservative alpha level in large-scale studies. And, as we have been discussing in this chapter, the odds of Type I error are compounded with each statistical test we conduct.

new85743_05_c05_213-280.indd 263 6/18/13 12:54 PM

264

CHAPTER 5Section 5.6 Wrap-Up: Avoiding Error

This means that Type I error is primarily under our control during statistical analysis— the smarter our statistics, the lower our odds of Type I error. We have covered several examples of “smart” statistics in this chapter: Instead of conducting lots of t-tests, we use an ANOVA to simultaneously test for differences across the entire design. Instead of conducting t-tests to compare means after an ANOVA, we use a mix of planned contrasts (for comparisons that we predicted) and post hoc tests (for other comparisons we want to explore). There are also more advanced statistical techniques that take this a step further. For example, the multivariate analysis of variance (MANOVA) statistic analyzes sets of dependent variables to further reduce the number of individual tests. This approach is used when dependent variables represent different measures of a related concept, such as using heart rate, blood pressure, and muscle tension to capture the stress response. The MANOVA works, broadly speaking, by computing a weighted sum of these sepa- rate dependent variables (called a canonical variable) and using this new variable as the dependent variable. If you are interested in learning more about this and other advanced statistical techniques, check out the excellent volume by James Stevens (2002), Applied Multivariate Statistics for the Social Sciences, http://goo.gl/zgGDi.

Avoiding Type II Error

Type II errors occur when the results are significant but are mistakenly credited to chance. The primary sources of this error are small samples and bad design. Small samples may fail to capture enough variability and may therefore lead to nonsignificant p values in testing an otherwise significant effect. Both large and small mistakes in experimental designs can add noise to the dataset, making it difficult to detect the real effects of inde- pendent variables.

This means that Type II error is primarily under the researcher’s control during the design process—the smarter the research designs, the lower the odds of Type II error. First, as we discussed in Chapter 2, it is relatively simple to estimate the sample size needed for our research using a power calculator. These tools take basic information about the number of conditions in the research design and the estimated size of the effect and then estimate the number of people needed to detect this effect.

Second, as we have discussed in every chapter, it is the experimenter’s responsibility to take steps to minimize extraneous variables that might interfere with the hypothesis test. Whether conducting an observation, a survey study, or an experiment, the overall goal is to ensure that the variables of interest are the main cause of changes in the depen- dent variable. This is perhaps easiest in an experimental context because these designs are usually conducted in a controlled setting where the experimenter has control over the independent variables. Nonetheless, as discussed earlier in this chapter, many factors can threaten the internal validity of an experiment—from confounds to sample bias to expec- tancy effects. In essence, the more the researcher can control the influence of these extrane- ous variables, the more confidence he or she can have in the results of the hypothesis test.

Table 5.8 presents a summary of the information in this section, listing the primary sources of Type I and Type II errors, as well as the time period when these are under experimenter control.

new85743_05_c05_213-280.indd 264 6/18/13 12:54 PM

265

CHAPTER 5Section 5.7 Critiquing a Quantitative Study

Table 5.8: Summary—Avoiding error

Error Definition Main Source When Researcher Can Control

Type I False positive Lots of tests; lots of people

Conducting statistics

Type II False negative Bad measures; not enough people

Designing experiments

5.7 Critiquing a Quantitative Study

As discussed in Chapter 1 (Section 1.5, Searching the Literature) and Chapter 3 (Section 3.3, Critiquing a Qualitative Study), the methods and guidelines for evaluating research studies are fairly detailed for both quantitative and qualita- tive studies. For any type of study, it is important to read the entire article so that you can determine which parts of the study are valid, which parts can be considered accept- able but with caution, and which parts have significant limitations. As mentioned in Chapter 3, a study does not necessarily have valid and reliable results even if it claims that it does. Not all studies are worthy of being taken seriously, and it is important that you, as a consumer of psychological and other types of research, are able to identify potential problems.

There are several factors to consider when evaluating the sections of a quantitative research study or proposal. The following sections discuss how to critically evaluate the literature review, the purpose statement, the sampling methods, the procedures, the instruments, the results, and the discussion.

Evaluating the Literature Review Section

Similar to a qualitative study, the literature review section of a quantitative study is meant to provide an adequate rationale for conducting the study and to show what dif- ference the study hopes to make in the field. Literature reviews should be thorough and comprehensive and contain relevant research on the topic being studied. They should also be objective, unbiased in the selection of articles being reviewed, and include previ- ous research that relates to the current study. Considering these daunting requirements, the following questions, adapted from Houser (2009), can help you to evaluate the litera- ture review:

• Do the researchers present an adequate rationale for conducting the study? • What is the significance of the study? What difference will it make to the field? • Is the literature review thorough and comprehensive? • Do the researchers demonstrate any potential biases in the literature review? • Are all important concepts clearly defined by the researchers? • Do the researchers clearly describe previous methods that are relevant to under-

standing the purpose for conducting this study?

new85743_05_c05_213-280.indd 265 6/18/13 12:54 PM

266

CHAPTER 5Section 5.7 Critiquing a Quantitative Study

Evaluating the Purpose Statement and Hypotheses

As discussed in Chapter 3, the purpose statement does the following: provides the aim or intent of the study; the focus, methodology, population, and geographical location of the study; the specific area of the research and the central concepts or ideas of the study; jus- tification for the method of inquiry (qualitative, quantitative, or mixed); and a description of possible conclusions and their implications. It is generally found in the Introduction, as the last paragraph before the Literature Review section. Purpose statements can be writ- ten in declarative form or in the form of a question or questions. They should include the type of research methodology and design used, as well as a description of the variables and population studied. When evaluating the purpose statement, it is important to exam- ine whether the purpose and research problem are researchable. Purpose statements and research problems are researchable when the variables of interest can be operationalized, or defined in a measurable and objective way.

It is important to note that not all researchers use purpose statements in quantitative stud- ies, but instead generally provide only hypotheses. It is not typical for a researcher to include the null hypothesis, as it can be inferred that every researcher has started out with one. Rather, alternative types of research hypotheses are given. For example, some research will include alternative nondirectional hypotheses. These indicate that there is enough evidence from the literature to suggest a difference, but not enough to predict the direction of that difference. Another hypothesis is the alternative directional hypothesis, which suggests a difference and the direction of that difference.

Hypotheses guide researchers in selecting appropriate statistical analyses, so there should be consistency between the hypotheses and statistical tests chosen. The follow- ing questions, adapted from Houser (2009), are provided to help you to evaluate the purpose statement:

• Does the article clearly state the purpose statement? • Is the purpose statement clearly based on the argument developed in the litera-

ture review? • Are the variables of interest (i.e., independent and dependent) clearly identified

in the problem statement? • What type of hypothesis is presented? • Is there a clear rationale for the direction of the hypothesis, based on the litera-

ture review? • Is the hypothesis testable and falsifiable? • Is the hypothesis consistent with current theories or research?

Evaluating the Methods Section: Sampling

The Sampling section includes thorough, detailed information about the sample that was chosen and the sampling techniques or methods that were used to select it. Descrip- tions of the sample should include all relevant demographic characteristics (e.g., age, ethnicity, gender) as well as the size. Unlike qualitative approaches, which do not have hard-and-fast rules regarding sample size, quantitative approaches often require much larger samples. As noted in Chapter 4 (Section 4.3), Gay, Mills, and Airasian (2009) pro- vided the following guidelines for determining a sufficient sample size based on the size of the population:

new85743_05_c05_213-280.indd 266 6/18/13 12:54 PM

267

CHAPTER 5Section 5.7 Critiquing a Quantitative Study

• For populations that include 100 or fewer individuals, the entire population should be sampled.

• For populations that include 400–600 individuals, 50% of the population should be sampled.

• For populations that include 1,500 individuals, 20% of the population should be sampled.

• For populations that are 5,000 or larger, about 8% of the population should be sampled. (Note, however, that it is possible to have a sample that is so large that it does not justify the increased costs, and may produce significant differences that are too small to be meaningful).

The sampling techniques employed should be discussed, including detailed information about how the sample was selected and what methods were used (e.g., simple random sampling, stratified sampling, cluster sampling). The following are a few of the most com- mon sampling procedures used in quantitative studies. These are discussed in more detail in Chapter 4, Section 4.3, Sampling From the Population.

• Simple Random Sampling: The researcher randomly picks study participants from a list of everyone in the population.

• Stratified Random Sampling: The researcher breaks the sampling frame into sub- groups and then samples a random number from each subgroup.

• Cluster Sampling: The researcher randomly selects intact groups. • Systematic Random Sampling: The researcher selects every nth participant from

a finite list of those in the population.

Sampling errors and sampling biases should also be considered. Sampling errors can occur when the sample selected is not representative of the population. This generally occurs when smaller sample sizes are used. Sampling bias occurs when the researcher purposely selects participants in a particular way that introduces bias into the sample. Convenience samples typically contain sampling bias because the samples are not usually representative of the entire population.

The following questions, adapted from Houser (2009), can help you evaluate the sampling methods section:

• What type of sampling method is used? • Are the sampling procedures consistent with the purpose and hypotheses? • Are relevant demographic characteristics of the sample clearly identified? • Do the methods of sample selection provide a good representative sample, based

on the population? • Are there any apparent biases in the selection of the sample? • Is the sample large enough for the study proposed? How has size been justified? • To whom can study results be generalized?

Evaluating the Methods Section: Procedures

The Procedures section describes in detail everything that was conducted in the study. For quantitative studies, it discusses the study variables, how informed consent was obtained, how the participants were assigned to groups, and how the researcher controlled for

new85743_05_c05_213-280.indd 267 6/18/13 12:54 PM

268

CHAPTER 5Section 5.7 Critiquing a Quantitative Study

extraneous or confounding variables. This section should clearly describe the indepen- dent variable as well as give details about whether a control group existed, and if so, how it was treated. Additionally, it should cover how the participants were randomly assigned to the experimental group and control group. When evaluating the procedures section, one should assess whether there are any threats to the internal or external validity of the study. Furthermore, it is important to examine whether the research design was appropri- ate for the study and whether it was consistent with the purpose and research questions. The following questions, adapted from Houser (2009), can assist in the evaluation of the Procedures section:

• What type of research design was used? • Was the research design consistent with the purpose and hypotheses? • Did the researcher provide a detailed description of what was conducted? • Did the researcher introduce any bias in the procedures used? • Was the independent variable clearly defined? • Were there any threats to the internal validity of the study? • Were there any threats to the external validity of the study?

Evaluating the Methods Section: Instruments

The Instruments section describes in detail the types of instruments and measures that were used to collect the data. In quantitative research, a variety of common instruments are used: direct observation and behavioral measures, self-report inventories, ratings of peoples’ behaviors, questionnaires, physiological approaches, interviews, and standard- ized assessments.

When evaluating the Instruments section, consider whether the instruments were appro- priate for the study and the sample and whether there were any limitations in the types of instruments selected. As noted by Houser (2009), “An important consideration in assess- ing the objectivity and quality of a dependent measure is the extent to which the proce- dures and scoring methods are standardized” (p. 167). A test is not highly reliable or valid simply because it is standardized. Whenever evaluating instruments, it is important to assess the test’s reliability and validity as well as its use in various populations. It is also important to assess whether the tests selected were consistent with the sample selected, the research problem, and the hypotheses. Studies that employ standardized assessments or standardized administration and scoring procedures generally obtain more reliable and valid data collection and results. The following questions, adapted from Houser (2009), can help guide the evaluation of the Instruments section:

• Is there a clear and adequate description of the instrument (data collection mea- sures) used?

• What types of instruments were used in the study (self-report inventories, rat- ings, etc.)?

• Are the instruments standardized? • What are some of the potential problems or limitations with the types of instru-

ments used? • Does the instrument appear to be appropriate for the sample? • Is the instrument consistent with the hypotheses? • Is there a discussion about the reliability and validity of the instruments used?

new85743_05_c05_213-280.indd 268 6/18/13 12:54 PM

269

CHAPTER 5Section 5.8 Mixed-Methods Research Designs

Evaluating the Results Section

The Results section describes the findings of the study. Unlike qualitative studies, which focus more on descriptions about the findings, quantitative studies are usually presented mathematically through statistical analyses.

Two forms of statistics are presented in the Results section: descriptive and inferential. Descriptive statistics summarize the data collected from the sample, such as demographic information. Inferential statistics use statistical methods or analyses to test hypotheses. When evaluating the Results section, consider whether appropriate statistics were used and whether those statistics were consistent with the purpose of the study and hypoth- eses. The following questions, adapted from Houser (2009), will help:

• What descriptive statistics were used? • Do the descriptive statistics adequately describe the sample? • What types of inferential statistics were used? • Are the inferential statistics used adequate to answer the hypotheses? • Is the research design consistent with the statistical methods used?

Evaluating the Discussion Section

Whether evaluating a quantitative or qualitative study, the Discussion section summa- rizes the purpose of the research, as well as the findings’ implications for future research and actual practice. Additionally, the Discussion section includes alternative explanations and potential limitations of the findings. The following questions, adapted from Houser (2009), can shed light on the Discussion section:

• Did the researchers clearly restate the purpose and hypotheses? • Did the researchers clearly discuss the implications of the findings and how they

relate to theories, other findings, and actual practice? • Did the researchers provide alternative explanations of the findings? • Has there been appropriate generalization of significant findings beyond the

sample to the population? • Did the researchers identify potential limitations of the study and the results? • Did the researchers identify possible directions for future research?

5.8 Mixed-Methods Research Designs

This chapter and the previous three have discussed the differences between quali-tative and quantitative research methods. From your readings, it may appear that these methods are totally exclusive and are never mixed or combined in a study. However, this is not actually the case. In recent years, researchers have started to utilize both approaches in their designs to provide a more comprehensive understanding of research problems. Mixed methods research is fairly new, and the integration and mix- ing of various qualitative and quantitative approaches can be tricky. As a result, books and articles have been published that provide comprehensive overviews of this type of inquiry.

new85743_05_c05_213-280.indd 269 6/18/13 12:54 PM

270

CHAPTER 5Section 5.8 Mixed-Methods Research Designs

Mixed-methods research uses the process of triangulation, or the combination of mul- tiple research methods to study a research problem. In order to successfully achieve this, several approaches can be employed, including scaling, reliability, and convergent valida- tion. Scaling involves converting qualitative data into quantitative or numeric form. This involves creating codes and themes in the qualitative data collected and then counting the number of times those codes or themes occur (Creswell, 2009). This approach allows researchers to compare the qualitative results with the quantitative ones. Reliability involves the use of similar methods to measure the same thing. For example, researchers might utilize unstructured (qualitative) and structured (quantitative) interviews to exam- ine the experiences of dementia caregivers. Finally, convergent validation involves utiliz- ing different but complementary methods (Houser, 2009). For example, a researcher might use a survey to collect data about AIDS prevention knowledge and then follow up with an unstructured or semistructured interview. In each of these triangulation methods, the researcher is combining qualitative and quantitative approaches to gain a more thorough picture of the research problem.

Mixed-Methods Strategies

Mixed-methods designs generally utilize one of six mixed strategies or models: sequential explanatory, sequential exploratory, sequential transformative, concurrent triangulation, concurrent embedded, or concurrent transformative.

The sequential explanatory strategy is one of the most popular mixed methods used, especially among quantitative researchers. This type of strategy includes two phases: The first phase involves the quantitative collection and analysis of data, and the second phase involves the qualitative collection and analysis of data that build on the results of the quantitative data. For example, a researcher might administer a standardized instrument to measure depression symptoms and then follow up with a semistructured interview to explore the participants’ personal experiences with depression. In this way, the interview would supply additional information about why the participants responded certain ways on the standardized instrument. This type of strategy is generally used to further explain and interpret the quantitative results.

The sequential exploratory strategy is similar to the sequential explanatory strategy except that the phases are switched. In this type of strategy, the first phase is the qualita- tive collection and analysis of data, whereas the second phase is the quantitative collection and analysis of data that build upon the qualitative results. Thus, the quantitative results are used to further explain and interpret the qualitative results.

The sequential transformative strategy is similar to the other two sequential strategies in that it includes two phases; however, the sequence of the phases is decided by the needs of the researcher. Thus, the researcher selects methods that will best support his or her theoretical perspective.

The concurrent triangulation strategy involves the researcher collecting both qualita- tive and quantitative data concurrently and then comparing them for similarities and

new85743_05_c05_213-280.indd 270 6/18/13 12:54 PM

271

CHAPTER 5Section 5.8 Mixed-Methods Research Designs

differences (Creswell, 2009). Concurrent triangulation is often used to support or discon- firm quantitative results. Similar to concurrent triangulation, the concurrent embedded strategy includes collecting qualitative and quantitative data concurrently; however, a primary method guides the research, and a secondary method provides a supportive role. The secondary method is given less priority and is embedded or “nested” within the primary method. For example, a primary method might include assessing treatment out- comes (a quantitative approach) while a secondary method explores the processes people experience during the study (a qualitative approach) (Creswell, 2009).

The last model is concurrent transformative strategy. It resembles the sequential trans- formative strategy in that the methods it employs are based on the theoretical perspective of the researcher, but the data are collected concurrently. And, similar to the concurrent embedded strategy, this strategy also utilizes a primary method and a secondary method that is embedded within the primary method.

The following are examples of mixed-methods designs.

Example 1: Sequential Explanatory Design Celestine Pea (2012) conducted research to examine whether environmental factors (e.g., human, sociocultural, design) affect inquiry-based teaching. She employed a mixed meth- ods, sequential explanatory design with a multicase research approach to conduct the study. A survey of middle-school science teachers was followed by case studies of three teachers from the survey pool (two females and one male). Here is an excerpt describing the study’s methods:

Data Collection and Analysis Quantitative methods: Data collection and analysis. An open-ended Likert- style survey was used to collect quantitative data. The survey included a section for demographics information and individual variables that included gender, years of teaching science, degree, certification in subject taught, and semester credits in science. The survey data were prepared for computation. Then, descriptive statistics, a paired i-test, and analysis of variance (ANOVA) were used to compute the survey data. The outcomes were analyzed and interpreted to build a cross-case report.

Qualitative methods: Data collection and analysis. Interviews and observa- tions were used to collect qualitative data. Protocols were developed for each of these methods. Each teacher was interviewed before, during, and after being observed using a semi-structured approach. An exit interview was also conducted at the end of the observation sessions. The interviews lasted 30–60 minutes. (Pea, 2012, pp. 36–37)

Notice how the data collection section is organized into quantitative methods and qualita- tive methods subsections. This is common practice in sequential studies, whose aim is to conduct separate data collections, with the latter elaborating on either the original qualita- tive or quantitative findings.

new85743_05_c05_213-280.indd 271 6/18/13 12:54 PM

272

CHAPTER 5Section 5.8 Mixed-Methods Research Designs

Example 2: Concurrent Embedded Design Brunetto and Farr-Wharton (2008) conducted a study to . . .

compare the impact of the quality of employee enablers (in this case mea- sured by the quality of the communication relationship between local gov- ernment employees and supervisors) with the ideal “high quality social capital network” on the quality of outputs (such as employees’ level of ambiguity in relation to customers and/or their level of productivity). (p. 37)

The following is an excerpt describing the study’s methods.

Methodology This study uses mixed methods—hence, both qualitative and quantitative research tools are used (Rocco et al., 2003). However, a concurrent nested strategy is used in that the primary source of evidence used for the find- ings is quantitative—in both data collection and analysis (Cresswell, 2003). Therefore data obtained from qualitative research methods is used only to support trends identified using quantitative analysis and aid triangulation, which in turn, increases the external validity and achieves greater under- standing of the phenomenon (Rocco et al., 2003). Quantitative data was obtained using a cross-sectional, survey-based, self-report strategy. While there is some criticism of this method (because of the problems of common method variance and same source bias), it is in this case, the best way of identifying the intangible values of the organisation by asking public sec- tor local government employees about their perceptions of the quality of information-sharing. On the other hand, the role of qualitative research methods (such as interviews) is to generate data that can explain the rea- sons for the trends evident from analyzing the quantitative results (Yin, 1994; Rocco et al., 2003). Once the data was collected, the findings were compared with the findings of previous research analysed in the literature review, that is, using a pattern-matching logic (Yin, 1994). (Brunetto & Farr- Wharton, 2008, p. 44 [footnotes omitted])

Notice how the data in this concurrent embedded study were not described in sep- arate quantitative and qualitative sections as in the sequential study in Example 1. This is because concurrent studies seek to converge or find similarities among the data and results.

Regardless of the type of strategy, it is important to remember that utilizing both quan- titative and qualitative methods can be time-consuming and expensive. However, it also can provide more rigor, validity, and utility to a study. For further information on mixed methods designs, review the Handbook of Mixed Methods in the Social and Behavioral Sciences by Tashakkori and Teddlie (2003) or the Journal of Mixed Methods Research.

new85743_05_c05_213-280.indd 272 6/18/13 12:54 PM

273

CHAPTER 5Summary

Summary

In this chapter, we have focused on experimental designs, the last group of specific research designs, and the highest point on the continuum of control. The primary goal of experimental designs is to explain behavior in terms of cause and effect. We began with an overview of experimental terminology and the key features of experi- ments. Experiments can be distinguished from other research designs by three key fea- tures. First, the researcher manipulates a variable, giving him or her a fair amount of confidence that the independent variable causes changes in the dependent variable. Sec- ond, the researcher controls the environment, ensuring that everything about the experi- mental context is the same for different groups of participants—except for the level of the independent variable. Third, the researcher has the power to assign participants to con- ditions, using random assignment. This process helps to ensure that standing differences among participants (e.g., in mood, motivation, intelligence) are balanced out across the experimental conditions.

Next, we covered the concept of experimental validity. In evaluating our experiments, we have to take into account both internal validity (the extent to which the independent vari- able is the cause of changes in the dependent variable) and external validity (the extent to which the results can be generalized beyond the specific laboratory setting). Several factors can threaten internal validity, including experimental confounds, selection bias, and expectancy effects. The common thread among these threats is that they add noise to the hypothesis test and cast doubt on the direct connection between independent and dependent variable. External validity involves two components: the realism of the study and the generalizability of the findings. Psychology experiments are designed to study real-world phenomena, but sometimes compromises have to be made to study these phe- nomena in the laboratory. The balance is often achieved via mundane realism, or replicating the psychological conditions of the real phenomenon. Finally, we have more confidence in the findings of our study when they can be replicated, or repeated in different settings with different measures.

We also discussed the three types of experimental designs: pre-experiments, quasi- experiments, and true experiments. In true experiments, researchers have the most con- trol over extraneous or confounding variables and therefore are more likely to uncover cause-and-effect relationships. In addition, true experiments always include random assignment and manipulation of the independent variable, whereas quasi-experiments and pre-experiments do not include random assignment.

In designing the nuts and bolts of true experiments, we have to make decisions about both the nature and number of independent variables. First, our designs can be described as between-subject, within-subject, or mixed. In a between-subject design, participants are subjected to only one experimental condition and receive only one combination of the independent variables. In a within-subject design, participants find themselves in all experimental conditions and receive all combinations of the independent variables. A mixed design combines between- and within-subject variables. Second, our designs can be described as either one-way or factorial. One-way designs consist of only one indepen- dent variable with at least two levels; factorial designs consist of at least two independent variables, each having at least two levels. In a factorial design, there are several results to examine: the main effect of each independent variable plus the interaction, or combina- tion, of the independent variables.

new85743_05_c05_213-280.indd 273 6/18/13 12:54 PM

274

CHAPTER 5Key Terms

between-group variance The variance explained by differences in the indepen- dent variables.

between-subject design Experimental design in which each group of participants is exposed to only one level of the inde- pendent variable.

Bonferroni correction A post hoc test that involves adjusting the alpha level by the number of comparisons to set a more con- servative cutoff point.

carryover effect When effects of one level of independent variable are present when another level is introduced, making it difficult to separate the effects of different levels.

conceptual replication Testing the rela- tionship between conceptual variables using new operational definitions.

concurrent embedded strategy A mixed methods strategy that includes collecting qualitative and quantitative data concur- rently; however, a primary method guides the research, and a secondary method plays a supportive role.

concurrent transformative strategy A mixed methods strategy that resembles the sequential transformative strategy in that the methods are based on the theoretical perspective of the researcher, but the data is collected concurrently. Also similar to the concurrent embedded strategy, this strategy utilizes a primary method and a secondary method that is embedded within the primary method.

We also discussed the process of analyzing experimental data, using the analysis of vari- ance (ANOVA) statistic. This test works by simultaneously comparing sources of variance and, therefore, avoids the risk of inflated Type I error. The ANOVA (or F) is calculated as a ratio of systematic variance to error variance, or, more specifically, of between-group variance to within-group variance. The bigger this ratio, the more experimental manipu- lations contribute to overall variability in scores. However, the F statistic only tells us that differences exist in the design, and further analyses are necessary to explore these differences. We walked through an example from a published study, discussing the pro- cess of comparing means and calculating effect sizes. In comparing means, we use a mix of planned contrasts (for comparisons that we predicted) and post hoc tests (for other comparisons we want to explore).

We then covered two recurring concepts: Type I error (false positive) and Type II error (false negative) and reviewed ways to minimize such errors. These errors interfere with our broad goal of making correct decisions about the status of our hypothesis. Type I errors are inflated primarily by large samples and performing lots of statistical analy- ses. Consequently, these errors are under the experimenter’s control at the data analysis stage. Type II errors are inflated primarily by small samples and flaws in the experimental design. Consequently, these errors are under the experimenter’s control at the design and planning stage.

An explanation of how to critique a quantitative study was also provided. Finally, we con- cluded the chapter with a review on mixed methods designs and the six strategies used to incorporate both qualitative and quantitative techniques. Although utilizing mixed meth- ods approaches can be time-consuming and costly, they can also increase the validity and utility of a study.

Key Terms

new85743_05_c05_213-280.indd 274 6/18/13 12:54 PM

275

CHAPTER 5Key Terms

concurrent triangulation strategy A mixed methods strategy that involves the researcher collecting both qualitative and quantitative data concurrently and then comparing the data to see whether there are any similarities or differences.

condition One of the versions of an inde- pendent variable, forming different groups in the experiment; in a factorial design, refers to the groups formed by combina- tions of independent variables.

confounding variable (confound) A vari- able that changes systematically with the independent variable; in the case of corre- lation studies, known as the third variable.

constructive replication Re-creation of the original experiment that adds elements to the design; usually designed to rule out alternative explanations or extend knowl- edge about the variables under study.

control condition Group within the exper- iment that does not receive the experimen- tal treatment.

convergent validation A form of triangu- lation that involves utilizing different but complementary methods to measure the same construct of interest.

counterbalancing Variation of the order of presentation among participants to reduce order effects.

cover story A misleading statement to participants about what is being studied to prevent effects of demand characteristics.

demand characteristic Cue in the study that leads participants to guess the hypothesis.

differential attrition (mortality threat) Loss of participants, who drop out of experimental groups for different reasons.

environmental manipulation Changing some aspect of the experimental setting.

exact replication Re-creation of the origi- nal experiment as closely as possible to verify the findings.

experimental condition Group within the experiment that receives a treatment designed to test a hypothesis.

experimental design Design whose pri- mary goal is to explain causes of behavior.

experimenter expectancies When research- ers see what they expect to see, leading to subtle bias in favor of their hypotheses; threat to internal validity.

external validity A metric that assesses generalizability of results beyond the spe- cific conditions of the experiment.

extraneous variable Variable that adds noise to a hypothesis test.

factorial design A design that has two or more independent variables, each with two or more levels.

factorial notation A system for describ- ing the number of variables and levels in experimental designs.

fatigue effect Decline of participants’ per- formance as a result of repeated testing.

generalizability The extent to which results extend to other studies, using a wide variety of populations and of opera- tional definitions; sometimes referred to as population validity.

history effect The influence on research participants by events in the outside world, such as political upheavals and natural disasters.

new85743_05_c05_213-280.indd 275 6/18/13 12:54 PM

276

CHAPTER 5Key Terms

instructional manipulation Changing the way a task is described to change partici- pants’ mind-sets.

interaction The combined effect of vari- ables in a factorial design; when the effects of one independent variable are different depending on the levels of the other inde- pendent variable.

internal validity A metric that assesses the degree to which results can be attrib- uted to independent variables.

invasive manipulation Taking measures to change internal, physiological processes; usually conducted in medical settings.

level Another way to describe the ver- sions of an independent variable; describes the specific circumstances created by manipulating a variable.

main effect The effect of each indepen- dent variable on the dependent vari- able, collapsing across the levels of other variables.

marginal mean The combined mean of one factor across levels of another factor.

matched random assignment A variation on random assignments; ensures that an important variable is equally distributed between or among the groups; the experi- menter obtains scores on an important matching variable, ranks participants on this variable, and then randomly assigns participants to conditions.

mean square A component of the F statistic; corrects the sum of squares for sample size by dividing by the appropriate degrees of freedom; resulting values repre- sent the average deviation from the sample mean, corrected for sample size.

mixed design Experimental design that contains at least one between-subject variable and at least one within-subject variable.

multivariate analysis of variance (MANOVA) A statistic that analyzes sets of dependent variables to reduce the num- ber of individual tests.

mundane realism Research that replicates the psychological conditions of the real- world phenomenon; criterion for judging external validity. Sometimes referred to as ecological validity.

one-shot case study A type of pre- experimental design that involves using one group, introducing an intervention, and then administering a posttest obser- vation to determine the effects of the intervention.

one-way design A design that has only one independent variable, with two or more levels to the variable.

order effect Moderation of the effects because of the order in which levels occur.

participant replication Repetition of the study with a new population of partici- pants; usually driven by a compelling the- ory as to why the two populations differ.

placebo control Group added to a study to separate placebo effects from the actual effects of a drug; mimics the experimental condition in every way but one.

placebo effect Change resulting from the mere expectation that change will occur.

planned (a priori) comparison Com- parison that involves comparing only the means for which differences were pre- dicted by the hypothesis.

new85743_05_c05_213-280.indd 276 6/18/13 12:54 PM

277

CHAPTER 5Key Terms

post hoc comparison Comparison that controls the overall alpha by taking into account that multiple comparisons are being performed; usually allowed only if the overall F test result is significant.

practice effect Improvement of partici- pants’ performance as a result of repeated testing.

pre-experimental design Research study design that confers virtually no control and involves studying a single group or unbalanced groups and then introducing an intervention or treatment.

quasi-experimental design Research study design that gives researchers some control and may or may not include a con- trol group but in which the participants are not randomly assigned to groups.

quasi-independent variable Existent difference used to divide participants in an experimental context; referred to as “quasi” because variables are being measured, not manipulated, by the experimenter.

random assignment A technique for assigning participants to conditions; before participants arrive, the experimenter makes a random decision about each par- ticipant’s placement in a group.

reliability (qualitative) A form of trian- gulation that involves the use of similar methods to measure the same construct of interest.

replication Repetition of research results in different contexts and/or different laboratories.

selection bias When groups are different before experimental manipulation; prob- lematic because existent differences might be the driving factor behind the results.

sequential explanatory strategy A mixed methods strategy that includes two phases: the first phase involving the quantitative collection and analysis of data and the sec- ond phase involving the qualitative collec- tion and analysis of data that build on the results of the quantitative data.

sequential exploratory strategy A mixed methods strategy similar to the sequential explanatory strategy except that the phases are switched. The first phase involves the qualitative collection and analysis of data, whereas the second phase involves the quantitative collection and analysis of data that build upon the qualitative results.

sequential transformative strategy A mixed methods strategy that includes two phases; however, the sequence of the phases is decided by the needs of the researcher. Thus, the researcher selects methods that will best support his or her theoretical perspective.

sum of squares The sum of the squared deviations between individual scores and the overall sample mean.

third variable See confounding variable.

triangulation The combination of mul- tiple research methods to study a research problem.

true experimental design Research study design that gives researchers maximum control; involves randomly assigning participants and manipulating the inde- pendent variable; includes randomized group selection and assignments, a control group, and a large degree of control over confounding variables.

new85743_05_c05_213-280.indd 277 6/18/13 12:54 PM

278

CHAPTER 5Apply Your Knowledge

Apply Your Knowledge

1. When ANOVA is used to analyze data from experiments with two independent variables (A and B), the total variability (MS

TOTAL ) is composed of four parts, or

components. What are they? (Writing the equation is fine.)

2. List and briefly describe the three distinguishing features of an experiment. a. b. c.

3. List the three types of expectancy effect that can affect experimental results, and name one way to avoid each type. a. b. c.

4. The following designs are described using factorial notation. For each one, state (a) the number of variables in the design, (b) the number of levels each variable has, and (c) the total number of experimental conditions.

3 3 3 3 3

a. b. c.

2 3 3 3 4

a. b. c.

Tukey’s HSD (Honestly Significant Dif- ference) A post hoc test that calculates a critical value for mean comparisons (the HSD) and then uses this critical value to evaluate whether mean comparisons are significantly different.

unrelated-experiments technique An experimental technique for preventing effects of demand characteristics; lead- ing participants to believe that they are completing two experiments during one session; experimenter can use this method to present the independent variable dur- ing the first experiment and measure the dependent variable during the second experiment.

within-group variance The total variance that occurs across conditions.

within-subject design Experimental design in which each group of participants is exposed to all levels of the independent variable.

new85743_05_c05_213-280.indd 278 6/18/13 12:54 PM

279

CHAPTER 5Apply Your Knowledge

4 3 4

a. b. c.

2 3 2 3 2 3 2

a. b. c.

5. Forty students were asked to rate two authors according to their knowledge of certain topic areas. Each student was given two passages to read. In one passage (“Brain”), the author discussed the roles of various brain structures in perceptual–motor coordination. In the second passage (“Motivation”), the author described ways to enhance motivation in preschool children. For half the students, both passages were written by male authors. For the other half of the students, both passages were written by a female author. After reading the passages, students rated the authors’ knowledge of their topic areas on a scale ranging from 1 (Displays very little knowledge) to 10 (Displays a thorough knowledge).

Male Author Female Author

Brain 9 4

Motivation 6 7

a. Identify the following information about the design: (1). Describe the design using factorial notation (e.g., 4 3 3). (2). Identify the total number of conditions. (3). Identify the design (circle one): between-subject within-subject mixed

b. The data from this study are analyzed and the results indicate the following F values:

(1). Topic Area: F 5 .91 (p 5 .87). (2). Author’s Gender: F 5 3.14 (p , .05). (3). Topic Area 3 Author’s Gender: F 5 12.46 (p , .01).

c. Interpret these results in terms of the three possible effects. That is, for each effect, state whether it is significant and then interpret the statistics in English.

(1). Is there a main effect of Topic Area? (circle one) YES NO Interpretation in English: (2). Is there a main effect of Author Gender? (circle one) YES NO Interpretation in English: (3). Is there an interaction between Topic Area and Author Gender?

new85743_05_c05_213-280.indd 279 6/18/13 12:54 PM

280

CHAPTER 5Critical Thinking & Discussion Questions

YES NO Interpretation in English:

6. For each of the following scenarios, identify what a Type I error and a Type II error would look like. Then, determine which type would be a bigger problem for that scenario. a. A large international airport has received a bomb threat. In response, the

airport police have tightened security and now check every piece of luggage manually.

(1). Type I: (2). Type II: (3). Bigger problem:

b. Your friend purchases a pregnancy test.

(1). Type I: (2). Type II: (3). Bigger problem:

Critical Thinking & Discussion Questions

1. Explain the advantages and disadvantages of a within-subject design. 2. Compare and contrast pre-experiments, quasi-experiments, and true experi-

ments. What is the primary difference between true experiments and both pre-experiments and quasi-experiments?

3. Compare and contrast the following terms. Your answers should demonstrate that you understand each term. Be sure to give some kind of context (e.g., “both are types of . . .”) or provide an example, and state how they are different. a. internal versus external validity b. between-subject versus within-subject design c. level versus condition

4. Explain the difference between Type I and Type II errors. How can each type of error be minimized?

5. Describe a research study that would require a mixed methods design.

new85743_05_c05_213-280.indd 280 6/18/13 12:54 PM