week5
jery.b
psy326_chapter05.pdf
chapter 5
Experimental Designs— Explaining Behavior
Chapter Contents
• Experiment Terminology • Key Features of Experiments • Experimental Validity • Experimental Design • Analyzing Experiments • Wrap-Up: Avoiding Error
CO_
new66480_05_c05_p173-222.indd 173 10/31/11 9:31 AM
CHAPTER 5Introduction
One of the oldest debates within psychology concerns the relative contributions of biology and the environment in shaping our thoughts, feelings, and behaviors. Do we become who we are because it is hard-wired into our DNA or in response to early experiences? Do people take on their parents’ personality quirks because they carry their parents’ genes or because they grew up in their parents’ homes? There are, in fact, several ways to address these types of question. In fact, a consortium of researchers at the University of Minnesota has spent the past two decades comparing pairs of identical and fraternal twins to tease apart the contributions of genes and environment. You can read more at the research group’s website, http://mctfr.psych.umn.edu/
An alternative to using twin pairs to separate genetic and environmental influence is through the use of experimental designs, which have the primary goal of explaining the causes of behav- ior. Recall from the design overview in Chapter 2 that experiments can speak to cause because the experimenter has control over the environment as well as over the manipulation of variables. One particularly ingenious example comes from the laboratory of Michael Meaney, a professor of psy- chiatry and neurology at McGill University, using female rats as experimental subjects (Francis, Dioro, Liu, & Meaney, 1999). Meaney’s research revealed that the parenting ability of female rats can be reliably classified based on how attentive they are to their rat pups, as well as how much time they spend grooming the pups. The question
tackled in this study was whether these behaviors were learned from the rats’ own moth- ers or transmitted genetically. To answer this question experimentally, Meaney and col- leagues had to think very carefully about the comparisons they wanted to make. It would have been insufficient to simply compare the offspring of good and bad mothers—this approach could not distinguish between genetic and environmental pathways.
Instead, Meaney decided to use a technique called cross-fostering, or switching rat pups from one mother to another as soon as they were born. This resulted in four combinations of rats: (1) those born to inattentive mothers but raised by attentive ones, (2) those born to attentive mothers but raised by inattentive ones, (3) those born and raised by attentive mothers, and (4) those born and raised by inattentive mothers. Meaney then tested the rat pups several months later and observed the way they behaved with their own offspring. The setup of this experiment allowed Meaney to make clear comparisons between the influ- ence of birth mothers and the rearing process. At the end of the study, the conclusion was crystal clear: Maternal behavior is all about the environment. Those rat pups that ultimately grew up to be inattentive mothers were those who had been raised by inattentive mothers.
This final chapter is dedicated to experimental designs, in which the primary goal is to explain behavior. Experimental designs rank highest on the continuum of control (see Figure 5.1) because the experimenter can manipulate variables, minimize extraneous vari- ables, and assign participants to conditions. The chapter begins with an overview of the key features of experiments and then covers the importance of both internal and external
Creatas Images/Thinkstock
Researchers at the University of Minnesota work with twins in order to study the impact of genetics versus upbringing on personality traits.
TX_
TX
new66480_05_c05_p173-222.indd 174 10/31/11 9:31 AM
CHAPTER 5Section 5.1 Experiment Terminology
validity of experiments. From there, the discussion moves to the process of designing and analyzing experiments, and it concludes with a summary of strategies for minimizing error in experiments.
5.1 Experiment Terminology
Before we dive into the details, it is important to cover the terminology that we will use to describe different aspects of experimental designs. Much of this will be famil-iar from previous chapters, with a few new additions. First, let’s review the basics. Recall that a variable is any factor that has more than one value. For example, height is a variable because people can be short, tall, or anywhere in between. Depression is a vari- able because people can experience a wide range of symptoms, from mild to severe. The independent variable (IV) is the variable that is manipulated by the experimenter in order to test hypotheses about cause. The dependent variable (DV) is the variable that is measured by the experimenter in order to assess the effects of the independent vari- able. For example, in an experiment testing the hypothesis that fear causes prejudice, fear would be the independent variable and prejudice would be the dependent vari- able. To keep these terms straight, it is helpful to think of the main goal of experimen- tal designs. That is, we test hypotheses about cause by manipulating an independent variable and then looking for changes in a dependent variable. Thus, our independent variable causes changes in the dependent variable; for example, fear is hypothesized to cause changes in prejudice.
Any manipulation of independent variables results in two or more versions of the vari- able. One common way to describe the versions of the IV is in terms of different groups, or conditions. The most basic experiments have two conditions: The experimental condi- tion receives treatment designed to test the hypothesis, while the control condition does not receive this treatment. In our fear and prejudice example above, the participants who make up the experimental condition would be made to feel afraid, while the participants
Increasing Control . . .Increasing Control . . .
• Case Study • Archival Research • Observation
Descriptive Methods
• Survey Research
Predictive Methods
• Quasi-experiments • “True” Experiments
Experimental Methods
Figure 5.1: Experimental Designs on the Continuum of Control
new66480_05_c05_p173-222.indd 175 10/31/11 9:31 AM
CHAPTER 5Section 5.2 Key Features of Experiments
who make up the control condition would not. This setup allows us to test whether intro- ducing fear to one group of participants leads them to express more prejudice than the other group of participants, who are not made fearful.
Another common way to describe these versions is in terms of levels of the indepen- dent variable. Levels describe the specific set of circumstances created by manipulating a variable. For example, in the fear and prejudice experiment, the variable of fear would have two levels—afraid and not afraid. There are countless ways to introduce fear into the experiment. One option would be to adopt the technique used by the Stanford social psychologist Stanley Schachter (1959), who led participants to believe they would be exposed to a series of painful electric shocks. In Schachter ’s study, the painful shocks never happened, but they did induce a fearful state as people anticipated them. So, those at the “afraid” level of the independent variable might be told to expect these shocks, while those at the “not afraid” level of the independent variable would not be given this expectation.
At this stage, it may seem odd to have two sets of vocabulary terms—”levels” and “con- ditions”—for the same concept. However, there is a subtle difference in how these terms are used once we get into advanced experimental designs. As the designs become more complex, it is often necessary to expand IVs to include several groups and multiple vari- ables. Once this happens, we will need different terminology to distinguish between the versions of one variable and the combinations of multiple variables. We will return to this complexity later in the chapter, in the section “Experimental Design.”
5.2 Key Features of Experiments
The overview of designs in Chapter 2 described the overall process of experiments in the following way: a researcher controls the environment as much as possible so that all participants have the same experience. She then manipulates, or changes, one key variable, and then she measures the outcomes in another key variable. In this section, we will examine this process of control in more detail. Experiments can be distin- guished from all other designs by three key features: manipulating variables, controlling the environment, and assigning people to groups.
Manipulating Variables
The most crucial element to an experiment is that the researcher must manipulate, or change, some key variable. To study the effects of hunger, for example, a researcher could manipulate the amount of food given to the participants. Or, to study the effects of tem- perature, the experimenter could raise and lower the temperature of the thermostat in the laboratory. Because these factors are under your direct control, you can feel more confi- dent that changing them contributes to changes in the dependent variables.
In Chapter 2 we discussed the main shortcoming of correlational research: These designs do not allow us to make causal statements. As you’ll recall from that chapter (as well as from Chapter 4), correlational research is designed to predict one variable from another.
new66480_05_c05_p173-222.indd 176 10/31/11 9:31 AM
CHAPTER 5Section 5.2 Key Features of Experiments
One of the examples in that chapter concerned the correlation between income levels and happi- ness, with the goal of trying to predict happiness levels based on knowing people’s income level. If we measure these as they occur in the real world, we cannot say for sure which variable causes the other. However, we could settle this question rel- atively quickly with the right experiment. Let’s say we bring two groups into the laboratory, give one group $100 and a second group nothing. If the first group were happier at the end of the study, this would support the idea that money really does buy happiness. Of course, this is a rather simplistic look at the connection between money and happiness, but because we manipu- late levels of money, this study would bring us closer to making causal statements about the effects of money.
To manipulate variables, it is necessary to have at least two versions of the variable. That is, to study the effects of money, we need a comparison group that does not receive money. To study the effects of hunger, we would need both a hungry and a not hungry group. Having two versions of the variable distinguishes experimental designs from the structured observations discussed in Chapter 3, in which all participants received the same set of conditions in the laboratory. Even the most basic experiment must have two sets of conditions, which are often an experimental group and a control group. But, as we will see later in this chapter, experi- ments can become much more complex. You might have one experimental group and two control groups, or five degrees of food deprivation, ranging from 0 to 12 hours without food. Your decisions about the number and nature of these groups will depend on con- sideration of both your hypotheses and previous literature.
When it comes to the manipulation of variables, there are three options available. First, environmental manipulations involve changing some aspect of the setting. Environmen- tal manipulations are perhaps the most common in psychology studies, and they include everything from varying the temperature to varying the amount of money people receive. The key is to change the way different groups of people experience their time in the lab- oratory—it is either hot or cold, and they either receive or don’t receive $100. Second, instructional manipulations involve changing the way a task is described in order to change participants’ mind-sets. For example, you could give all participants the same math test but describe it as an intelligence test for one group and a problem-solving task for another. Because an intelligence test is thought to have implications for life success, you might expect participants in this group to be more nervous about their scores. Finally, an invasive manipulation involves taking measures to change internal, physiological processes and is usually conducted in medical settings. For example, studies of new drugs involve administering the drug to volunteers to determine whether it has an effect on
iStockphoto/thinkstock
Variables such as temperature or mood can be manipulated during an experiment.
new66480_05_c05_p173-222.indd 177 10/31/11 9:31 AM
CHAPTER 5Section 5.2 Key Features of Experiments
some physical or psychological symptom. Or, for example, studies of cardiovascular health often involve having participants run on a treadmill to measure how the heart functions under stress.
Finally, there is one qualification to the rule that we must manipulate a variable. In many experi- ments, researchers divide participants based on a preexisting difference (e.g., gender) or person- ality measures (e.g., self-esteem or neuroticism) that capture stable individual differences among people. The idea behind these personality mea- sures is that someone scoring high on a measure of neuroticism (for example) would be expected to be more neurotic across situations than some- one scoring lower on the measure. Using this technique allows us to compare how, for example, men and women, or people with high and low self-esteem, respond to manipulations. When pre- existing differences are used in an experimental context, they are referred to as quasi-independent variables—“quasi,” or “nearly,” because they are being measured, not manipulated, by the experi- menter, and thus do not meet the criteria for a regular independent variable. Because these vari- ables are not manipulated, an experimenter cannot make causal statements about them. In order for a study to count as an experiment, these quasi-inde-
pendent variables would have to be combined with a true independent variable. This could be as simple as comparing how men and women respond to a new antidepressant drug— gender would be quasi-independent while drug type would be a true independent variable.
Controlling the Environment
The second important element of experimental designs is that the researcher has a high degree of control over the environment. In addition to manipulating variables, a researcher conducting an experiment ensures that the other aspects of the environment are the same for all participants. For instance, if you were interested in the effects of temperature on people’s mood, you could manipulate temperature levels in the laboratory so that some people experienced warmer temperatures and other people cooler temperatures. But it is equally important to make sure that other potential influences on mood are the same for both groups. That is, you would want to make sure that the “warm” and “cool” groups were tested in the same room, around the same time of day, and by the same experimenter.
The overall goal, then, is to control extraneous variables, or variables that add noise to your hypothesis test. In essence, the more you are able to control extraneous variables, the more confidence you can have in the results of your hypothesis test. As we will discuss in the section “Validity and Control,” the impact of extraneous variables can vary in a study. Let’s say we conduct the study on temperature and mood and all of our participants are
iStockphoto/thinkstock
Having a patient run on a treadmill to measure cardiovascular stress is an example of invasive manipulation.
new66480_05_c05_p173-222.indd 178 10/31/11 9:31 AM
CHAPTER 5Section 5.2 Key Features of Experiments
in a windowless room with a flickering fluorescent light. This would likely have an influ- ence on mood—making everyone a little bit grumpy—but causes few problems for our hypothesis test because it affects everyone equally. Table 5.1 shows hypothetical data from two variations of this study, using a 10-point scale to measure mood ratings. In the top row, participants were in a well-lit room; we can see that participants in the cooler room reported being in a better mood (i.e., an 8 versus a 5). In the bottom row, all participants were in the windowless room with flickering lights. These numbers suggest that people were still in a better mood in the cooler room (5) than a warm room (2), but the flickering fluorescent light had a constant dampening effect on everyone’s mood.
Table 5.1: Influence of an Extraneous Variable
Cool Room Warm Room
Variation 1: Well-Lit 8 5
Variation 2: Flickering Fluorescent 5 2
Assigning People to Conditions
The third key feature of experimental designs is that the researcher can assign people to receive different conditions, or versions, of the independent variable. This is an important piece of the experimental process: The experimenter not only controls the options—warm vs. cool room; $100 vs. no money etc.—he or she also gets to control which participants get each option. Whereas a correlational design might assess the relationship between current mood and choosing the warm room, an experimental design will have some participants assigned to the warm room and then measure the effects on their mood. In other words, an experimenter is able to make causal statements because she causes things to happen.
The most common, and most preferable, way to assign people to conditions is through a process called random assignment. An experimenter who uses random assignment makes a separate decision for each participant as to which group he or she will be assigned to before the participant arrives. As the term implies, this decision is made randomly—by flipping a coin, using a random number table (for an example, see http://stattrek.com/ tables/random.aspx), drawing numbers out of an envelope, or some other random pro- cess. The overall goal is to try to balance out preexisting differences among people, as illustrated in Figure 5.2. So, for example, some people might generally be more comfort- able in warm rooms, while others might be more comfortable in cold rooms. If each per- son who shows up for the study has an equal chance of being in either group, then the groups in the sample should reflect the same distribution of differences as the population.
Another significant advantage of forming groups through random assignment is that it helps to avoid bias in the selection and assignment of subjects. For example, it would be a bad idea to assign people to groups based on a first impression of them because partici- pants might be placed in the cold room if they arrived at the laboratory dressed in warm clothing. Experimenters who make decisions about condition assignments ahead of time can be more confident that the independent variable is responsible for changes in the dependent variable.
new66480_05_c05_p173-222.indd 179 10/31/11 9:31 AM
CHAPTER 5Section 5.2 Key Features of Experiments
The 25 participants in our sample consist of a mix of happy and sad people. The goal of random assignment is to have these diffrences distributed equally across the experimental conditions. Thus, the
two groups on the right each consist of six happy and six sad people, and our random assigment was successful.
Figure 5.2: Random Assignment
It is worth highlighting the difference here between random selection and random assign- ment (discussed in Chapter 4). Random selection means that the sample of participants is chosen at random from the population, as with the probability sampling methods dis- cussed in the last chapter. However, most psychology experiments use a convenience sample of individuals who volunteer to complete the study. This means that the sample is often far from fully random. However, a researcher can still make sure that the group assignments are random so that each condition contains an equal representation of the sample.
In some cases—most notably, when samples are small—random assignment may not be sufficient to balance an important characteristic that might affect the results of a particular study. Imagine conducting a study that compares two strategies for teaching students com- plex math skills. In this example, it would be especially important to make sure that both groups contained a mix of individuals with, say, average and above-average intelligence. For this reason, it would be necessary to take extra steps to ensure that intelligence was equally distributed between the groups, which can be accomplished with a variation on random assignment called matched random assignment. This requires the experimenter to obtain scores on an important matching variable—in this case, intelligence—rank par- ticipants based on the matching variable, and then randomly assign people to conditions. Figure 5.3 shows how this process would unfold in our math skills study. First, partici- pants are given an IQ test to measure preexisting differences in intelligence. Second, the experimenter ranks participants based on these scores, from highest to lowest. Third, the experimenter would move down this list in order, and randomly assign each participant to one of the conditions. This process still contains an element of random assignment, but adding the extra step of rank ordering ensures a more balanced distribution of intelligence test scores across the conditions.
new66480_05_c05_p173-222.indd 180 10/31/11 9:31 AM
CHAPTER 5Section 5.2 Key Features of Experiments
9 9 9 8 9 7 9 6 9 6 9 4
9 3 8 0 7 9 7 8 7 7 7 6
7 5 74 7 3 5 0 4 9 4 8
4 7 4 6
9 9 9 7
9 5 9 3
7 9 7 7
7 5 7 3
4 9 4 7
9 8 9 6
9 4 8 0
7 8 7 6
74 7 5
4 8 4 6
The 20 participants in our sample represent a mix of very high, average, and very low intelligence test scores (measured 1-100). The goal of matched random assignment is to to ensure that this variation is distributed equally across the two conditions. The experimenter would first rank participants by intelligence test scores (top box), and then distribute these participants alternately between the conditions. The end result is that both groups (lower boxes) contain a good mix of high, average, and low scores.
Group A Group B
Figure 5.3
Research: Making an Impact The Stanford Prison Experiment
The landmark 1971 Stanford Prison Experiment had an extremely widespread impact on multiple fields and real-life settings such as prison reform, the ethics of human research, and terrorism and torture tactics (Clements, 1999; Haney, 2002). The study, conducted by Phillip Zimbardo and his col- leagues at Stanford University, placed volunteer participants in a simulated prison environment, and randomly assigned them to play the roles of “guards” and “prisoners.” These twenty-four participants had been selected based on their personality traits that marked them as “good apples,” and who had no previous evidence of antisocial behavior (Haney, Banks & Zimbardo, 1973). Nevertheless, the simulated prison quickly took on the characteristics of a real prison, with simulated situations of dom- inance, dehumanization, severe psychological distress, and the unexpected phenomenon of social control, obedience, and effects of power (Haney, 2002; Movahedi & Banuazizi, 1975). The prisoners were subjected to humiliation similar to what has been seen in prison scandals such as Abu Ghraib: nakedness, sexual humiliation, verbal torment, chains and bags over prisoners’ heads. Although planned to run for two weeks, the experiment was stopped after only six days due to the severe psychological harm it was causing the prisoners and the unexpected behavior of the prison guards.
The study led to major reform in the ethical guidelines for psychological research and human treat- ment, and in fact has never been replicated in a scientific setting. Following publication (continued)
new66480_05_c05_p173-222.indd 181 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
5.3 Experimental Validity
Chapter 2 discussed the concept of validity, or the degree to which measures capture the constructs that they were designed to capture. That is, a measure of happiness needs to actually capture differences in people’s levels of happiness. In this section, we return to the subject of validity in an experimental context. Similar to our earlier dis- cussion, validity refers here to whether the experimental results are demonstrating what we think they are demonstrating. We will cover two types of validity that are relevant to experimental designs. The first is internal validity, which assesses the degree to which results can be attributed to independent variables. The second is external validity, which assesses how well the results generalize to situations beyond the specific conditions laid out in the experiment. Taken together, internal and external validity provide a way to assess the merits of an experiment. However, each of these has its own threats and rem- edies, as discussed in the following sections.
Internal Validity
In order to have a high degree of internal validity, experimenters strive for maximum control over extraneous variables. That is, they try to design experiments so that the inde- pendent variable is the only cause of differences between groups. But, of course, no study is ever perfect, and there will always be some degree of error. In many cases, errors are the result of unavoidable causes, such as the health or mood of the participants on the day of the experiment. In other cases, errors are due to factors that are, in fact, under the experimenter’s control. In this section, we will focus on several of these more manageable threats to internal validity and discuss strategies for reducing their influence.
Experimental Confounds To avoid threats to the internal validity of an experiment, it is important to control and minimize the influence of extraneous variables that might add noise to a hypothesis test. In many cases, extraneous variables can be considered relatively minor nuisances, as when our mood experiment was accidentally run in a depressing room. But now, let’s say we run our study on temperature and mood, and due to a lack of careful planning,
of this experiment, the Supreme Court saw an influx of cases regarding prisoner treatment and the structure of the prison system. In an interesting twist, one of the original researchers, Craig Haney, was inspired to pursue a career in prison reform based on what he learned from this study. Zimbardo himself went on to testify on behalf of the soldiers accused of abuse at Abu Ghraib, highlighting the powerful influence of social roles (e.g., prison guard) on our behavior. An overwhelming majority of social and psychological research has found the punitive punishment system to not only be inef- fective but actually deleterious to prisoner behavior and recidivism. The suggestions from current research include calls to restructure prison “power” dynamics in an effort to increase prisoner safety and reduce guard brutality (e.g., Jacobs, 2004).
Research: Making an Impact (continued)
new66480_05_c05_p173-222.indd 182 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
we accidentally place all of the warm-room par- ticipants in a sunny room, and the cool-room par- ticipants in a windowless room. We might very well find that the warm-room participants were in a much better mood. But is this the result of warm temperatures or the result of exposure to sunshine? Unfortunately, we would be unable to tell the difference because of a confounding variable (or confound)—a variable that changes systematically with the independent variable. In this example, room lighting is confounded with room temperature because all of the warm-room participants are also exposed to sunshine, and all of the cool-room participants are not. This combi- nation of variables leaves us unable to determine which variable actually has the effect on mood. The result is that our groups differ in more than one way, which seriously hinders our ability to say that the independent variable (the room) caused the dependent variable (mood) to change.
It may sound like an oversimplification, but the way to avoid confounds is to be very careful in designing experiments. By ensuring that groups are alike in every way but the experimental condition, one can generally prevent confounds. This is somewhat easier said than done because confounds can come from unexpected places. For example, most studies involve the use of multiple research assistants who manage data collection and interact with participants. Some of these assistants might be more or less friendly than others, so it is important to make sure each of them interacts with participants in all con- ditions. If your friendliest assistant works with everyone in the warm-room group, for example, it would result in a confounding variable (friendly versus unfriendly assistants) between room and research assistant. Consequently, you would be unable to separate the influence of your independent variable (the room) from that of the confound (your research assistant).
Selection Bias Internal validity can also be threatened when groups are different before the manipu- lation, which is known as selection bias. Selection bias causes problems because these preexisting differences might be the driving factor behind the results. Imagine you are testing a new program that will help people stop smoking. You might decide to ask for volunteers who are ready to quit smoking and put them through a 6-week program. But by asking for volunteers—a remarkably common error—you gather a group of people who are already somewhat motivated to stop smoking. Thus, it is difficult to separate the effects of your new program from the effects of this preexisting motivation.
One easy way to avoid this problem is through either random or matched-random assign- ment. In the stop-smoking example, you could still ask for volunteers, but then randomly assign these volunteers to one of the two programs. Because both groups would consist of people motivated to quit smoking, this would help to cancel out the effects of motivation. Another way to minimize selection bias is to use the same people in both conditions so that
Digital Vision/thinkstock
Friendliness of the research assistant is a variable that can affect the outcome of an experiment.
new66480_05_c05_p173-222.indd 183 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
they serve as their own control. In the stop-smoking example, you could assign volunteers first to one program and then to the other. However, you might run into a problem with this approach—participants who successfully quit smoking in the first program would not benefit from the second program. This technique is known as a within-subject design, and we will discuss its advantages and disadvantages in the section “Within-Subject Designs.”
Differential Attrition Despite your best efforts at random assignment, you could still have a biased sample at the end of a study as a result of differential attrition. The problem of differential attri- tion (sometimes called the mortality threat) occurs when subjects drop out of experimental groups for different reasons. Let’s say you’re conducting a study of the effects of exercise on depression levels. You manage to randomly assign people to either 1 week of regular exer- cise or 1 week of regular therapy. At first glance, it appears that the exercise group shows a dramatic drop in depression symptoms. But then you notice that about one third of the people in this group dropped out before completing the study. Chances are you are left with those who are most motivated to exercise, or to overcome their depression, or both. Thus, you are unable to isolate the effects of your independent variable on depression symptoms. While you cannot prevent people from dropping out of your study, you can look carefully at those who do. In many cases, you can spot a pattern and use it to guide future research. For example, it may be possible to discover a profile of people who dropped out of the exercise study and use this knowledge to increase retention for the next attempt.
Outside Events As much as we strive to control the laboratory environment, participants are often influ- enced by events in the outside world. These events—sometimes called history effects—are often large-scale events such as political upheavals and natural disasters. The threat to research is that it becomes difficult to tell whether participants’ responses are due to the independent variable or to the historical event(s). One great example of this comes from a paper published by social psychologist Ryan Brown, now a professor at the University of Oklahoma, on the effects of receiving different types of affirmative action as people were selected for a leadership position. The goal was to determine the best way to frame affir- mative action in order to avoid undermining the recipient’s confidence (Brown, Charn-
sangavej, Keough, Newman, & Rentfrow, 2000). For about a week during the data collection pro- cess, students at the University of Texas, where the study was being conducted, were protesting on the main lawn about a controversial lawsuit regarding affirmative action policies. The result was that par- ticipants arriving for this laboratory study had to pass through a swarm of people holding signs that either denounced or supported affirmative action! These types of outside event are difficult, if not impossible, to control. But, because these research- ers were aware of the protests, they made a deci- sion to exclude data gathered from participants during the week of the protests from the study, thus minimizing the effects of outside events.
Photodisc/thinkstock
Events that occur in the outside world can affect the results of your experiments.
new66480_05_c05_p173-222.indd 184 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
Expectancy Effects One final set of threats to internal validity results from the influence of expectancies on people’s behavior. This can cause trouble for experimental designs in three related ways. First, experimenter expectancies can cause researchers to see what they expect to see, leading to subtle bias in favor of their hypotheses. In a clever demonstration of this phe- nomenon, the psychologist Robert Rosenthal asked his graduate students at Harvard University to train groups of rats to run a maze (Rosenthal & Fode, 1963). He also told them that based on a pretest, the rats had been classified as either bright or dull. As you might have guessed, these labels were pure fiction, but they still influenced the way that the students treated the rats. Those labeled bright were given more encouragement and learned the maze much more quickly than rats labeled dull. Rosenthal later extended this line of work to teachers’ expectations of their students (Rosenthal & Jacobson, 1992) and found support for the same conclusion: People often bring about the results they expect by behaving in a particular way.
One common way to avoid experimenter expectancies is to have participants interact with a researcher who is “blind” (i.e., unaware) to the condition that each participant is in. The researcher may be fully aware of the research hypothesis, but her behavior is unlikely to affect the results. In the Rosenthal and Fode (1963) study, the graduate students’ behavior only influenced the rats’ learning speed because they were aware of the labels bright and dull. If these had not been assigned, the rats would have been treated fairly equally across the conditions.
Second, participants in a research study often behave differently based on their own expectan- cies about the goals of the study. These expectan- cies often develop in response to demand charac- teristics, or cues in the study that lead participants to guess the hypothesis. In a well-known study conducted at the University of Wisconsin, psy- chologists Leonard Berkowitz and Anthony LeP- age found that participants would behave more aggressively—by delivering electric shocks to another participant—if a gun was in the room than if there were no gun present (Berkowitz & LePage, 1967). This finding has some clear implications for gun control policies, suggesting that the mere pres- ence of guns increases the likelihood of gun vio- lence. However, a common critique of this study is that participants may have quickly clued in to its purpose and figured out how they were “supposed” to behave. That is, the gun served as a demand characteristic, possibly making participants act more aggressively because they thought it was expected of them.
To minimize demand characteristics, researchers use a variety of techniques, all of which attempt to hide the true purpose of the study from participants. One common strategy is to use a cover story, or a misleading statement about what is being studied. In Chapter 1, we discussed Milgram’s famous obedience studies, which discovered that people were willing to obey orders to deliver dangerous levels of electric shocks to other people. In order to disguise the purpose of the study, Milgram described it to people as a study
Hemera/thinkstock
People tend to act more aggressively if there is a weapon present.
new66480_05_c05_p173-222.indd 185 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
of punishment and learning. And the affirmative action study by Ryan Brown and col- leagues was presented as a study of leadership styles. The goal in using these cover stories is to give participants a compelling explanation for what they experience during the study and to direct their attention away from the research hypothesis.
Another strategy is to use the unrelated-experiments technique, which leads partici- pants to believe that they are completing two different experiments during one labora- tory session. The experimenter can use this bit of deception to present the independent variable during the first experiment and then measure the dependent variable during the second experiment. For example, a study by Harvard psychologist Margaret Shih
and colleagues (Shih, Pittinsky, & Ambady, 1999) recruited Asian-American females and asked them to complete two supposedly unrelated studies. In the first, they were asked to read and form impressions of one of two magazine articles; these articles were designed to make them focus on either their Asian-American identity or their female identity. In the second experiment, they were asked to complete a math test as quickly as possible. The goal of this study was to examine the effects of priming different aspects of identity on math performance. Based on previous research, these authors predicted that priming an Asian- American identity would remind participants of positive stereotypes regarding Asians and math performance, whereas priming a female identity would remind participants of negative stereo- types regarding women and math performance. As expected, priming an Asian-American iden- tity led this group of participants to do better on a math test than did priming a female identity. The unrelated-experiments technique was espe- cially useful for this study because it kept par- ticipants from connecting the independent vari- able (magazine article prime) with the dependent variable (math test).
A final way in which expectancies shape behavior is the placebo effect, meaning that change can result from the mere expectation that change will occur. Imagine you wanted to test the hypothesis that alcohol causes people to become aggressive. One relatively easy way to do this would be to give alcohol to a group of volunteers (aged 21 and older) and then measure how aggressive they were in response to being provoked. The problem with this approach is that people also expect alcohol to change their behavior, and so you might see changes in aggression simply because of these expectations. Fortunately, there is an easy solution: add a placebo control group to your study that mimics the experimental condition in every way but one. In this case, you might tell all participants that they will be drinking a mix of vodka and orange juice but only add vodka to half of the partici- pants’ drinks. The orange-juice-only group serves as our placebo control—any differences between this group and the alcohol group can be attributed to the alcohol itself.
Digital Vision/thinkstock
The placebo effect can test whether alcohol affects behavior, or whether people just expect it to and exhibit changed behavior based on their expectations.
new66480_05_c05_p173-222.indd 186 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
External Validity
In order to have a high degree of external validity in our experiments, we strive for maxi- mum realism in the laboratory environment. External validity means that the results extend beyond the particular set of circumstances created in a single study. Recall that science is a cumulative discipline and that knowledge grows one study at a time. Thus, each study is more meaningful to the extent that it sheds light on a real phenomenon and to the extent that the results generalize to other studies. Let’s examine each of these criteria separately.
Mundane Realism The first component of external validity is the extent to which an experiment captures the real-world phenomenon under study. One popular question in the area of aggression research is whether rejection by a peer group leads to aggression. That is, when people are rejected from a group, do they lash out and behave aggressively toward the members of that group? Researchers must find realistic ways to manipulate rejection and measure aggression without infringing on participants’ welfare. Given the need to strike this bal- ance, how real can things get in the laboratory? How do we study real-world phenomena without sacrificing internal validity?
The answer is to strive for mundane realism, meaning that the research replicates the psychological conditions of the real-world phenomenon (sometimes referred to as eco- logical validity). In other words, we need not recreate the phenomenon down to the last detail; instead, we aim to make the laboratory setting feel like the real-world phenome- non. Researchers studying aggressive behavior and rejection have developed some rather clever ways of doing this, including allowing participants to administer loud noise blasts or serve large quantities of hot sauce to those who rejected them. Psychologically, these acts feel like aggressive revenge because participants were able to lash out against those who rejected them, with the intent of causing harm, even though the behaviors them- selves may differ from the ways people exact revenge in the real world.
In a 1996 study, Tara MacDonald and her colleagues at Queen’s University in Ontario, Canada, examined the relationship between alcohol and condom use (MacDonald, Zanna, & Fong, 1996). The authors pointed out a puzzling set of real-world data: Most people reported that they would use condoms when engaging in casual sex, but the rates of unpro- tected sex (i.e., having sexual intercourse without a condom) were also remarkably high. In this study, the authors found that alcohol was a key factor in causing “common sense to go out the window” (p. 763), resulting in a decreased likelihood of condom use. But how on earth might they study this phenomenon in the laboratory? In the authors’ words, “even the most ambitious of scientists would have to conclude that it is impossible to observe the effects of intoxication on actual condom use in a controlled laboratory setting” (p. 765).
To solve this dilemma, MacDonald and colleagues developed a clever technique for study- ing people’s intentions to use condoms. Participants were randomly assigned to either an alcohol or placebo condition, and then they viewed a video depicting a young couple that was faced with the dilemma of whether to have unprotected sex. At the key decision point in the video, the tape was stopped and participants were asked what they would do in the situation. As predicted, participants who were randomly assigned to consume alcohol said they would be more willing to proceed with unprotected sex. While this laboratory
new66480_05_c05_p173-222.indd 187 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
study does not capture the full experience of making decisions about casual sex, it does a nice job of capturing the psychological conditions involved.
Generalizing Results The second component of external validity is the extent to which research findings gen- eralize to other studies. Generalization refers to the extent to which the results extend to other studies, using a wide variety of populations and a wide variety of operational defi- nitions (sometimes referred to as population validity). If we conclude that rejection causes people to become more aggressive, for example, this conclusion should ideally carry over to other studies of the same phenomenon, using different ways of manipulating rejection, and different ways of measuring aggression. If we want to conclude that alcohol reduces intentions to use condoms, we would need to test this relationship in a variety of set- tings—from laboratories to nightclubs—using different measures of intentions.
Thus, each study that we conduct is limited in its conclusions. In order for your particular idea to take hold in the scientific literature, it must be replicated, or repeated in different con- texts. These replications can take one of four forms. First, exact replication involves trying to recreate the original experiment as closely as possible in order to verify the findings. This type of replication is often the first step following a surprising result, and it helps research- ers to gain more confidence in the patterns. The second and much more common method, conceptual replication involves testing the relationship between conceptual variables using new operational definitions. Conceptual replications would include testing our aggression hypotheses using new measures or examining the link between alcohol and condom use in different settings. For example, rejection might be operationalized in one study by having participants be chosen last for a group project. A conceptual replication might take a differ- ent approach, operationalizing rejection by having participants be ignored during a group conversation or voted out of the group. Likewise, a conceptual replication might change the operationalization of aggression, with one study measuring the delivery of loud blasts of noise and another measuring the amount of hot sauce that people give to their rejecters. Each variation studies the same concept (aggression or rejection) but uses slightly different opera- tionalizations. If all of these variations yield similar results, this provides further evidence of the underlying ideas—in this case, that rejection causes people to be more aggressive.
The third method, participant replication, involves repeating the study with a new popula- tion of participants. These types of replication are usually driven by a compelling theory as to why the two populations differ. For example, you might reasonably hypothesize that the decision to use condoms is guided by a different set of considerations among college stu- dents than among older, single adults. Finally, constructive replication re-creates the original experiment but adds elements to the design. These additions are typically designed to either rule out alternative explanations or extend knowledge about the variables under study. In our rejection and aggression example, you might test whether males and females respond the same way or perhaps compare the impact of being rejected by a group versus an individual.
Internal versus External Validity
We have focused on two ways to assess validity in the context of experimental designs. Internal validity assesses the degree to which results can be attributed to independent vari- ables; external validity assesses how well results generalize beyond the specific conditions
new66480_05_c05_p173-222.indd 188 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
of the experiment. In an ideal world, studies would have a high degree of both of these. That is, we would feel completely confident that our independent variable was the only cause of differences in our dependent variable, and our experimental paradigm would perfectly capture the real-world phenomenon under study.
In reality, though, there is often a trade-off between internal and external validity. In Mac- Donald et al.’s study on condom use, the researchers sacrificed some realism in order to conduct a tightly controlled study of participants’ intentions. In Berkowitz and LePage’s study on the effect of weapons, the researchers risked the presence of a demand charac- teristic in order to study reactions to actual weapons. These types of trade-offs are always made based on the goals of the experiment. To give you a better sense of how researchers make these compromises, let’s evaluate three fictional examples:
Scenario 1—Time Pressure and Stereotyping Dr. Bob is interested in whether people are more likely to rely on stereotypes when they are in a hurry. In a well-controlled laboratory experiment, participants are asked to categorize ambiguous shapes as either squares or circles, and half of these participants are given a short time limit to accomplish the task. The independent variable is the presence or absence of time pressure, and the dependent variable is the extent to which people use stereotypes in their classification of ambiguous shapes. Dr. Bob hypothesizes that people will be more likely to use stereotypes when they are in a hurry because they will have fewer cognitive resources to carefully consider all aspects of the situation. Dr. Bob takes great care to have all participants meet in the same room. He uses the same research assistant every time, and the study is always conducted in the morning. Consistent with his hypothesis, Dr. Bob finds that people seem to use shape stereotypes more under time pressure.
The internal validity of this study appears high—Dr. Bob has controlled for other influ- ences on participants’ attention span by collecting all of his data in the morning. He has also minimized error variance by using the same room and the same research assistant. In addition, Dr. Bob has created a tightly controlled study of stereotyping through the use of circles and squares. Had he used photographs of people (rather than shapes), the attractiveness of these people might have influenced participants’ judgments. But here’s the trade-off: By studying the social phenomenon of stereotyping using geometric shapes, Bob has removed the social element of the study, thereby posing a threat to mundane real- ism. The psychological meaning of stereotyping shapes is rather different from the mean- ing of stereotyping people, which makes this study relatively low in external validity.
Scenario 2—Hunger and Mood Dr. Jen is interested in the effects of hunger on mood; not surprisingly, she predicts that people will be happier when they are well fed. She tests this hypothesis with a lengthy laboratory experiment, requiring participants to be confined to a laboratory room for 12 hours with very few distractions. Participants have access to a small pile of magazines to help pass the time. Half of the participants are allowed to eat during this time, and the other half is deprived of food for the full 12 hours. Dr. Jen—a naturally friendly person— collects data from the food-deprivation groups on a Saturday afternoon, while her grumpy research assistant, Mike, collects data from the well-fed group on a Monday morning. Her independent variable is food deprivation, with participants either not deprived of food or deprived for 12 hours. Her dependent variable consists of participants’ self-reported
new66480_05_c05_p173-222.indd 189 10/31/11 9:31 AM
CHAPTER 5Section 5.3 Experimental Validity
mood ratings. When Dr. Jen analyzes the data, she is shocked to discover that participants in the food-deprivation group are much happier than those in the well-fed group.
Compared to our first scenario, this study seems high on external validity. To test her pre- dictions about food deprivation, Dr. Jen actually deprives her participants of food. One possible problem with external validity is that participants are confined to a laboratory set- ting during the deprivation period with only a small pile of magazines to read. That is, participants may be more affected by hunger when they do not have other things to distract them. In the real world, people are often hungry but distracted by paying attention to work, family, or leisure activities. But Dr. Jen has sacrificed some external validity for the sake of controlling how participants spend their time during the deprivation period. The larger problem with her study has to do with internal validity. Dr. Jen has accidentally confounded two additional variables with her independent variable: Participants in the deprivation group have a different experimenter and data are collected at a different time of day. Thus, Dr. Jen’s surprising results most likely reflect that everyone is in a better mood on Saturday than on Monday and that Dr. Jen is more pleasant to spend 12 hours with than Mike is.
Scenario 3—Math Tutoring and Graduation Rates Dr. Liz is interested in whether specialized math tutoring can help increase graduation rates among female math majors. To test her hypothesis, she solicits female volunteers for a math skills work- shop by placing fliers around campus, as well as by sending email announcements to all math majors. The independent variable is whether participants are in the math skills workshop, and the dependent variable is whether participants graduate with a math degree. Those who volunteer for the work- shop are given weekly skills tutoring, along with informal discussion groups designed to provide encouragement and increase motivation. At the end of the study, Liz is pleased to see that partici- pants in the workshops are twice as likely as non- participants to stick with the major and graduate.
The obvious strength of this study is its external validity. Dr. Liz has provided math tutor- ing to math majors, and she has observed a difference in graduation rates. Thus, this study is very much embedded in the real world. But, as you might expect, this external validity comes at a cost to internal validity. The biggest flaw is that Dr. Liz has recruited volunteers for her workshops, resulting in selection bias for her sample. People who volunteer for extra math tutoring are likely to be more invested in completing their degree and might also have more time available to dedicate to their education. Dr. Liz would also need to be mindful of how many people drop out of her study. If significant numbers of participants withdrew, she could have a problem with differential attrition, so that the most motivated people stayed with the workshops. One relatively easy fix for this study would have been to ask for volunteers more generally, and then randomly assign these volunteers to take part in either the math tutoring workshops or a different type of workshop. While the sample might still be less than random, Dr. Liz would at least have the power to assign participants to different groups.
Photodisc/thinkstock
Recruiting volunteers for a study on the effects of tutoring may compromise the internal validity of the experiment.
new66480_05_c05_p173-222.indd 190 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
5.4 Experimental Design
The process of designing experiments boils down to deciding what to manipulate and how to do it. In this section, we will cover two broad issues related to experimental design: deciding how to structure the levels, or different versions of an independent variable, and deciding on the number of independent variables necessary to test the hypoth- eses. While these decisions may seem tedious, they are at the crux of designing successful experiments, and, therefore, are the key to performing successful tests of our hypotheses.
Levels of the Independent Variable
Between-Subject Designs The primary goal in designing experiments is to ensure that the levels of independent variables are equivalent in every way but one. This allows us to make causal statements about the effects of that single change. In most of the examples we have discussed so far, the levels of our independent variables have represented two distinct groups—par- ticipants are in either the control group or the experimental group. This type of design is referred to as a between-subject design because the levels differ between one subject and the next. Each participant who enrolls in the experiment would be exposed to only one level of the independent variable. That is, an individual participant might be in either the experimental or the control group. Most of our examples so far have been illustrations of between-subject designs: participants receive either alcohol or placebo; students read an article designed to prime either their Asian or their female identity; and graduate students train rats that are falsely labeled either bright or dull. This approach is very common and has the advantage of using distinct groups to represent each level of the independent variable. In other words, participants who are asked to consume alcohol are completely distinct from those asked to consume the placebo drink. However, this is only one option for structuring the levels of the independent variable. In this section, we will examine two additional ways to structure these levels.
Within-Subject Designs In some cases, the levels of the independent variable represent the same participants at dif- ferent time periods. This type of design is referred to as a within-subject design because the levels differ within people. Each participant who enrolls in the experiment would be exposed to both or all levels of the independent variable. That is, every participant would be in both the experimental and the control group. Within-subject designs are often used to compare changes over time in response to various stimuli. For example, you might measure anxiety symptoms before and after people are locked in a room with a spider or measure depression symptoms before and after people undergo drug treatment.
Within-subject designs have two main advantages over between-subject designs. First, because the same people comprise both levels of the IV, these designs require fewer par- ticipants. Let’s say you decide to collect data from 20 participants at each level of your IV. In a between-subject design with three levels, you would need 60 people. However, if you run the same experiment as a within-subject design—exposing the same group of people to three different sets of circumstances—you would only need 20 people. Thus, within- subject designs are often a good way to conserve resources.
new66480_05_c05_p173-222.indd 191 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
Second, participants also serve as their own control group, allowing you to minimize a big source of error variance. Remember that one key feature of experimental design is the researcher’s power to assign people to groups; this is done to randomly distribute subject differences across the levels of the IV. Using a within-subject design solves the problem of subject differences in another way—by examining changes within people. For instance, in our example regarding spiders and anxiety, some participants are likely to have higher baseline anxiety than others. By measuring changes in anxiety in the same group of people before and after spider exposure, we are able to minimize the effects of individual differences.
Problems with Within-Subject Designs Within-subject designs also have two clear disadvantages compared to between-subject designs. First, there is the risk of carryover effects, in which the effects of one level are still present when another level is introduced. Because the same people are exposed to all levels of the IV, it can be difficult to separate the effects of one level from the effects of the others. One common paradigm in emotion research is to show participants several film clips that elicit different types of emotion. People might view one clip showing a puppy playing with a blanket, another showing a child crying, and another showing a surgical amputation. Even without seeing these in full color, you can imagine that it would be hard to shake off the disgust triggered by the amputation in order to experience the joy trig- gered by the puppy.
When researchers use a within-subject design, they take steps to minimize carryover effects. In studies of emotion, for example, researchers typically show a brief neutral clip, such as waves rolling onto a beach, between emotional clips so that participants experi-
ence each emotion after viewing a benign image. Another simple technique is to collect data from the control condition first whenever possible. In our study of spiders and anxiety, it would be important to measure baseline anxiety at the start of the experiment before exposing people to spi- ders. Once people have been surprised by a spi- der, it will be hard to get them to relax enough to collect control ratings of anxiety.
Second, there is a risk of order effects, meaning that the order in which levels are presented can moderate their effects. Order effects fall into two categories. The practice effect happens when par- ticipants’ performance improves over time sim- ply due to repeated attempts. This is a particular problem in studies that examine learning. Let’s say you use a within-subject design to compare two techniques for teaching people to solve logic problems. Participants would learn technique A, then take a logic test, then learn technique B, and then take a second logic test. The possible problem is that participants will have had more
Ryan McVay/Photodisc/Thinkstock
Carryover effects can be understood through the example of mointoring people's reactions to different film clips. How they feel about one image may influence how they react to the next image.
new66480_05_c05_p173-222.indd 192 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
opportunities to practice logic problems by the time they take the second test. This makes it difficult to separate the effects of practice from the effects of different teaching techniques.
The flipside of practice effects is the phenomenon of the fatigue effect, which happens when participants’ performance decreases over time due to repeated testing. Let’s say you run a variation of the above experiment, attempting to teach people to improve their reaction time. Participants might learn each technique and have their reaction time tested several times after each one. The problem is that people would gradually start to become tired, and their reaction times would slow down due to fatigue. Thus, it would be difficult to separate the effects of fatigue from the effects of the different teaching techniques.
The result of both of these problems is that the order of presentation becomes confounded with the level of the independent variable. Fortunately, there is a relatively easy way to avoid both carryover and fatigue effects by using a process called counterbalancing. Counterbalancing involves varying the order of presentation to groups of participants. The simplest approach is to divide participants into as many groups as there are com- binations of levels in the experiment. That is, we create a group for each possible order, allowing us to identify the effects of encountering the conditions in different orders. In our examples above, the learning experiments involved two techniques, A and B. To coun- terbalance these techniques across the study, we would divide the participants into two groups. One group would be exposed to A and then B; the other group would be exposed to B and then A. When it came time to analyze the data, we would be able to examine the effects of both presentation order and teaching technique. If the order of presentation made a difference, then the A/B group would differ from the B/A group in some way.
Mixed Designs The third common way to structure the levels of an IV is called a mixed design, which contains at least one between-subject variable and at least one within-subject variable. So, in the example we’ve just discussed, our participants would be exposed to both teaching techniques (A and B) but in only one of two possible orders of presentation. In this case, teaching technique is a within-subject variable because participants experience both levels. And presentation order is a between-subject variable because participants experience only one level. Because we have one of each in the overall experiment, this is a mixed design.
One common use of mixed designs is in studies that compare the effects of different drugs. Imagine you wanted to compare three new drugs—Drug X, Drug Y, and a placebo con- trol—to determine which had the strongest effects on reducing depression symptoms. To do this study, you would want to measure depression symptoms on at least three occa- sions: before starting drug treatment, after a few months of taking the drug, and then again after a few months of stopping the drug (to assess relapse rates). So, our participants would be given one of three possible drugs, and then measured at each of three time periods. In this mixed design, measurement time is a within-subject variable because par- ticipants experience all three levels, while the drug is a between-subject variable because participants experience only one of three possible levels.
The hypothetical results of this study are shown in Figure 5.4. You can see that the placebo pill has no effect on depression symptoms; depression scores in this group are the same at all three measurements. Drug X appears to cause significant improvement in depression
new66480_05_c05_p173-222.indd 193 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
symptoms; depression scores drop steadily across measurements in this group. And, strangely, Drug Y seems to make depression worse; depression scores increase steadily across measurements in this group. The mixed design allows us both to track people over time and to compare different drugs in one study.
D e p
re s s io
n S
c o
re
30
35
Baseline Time 1 Time 2
Time of Measurement (Within–Subject)
Drug Y
Placebo
Drug X
25
20
15
10
5
0
Figure 5.4: Example of a Mixed-Subjects Design
Research: Thinking Critically Outwalking Depression
By Michael Otto
There is a great deal of evidence that exercise not only improves mood and enhances well being but also is an effective intervention for depression. That is, well-conducted clinical trials have repeatedly shown mood benefit from exercise in adults with clinical depression. Indeed, there is evidence that exercise provides benefits at levels similar to that found for antidepressant medication.
Now, there is new evidence for the power of exercise when other treatments for depression have not provided adequate help. A recent study, published in the August, 2011, issue of the Journal of Psychi- atric Research provides data that exercise can provide benefits when medication alone does not. Spe- cifically, researchers in Portugal examined the effects of exercise—in this case programmed episodes of walking for 30-45 minutes five times per week-on depressed adults who failed to respond to two previous trials of antidepressant medication. In the study, all patients remained on their antidepres- sant medication, and two-thirds of the sample received the program of regular (walking) exercise.
The results were dramatic. Of those patients who received only medication, no average changes in depression mood ratings were seen over the next 12 weeks. In contrast, clear improvement was seen in those who exercised, with 10 of the 19 patients who exercised showing a response or full remission in symptoms during this time.
Although this was a small study, the results were consistent with previous exercise studies. These studies underscore the importance of considering combined treatment strategies for (continued)
new66480_05_c05_p173-222.indd 194 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
patients who do not respond to one type of intervention—if one treatment does not work, con- sider using other resources with evidence for success. In the case of treatment of depression, these resources prominently include psychotherapy (with a wealth of studies indicating that specific forms of psychotherapy like Cognitive Behavior Therapy or Interpersonal Psychotherapy can bring about timely relief of depression), a range of medication treatments, or, with increasing evidence, regular exercise. Within these choices there is not a clear cure-all option, but, importantly, there are a range of options to be pursued to try to find the right fit for any particular person suffering from depression.
A number of important features of the Portuguese study deserve additional note. First, the study provided additional evidence that high intensities of exercise are not required to bring about mood changes. Moderate exercise is adequate for mood benefit (see my next blog for information on why moderate rather than high-intensity exercise may be important for starting and keeping the exercise habit over time). Second, the study used an exercise program that relied on the individual efforts of the patient. Exercise was performed on a treadmill at the treatment center for one out of the five weekly sessions; the rest of the sessions were up to the individual. This individual program of exercise helped show that the mood effects of exercise were not simply due to the social contact from a group of patients exercising together. It also showed that moderate exercise is an accessible option even for chronically depressed individuals. Third, the exercise prescription included the use of regular reminders for exercise: keeping walking shoes in a visible location, having support for the walking program from family, or arranging for written or cell phone reminders for walking session. As we will talk about in subsequent blogs, simple reminders like these can have a powerful effect on helping people keep up with any new habit.
Depression is an insidious disorder that harms our mood, our goals, and our relationships. Having another tool for intervening with depression is deeply important. Exercise is an intervention with particular broad reach. It is not only for those who are focused on keeping a mind-body balance; it is an important option to be considered as an addition to psychotherapy or medication treatment, as well as an option for anyone with mood challenges who is looking for help. As always with exer- cise, it is important to start slow and to meet with your physician to find out which type and level of exercise is right for you.
Think about it
1. Identify the following essential aspects of this experimental design: a. What are the IV and DV in this study? b. How many levels does the IV have? c. Is this a between-subjects, within-subjects, or mixed design? d. Draw a simple table labeling each condition.
2 a. What preexisting differences between groups should the researchers be sure to take into account? Name as many as you can.
b. How should the researchers assign participants to the conditions in order to ensure that pre- existing differences cannot account for the results?
3. How might expectancy effects influence the results of this study? Can you think of any ways to control for this?
4. Briefly state how you would replicate this study in each of the following ways: a. exact replication b. conceptual replication c. participant replication d. constructive replication
Research: Thinking Critically (continued)
new66480_05_c05_p173-222.indd 195 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
One-Way versus Factorial Designs
The second big issue in creating experimental designs is to decide on the number of inde- pendent variables to manipulate. In some cases, we can test our hypotheses by manipu- lating a single IV and measuring the outcome—such as giving people either alcohol or a placebo drink and measuring the intention to use condoms. In other cases, hypotheses involve more complex combinations of variables. Earlier in the chapter, we discussed research findings that people tend to act more aggressively after a peer group has rejected them. But we could extend this and ask what happens when people are rejected by mem- bers of the same sex versus members of the opposite sex. And we could go one more step and test whether the attractiveness of the rejecters matters, for a total of three independent variables. These examples illustrate two broad categories of experimental design, known as one-way and factorial designs.
One-Way Designs If a study involves giving people either alcohol or a placebo and then measuring outcomes, it has a one-way design, or a design that has only one independent variable with two or more levels to the variable. These tend to be the simplest experiments and have the advan- tage of testing manipulations in isolation. One-way designs are used in the majority of drug studies. These types of study compare the effects on medical outcomes for people randomly assigned, for instance, to take the antidepressant drug Prozac or a placebo. Note that a one- way design can still have multiple levels—in many cases it is preferable to test several differ- ent doses of a drug. So, for example, we might test the effects of Prozac by assigning people to take doses of 5 mg, 10 mg, 20 mg, or a placebo control. Our independent variable would be the drug dose, and the dependent variable would be a change in depression symptoms. This design would allow us to compare all three of the drug doses to a placebo control, as well as to compare the effects of each dose to those of the other drugs. Figure 5.5 shows hypothetical results from this study. We can see that even those receiving the placebo showed a drop in depression symptoms, with the maximum benefit caused by the 10-mg dose of Prozac.
Decrease in Depression Symptoms
Placebo 5mg 10mg 20mg 0
-5
-10
-15
-20
-25
Figure 5.5: Comparing Drug Doses in a One-Way Design
new66480_05_c05_p173-222.indd 196 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
Factorial Designs Despite the appealing simplicity of one-way designs, experiments conducted in the field of psychology with only one IV are relatively rare. The real world is much more compli- cated, which means that studies that focus on people’s thoughts, feelings, and behaviors have to somehow capture the associated complexity. Thus, our rejection and aggression example above is not that far-fetched. If a researcher wanted to manipulate the presence of rejection, the sex of the rejecters, and the attractiveness of the rejecters in a single study, the experiment would have a factorial design. Factorial designs are those that have two or more independent variables, each of which has two or more levels. When using a factorial design, the purpose is to observe both the effects of individual variables and the combined effects of multiple variables.
Factorial designs have their own terminology to reflect the fact that they have both individ- ual variables and combinations of variables. At the beginning of this chapter, we learned that the versions of an independent variable are referred to as both levels and conditions, with a subtle difference between the two. This difference comes into play when we start discussing factorial designs. Specifically, levels refer to the versions of each IV, while condi- tions refer to the groups formed by combinations of IVs. Let’s walk through one variation of our rejection and aggression example from this perspective: Our first IV has two levels because participants are either rejected or not rejected. And our second IV has two levels because members of the same sex or the opposite sex do the rejection. To determine the number of conditions in this study, we calculate the number of different experiences that participants can have in the study. This is a simple matter of multiplying the levels of separate variables, so 2 multiplied by 2, for a total of four conditions.
Researchers also have a way to quickly describe the number of variables in their design: A two-way design has two independent variables; a three-way design has three indepen- dent variables; an eight-way design has eight independent variables, and so on. Even more useful, the system of factorial notation offers a simple way to describe both the number of variables and the number of levels in experimental designs. For instance, we might describe our design as a 2 3 2 (pronounced “two by two”), which instantly com- municates two things: (1) we have two independent variables, indicated by the presence of two separate numbers and (2) each IV has two levels, indicated by the number 2 listed for each one.
The 2 3 2 Design One of the most common factorial designs also happens to be the simplest one—the 2 3 2 design. These designs have two independent variables, with two levels each, for a total of four experimental conditions. The simplicity of these designs makes them a useful way to become more comfortable with some of the basic concepts of experiments. In this section, we will walk through an example of a 2 3 2 and analyze it in detail.
Beginning in the late 1960s, social psychologists developed a keen interest in under- standing the predictors of helping behavior. This was inspired, in large part, by the trag- edy of Kitty Genovese, who was killed outside her apartment building while none of her neighbors called the police (Gansberg, 1964). In one representative study, Princeton
new66480_05_c05_p173-222.indd 197 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
psychologists John Darley and Bibb Latané examined people’s likelihood of respond- ing to a staged emergency. Participants were led to believe that they were taking part in a group discussion over an intercom system, but in reality, all of the other partici- pants were prerecorded. The key independent variable was the number of other people supposedly present, ranging from two to six. A few minutes into the conversation, one participant appeared to have a seizure. The recording went like this (actual transcript; Darley & Latané, 1968):
I could really-er-use some help so if someone would-er-give me a little h-help- uh-er-er-er c-could somebody er-er-hel-er-uh-uh-uh [choking sounds] . . . I’m gonna die-er-er-I’m . . . gonna die-er-hel-er-er-seizure-er [chokes, then quiet].
What do people do in this situation? Do they help? How long does it take? Darley and Latané discovered that two things happen as the group became larger: People were less likely to help at all, and those who did help took considerably longer to do so. One of the primary conclusions to come of out this and other studies is that people are less likely to help when other people are present because the responsibility for helping is diffused among the members of the crowd (Darley & Latané, 1968).
Building on this conclusion, the sociologist Jane Piliavin and her colleages (Piliavin, Pilia- vin, & Rodin, 1975) explored the influence of two additional variables on helping behavior by staging an emergency on a New York City subway train, in which a person who was in on the study appeared to collapse in pain. Specifically, the researchers manipulated two variables in their staged emergency. The first independent variable was whether there was a medical intern nearby, who could be easily identified by wearing blue scrubs. The second independent variable was whether the victim had a large disfiguring scar on his face. The combination of these variables resulted in four conditions, as shown in Table 5.2. The dependent variable in this study was the percentage of people taking action to help the confederate.
Table 5.2: 2 3 2 Design of the Piliavin et al. Study
No intern Intern
No scar 1 2
Scar 3 4
The authors predicted that bystanders would be less likely to help if there was a per- ceived medical professional nearby since he or she was considered more qualified to help the victim. They also predicted that people would be less likely to help when the confederate had a large scar because previous work had demonstrated convincingly that people avoid contact with those who are disfigured or have other stigmatizing con- ditions (e.g., Goffman, 1963). As seen in Figure 5.6, these hypotheses were supported by the results. Both the presence of a scar and the presence of a perceived medical profes- sional reduced the percentage of people who came to help. But there’s something else going on in these results: When the confederate is not scarred, having an intern nearby
new66480_05_c05_p173-222.indd 198 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
leads to a small decrease in helping (from 88% to 84%). However, when the confederate had a large facial scar, having an intern nearby decreased helping from 72% to 48%! In other words, it seems these variables are having a combined effect on helping behavior. We will examine these combined effects more closely in the next section.
Other Factorial Designs Experimental designs can often be more complex than a simple 2 3 2. In another variation of our rejection and aggression study, we proposed having participants in one of eight cells—they are either rejected or not, by a person of either the same sex or the opposite sex, who happens to be either attractive or unattractive. This design would have eight conditions (2 3 2 3 2), as shown in Table 5.3a, with the separate conditions numbered for illustration purposes.
Table 5.3a: Levels versus Conditions in a 2 3 2 3 2 Design
Rejected Not rejected
Rejecter Attractive Unattractive Attractive Unattractive
Same sex 1 2 5 6
Opposite sex 3 4 7 8
This is a 2 3 2 3 2 design for a total of eight conditions (numbered 1–8) “Rejection” has two levels: rejected and not rejected. “Attractiveness” has two levels: attractive and unattractive. “Sex” has two levels: the same sex and the opposite sex rejecter.
% H
e lp
in g
100
No Scar Scar
90
80
70
60
50
40
30
10
20
0
No Intern Intern
Figure 5.6: Sample 2 3 2 Design: Results from Piliavan et al. (1975)
new66480_05_c05_p173-222.indd 199 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
Table 5.3b: Levels versus Conditions in a 2 3 3 Design
No Hunger Mild Hunger Extreme Hunger
Pain 1 2 3
No Pain 4 5 6
This is a 2 3 3 design for a total of six conditions (numbered 1–6). “Pain” has two levels: presence and absence of pain. “Hunger” has three levels: no hunger, mild hunger, and extreme hunger.
In a different study, we might describe a design as a 2 3 3, which would tell people two things about this design: We have two independent variables and one variable has two levels (the 2), while the other variable has three levels (the 3). One possibility for this 2 3 3 design is illustrated in Table 5.3b. For this hypothetical study, the researcher was interested in the effects of discomfort on mood. She manipulated both hunger, with three levels, and pain, with two levels. Participants were asked to abstain from eating for 4 hours (“mild hunger”) or 24 hours (“extreme hunger”), or they were allowed to eat before the study (“no hunger”). During the laboratory session, participants were asked either to hold their hand in a bucket of ice water (“pain”) or not (“no pain”). As you can see in Table 5.3b, this results in six possible conditions for her participants to experience.
Main Effects and Interactions
When experiments involve only one independent variable, the analyses can be as simple as comparing two group means—as we did with our example in Chapter 1, comparing the happiness levels of couples with and without children. But what about cases where our design has more than one independent variable?
In a factorial design, we have two types of effect: A main effect refers to the effect of each independent variable on the dependent variable, averaging values across the levels of other variables. A 2 3 2 design has two main effects; a 2 3 2 3 2 design has three main effects because there are three IVs. An interaction occurs when the variables have a com- bined effect; that is, the effects of one IV are different depending on the levels of the other IV. So, applying this new terminology to the Pilavin et al. (1975) “subway emergency” study, we have three possible results (possible because we’ll eventually use statistical analyses to verify them):
1. The main effect of scar: Does the presence of a scar affect helping behavior? Yes. More people help in absence of a facial scar. In Figure 5.6, you can see that the
bars on the left (no scar) are, on average, higher than those on the right (scar).
2. The main effect of intern: Does the presence of an intern affect helping behavior? Yes. More people help when there is no medical intern on hand. In Figure 5.6,
you can see that the red bars (no intern) are, on average, higher than the blue bars (intern).
new66480_05_c05_p173-222.indd 200 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
3. The interaction between scar and intern: Does the effect of one variable depend on the effect of another variable?
Yes. If you refer to Figure 5.6, you can see that the presence of a medical intern matters more when the victim has a facial scar. In visual terms, the gap between red and blue bars is much larger in the bars on the right. This indicates an interac- tion between scar and intern.
Let’s walk through a fictional example. Imagine you were interested in people’s percep- tions of actors in different types of movies. You might predict that some actors are better suited to comedy and others are better suited to action movies. One simple way to test this hypothesis would be an experiment that showed four movies in a 2 3 2 design, using the same two actors in two movies (for a total of four conditions). Our first IV would be the movie type, with two levels: action and comedy. Our second IV would be the actor, with two levels: Will Smith and Arnold Schwarzenegger. Our dependent variable would be the ratings of each movie on a 10-point scale. This gives us three possible results:
1. The main effect of actor: Do people generally prefer Will Smith or Arnold Schwar- zenegger, regardless of the movie?
2. The main effect of movie type: Do people generally prefer action or comedy movies, regardless of the actor?
3 The interaction between actor and movie type: Do people prefer each actor in a differ- ent kind of movie? (i.e., are ratings affected by the combination of actor and movie type?)
After collecting data from a sample of participants, we end up with the following average ratings for each movie, shown in Table 5.4.
Table 5.4. Main Effects and Marginal Means: The Actor Study
Arnold 6 5
5.5
Will 1.5 8
4.75
Marginal Mean 3.75 6.5
Ac�on Comedy
Marginal Mean
These two means let us compare actor ra�ngs, ignoring the movie type. People have a slight preference for Arnold (5.5)
over Will (4.75).
These two means let us compare movie ra�ngs, ignoring the actor.
People have a slight preference for Comedy (6.5) over Ac�on (3.75)
Remember that main effects represent the effects of one IV, averaging across the levels of the other IV. To average across levels, we calculate the marginal means, or the combined mean across levels of another factor. In other words, the marginal mean for action movies is calculated by averaging together the ratings of both Arnold Schwarzenegger and Will Smith in action movies. The marginal mean for Arnold Schwarzenegger is calculated by averaging together ratings of Arnold Schwarzenegger in both action and comedy movies. Doing this for our 2 3 2 design results in four marginal means, which are presented
new66480_05_c05_p173-222.indd 201 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
alongside the participant ratings in Table 5.4. We would need to verify these patterns with statistical analyses, but it appears we have two main effects, based on these marginal means: People have a slight preference for comedy over action movies, as well as a slight prefer- ence for Arnold Schwarzenegger’s acting over Will Smith’s acting, regardless of the movie.
What about the interaction? Our main hypothesis here is that some actors perform best in some genres of movies (e.g., action or comedy) than they do in other genres. This suggests that the actor and the movie type combine to influence people’s ratings of the movies. We can get a sense of this from examining the means in Table 5.4, but it is even easier to appreciate in a graph. Figure 5.7 shows the mean of participants’ ratings across the four conditions. If we focus first on the ratings of Arnold Schwarzenegger, you can see that participants did have a slight preference for him in action (6) versus comedy (5) roles. Then, examining ratings of Will Smith, you can see that participants had a strong prefer- ence for him in comedy (8) versus action (1.5) roles. Together, this set of means indicates an interaction between actor and movie type because the effects of one variable depend on another. Or, in plain English: People’s perceptions of the actor depend on the type of movie he is in. This pattern of results is a nice fit for our hypothesis that certain actors are better suited to certain types of movie: Arnold should probably stick to action movies, and Will should definitely stick to comedies.
Before we move on to analyses, let’s look at one more example from a published experi- ment. A large body of research in social psychology suggests that stereotypes can negatively impact performance on cognitive tasks (e.g., tests of math and verbal skills). According to Stanford social psychologist Claude Steele and his colleagues, the fear of confirming nega- tive stereotypes about one’s group acts as a distraction. This distraction—which they term stereotype threat—makes it hard to concentrate and perform well, and thus leads to lower scores on a cognitive test (Steele, 1997). One of the primary implications of this research is that
F re
q u
e n
c y
Arnold Will
9
8
7
6
5
4
3
1
2
0
Action
Comedy
Figure 5.7: Interaction in the Actor Study
new66480_05_c05_p173-222.indd 202 10/31/11 9:31 AM
CHAPTER 5Section 5.4 Experimental Design
ethnic differences in standardized test scores can be viewed as a situational phenomenon— if we change the situation, the differences go away. In the first published study of stereotype threat, Claude Steele and Josh Aronson (1995) found that when African-American students at Stanford were asked to indicate their race before taking a standardized test, this was enough to remind them of negative stereotypes, and they performed poorly. But when the testing situation was changed, and participants were no longer asked their race, these stu- dents performed at the same level as Caucasian students. It is worth emphasizing that these were Stanford students and had therefore met admissions standards for one of the best universities in the nation. Even this group of elite students was susceptible to situational pressure but performed at their best when the pressure was eliminated.
In a great application of stereotype threat, social psychologist Jeff Stone at the University of Ari- zona asked both African-American and Caucasian college students to try their hands at putting on a golf course (Stone, Lynch, Sjomeling, & Darley, 1999). Putting was described as a test of natural athletic ability to half of the participants and as a test of sports intelligence to the other half. Thus, there were two independent variables: the race of the participants (African American or Caucasian) and the description of the task (“athletic ability” or “sports intelligence”). Note that “race” in this study is technically a quasi-independent variable because it is not manipulated. This resulted in a total of four conditions, and the dependent vari- able was the number of putts that participants managed to make. Stone and colleagues hypoth- esized that describing the task as a test of athletic ability would lead Caucasian participants to worry about the stereotypes regarding their poor athletic ability. In contrast, describing the task as a test of intelligence would lead African-American par- ticipants to worry about the stereotypes regarding their lower intelligence.
Consistent with their hypotheses, Stone and colleagues found an interaction between race and task description but no main effects. That is, neither race was better at the putting task overall, and neither task description had an overall effect on putting performance. But the combination of these variables proved fascinating. When the task was described as measuring sports intelligence, the African-American participants did poorly due to fear of confirming negative stereotypes about their overall intelligence. But when the task was described as measuring natural athletic ability, the Caucasian participants did poorly due to fear of confirming negative stereotypes about their athleticism. This is a beautiful illus- tration of an interaction; the effects of one variable (task description) depend on the effects of another (race of participants). And this is further confirmation of the power of the situa- tion: Neither group did better or worse overall, but both were responsive to a situationally induced fear of confirming negative stereotypes.
Comstock/thinkstock
Skill on the golf course used to study stereotypes was conducted by Jeff Stone at the University of Arizona.
new66480_05_c05_p173-222.indd 203 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
5.5 Analyzing Experiments
So far, we have been drawing conclusions about our experimental findings in concep-tual terms. But naturally, before we actually make a decision about the status of our hypotheses, we have to conduct statistical analyses. In this section, we will cover the most common statistical techniques for analyzing experimental data.
Dealing with Multiple Groups
You may find yourself wondering why we need a special technique for experimental designs. After all, we learned in Chapter 2 that we can compare two pairs of means using a t-test; why not just use several t-tests to analyze our experimental designs? In our movie ratings study, we could analyze the data using a total of six t-tests to capture every pos- sible pair of means: Arnold Schwarzenegger in a comedy versus Will Smith in a comedy; Arnold Schwarzenegger in an action movie versus Will Smith in an action movie; Arnold Schwarzenegger in a comedy versus an action movie; Will Smith in a comedy versus an action movie; Will Smith in a comedy versus Arnold Schwarzenegger in an action movie; and finally Will Smith in an action movie versus Arnold Schwarzenegger in a comedy.
But there is a problem with this approach: The odds of developing a Type I error (a false positive) are increased with every statistical test. We typically set our alpha level at .05 for a t-test, meaning that we’re comfortable with a 5% chance of a Type I error. Unfortunately, if we conduct six t-tests, each one has a 5% chance of a Type I error, meaning that we’re now rather likely to draw conclusions about a false positive. The moral of this story is that we need a statistical approach that reduces the number of comparisons we perform. Fortunately, this can be accomplished using a statistical technique called the analysis of variance (ANOVA), which tests for differences by comparing the amount of variance explained by the independent variables to the variance explained by error. The following sections explore ANOVA in more detail.
The Logic of ANOVA
The logic behind the analysis of variance is rather straightforward. As we have discussed throughout this course, the variability in data can be divided into systematic and error variance. That is, we can attribute some of the variability to the factors we are studying, but there will always be a degree of random error. In our movie ratings study, some of the variability in these ratings can be attributed to our independent variables (differences in actors and movie types), while some of the variability is due to other factors—perhaps some people simply like movies more than other people.
The ANOVA works by comparing the influence of these different sources of variance. We always want to explain as much of the variance as possible through the independent variables. If the independent variables have more influence than random error does, this is good news. But if error variance has more influence than the independent variables, this is bad news for the hypotheses. You can get a sense of this by comparing the three pie charts in Figure 5.8. The proportion of variance explained by our independent variables is shaded in blue, while the proportion explained by error is shaded in red. In the top graph, approximately 80% of the variance can be explained by our independent variables, which can be viewed as a good result. But in the middle graphs, variance is explained equally by
new66480_05_c05_p173-222.indd 204 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
the independent variables and by error, and in the bottom graph, only 20% of the variance is explained by the independent variables. Thus, in the latter two graphs, our independent variables do no better than random error at explaining our results.
One more analogy may be helpful. In the field of engineering, the term signal-to-noise ratio is used to describe the amount of light or sound or energy, and so forth that is detectable above and beyond background noise. This ratio is high when the signal comes through clearly and low when it is mixed in with static or other interference. Likewise, when you try to tune in your favorite radio station, the goal is to find a clear signal that is not cov- ered up by static. Believe it or not, the ANOVA statistic is doing the same thing. That is, the analysis tells us whether differences in experimental conditions (signal) are detectable above and beyond error variance (noise). In the next section, we’ll take a closer look at how ANOVA makes these distinctions.
Calculating ANOVA
The ANOVA statistic—abbreviated with the capital letter F—works by comparing the variance that is explained by group differences to the variance explained by error. This is really just a minor adaptation, with new terminology, of the formula we have encountered before, describing the components of total variance in the dataset:
Total Variance 5 Systematic Variance 1 Error Variance
In conducting an ANOVA, these two components will have special names, but the idea is the same. Specifically, we refer to systematic variance as between-groups variance, or the variance explained by differences in our IVs. In the movie ratings study, between- groups variance represents the variability in ratings that can be attributed to different
Explained by Ivs
Explained by Error
Explained by Ivs
Explained by Error
Explained by Ivs
Explained by Error
Figure 5.8: Comparing Sources of Variance
new66480_05_c05_p173-222.indd 205 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
actors, different movies, and the combinations of these IVs. In addition, we refer to error variance as within-groups variance, or the total variance that occurs across each condi- tion. In the movie ratings study, within-groups variance would represent the sum total of all other influences on movie ratings (i.e., the error).
In this day and age, ANOVA (F) is calculated using one of the numerous statistical soft- ware programs; these programs take columns of data and churn out significance tests to test hypotheses. But in order to understand what these significance tests mean, it is worth walking through a conceptual overview of the pieces of the ANOVA formula. The start- ing point for the F statistic is the calculation of several sums of squares, or the sum of the squared deviations between individual scores and the overall sample mean. We encoun- tered these sums of squares (abbreviated SS) in Chapter 3, in our discussion of calculating the standard deviation.
Using this notation, our components of variance formula can be rewritten:
SST 5 SSBG 1 SSWG
SST 5 sum of squares total SSBG 5 sum of squares between groups SSWG 5 sum of squares within groups
In conceptual terms, SSBG represents the deviation between each condition’s mean and the overall mean of the sample. If you refer to the data from our movie ratings study in Table 5.4, you’ll see the four condition means of 6, 5, 1.5, and 8. The overall mean of movie ratings—across conditions—is 5.13 (i.e., 6 1 5 1 1.5 1 8 4 4). To compute the SSBG, we subtract each of the four condition means from 5.13, square the difference, and add up these squared deviation scores.
In conceptual terms, SSWG represents the deviation between each individual score and the condition mean for that individual. To compute SSWG in our movie ratings study, we sub- tract each individual score from the relevant condition mean, square the difference, and add up the squared deviation scores.
That’s a lot of math, so let’s take a step back. In essence, these terms are calculating and comparing two sources of variation around the overall mean. When we combine all the participants’ movie ratings, and ignore everything about what they were rating, we get an overall mean of 5.13. But, naturally, individual participants deviate from this mean— some people gave higher ratings and some gave lower ratings. The point of calculating SS terms is to understand these deviations. Our between-groups SS gives us an index of how much each condition deviates from the overall mean, while our within-groups SS gives us an index of how much individuals differ from the mean of their condition. If the former is higher, it means that the groups matter; if the latter is higher, it means that individual quirks mean more than our independent variables.
These sums of squares are the first step toward calculating our F statistic, but they have a flaw that has to be addressed. Namely, sum of squares is an imprecise measure because it does not take sample size into account. You’ll notice that the SS terms are calculated by add- ing up the squared deviation scores for each participant (or each condition). If our experi- ment happened to have a large sample, the SSWG would automatically be higher because
new66480_05_c05_p173-222.indd 206 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
there would be a greater number of squared deviation scores. Likewise, if our experiment happened to have several conditions, the SSBG would be inflated. Fortunately, this flaw is easily corrected by calculating a value called the mean square, which corrects the sum of squares for sample size by dividing by the appropriate degrees of freedom. The resulting values represent the average deviation from the sample mean, corrected for the number of scores that went into the calculation—much like the final step in our calculation of standard deviations in Chapter 3. So, our components of variance formula is altered one more time:
MST 5 MSBG 1 MSWG
MST 5 mean square total MSBG 5 mean square between groups MSWG 5 mean square within groups
We now have all the information we need to calculate the F statistic (ANOVA), using the following formula:
F 5 between-groups variability within-groups variability
5 MSBG MSWG
This is the formula statistical software programs use to calculate an F value. F is expressed as a ratio of variances, expressing group differences relative to error. Thus, the bigger the F value, the higher the ratio of group differences to error, and the bigger the influence of inde- pendent variables. In other words, we want F values to be as large as possible because this indicates that our experimental manipulations make a difference on the outcome variable.
We can also frame the components of this ratio in terms of the null hypothesis versus the experimental hypothesis. If our experimental hypothesis is true, MSWG represents random variation and MSBG represents meaningful differences between groups. Thus, MSBG . MSWG, and F will be a large value. In contrast, if the null hypothesis is true, both MSWG and MSBG represent random variation because our experimental manipulations do not have a mean- ingful effect. Thus, MSBG 5 MSWG, and the value of F is 1. One final possibility here is that our value of F can occasionally be less than 1. This occurs when results are more influ- enced by random variation than they are by the independent variables; in other words, MSWG . MSBG.
One-Way versus Factorial ANOVA
Before we move on from the pieces of our ANOVA, there is one more level of complexity to explore. The formula we have been discussing so far only describes an ANOVA for a one- way design. When there is one independent variable, MST 5 MSBG 1 MSWG. However, a factorial design has multiple independent variables, and the between-groups variance term (MSBG) is divided into several pieces. As we discussed, a 2 3 2 design has three possible results: a main effect of variable A; the main effect of B; and the interaction of A and B. To account for these effects, we split our between-groups variance into three new components:
MSBG 5 MSA 1 MSB 1 MSAB
MSA 5 MSB 1 MSAB
new66480_05_c05_p173-222.indd 207 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
MSA 5 mean square for the main effect of A MSB 5 mean square for the main effect of B MSAB 5 mean square for the interaction of A and B
This results in another update to our components of variance formula, replacing the MSBG term with these three components:
MST 5 (MSA 1 MSB 1 MSAB) 1 MSWG
This also results in three F values, expressing each mean square relative to within-group variance. These are illustrated below using our movie ratings study. Our 2 3 2 design in that study yields a main effect of actor, a main effect of movie type, and the actor * movie interaction:
FACTOR 5 MSACTOR
MSWG
FMOVIE 5 MSMOVIE
MSWG
FACTOR*MOVIE 5 MSACTOR*MOVIE
MSWG For illustration purposes, consider the following formula, representing the components of variance in a three-way design (which, incidentally, would yield seven F values!). As our designs become more complex, it becomes easier to appreciate the benefits of statistical software packages.
MST 5 (MSA 1 MSB 1 MSC 1 MSAB 1 MSBC 1 MSAC 1 MSABC) 1 MSWG
MSA 5 mean square for the main effect of A MSB 5 mean square for the main effect of B MSAB 5 mean square for the interaction of A and B MSBC 5 mean square for the interaction of B and C MSAC 5 mean square for the interaction of A and C MSABC 5 mean square for the three-way interaction of A, B, and C
Research: Thinking Critically Love Ballad Leaves Women More Open to a Date
Medical News Today
If you’re having trouble getting a date, French researchers suggest that picking the right soundtrack could improve the odds. Women were more prepared to give their number to an ‘average’ young man after listening to romantic background music, according to research that appears in the journal Psychology of Music, published by SAGE.
There’s plenty of research indicating that the media affects our behaviour. Violent video games or music with aggressive lyrics increase the likelihood of aggressive behaviour, thoughts and (continued)
new66480_05_c05_p173-222.indd 208 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
feelings—but do romantic songs have any effect? This question prompted researchers Nicolas Guéguen and Céline Jacob from the Université de Bretagne-Sud along with Lubomir Lamy from Université de Paris-Sud to test the power of romantic lyrics on 18-20 year old single females. And it turns out that at least one romantic love song did make a difference.
Guéguen and Jacob were part of a research team that had already shown how romantic music played in a flower shop led to male customers spending more money. This time the researchers used questionnaires to pinpoint agreed-upon neutral and romantic songs. They chose ‘Je l’aime à mourir,’ a well-known love song by French songwriter Francis Cabrel, and the neutral song ‘L’heure du thé,’ by Vincent Delerm. A group of young women separate from the main study rated 12 young male volunteers for attractiveness, and the research- ers picked the one rated closest to ‘average’ to help with the experiment.
The researchers then set up a scenario where the 87 females each spent time in a waiting room with back- ground music playing, before moving to a different room where the experimenter instructed her to discuss the difference between two food products with the young man. Once the experimenter returned, she asked them to wait for a few moments alone, and this gave the ‘average’ male a chance to use his standard chat up line: “My name is Antoine, as you know, I think you are very nice and I was wondering if you would give me your phone number. I’ll phone you later and we can have a drink together somewhere next week.’
The love song in the waiting room almost doubled Antoine’s chances of getting a woman’s number—52% of participants responded to his advances under the influence of Francis Cabrel, whereas only 28% of those who had heard the ‘neutral’ song by Vincent Delerm offered their details.
“Our results confirm that the effect of exposure to media content is not limited to violence and could have the potential to influence a high spectrum of behaviour,” says Guéguen. “The results are interesting for sci- entists who work on the effect of background music on individuals’ behaviour.”
The results also add weight to a general learning model proposed by Buckley and Anderson in 2006 to explain the effect of media exposure. Their model states that media exposure in general, and not only aggressive or violent media, affects individuals’ internal states, which explains why prosocial media fosters prosocial outcomes.
Why did the music have this effect? It may be that, as shown in earlier research, the music induced positive affect (in psychological terms, affect is the experience of feeling or emotion). Positive affect is associated with being more receptive to courtship requests. Alternatively, the romantic content of the song may have acted as a prime that then led to displays of behaviour associated with that prime. In either case, further research is needed before the researchers will commit to wider generalisations on the targeted use of love songs. But if you’re a hopeful single, awareness of the background music certainly won’t do any harm.
Think about it
1. In this experiment, the type of song (love song or neutral song) is confounded with at least one other variable. Try to identify one. Do you think that this confounded variable would make a difference? How would you design a study that overcomes this?
2. Describe how demand characteristics might compromise the internal validity of this study. Can you think of any ways around this?
3. Toward the end of the article, the authors suggest that one explanation for these results is that the romantic music put the women into a more positive mood, and that this in turn made them more receptive to the man. How could you design a study that tests this hypothesis?
4. Given the nature of the DV in this study, would an ANOVA test be appropriate? What would be the more appropriate statistical test, and why?
Research: Thinking Critically (continued)
new66480_05_c05_p173-222.indd 209 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
Exploring Your Data
Now that we have F values, what do we do with them? At the end of conducting an ANOVA, we have a yes-or-no answer to the following question: Do our experimental groups have a systematic effect on the dependent variable? The answer lets us decide whether to reject the null hypothesis, but it does not tell us everything we want to know about the data. In essence, a significant F value tells us that there is a significant difference between the groups, but it does not tell us what the difference is. If we conducted an ANOVA on our movie ratings study, we would see a significant interaction between actor and movie, but we would need to take additional steps to determine the meaning of this interaction.
In this section, we will walk through the process of exploring and interpreting ANOVA results in order to make sense of the data. Our example is drawn from a published study by Newman, Josephs, and Sellers (2005), which was designed to explore the effects of testosterone on cognitive performance. Previous research had suggested that testosterone was involved in two types of complex human behavior. On one hand, people with higher testosterone tend to perform better on tests of spatial skills, such as having to rotate objects mentally, and perform worse on tests of verbal skills, such as listing all the synonyms for a particular word. These patterns are thought to reflect the influence of testosterone on developing brain structures. On the other hand, people with higher testosterone are also more concerned with gaining and maintaining high status relative to other people. Tes- tosterone is correlated with a person’s position in the hierarchy and tends to rise and fall when people win and lose competitions, respectively. In one study by the sociologist Alan Mazur and his colleagues, testosterone levels were measured before, during, and after a series of professional chess matches. They found that testosterone rose in both players in anticipation of the competition, then rose even further in the winners, but plummeted in the losers (Mazur, Booth, & Dabbs, 1992).
Newman and colleagues (2005) set out to test the combination of these variables. Based on previous research, they hypothesized that people with higher testosterone would be uncomfortable when they were placed in a low-status position, leading them to perform worse on cognitive tasks. The researchers tested this hypothesis by randomly assigning people to a high status, low status, or control condition, and then administering a spatial and a verbal test. This resulted in a 2 (testosterone: high or low) 3 3 (condition: high status, low status, control) between-subjects design, for a total of six groups. Note that “testosterone” in this study is a quasi-independent variable, because it is measured rather than manipulated by the experimenters.
Once the results were in, the ANOVA revealed an interaction between testosterone and status but no main effects. The results of the study are shown in Figure 5.9. These bars represent z scores that combine the spatial and verbal tests into one number. So, what do these numbers mean? How do we make sense out of the patterns? This involves a combi- nation of comparing means and calculating effect sizes, as discussed next.
new66480_05_c05_p173-222.indd 210 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
Mean Comparisons The first step in interpreting our results is to compare the various pairs of means within the design. This might seem counterintuitive, since the whole point of the ANOVA was to test for effects without comparing individual means. Our goal, therefore, is to somehow explore differences in conditions without inflating Type I error rates. There are two strate- gies for achieving this balance.
Planned comparisons (also called a priori comparisons) involve comparing only the means for which differences were predicted by the hypothesis. In the experiment by New- man et al. (2005), the hypothesis explicitly stated that high-testosterone people should perform better in a high status position than a low status position. So, a planned compari- son for this prediction would involve comparing two means with a t-test: high T, high status (the highest red bar); and high T, low status (the lowest yellow bar). Consistent with the researchers’ hypothesis, high-testosterone people did perform higher on both tests, t (27) 5 2.35, p 5 0.01, but only in a high status position. Type I errors are of less concern with planned comparisons because only a small number of theoretically driven compari- sons are being conducted.
If you refer to the graph of these results in Figure 5.9 and compare high- with low-testosterone people, you’ll notice another interesting pattern: In a high status position, high-testosterone people do better than low-testosterone people. But in a low status position, this pattern is reversed, and high-testosterone people do worse. However, the researchers did not predict these mean comparisons, so it would be cheating to do planned contrasts. Instead, they would use a second strategy called a post hoc comparison, which controls the overall alpha by taking into account the fact that multiple comparisons are being performed. In most cases, post hoc tests are only allowed if the overall F test is significant.
C o
m b
in e d
C o
g n
it iv
e T
e s t
S c o
re
High Testosterone Low Testosterone
T Level
Leader
Control
Follower
–0.6
–0.4
–0.2
0
0.2
0.4
0.6
Figure 5.9: Exploring Your Data: Results from Newman et al. (2005)
new66480_05_c05_p173-222.indd 211 10/31/11 9:31 AM
CHAPTER 5Section 5.5 Analyzing Experiments
One popular way to conduct post hoc tests while minimizing the error rate is to use a tech- nique called a Bonferroni correction. This technique, named after the Italian mathematician who developed it, involves simply adjusting the alpha level by the number of comparisons that are performed. For example, imagine you want to conduct 10 follow-up post hoc tests to explore your data. The Bonferroni correction would involve dividing the alpha level (.05) by the number of comparisons (10), for a corrected alpha level of .005. Then, rather than using a cutoff of .05 for each test, you use this more conservative Bonferroni-corrected value. Translation: Rather than accepting a Type I error rate of 5%, we are moving to a more conservative 0.5% cutoff to correct for the number of comparisons that we are performing.
Another popular alternative to the Bonferroni correction is called Tukey’s HSD (for Hon- estly Significant Difference). This test works by calculating a critical value for mean com- parisons (the HSD), and then using this critical value to evaluate whether mean compari- sons are significantly different. This manages to avoid inflating Type I error because the HSD is calculated based on the sample size, the number of experimental conditions, and the MSWG, which essentially tests all the comparisons at once. In the study by Newman et al. (2005), both of these post hoc tests were significant: Compared to those low in testoster- one, high-testosterone people did better in a high status position but worse in a low status position. This suggests that high testosterone magnifies the effect of testing situations on cognitive performance.
Effect Size As you’ll remember from our discussion in Chapter 2, statistical significance is only part of the story; researchers also want to know how big the effects of their independent variables are. There are several ways to calculate effect size, but in general, bigger values mean a stronger effect. One of these statistics, Cohen’s d, is calculated as the difference between two means divided by their pooled standard deviation. The resulting values can therefore be expressed in terms of standard deviations; a d of 1 means that the means are one stan- dard deviation apart. How big should we expect our effects to be? Based on his analyses of typical effect sizes in the social sciences, Cohen suggests the following benchmarks: d 5 .20 is a small effect; d 5 .40 is a moderate effect; and d 5 .60 is a large effect. In addition to these qualitative categories, effect size values can be interpreted in terms of standard deviation units. So, a d of 1 is equivalent to a standard deviation of 1. In other words, a large effect in social and behavioral sciences accounts for a little more than half of a standard deviation.
In interpreting the results of their testosterone experiment, Newman and colleagues (2005) computed effect size measurements for two of the key mean comparisons. First, they compared high-testosterone people in the high and low status conditions; the size of this effect was a d 5 .78. Second, they compared the high- and low-testosterone people in the low status condition; the size of this effect was a d 5 .61. Both of these effects fall in the “large” range based on Cohen’s benchmarks. More important, taken together with the mean comparisons, they help us to understand the way testosterone affects behavior. The authors conclude that cognitive performance stems from an interaction between biology (testosterone) and environment (assigned status) such that high-testosterone people are more responsive to their status in a given situation. When they are placed in a high sta- tus position, they relax and perform well. But when placed in a low status position, they become distracted and perform poorly. This nuanced conclusion is only possible through an exploration of the data, using mean comparisons and effect size measures.
new66480_05_c05_p173-222.indd 212 10/31/11 9:31 AM
CHAPTER 5Section 5.6 Wrap-Up: Avoiding Error
5.6 Wrap-Up: Avoiding Error
As we wrap up our final chapter, it is worth thinking back to one of the key concepts in Chapter 2, Type I and Type II errors. Regardless of the research question, the hypothesis, or the particulars of the research design, all studies have the goal of making accurate decisions about the hypotheses. That is, we need to be able to correctly reject the null hypothesis when it is false and to fail to reject the null when it is true. But, from time to time, and despite our best efforts, we make mistakes when we draw conclu- sions about our hypotheses, as summarized in Table 5.5. A Type I error, or “false positive,” involves falsely rejecting a null hypothesis and getting excited about an effect that is due to chance. A Type II error, or “false negative,” involves failing to reject the null hypothesis and missing an effect that is real and interesting. (If you need more of a refresher on these terms, refer back to Chapter 2.)
Table 5.5: Review of Type I and Type II Errors
Researcher's Decision
Reject Null Fail to Reject Null
Null is FALSE Correct Decision Type II Error
Null is TRUE Type I Error Correct Decision
In this section, we take a problem-solving approach to minimizing both of these errors in an experimental context. It turns out that each error is primarily under the researcher’s control at different stages in the research process, which means there are different strate- gies for reducing each error.
Avoiding Type I Error
Type I errors occur when results are due to chance but are mistakenly interpreted as sig- nificant. We can generally reduce the odds of this happening by setting our alpha level at p , .05, meaning that we will only get excited about results that have less than a 5% chance of Type I error. However, Type I errors can still occur as a result of either extremely large samples or large numbers of statistical comparisons. Large samples can make small effects seem highly significant, so it is important to set a more conservative alpha level in large- scale studies. And, as we have been discussing in this chapter, the odds of Type I error are compounded with each statistical test we conduct.
This means that Type I error is primarily under our control during statistical analysis—the smarter our statistics, the lower our odds of Type I error. We have covered several examples of “smart” statistics in this chapter: Instead of conducting lots of t-tests, we use an ANOVA to simultaneously test for differences across the entire design. Instead of conducting t-tests to compare means after an ANOVA, we use a mix of planned contrasts (for comparisons that we predicted) and post hoc tests (for other comparisons we want to explore). There are also more advanced statistical techniques that take this a step further. For example, the multivariate analysis of variance (MANOVA) statistic analyzes sets of dependent variables
new66480_05_c05_p173-222.indd 213 10/31/11 9:31 AM
CHAPTER 5Section 5.6 Wrap-Up: Avoiding Error
to further reduce the number of individual tests. This approach is used when dependent variables represent different measures of a related concept, such as using heart rate, blood pressure, and muscle tension to capture the stress response. The MANOVA works, broadly speaking, by computing a weighted sum of these separate DVs (called a canonical variable), and using this new variable as the dependent variable. If you are interested in learning more about this and other advanced statistical techniques, check out the excellent volume by James Stevens (2002), Applied Multivariate Statistics, http://goo.gl/zgGDi.
Avoiding Type II Error
Type II errors occur when the results are significant but are mistakenly credited to chance. The primary sources of this error are small samples and bad design. Small samples may fail to capture enough variability and may therefore lead to nonsignificant p values in testing an otherwise significant effect. Both large and small mistakes in experimental designs can add noise to the dataset, making it difficult to detect the real effects of independent variables.
This means that Type II error is primarily under the researcher’s control during the design process—the smarter the research designs, the lower the odds of Type II error. First, as we discussed in Chapter 2, it is relatively simple to estimate the sample size needed for our research using a power calculator. These tools take basic information about the number of conditions in the research design and the estimated size of the effect and then estimate the number of people needed to detect this effect. (See Chapter 2, Figure 2.5, for an annotated example using one of these online calculators.)
Second, as we have discussed in every chapter, it is the experimenter’s responsibility to take steps to minimize extraneous variables that might interfere with the hypothesis test. Whether you are conducting an observation, a survey study, or an experiment, the overall goal is to ensure that the variables of interest are the main cause of changes in your depen- dent variable. This is perhaps easiest in an experimental context because these designs are usually conducted in a controlled setting where the experimenter has control over the independent variables. Nonetheless, as we discussed earlier in this chapter, many factors can threaten the internal validity of an experiment—from confounds to sample bias to expectancy effects. In essence, the more you can control the influence of these extraneous variables, the more confidence you can have in the results of your hypothesis test.
Table 5.6 presents a summary of the information in this section, listing the primary sources of Type I and Type II errors, as well as the time period when these are under experimenter control.
Table 5.6: Summary—Avoiding Error
Error Definition Main Source When You Can Control
Type I False-positive Lots of tests; lots of people
Conducting stats
Type II False-negative Bad measures; not enough people
Designing experiments
new66480_05_c05_p173-222.indd 214 10/31/11 9:31 AM
CHAPTER 5Summary
Summary
In this chapter, we have focused on experimental designs, the last group of specific research designs, and the highest point on the continuum of control. The primary goal of experimental designs is to explain behavior in causal terms. This chapter began with an overview of experimental terminology and the key features of experiments. Experi- ments can be distinguished from other research designs by three key features. First, the researcher manipulates a variable, giving him or her a fair amount of confidence that the independent variable causes changes in the dependent variable. Second, the researcher controls the environment, ensuring that everything about the experimental context is the same for different groups of participants—except for the level of the independent variable. Finally, the researcher has the power to assign participants to conditions, using random assignment. This process helps to ensure that preexisting differences among participants (e.g., in mood, motivation, intelligence, etc.) are balanced out across the experimental conditions.
Next, we covered the concept of experimental validity. In evaluating our experiments, we have to take into account both internal validity—or the extent to which the IV is the cause of changes in the DV—and external validity—or the extent to which the results general- ize beyond the specific laboratory setting. Several factors can threaten internal validity, including experimental confounds, selection bias, and expectancy effects. The common thread among these threats is that they add noise to the hypothesis test and cast doubt on the direct connection between IV and DV. External validity involves two components, the realism of the study and the generalizability of the findings. Psychology experiments are designed to study real-world phenomena, but sometimes compromises have to be made to study these phenomena in the laboratory. The balance is often achieved via mundane realism, or replicating the psychological conditions of the real phenomenon. Last, we have more confidence in the findings of our study when they can be replicated, or repeated in different settings with different measures.
In designing the nuts and bolts of our experiments, we have to make decisions about both the nature and number of independent variables. First, our designs can be described as between-subject, within-subject, or mixed. In a between-subject design, participants are in only one experimental condition and receive only one combination of the indepen- dent variables. In a within-subject design, participants are in all experimental conditions and receive all combinations of the independent variables. Finally, a mixed design con- tains a combination of between- and within-subject variables. Second, our designs can be described as either one-way or factorial. One-way designs consist of only one IV with at least two levels; factorial designs consist of at least two IVs, each having at least two levels. In a factorial design, there are several results to examine: the main effect of each IV, plus the interaction, or combination, of the IVs.
We also discussed the process of analyzing experimental data, using the analysis of vari- ance (ANOVA) statistic. This test works by simultaneously comparing sources of variance and, therefore, avoids the risk of inflated Type I error. The ANOVA (or F) is calculated as a ratio of systematic variance to error variance, or, more specifically, of between-groups variance to within-groups variance. The bigger this ratio, the more experimental manipu- lations contribute to overall variability in scores. However, the F statistic only tells us that differences exist in the design, and further analyses are necessary to explore these
new66480_05_c05_p173-222.indd 215 10/31/11 9:31 AM
CHAPTER 5Summary
differences. We walked through an example from a published study, discussing the pro- cess of comparing means and calculating effect sizes. In comparing means, we use a mix of planned contrasts (for comparisons that we predicted) and post hoc tests (for other comparisons we want to explore).
Finally, we concluded the chapter by referring to two recurring concepts, Type I error (false positive) and Type II error (false negative). These errors interfere with our broad goal of making correct decisions about the status of our hypothesis. Thus, the goal of this final section was to review ways to minimize errors. Type I errors are primarily inflated by large samples and lots of statistical analyses. Consequently, this error is under the experimenter’s control at the data analysis stage. Type II errors are primarily inflated by small samples and flaws in the experimental design. Consequently, this error is under the experimenter’s control at the design and planning stage.
Key Terms
experimental design design whose pri- mary goal is to explain causes of behavior
independent variable (IV) the variable manipulated by the experimenter to test hypotheses about cause
dependent variable (DV) the variable measured by the experimenter to assess the effects of the independent variable
condition one of the versions of an inde- pendent variable, forming different groups in the experiment; in a factorial design, refers to the groups formed by combina- tions of IVs
experimental condition group within the experiment that receives a treatment designed to test a hypothesis
control condition group within the exper- iment that does not receive the experimen- tal treatment
level another way to describe the versions of an independent variable; describes the specific circumstances created by manipu- lating a variable
environmental manipulation changing some aspect of the experimental setting
instructional manipulation changing the way a task is described to change partici- pants’ mind-sets
invasive manipulation taking measures to change internal, physiological processes; usually conducted in medical settings
quasi-independent variable preexist- ing difference used to divide participants in an experimental context; referred to as “quasi” because variables are being measured, not manipulated, by the experimenter
extraneous variable variable that adds noise to a hypothesis test
random assignment a technique for assigning participants to conditions; before participants arrive, the experimenter makes a random decision for each partici- pant’s placement in a group
matched random assignment a variation on random assignments; ensures that an important variable is equally distributed between or among the groups; the experi- menter obtains scores on an important matching variable, ranks participants on this variable, and then randomly assigns participants to conditions
new66480_05_c05_p173-222.indd 216 10/31/11 9:31 AM
CHAPTER 5Summary
internal validity a metric that assesses the degree to which results can be attributed to independent variables
external validity a metric that assesses generalizability of results beyond the spe- cific conditions of the experiment
confounding variable (or confound) a variable that changes systematically with the independent variable
selection bias occurs when groups are dif- ferent before the manipulation; problem- atic because preexisting differences might be the driving factor behind the results
differential attrition loss of participants, who drop out of experimental groups for different reasons
experimenter expectancy researchers see what they expect to see, leading to subtle bias in favor of their hypotheses; threat to internal validity
demand characteristic cue in the study that leads participants to guess the hypothesis
cover story a misleading statement to participants about what is being studied to prevent effects of demand characteristics
unrelated-experiments technique to prevent effects of demand characteristics, leading participants to believe that they are completing two experiments during one session; experimenter can use this to present the independent variable dur- ing the first experiment and measure the dependent variable during the second experiment
placebo effect change resulting from the mere expectation that change will occur
placebo control group added to a study to reduce placebo effects; mimics the experi- mental condition in every way but one
mundane realism research that replicates the psychological conditions of the real- world phenomenon; criterion for judging external validity
generalizability the extent to which results extend to other studies, using a wide variety of populations and of opera- tional definitions
replication repetition of research results in different contexts and/or different laboratories
exact replication re-creation of the origi- nal experiment as closely as possible to verify the findings
conceptual replication testing the rela- tionship between conceptual variables using new operational definitions
participant replication repetition of the study with a new population of partici- pants; usually driven by a compelling the- ory as to why the two populations differ
constructive replication re-creation of the original experiment that adds elements to the design; usually designed to rule out alternative explanations or extend knowl- edge about the variables under study
between-subject design experimental design in which each group of participants is exposed to only one level of the inde- pendent variable
within-subject design experimental design in which each group of participants is exposed to all levels of the independent variable
new66480_05_c05_p173-222.indd 217 10/31/11 9:31 AM
CHAPTER 5Summary
carryover effect effects of one level are present when another level is introduced, making it difficult to separate the effects of different levels
order effect moderation of the effects because of the order in which levels occur
practice effect improvement of partici- pants’ performance as a result of repeated testing
fatigue effect decline of participants’ per- formance as a result of repeated testing
counterbalancing variation of the order of presentation among participants to reduce order effects
mixed design experimental design that contains at least one between-subject variable and at least one within-subject variable
one-way design a design that has only one independent variable, with two or more levels to the variable
factorial design a design that has two or more independent variables, each with two or more levels
factorial notation a system for describing the number of variables and the number of levels in experimental designs
main effect the effect of each independent variable on the dependent variable, col- lapsing across the levels of other variables
interaction the combined effect of vari- ables in a factorial design; the effects of one IV are different depending on the levels of the other IV
marginal mean the combined mean of one factor across levels of another factor
analysis of variance (ANOVA) a statisti- cal procedure that tests for differences by comparing the variance explained by sys- tematic factors to the variance explained by error
between-groups variance the variance explained by differences in the IVs
within-groups variance the total variance that occurs across conditions
sum of squares the sum of the squared deviations between individual scores and the overall sample mean
mean square a component of the F sta- tistic; corrects the sum of squares for sample size by dividing by the appropriate degrees of freedom; resulting values repre- sent the average deviation from the sample mean, corrected for sample size
planned comparison (or a priori compari- son) comparisons that involve comparing only the means for which differences were predicted by the hypothesis
post hoc comparison comparison that controls the overall alpha by taking into account that multiple comparisons are being performed; usually allowed only if the overall F test is significant
Bonferroni correction a post hoc test that involves adjusting the alpha level by the number of comparisons to set a more con- servative cutoff
Tukey’s HSD (Honestly Significant Dif- ference) a post hoc test that calculates a critical value for mean comparisons (the HSD) and then uses this critical value to evaluate whether mean comparisons are significantly different
multivariate analysis of variance (MANOVA) a statistic that analyzes sets of dependent variables to reduce the num- ber of individual tests
new66480_05_c05_p173-222.indd 218 10/31/11 9:31 AM
CHAPTER 5Summary
Apply Your Knowledge
1. When ANOVA is used to analyze data from experiments with two independent variables (A and B), the total variability (MSTOTAL) is composed of four parts, or components. What are they? (Writing the equation is fine.)
2. List and briefly describe the three distinguishing features of an experiment.
a. b. c.
3. List the three types of expectancy effect that can affect experimental results, and name one way to avoid each type.
a. b. c.
4. The following designs are described using factorial notation. For each one, state (a) the number of variables in the design, (b) the number of levels each variable has, and (c) the total number of experimental conditions.
3 3 3 3 3 a. b. c.
2 3 3 3 4 a. b. c.
4 3 4 a. b. c.
2 3 2 3 2 3 2 a. b. c.
5. Forty students were asked to rate two authors according to their knowledge of certain topic areas. Each student was given two passages to read. In one passage (“Brain”), the author discussed the roles of various brain structures in perceptual- motor coordination. In the second passage (“Motivation”), the author described ways to enhance motivation in preschool children. For half the students, both passages were written by male authors. For the other half of the students, both passages were written by a female author. After reading the passages, students rated the authors’ knowledge of their topic areas on a scale ranging from 1 (Dis- plays very little knowledge) to 10 (Displays a thorough knowledge).
new66480_05_c05_p173-222.indd 219 10/31/11 9:31 AM
CHAPTER 5Summary
Male Author Female Author
Brain 9 4
Motivation 6 7
A. Identify the following information about the design:
(1). Describe the design using factorial notation (e.g., 4 3 3). (2). Identify the total number of conditions . (3). Identify the design (circle one): between-subject within-subject mixed
B. The data from this study are analyzed and the results indicate the following F values:
(1). Topic Area: F 5 .91 (p 5 .87). (2). Author’s Gender: F 5 3.14 (p , .05). (3). Topic Area 3 Author’s Gender: F 5 12.46 (p , .01).
C. Interpret these results in terms of the three possible effects. That is, for each effect, state whether it is significant and then interpret the statistics in English.
(1). Is there a main effect of Topic Area? (circle one) YES NO interpretation in English:
(2). Is there a main effect of Author Gender? (circle one) YES NO interpretation in English:
(3). Is there an interaction between Topic Area and Author Gender? YES NO interpretation in English:
6. For each of the following scenarios, identify what a Type I error and a Type II error would look like. Then, determine which type would be a bigger problem for that scenario.
a. A large international airport has received a bomb threat. In response, the airport police have tightened security and now check every piece of luggage manually.
(1). Type I: (2). Type II: (3). Bigger problem:
b. Your friend purchases a pregnancy test. (1). Type I: (2). Type II: (3). Bigger problem:
Critical Thinking Questions
1. Explain the advantages and disadvantages of a within-subject design.
2. Compare and contrast the following terms. Your answers should demonstrate that you understand each term. Be sure to give some kind of context (e.g., “both are types of . . .”) or provide an example, and state how they are different.
new66480_05_c05_p173-222.indd 220 10/31/11 9:31 AM
CHAPTER 5Summary
a. internal vs. external validity b. between-subjects vs. within-subject design c. level vs. condition
3. Explain the difference between Type I and Type II errors. How can each type of error be minimized?
new66480_05_c05_p173-222.indd 221 10/31/11 9:31 AM
new66480_05_c05_p173-222.indd 222 10/31/11 9:31 AM
