Experimental Validity & Quantitative Research Designs

Researchers at the University of Minnesota work with twins in order to

Creatas Images/Thinkstock

Chapter 5 Experimental Designs—Determining Cause-and-Effect

Rela�onships

Cosmo Condina/Stone/Ge�y Images

Chapter Contents

Experiment Terminology (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.1#sec5.1) Key Features of Experiments (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.2#sec5.2) Experimental Validity (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.3#sec5.3) Experimental Designs (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.4#sec5.4) Analyzing Experiments (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.5#sec5.5) Wrap-Up: Avoiding Error (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.6#sec5.6) Cri�quing a Quan�ta�ve Study (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.7#sec5.7) Mixed Methods Research Designs (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec5.8#sec5.8)

One of the oldest debates within psychology concerns the rela�ve contribu�ons that biology and the environment make in shaping our thoughts, feelings, and behaviors. Do we become who we are because it is hard-wired into our DNA or in response to early experiences? Do people take on their parents’ personality quirks because they carry their parents’ genes or because they grew up in their parents’ homes? There are, in fact, several ways to address these types of ques�ons. In fact, a consor�um of researchers at the University of Minnesota has spent the past 2 decades comparing pairs of iden�cal and fraternal twins to tease apart the contribu�ons of genes and environment. You can read more at the research group’s website, Minnesota Center for Twin and Family Research, h�p://mc�r.psych.umn.edu/ (h�p://mc�r.psych.umn.edu/) .

An alterna�ve to using twin pairs to separate gene�c and environmental influence is through the use of experimental designs, which have the primary goal of explaining the causes of behavior. Recall from Chapter 2 (Sec�on 2.1 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.1#sec2.1) , Overview of Research Designs) that experiments can speak to cause and effect because the experimenter has control over the environment and is able to manipulate variables. One par�cularly ingenious example comes from the laboratory of Michael Meaney, a professor of psychiatry and neurology at McGill University, using female rats as experimental subjects (Francis, Dioro, Liu, & Meaney, 1999). Meaney’s research revealed that the paren�ng ability of female rats can be reliably classified based on how a�en�ve they are to their rat pups, as well as how much �me they spend grooming the pups. The ques�on tackled in this study was whether these behaviors were learned from the rats’ own mothers or transmi�ed gene�cally. To answer this ques�on experimentally, Meaney and colleagues had to think very carefully about the comparisons they wanted to make. It would have been

study the impact of gene�cs versus upbringing on personality traits.

insufficient to simply compare the offspring of good and bad mothers—this approach could not dis�nguish between gene�c and environmental pathways.

Instead, Meaney decided to use a technique called cross-fostering, or switching rat pups from one mother to another as soon as they were born. This resulted in four combina�ons of rats: (1) those born to ina�en�ve mothers but raised by a�en�ve ones, (2) those born to a�en�ve mothers but raised by ina�en�ve ones, (3) those born and raised by a�en�ve mothers, and (4) those born and raised by ina�en�ve mothers. Meaney then tested the rat pups several months later and observed the way they behaved with their own offspring. The setup of this experiment allowed Meaney to make clear comparisons between the influence of birth mothers and the rearing process. At the end of the study, the conclusion was crystal clear: Maternal behavior is all about the environment. Those rat pups that ul�mately grew up to be ina�en�ve mothers were those who had been raised by ina�en�ve mothers.

This final chapter is dedicated to experimental designs, in which the primary goal is to explain behavior. Experimental designs rank highest on the con�nuum of control (see Figure 5.1) because the experimenter can manipulate variables, minimize extraneous variables, and assign par�cipants to condi�ons. The chapter begins with an overview of the key features of experiments and then covers the importance of both internal and external validity of experiments. From there, the discussion moves to the process of designing and analyzing experiments and a summary of strategies for minimizing error in experiments. It concludes with guidelines for cri�quing a quan�ta�ve study.

Figure 5.1: Experimental designs on the continuum of control

A variable is any factor that has more than one value, such as height.

Flying Colours Ltd/Photodisc/Thinkstock

5.1 Experiment Terminology

Before we dive into the details, it is important to cover the terminology that we will use to describe different aspects of experimental designs. Much of this will be familiar from previous chapters, with a few new addi�ons. First, let’s review the basics.

Recall that a variable is any factor that has more than one value. For example, height is a variable because people can be short, tall, or anywhere in between. Depression is a variable because people can experience a wide range of symptoms, from mild to severe. The independent variable (IV) is the variable that is manipulated by the experimenter in order to test hypotheses about cause. The dependent variable (DV) is the variable that is measured by the experimenter in order to assess the effects of the independent variable. For example, in an experiment tes�ng the hypothesis that fear causes prejudice, fear would be the independent variable and prejudice would be the dependent variable. To keep these terms straight, it is helpful to think of the main goal of experimental designs. That is, we test hypotheses about cause by manipula�ng an independent variable and then looking for changes in a dependent variable. Thus, our independent variable causes changes in the dependent variable; for example, fear is hypothesized to cause changes in prejudice.

Any manipula�on of independent variables results in two or more versions of the variable. One common way to describe the versions of the independent variable is in terms of different groups, or condi�ons. The most basic experiments have two condi�ons: The experimental condi�on receives treatment designed to test the hypothesis, while the control condi�on does not receive this treatment. In our fear and prejudice example, the par�cipants who make up the experimental condi�on would be made to feel afraid, while the par�cipants who make up the control condi�on would not. This setup allows us to test whether introducing fear to one group of par�cipants leads them to express more prejudice than the other group of par�cipants, who are not made fearful.

Another common way to describe these versions is in terms of levels of the independent variable. Levels describe the specific set of circumstances created by manipula�ng a variable. For example, in the fear and prejudice experiment, the variable of fear would have two levels—afraid and not afraid. There are countless ways to introduce fear into the experiment. One op�on would be to adopt the technique used by the Stanford social psychologist Stanley Schachter (1959), who led par�cipants to believe they would be exposed to a series of painful electric shocks. In Schachter’s study, the painful shocks never happened, but they did induce a fearful state as people an�cipated them. So those at the “afraid” level of the independent variable might be told to expect these shocks, while those at the “not afraid” level of the independent variable would not be given this expecta�on.

At this stage, it may seem odd to have two sets of vocabulary terms—“levels” and “condi�ons”—for the same concept. However, there is a subtle difference in how these terms are used once we get into advanced experimental designs. As the designs become more complex, it is o�en necessary to expand independent variables to include several groups and mul�ple variables. Once this happens, we will need different terminology to dis�nguish between the versions of one variable and the combina�ons of mul�ple variables. We will return to this complexity later in the chapter, in Sec�on 5.4, Experimental Designs.

Medical studies o�en use invasive manipula�on to change internal, physiological processes.

iStockphoto/Thinkstock

5.2 Key Features of Experiments

The overview of research designs (Chapter 2, Sec�on 2.1 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.1#sec2.1) ) described the overall process of experiments in the following way: A researcher controls the environment as much as possible so that all par�cipants have the same experience. He or she then manipulates, or changes, one key variable, and then measures the outcomes in another key variable. In this sec�on, we will examine this process of control in more detail. Experiments can be dis�nguished from all other designs by three key features: manipula�ng variables, controlling the environment, and assigning people to groups comprising experimental and control condi�ons.

Manipulating Variables

The most crucial element of an experiment is that the researcher must manipulate, or change, some key variable. To study the effects of hunger, for example, a researcher could manipulate the amount of food given to the par�cipants. Or, to study the effects of temperature, the experimenter could raise and lower the temperature of the thermostat in the laboratory. Because these factors are under the researchers’ direct control, they can feel more confident that changing them contributes to changes in the dependent variables.

In Chapter 2 we discussed the main shortcoming of correla�onal research: These designs do not allow us to make causal statements. As you’ll recall from that chapter (as well as from Chapter 4), correla�onal research is designed to predict one variable from another.

One of the examples in Chapter 2 concerned the correla�on between income levels and happiness, with the goal of trying to predict happiness levels based on knowing people’s income level. If we measure these as they occur in the real world, we cannot say for sure which variable causes the other. However, we could se�le this ques�on rela�vely quickly with the right experiment. Let’s say we bring two groups into the laboratory, give one group $100 and a second group nothing. If the first group were happier at the end of the study, this would support the idea that money really does buy happiness. Of course, this is a rather simplis�c look at the connec�on between money and happiness, but because it manipulated levels of money, this study would bring us closer to making causal statements about the effects of money.

To manipulate variables, it is necessary to have at least two versions of the variable. That is, to study the effects of money, we need a comparison group that does not receive money. To study the effects of hunger, we would need both a hungry and a not-hungry group. Having two versions of the variable dis�nguishes experimental designs from the structured observa�ons discussed in Chapter 3 (Observa�onal Research), in which all par�cipants received the same set of condi�ons in the laboratory. Even the most basic experiment must have two sets of condi�ons, which are o�en an experimental group and a control group. But, as we will see later in this chapter, experiments can become much more complex. You might have one experimental group and two control groups, or five degrees of food depriva�on, ranging from 0 to 12 hours without food. Your decisions about the number and nature of these groups will depend on considera�on of both your hypotheses and previous literature.

When it comes to the manipula�on of variables, there are three op�ons available. First, environmental manipula�ons involve changing some aspect of the se�ng. Environmental manipula�ons are perhaps the most common in psychology studies, and they include everything from varying the temperature to varying the amount of money people receive. The key is to change the way different groups of people experience their �me in the laboratory—it is either hot or cold, and they either receive or don’t receive $100. Second, instruc�onal manipula�ons involve changing the way a task is described in order to change par�cipants’ mind-sets. For example, you could give all par�cipants the same math test but describe it as an intelligence test for one group and a problem-solving task for another. Because an intelligence test is thought to have implica�ons for life success, you might expect par�cipants in this group to be more nervous about their scores. Finally, an invasive manipula�on involves taking measures to change internal, physiological processes and is usually conducted in medical se�ngs. For example, studies of new drugs involve administering the drug to volunteers to determine whether it has an effect on some physical or psychological symptom. Or, for example, studies of cardiovascular health o�en involve having par�cipants run on a treadmill to measure how the heart func�ons under stress.

Finally, there is one qualifica�on to the rule that we must manipulate a variable. In many experiments, researchers divide up par�cipants based on an inherent difference (e.g., gender) or personality measures (e.g., self-esteem or neuro�cism) that capture stable individual characteris�cs among people. The idea behind these personality measures is that someone scoring high on a measure of neuro�cism (for example) would be expected to be more neuro�c across situa�ons than someone scoring lower on the measure. Using this technique allows us to compare how, for example, men and women, or people with high and low self-esteem, respond to manipula�ons. When existent differences are used in an experimental context, they are referred to as quasi-independent variables—“quasi,” or “nearly,” because they are being measured, not manipulated, by the experimenter, and thus do not meet the criteria for a regular independent variable. Because these variables are not manipulated, an experimenter cannot make causal statements about them. In order for a study to count as an experiment, these quasi-independent variables would have to be combined with a true independent variable. This could be as simple as comparing how men and women respond to a new an�depressant drug—gender would be quasi- independent while drug type would be a true independent variable.

Controlling the Environment

The second important element of experimental designs is that the researcher has a high degree of control over the environment. In addi�on to manipula�ng variables, a researcher conduc�ng an experiment ensures that the other aspects of the environment are the same for all par�cipants. For instance, if you were interested in the effects of temperature on people’s mood, you could manipulate temperature levels in the laboratory so that some people experienced warmer temperatures and other people cooler temperatures. But it would be equally important to make sure that other poten�al

The 25 par�cipants in our sample consist of a mix of happy and sad people. The goal of random assignment is to have these differences distributed equally across the experimental condi�ons. Thus, the two groups on the right each consist of six happy and six sad people, and our random assignment was successful.

influences on mood were the same for both groups. That is, you would want to make sure that the “warm” and “cool” groups were tested in the same room, around the same �me of day, and by the same experimenter.

The overall goal, then, is to control extraneous variables, or variables that add noise to your hypothesis test. In essence, the more you are able to control extraneous variables, the more confidence you can have in the results of your hypothesis test. As we will discuss in the sec�on “Experimental Validity,” the impact of extraneous variables can vary in a study. Let’s say we conduct the study on temperature and mood, and all of our par�cipants are in a windowless room with a flickering fluorescent light. This would likely have an influence on mood—making everyone a li�le bit grumpy—but cause few problems for our hypothesis test because it would affect everyone equally. Table 5.1 shows hypothe�cal data from two varia�ons of this study, using a 10-point scale to measure mood ra�ngs. In the top row, par�cipants were in a well-lit room; we can see that par�cipants in the cooler room reported being in a be�er mood (i.e., an 8 versus a 5). In the bo�om row, all par�cipants were in the windowless room with flickering lights. These numbers suggest that people were s�ll in a be�er mood in the cooler room (5) than a warm room (2), but the flickering fluorescent light had a constant dampening effect on everyone’s mood.

Table 5.1: Influence of an extraneous variable

Cool Room Warm Room

Varia�on 1: Well-Lit 8 5

Varia�on 2: Flickering Fluorescent 5 2

Assigning People to Conditions

The third key feature of experimental designs is that the researcher can assign people to receive different condi�ons, or versions, of the independent variable. This is an important piece of the experimental process: The experimenter not only controls the op�ons—warm versus cool room; $100 versus no money, etc.—he or she also gets to control which par�cipants get each op�on. Whereas a correla�onal design might assess the rela�onship between current mood and choosing the warm room, an experimental design will have some par�cipants assigned to the warm room and then measure the effects on their mood. In other words, an experimenter is able to make causal statements because that person causes things to happen.

The most common, and most preferable, way to assign people to condi�ons is through a process called random assignment. An experimenter who uses random assignment makes a separate decision for each par�cipant as to which group he or she will be assigned to before the par�cipant arrives. As the term implies, this decision is made randomly—by flipping a coin, using a random number table (for an example, click here (h�p://sta�rek.com/tables/random.aspx) ), drawing numbers out of an envelope, or some other random process. The overall goal is to try to balance out existent differences among people, as illustrated in Figure 5.2. So, for example, some people might generally be more comfortable in warm rooms, while others might be more comfortable in cold rooms. If each person who shows up for the study has an equal chance of being in either group, then the groups in the sample should reflect the same distribu�on of differences as the popula�on.

Figure 5.2: Random assignment

Another significant advantage of forming groups through random assignment is that it helps to avoid bias in the selec�on and assignment of subjects. For example, it would be a bad idea to assign people to groups based on a first impression of them because par�cipants might be placed in the cold room if they arrived at the laboratory dressed in warm clothing. Experimenters who make decisions about condi�on assignments ahead of �me can be more confident that the independent variable is responsible for changes in the dependent variable.

It is worth highligh�ng the difference here between random selec�on and random assignment (discussed in Chapter 4, Sec�on 4.3 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec4.3#sec4.3) , Sampling From the Popula�on). Random selec�on means that the sample of par�cipants is chosen at random from the popula�on, as with the probability sampling methods discussed in Chapter 4. However, most psychology experiments use a convenience sample of individuals who volunteer to complete the study. This means that the sample is o�en far from fully random. However, a researcher can s�ll make sure that the group assignments are random so that each condi�on contains an equal representa�on of the sample.

The 20 par�cipants in our sample represent a mix of very high, average, and very low intelligence test scores (measured 1–100). The goal of matched random assignment is to ensure that this varia�on is distributed equally across the two condi�ons. The experimenter would first rank par�cipants by intelligence test scores (top box) and then distribute these par�cipants alternately between the condi�ons. The result is that both groups (lower boxes) contain a good mix of high, average, and low scores.

In some cases—most notably, when samples are small—random assignment may not be sufficient to balance an important characteris�c that might affect the results of a par�cular study. Imagine conduc�ng a study that compared two strategies for teaching students complex math skills. In this example, it would be especially important to make sure that both groups contained a mix of individuals with, say, average and above-average intelligence. For this reason, it would be necessary to take extra steps to ensure that intelligence was equally distributed among the groups, which can be accomplished with a varia�on on random assignment called matched random assignment. This requires the experimenter to obtain scores on an important matching variable (in this case, intelligence), rank par�cipants based on the matching variable, and then randomly assign people to condi�ons. Figure 5.3 shows how this process would unfold in our math skills study. First, par�cipants are given an IQ test to measure exis�ng differences in intelligence. Second, the experimenter ranks par�cipants based on these scores, from highest to lowest. Third, the experimenter would move down this list in order and randomly assign each par�cipant to one of the condi�ons. This process s�ll contains an element of random assignment, but adding the extra step of rank ordering ensures a more balanced distribu�on of intelligence test scores across the condi�ons.

Figure 5.3

Research: Making an Impact

The Stanford Prison Experiment

The landmark 1971 Stanford Prison Experiment had an extremely widespread impact on mul�ple fields and real-life se�ngs such as prison reform, the ethics of human research, and terrorism and torture tac�cs (Clements, 1999; Haney & Zimbardo, 1998). The study, conducted by Phillip Zimbardo and his colleagues at Stanford University, placed volunteer par�cipants in a simulated prison environment and randomly assigned them to play the roles of “guards” and “prisoners.” These 24 par�cipants had been selected based on their personality traits that marked them as “good apples” and who had no previous evidence of an�social behavior (Haney, Banks, & Zimbardo, 1973). Nevertheless, the simulated prison quickly took on the characteris�cs of a real prison, with simulated situa�ons of dominance, dehumaniza�on, severe psychological distress, and the unexpected phenomena of social control, obedience, and effects of power (Haney, 2002; Banuazizi & Movahedi, 1975). The prisoners were subjected to humilia�on similar to what has been seen in prison scandals such as Abu Ghraib: nakedness, sexual humilia�on, verbal torment, chains, and bags over prisoners’ heads. Although planned

to run for 2 weeks, the experiment was stopped a�er only 6 days due to the severe psychological harm it was causing the prisoners and the unexpected behavior of the prison guards.

The study led to major reform in the ethical guidelines for psychological research and humane treatment, and in fact has never been replicated in a scien�fic se�ng. Following publica�on of this experiment, the Supreme Court saw an influx of cases regarding prisoner treatment and the structure of the prison system. In an interes�ng twist, one of the original researchers, Craig Haney, was inspired to pursue a career in prison reform based on what he learned from this study. Zimbardo himself went on to tes�fy on behalf of the soldiers accused of abuse at Abu Ghraib, highligh�ng the powerful influence of social roles (e.g., prison guard) on our behavior. An overwhelming majority of social and psychological research has found the puni�ve punishment system to not only be ineffec�ve but actually deleterious to prisoner behavior and recidivism. The sugges�ons from current research include calls to restructure prison “power” dynamics in an effort to increase prisoner safety and reduce guard brutality.

The demeanor of the person running a study may be a confounding variable.

Digital Vision/Thinkstock

5.3 Experimental Validity

Chapter 2 (Sec�on 2.2 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.2#sec2.2) ) discussed the concept of validity, or the degree to which measures capture the constructs that they were designed to capture. For example, a measure of happiness needs to actually capture differences in people’s levels of happiness. In this sec�on, we return to the subject of validity in an experimental context. Similar to our earlier discussion, validity refers here to whether the experimental results are demonstra�ng what we think they are demonstra�ng. We will cover two types of validity that are relevant to experimental designs. The first is internal validity, which assesses the degree to which results can be a�ributed to independent variables. The second is external validity, which assesses how well the results generalize to situa�ons beyond the specific condi�ons laid out in the experiment. Taken together, internal and external validity provide a way to assess the merits of an experiment. However, each of these has its own threats and remedies, as discussed in the following sec�ons.

Internal Validity

In order to have a high degree of internal validity, experimenters strive for maximum control over extraneous variables. That is, they try to design experiments so that the independent variable is the only cause of differences between groups. But, of course, no study is ever perfect, and there will always be some degree of error. In many cases, errors are the result of unavoidable causes, such as the health or mood of the par�cipants on the day of the experiment. In other cases, errors are caused by factors that are, in fact, under the experimenter’s control. In this sec�on, we will focus on several of these more manageable threats to internal validity and discuss strategies for reducing their influence.

Experimental Confounds

To avoid threats to the internal validity of an experiment, it is important to control and minimize the influence of extraneous variables that might add noise to a hypothesis test. In many cases, extraneous variables can be considered rela�vely minor nuisances, as when our mood experiment was accidentally run in a depressing room. But now, let’s say we ran our study on temperature and mood, and owing to a lack of careful planning, we accidentally placed all of the warm- room par�cipants in a sunny room, and the cool-room par�cipants in a windowless room. We might very well find that the warm-room par�cipants were in a much be�er mood. But would this be the result of warm temperatures or the result of exposure to sunshine? Unfortunately, we would be unable to tell the difference because of a confounding variable, or confound (in the case of correla�on studies, third variable). The confounding variable changes systema�cally with the independent variable. In this example, room ligh�ng would be confounded with room temperature because all of the warm-room par�cipants were also exposed to sunshine, and all of the cool-room par�cipants to ar�ficial ligh�ng. This combina�on of variables would leave us unable to determine which variable actually had the effect on mood. The result would be that our groups differed in more than one way, which would seriously hinder our ability to say that the independent variable (the room) caused the dependent variable (mood) to change.

It may sound like an oversimplifica�on, but the way to avoid confounds is to be very careful in designing experiments. By ensuring that groups are alike in every way but the experimental condi�on, one can generally prevent confounds. This is somewhat easier said than done because confounds can come from unexpected places. For example, most studies involve the use of mul�ple research assistants who manage data collec�on and interact with par�cipants. Some of these assistants might be more or less friendly than others, so it is important to make sure each of them interacts with par�cipants in all condi�ons. If your friendliest assistant works with everyone in the warm-room group, for example, it would result in a confounding variable (friendly versus unfriendly assistants) between room and research assistant. Consequently, you would be unable to separate the influence of your independent variable (the room) from that of the confound (your research assistant).

Selec�on Bias

Internal validity can also be threatened when groups are different before the manipula�on, which is known as selec�on bias. Selec�on bias causes problems because these inherent differences might be the driving factor behind the results. Imagine you are tes�ng a new program that will help people stop smoking. You might decide to ask for volunteers who are ready to quit smoking and put them through a 6-week program. But by asking for volunteers—a remarkably common error—you gather a group of people who are already somewhat mo�vated to stop smoking. Thus, it is difficult to separate the effects of your new program from the effects of this a priori mo�va�on.

One easy way to avoid this problem is through either random or matched-random assignment. In the stop-smoking example, you could s�ll ask for volunteers, but then randomly assign these volunteers to one of the two programs. Because both groups would consist of people mo�vated to quit smoking, this would help to cancel out the effects of mo�va�on. Another way to minimize selec�on bias is to use the same people in both condi�ons so that they serve as their own control. In the stop-smoking example, you could assign volunteers first to one program and then to the other. However, you might run into a problem with this approach— par�cipants who successfully quit smoking in the first program would not benefit from the second program. This technique is known as a within-subject design, and we will discuss its advantages and disadvantages in the subsec�on “Within-Subject Designs” in Sec�on 5.4, Experimental Designs.

Differen�al A�ri�on

Despite your best efforts at random assignment, you could s�ll have a biased sample at the end of a study as a result of differen�al a�ri�on. The problem of differen�al a�ri�on (some�mes called the mortality threat) occurs when subjects drop out of experimental groups for different reasons. Let’s say you’re conduc�ng a study of the effects of exercise on depression levels. You manage to randomly assign people to either 1 week of regular exercise or 1 week of regular therapy. At first glance, it appears that the exercise group shows a drama�c drop in depression symptoms. But then you no�ce that approximately one third of the people in this group dropped out before comple�ng the study. Chances are you are le� with those who are most mo�vated to exercise, to overcome their depression, or

both. Thus, you are unable to isolate the effects of your independent variable on depression symptoms. While you cannot prevent people from dropping out of your study, you can look carefully at those who do. In many cases, you can spot a pa�ern and use it to guide future research. For example, it may be possible to discover a profile of people who dropped out of the exercise study and use this knowledge to increase reten�on for the next a�empt.

Outside Events

As much as we strive to control the laboratory environment, par�cipants are o�en influenced by events in the outside world. These events—some�mes called history effects—are o�en large-scale events such as poli�cal upheavals and natural disasters. The threat to research is that it becomes difficult to tell whether par�cipants’ responses are the result of the independent variable or the historical event(s). One great example of this comes from a paper published by social psychologist Ryan Brown, now a professor at the University of Oklahoma, on the effects of receiving different types of affirma�ve ac�on as people were selected for a leadership posi�on. The goal was to determine the best way to frame affirma�ve ac�on in order to avoid undermining the recipient’s confidence (Brown, Charnsangavej, Keough, Newman, & Ren�row, 2000). For about a week during the data collec�on process, students at the University of Texas, where the study was being conducted, were protes�ng on the main lawn about a controversial lawsuit regarding affirma�ve ac�on policies. The result was that par�cipants arriving for this laboratory study had to pass through a swarm of people holding signs that either denounced or supported affirma�ve ac�on. These types of outside events are difficult, if not impossible, to control. But, because these researchers were aware of the protests, they made a decision to exclude from the study data gathered from par�cipants during the week of the protests, thus minimizing the effects of these outside events.

Expectancy Effects

One final set of threats to internal validity results from the influence of expectancies on people’s behavior. This can cause trouble for experimental designs in three related ways. First, experimenter expectancies can cause researchers to see what they expect to see, leading to subtle bias in favor of their hypotheses. In a clever demonstra�on of this phenomenon, the psychologist Robert Rosenthal asked his graduate students at Harvard University to train groups of rats to run a maze (Rosenthal & Fode, 1963). He also told them that based on a pretest, the rats had been classified as either bright or dull. As you might have guessed, these labels were pure fic�on, but they s�ll influenced the way that the students treated the rats. Rats labeled “bright” were given more encouragement and learned the maze much more quickly than rats labeled “dull.” Rosenthal later extended this line of work to teachers’ expecta�ons of their students (Rosenthal & Jacobson, 1968) and found support for the same conclusion: People o�en bring about the results they expect by behaving in a par�cular way.

One common way to avoid experimenter expectancies is to have par�cipants interact with a researcher who is “blind” (i.e., unaware) to the condi�on that each par�cipant is in. The researcher may be fully aware of the research hypothesis, but his or her behavior is unlikely to affect the results. In the Rosenthal and Fode (1963) study, the graduate students’ behavior influenced the rats’ learning speed only because they were aware of the labels “bright” and “dull.” If these had not been assigned, the rats would have been treated fairly equally across the condi�ons.

Second, par�cipants in a research study o�en behave differently based on their own expectancies about the goals of the study. These expectancies o�en develop in response to demand characteris�cs, or cues in the study that lead par�cipants to guess the hypothesis. In a well-known study conducted at the University of Wisconsin, psychologists Leonard Berkowitz and Anthony LePage found that par�cipants would behave more aggressively—by delivering electric shocks to another par�cipant—if a gun was in the room than if there were no gun present (Berkowitz & LePage, 1967). This finding has some clear implica�ons for gun control policies, sugges�ng that the mere presence of guns increases the likelihood of violence. However, a common cri�que of this study is that par�cipants may have quickly clued in to its purpose and figured out how they were “supposed” to behave. That is, the gun served as a demand characteris�c, possibly making par�cipants act more aggressively because they thought it was expected of them.

To minimize demand characteris�cs, researchers use a variety of techniques, all of which a�empt to hide the true purpose of the study from par�cipants. One common strategy is to use a cover story, or a misleading statement about what is being studied. In Chapter 1 (Sec�on 1.4 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.4#sec1.4) , Hypotheses and Theories, and Sec�on 1.7 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.7#sec1.7) , Ethics in Research), we discussed Milgram’s famous obedience studies, which discovered that people were willing to obey orders to deliver dangerous levels of electric shocks to other people. In order to disguise the purpose of the study, Milgram described it to people as a study of punishment and learning. And the affirma�ve ac�on study by Ryan Brown and colleagues (Brown et al., 2000) was presented as a study of leadership styles. The goal in using these cover stories is to give par�cipants a compelling explana�on for what they experience during the study and to direct their a�en�on away from the research hypothesis.

Another strategy is to use the unrelated-experiments technique, which leads par�cipants to believe that they are comple�ng two different experiments during one laboratory session. The experimenter can use this bit of decep�on to present the independent variable during the first experiment and then measure the dependent variable during the second experiment. For example, a study by Harvard psychologist Margaret Shih and colleagues (Shih, Pi�nsky, & Ambady, 1999) recruited Asian American females and asked them to complete two supposedly unrelated studies. In the first, they were asked to read and form impressions of one of two magazine ar�cles; these ar�cles were designed to make them focus on either their Asian American iden�ty or their female iden�ty. In the second experiment, they were asked to complete a math test as quickly as possible. The goal of this study was to examine the effects on math performance of priming different aspects of iden�ty. Based on previous research, these authors predicted that priming an Asian American iden�ty would remind par�cipants of posi�ve stereotypes regarding Asians and math performance, whereas priming a female iden�ty would remind par�cipants of nega�ve stereotypes regarding women and math performance. As expected, priming an Asian American iden�ty led this group of par�cipants to do be�er on a math test than did priming a female iden�ty. The unrelated-experiments technique was especially useful for this study because it kept par�cipants from connec�ng the independent variable (magazine ar�cle prime) with the dependent variable (math test).

A final way in which expectancies shape behavior is the placebo effect, meaning that change can result from the mere expecta�on that change will occur. Imagine you wanted to test the hypothesis that alcohol causes people to become aggressive. One rela�vely easy way to do this would be to give alcohol to a group of volunteers (aged 21 and older) and then measure how aggressive they became in response to being provoked. The problem with this approach is that people also

expect alcohol to change their behavior, so you might see changes in aggression simply because of these expecta�ons. Fortunately, there is an easy solu�on: Add a placebo control group to your study that mimics the experimental condi�on in every way but one. In this case, you might tell all par�cipants that they will be drinking a mix of vodka and orange juice but only add vodka to half of the par�cipants’ drinks. The orange-juice-only group serves as the placebo control, so any differences between this group and the alcohol group can be a�ributed to the alcohol itself.

External Validity

In order to a�ain a high degree of external validity in their experiments, researchers strive for maximum realism in the laboratory environment. External validity means that the results extend beyond the par�cular set of circumstances created in a single study. Recall that science is a cumula�ve discipline and that knowledge grows one study at a �me. Thus, each study is more meaningful to the extent that it sheds light on a real phenomenon and to the extent that the results generalize to other studies. Let’s examine each of these criteria separately.

Mundane Realism

The first component of external validity is the extent to which an experiment captures the real-world phenomenon under study. One popular ques�on in the area of aggression research is whether rejec�on by a peer group leads to aggression. That is, when people are rejected from a group, do they lash out and behave aggressively toward the members of that group? Researchers must find realis�c ways to manipulate rejec�on and measure aggression without infringing on par�cipants’ welfare. Given the need to strike this balance, how real can things get in the laboratory? How do we study real-world phenomena without sacrificing internal validity?

The answer is to strive for mundane realism, meaning that the research replicates the psychological condi�ons of the real-world phenomenon (some�mes referred to as ecological validity). In other words, we need not re-create the phenomenon down to the last detail; instead, we aim to make the laboratory se�ng feel like the real world. Researchers studying aggressive behavior and rejec�on have developed some rather clever ways of doing this, including allowing par�cipants to administer loud noise blasts or serve large quan��es of hot sauce to those who rejected them. Psychologically, these acts feel like aggressive revenge because par�cipants are able to lash out against those who rejected them, with the intent of causing harm, even though the behaviors themselves may differ from the ways people exact revenge in the real world.

In a 1996 study, Tara MacDonald and her colleagues at Queen’s University in Ontario, Canada, examined the rela�onship between alcohol and condom use (MacDonald, Zanna, & Fong, 1996). The authors pointed out a puzzling set of real-world data: Most people reported that they would use condoms when engaging in casual sex, but the rates of unprotected sex (i.e., having sexual intercourse without a condom) were also remarkably high. In this study, the authors found that alcohol was a key factor in causing “common sense to go out the window” (p. 763), resul�ng in a decreased likelihood of condom use. But how on earth might they study this phenomenon in the laboratory? In the authors’ words, “even the most ambi�ous of scien�sts would have to conclude that it is impossible to observe the effects of intoxica�on on actual condom use in a controlled laboratory se�ng” (p. 765).

To solve this dilemma, MacDonald and colleagues developed a clever technique for studying people’s inten�ons to use condoms. Par�cipants were randomly assigned to either an alcohol or placebo condi�on, and then they viewed a video depic�ng a young couple that was faced with the dilemma of whether to have unprotected sex. At the key decision point in the video, the tape was stopped and par�cipants were asked what they would do in the situa�on. As predicted, par�cipants who were randomly assigned to consume alcohol said they would be more willing to proceed with unprotected sex. While this laboratory study does not capture the full experience of making decisions about casual sex, it does a nice job of capturing the psychological condi�ons involved.

Generalizing Results

The second component of external validity is the extent to which research findings generalize to other studies. Generalizability refers to the extent to which the results extend to other studies, using a wide variety of popula�ons and a wide variety of opera�onal defini�ons (some�mes referred to as popula�on validity). If we conclude that rejec�on causes people to become more aggressive, for example, this conclusion should ideally carry over to other studies of the same phenomenon, using different ways of manipula�ng rejec�on and different ways of measuring aggression. If we want to conclude that alcohol reduces inten�ons to use condoms, we would need to test this rela�onship in a variety of se�ngs—from laboratories to nightclubs—using different measures of inten�ons.

Thus, each study that we conduct is limited in its conclusions. In order for your par�cular idea to take hold in the scien�fic literature, it must be replicated, or repeated in different contexts. These replica�ons can take one of four forms. First, exact replica�on involves trying to re-create the original experiment as closely as possible in order to verify the findings. This type of replica�on is o�en the first step following a surprising result, and it helps researchers to gain more confidence in the pa�erns. The second and much more common method, conceptual replica�on, involves tes�ng the rela�onship between conceptual variables using new opera�onal defini�ons. Conceptual replica�ons would include tes�ng our aggression hypotheses using new measures or examining the link between alcohol and condom use in different se�ngs. For example, rejec�on might be opera�onalized in one study by having par�cipants be chosen last for a group project. A conceptual replica�on might take a different approach: opera�onalizing rejec�on by having par�cipants be ignored during a group conversa�on or voted out of the group. Likewise, a conceptual replica�on might change the opera�onaliza�on of aggression by having one study measure the delivery of loud blasts of noise and another measure the amount of hot sauce that people give to their rejecters. Each varia�on studies the same concept (aggression or rejec�on) but uses slightly different opera�onaliza�ons. If all of these varia�ons yielded similar results, this would provide further evidence of the underlying ideas—in this case, that rejec�on causes people to be more aggressive.

The third method, par�cipant replica�on, involves repea�ng the study with a new popula�on of par�cipants. These types of replica�on are usually driven by a compelling theory as to why the two popula�ons differ. For example, you might reasonably hypothesize that the decision to use condoms is guided by a different set of considera�ons among college students than among older, single adults. Finally, construc�ve replica�on re- creates the original experiment but adds elements to the design. These addi�ons are typically designed to either rule out alterna�ve explana�ons or extend knowledge about the variables under study. In our

rejec�on and aggression example, you might test whether males and females respond the same way or perhaps compare the impact of being rejected by a group versus an individual.

Internal Versus External Validity

We have focused on two ways to assess validity in the context of experimental designs. Internal validity assesses the degree to which results can be a�ributed to independent variables; external validity assesses how well results generalize beyond the specific condi�ons of the experiment. In an ideal world, studies would have a high degree of both of these. That is, we would feel completely confident that our independent variable was the only cause of differences in our dependent variable, and our experimental paradigm would perfectly capture the real-world phenomenon under study.

In reality, though, there is o�en a trade-off between internal and external validity. In MacDonald and colleagues’ study on condom use (MacDonald et al., 1996), the researchers sacrificed some realism in order to conduct a �ghtly controlled study of par�cipants’ inten�ons. In Berkowitz and LePage’s (1967) study on the effect of weapons, the researchers risked the presence of a demand characteris�c in order to study reac�ons to actual weapons. These types of trade-offs are always made based on the goals of the experiment. To give you a be�er sense of how researchers make these compromises, let’s evaluate three fic�onal examples.

Scenario 1: Time Pressure and Stereotyping

Dr. Bob is interested in whether people are more likely to rely on stereotypes when they are in a hurry. In a well-controlled laboratory experiment, par�cipants are asked to categorize ambiguous shapes as either squares or circles, and half of these par�cipants are given a short �me limit to accomplish the task. The independent variable is the presence or absence of �me pressure, and the dependent variable is the extent to which people use stereotypes in their classifica�on of ambiguous shapes. Dr. Bob hypothesizes that people will be more likely to use stereotypes when they are in a hurry because they will have fewer cogni�ve resources to carefully consider all aspects of the situa�on. Dr. Bob takes great care to have all par�cipants meet in the same room. He uses the same research assistant every �me, and the study is always conducted in the morning. Consistent with his hypothesis, Dr. Bob finds that people seem to use shape stereotypes more under �me pressure.

The internal validity of this study appears high—Dr. Bob has controlled for other influences on par�cipants’ a�en�on span by collec�ng all of his data in the morning. He has also minimized error variance by using the same room and the same research assistant. In addi�on, Dr. Bob has created a �ghtly controlled study of stereotyping through the use of circles and squares. Had he used photographs of people (rather than shapes), the a�rac�veness of these people might have influenced par�cipants’ judgments. But here’s the trade-off: By studying the social phenomenon of stereotyping using geometric shapes, Bob has removed the social element of the study, thereby posing a threat to mundane realism. The psychological meaning of stereotyping shapes is rather different from the meaning of stereotyping people, which makes this study rela�vely low in external validity.

Scenario 2: Hunger and Mood

Dr. Jen is interested in the effects of hunger on mood; not surprisingly, she predicts that people will be happier when they are well fed. She tests this hypothesis with a lengthy laboratory experiment, requiring par�cipants to be confined to a laboratory room for 12 hours with very few distrac�ons. Par�cipants have access to a small pile of magazines to help pass the �me. Half of the par�cipants are allowed to eat during this �me, and the other half is deprived of food for the full 12 hours. Dr. Jen—a naturally friendly person—collects data from the food-depriva�on groups on a Saturday a�ernoon, while her grumpy research assistant, Mike, collects data from the well-fed group on a Monday morning. Her independent variable is food depriva�on, with par�cipants either not deprived of food or deprived for 12 hours. Her dependent variable consists of par�cipants’ self-reported mood ra�ngs. When Dr. Jen analyzes the data, she is shocked to discover that par�cipants in the food-depriva�on group were much happier than those in the well-fed group.

Compared with our first scenario, this study seems high on external validity. To test her predic�ons about food depriva�on, Dr. Jen actually deprives her par�cipants of food. One possible problem with external validity is that par�cipants are confined to a laboratory se�ng during the depriva�on period with only a small pile of magazines to read. That is, par�cipants may be more affected by hunger when they do not have other things to distract them. In the real world, people are o�en hungry but distracted by paying a�en�on to work, family, or leisure ac�vi�es. But Dr. Jen has sacrificed some external validity for the sake of controlling how par�cipants spend their �me during the depriva�on period. The larger problem with her study has to do with internal validity. Dr. Jen has accidentally confounded two addi�onal variables with her independent variable: Par�cipants in the depriva�on group have a different experimenter and data are collected at a different �me of day. Thus, Dr. Jen’s surprising results most likely reflect the fact that everyone is in a be�er mood on Saturday than on Monday and that Dr. Jen is more pleasant to spend 12 hours with than Mike is.

Scenario 3: Math Tutoring and Gradua�on Rates

Dr. Liz is interested in whether specialized math tutoring can help increase gradua�on rates among female math majors. To test her hypothesis, she solicits female volunteers for a math skills workshop by placing flyers around campus, as well as by sending email announcements to all math majors. The independent variable is whether par�cipants are in the math skills workshop, and the dependent variable is whether par�cipants graduate with a math degree. Those who volunteer for the workshop are given weekly skills tutoring, along with informal discussion groups designed to provide encouragement and increase mo�va�on. At the end of the study, Dr. Liz is pleased to see that par�cipants in the workshops are twice as likely as nonpar�cipants to s�ck with the major and graduate.

The obvious strength of this study is its external validity. Dr. Liz has provided math tutoring to math majors, and she has observed a difference in gradua�on rates. Thus, this study is very much embedded in the real world. But, as you might expect, this external validity comes at a cost to internal validity. The biggest flaw is that Dr. Liz has recruited volunteers for her workshops, resul�ng in selec�on bias for her sample. People who volunteer for extra math tutoring are likely to be more invested in comple�ng their degree and might also have more �me available to dedicate to their educa�on. Dr. Liz would also need to be mindful of how many people drop out of her study. If significant numbers of par�cipants withdrew, she could have a problem with differen�al a�ri�on, so that the most mo�vated people stayed with the workshops. One rela�vely easy fix for this study would have been to ask for volunteers more generally, and then randomly assign these

volunteers to take part in either the math tutoring workshops or a different type of workshop. While the sample might s�ll have been less than random, Dr. Liz would at least have had the power to assign par�cipants to different groups.

A Note on Qualitative Research Validity and Reliability

As discussed in Chapter 3, the validity of a quan�ta�ve study hinges on whether the experimental results demonstrate what we think they are demonstra�ng; reliability refers to whether the experimental results will yield the same or similar results in other experiments. The concepts of validity and reliability in qualita�ve research do not carry the same meanings as they do in quan�ta�ve research, nor is the goal to generalize the results to individuals, sites, or places outside of those under study. As Creswell (2009) notes, qualita�ve validity “means the researcher checks for the accuracy of the findings by employing certain procedures, while qualita�ve reliability indicates that the researcher’s approach is consistent across different researchers and different projects” (p. 190). Because qualita�ve research does not include experimental results, numerical output, and data analyses, many qualita�ve researchers argue that they must evaluate the quality of their results differently and focus more on the trustworthiness or overall worth of their data. Thus, they ask, are the findings worthy of a�en�on? And, how do you evaluate them?

To evaluate the trustworthiness or the validity and reliability of qualita�ve studies, Guba and Lincoln (1994) proposed the following alterna�ve criteria outlined in Table 5.2 that are u�lized by many qualita�ve researchers.

Table 5.2: Criteria for evalua�ng quan�ta�ve research and qualita�ve research

Criteria for evalua�ng quan�ta�ve research Alterna�ve criteria for evalua�ng qualita�ve research

Internal validity

Assesses whether the independent variable is the only possible explana�on for the dependent variable

Credibility

Used to assess “the accuracy of the iden�fica�on and descrip�on of the subject of the study” (Smith & Davis, 2010, p. 51) Examines whether the research is credible or believable from the perspec�ve of the par�cipant

External validity

Evaluates whether the results can be applied to different popula�ons and se�ngs

Transferability

Focuses on the transferability of findings to other se�ngs and groups Transferability is enhanced by providing thorough and clear reports so that the results can be transferred to a different context

Objec�vity

Empirical findings that can be confirmed by others and corrected through subsequent research

Confirmability

The extent to which the qualita�ve report “is accurate, unbiased, and can be confirmed by others” (Smith & Davis, 2010, p. 51) Confirmability is enhanced by having other researchers review dra�s and point out inconsistencies, contradic�ons, and biases Confirmability is also enhanced by providing thorough reports regarding the procedures that were used to check and recheck the data

Reliability

Assesses whether the methods will yield similar results in other studies

Dependability

“The extent to which the researcher believes the same results would be produced if the study were replicated” (Smith & Davis, 2010, p. 51) Emphasizes the need for the researcher to account for and describe the changes that occur in the se�ng and how these changes affected the way the researcher approached the study

There have been lengthy debates about the value of including alterna�ve sets of criteria for judging qualita�ve research. Although the criteria for evalua�ng validity and reliability in quan�ta�ve and qualita�ve research may seem similar and appear to be mere relabeling of concepts, it should be noted that the procedures u�lized to assess them are not. For example, to ensure validity in qualita�ve studies, researchers employ one or more of the following strategies:

Triangula�on: Using mul�ple sources of data collec�on to build jus�fica�on for themes. If mul�ple sources of data confirm themes, this process can be considered to add to the validity of the study. Member checking: Asking par�cipants to review the final report and confirm whether the descrip�ons or themes are accurate. Providing rich, thick descrip�ons: Describing in detail the se�ng, par�cipants, and procedures. This process can add to the validity of the findings. Clarifying researcher bias: Self-reflec�ons on any bias the researcher brings to the study helps create an open and honest report. Presen�ng nega�ve or discrepant informa�on: The researcher discussing any informa�on that runs counter to the themes. Peer debriefing: U�lizing peer debriefers that review and ask ques�ons about the study. Spending prolonged �me in the field: Spending long periods of �me in the field allows the researcher to develop in-depth understandings of the phenomenon of interest. The more experience a researcher has with par�cipants in their natural se�ng, the more valid the findings will be.

External auditors: Employing an independent reviewer who is not familiar with the research or project who can provide an objec�ve assessment of the study.

To determine whether qualita�ve research approaches are consistent or reliable, researchers must first ensure that all steps of the procedures are documented thoroughly. Gibbs (2007) suggests the following strategies:

Checking transcripts: The researcher checks wri�en transcripts against tape-recorded informa�on to ensure that mistakes were not made during transcrip�on. Ensuring codes are stable: Verifying that a shi� in the meaning of codes did not occur during the process of coding. This is accomplished by constantly comparing data with codes and providing detailed descrip�ons of the codes. Coordina�ng communica�on: The researcher communicates the analyses to coders through regular documented mee�ngs. Cross-checking codes: The researcher cross-checks codes developed by other researchers and compares the results with his or her own.

5.4 Experimental Designs

There are three types of experimental designs that can be used in research studies: pre-experimental designs, quasi-experimental designs, and true experimental designs. Each design differs in the degree to which it controls for confounding or hidden variables; in turn, the degree of control affects the internal validity of the study. Pre-experimental designs have li�le to almost no control; instead, they involve studying a single group or unbalanced groups that are not randomly assigned and then introducing an interven�on or treatment during the study. Quasi-experimental designs offer some control and may or may not include a control group, but the par�cipants are not randomly assigned to groups. True experimental designs give the researcher maximum control and involve randomly assigning par�cipants and manipula�ng the independent variable. As we will see later in this discussion, true experiments include randomized group selec�on and assignments, a control group, and a large degree of control over confounding variables.

The best types of designs are those that offer control over confounding variables and include random assignment of the par�cipants. Although true experiments provide a much stronger and more valid design, some�mes they cannot be used for a study for par�cular reasons. The following sec�ons will discuss examples of the various types of designs. This is not an exhaus�ve list, and researchers can modify designs or combine them in various ways.

Pre-Experimental Designs

Pre-experimental designs follow basic experimental procedures but cannot show cause-and-effect rela�onships. Some researchers argue that pre-experimental designs confer so li�le control that they have minimal scien�fic value. As a result, these designs should be used only when making tenta�ve hypotheses and should be followed up with more controlled research (Leedy & Ormrod, 2010). As we will see in the following three sec�ons, pre-experimental designs generally enroll a single group, although some designs include experimental and control groups that are not randomly selected.

The three pre-experimental designs we describe next are very limited in terms of the conclusions that can be drawn from them. In comparison, quasi-experimental designs (described later in this sec�on) confer more control over confounding variables.

One-Shot Case Study

One-shot case studies are the most basic type of experimental designs. Because they do not include a control or comparison group, it is impossible to know whether the outcome results would have been be�er if the interven�on had not been provided. This type of pre-experimental design involves using one group (Group A), introducing an interven�on (X), and then administering a pos�est observa�on (O) to determine the effects of the interven�on, as shown below:

Group A: X_____________O

Addi�onally, since this type of design does not use a pretest, it is impossible to determine whether changes within the group have actually taken place. As a result, one-shot case studies have low internal validity.

In addi�on, because one-shot case studies confer li�le to no control over confounding variables, variables such as matura�on or other environmental condi�ons could have influenced the par�cipant’s performance. For instance, let’s say a researcher wants to measure whether praising first-grade students makes a difference in their reading scores. The researcher selects a group of students and increasingly praises those students. At the end of the study, the researcher finds that reading scores have improved. However, the researcher cannot be sure that the scores improved because of the praise, as it is possible that the students selected were already good in reading or that their parents were helping them to learn reading strategies at home. While this type of design is easy to carry out, its results are nearly meaningless.

One-Group Pretest–Pos�est Design

The one-group pretest–pos�est design involves one group but includes both a pretest (O1) before the interven�on (X) is given and a pos�est (O2) a�er the interven�on is provided, as shown below:

Group A: O1_____________X_____________O2

This design gives the researcher a li�le more control over the study than a one-shot case study, as it includes a pretest to measure performance before the interven�on is provided. Thus, a�er the pos�est is administered, one can know that some sort of change has taken place. However, even though it includes a pretest and a pos�est, the researcher s�ll cannot be sure whether some sort of confounding variable contributed to the results.

Suppose, for example, that a researcher wanted to know whether having a group of students take an 8-week course on GRE prepara�on would improve their scores. The researcher would give the students a form of the GRE test, have all of them par�cipate in the 8-week prepara�on course, and then give the students an alternate form of the GRE. If the scores improved a�er the 8-week course, the researcher might conclude that the course produced an increase in GRE scores. However, this conclusion may or may not be accurate, and here is why. Let’s say some of the students studied GRE material at home or par�cipated in another GRE study group at the same �me. How could we say that the results were the result of only the 8-week GRE prepara�on course? The answer is, we cannot.

Sta�c Group Comparison or Pos�est-Only With Nonequivalent Groups

Sta�c group comparison (or pos�est only, with nonequivalent groups’ designs) involves both an experimental group and a control group, although par�cipants are not randomly assigned to either group. This type of design is similar to the one-shot case study design, in that an interven�on (X) is provided to one group (Group A) and then a pos�est is used to measure the effects of the interven�on (O). However, this design includes a second control group (Group B) that does not receive the interven�on. The idea here is to include the second group to see whether receiving the interven�on really affects the par�cipants’ performance (O). Using our

reading score example from the one-shot case study sec�on, the researcher would include a group of students (Group A) who are praised and a group of students (Group B) who are not praised. Since the par�cipants were not randomly assigned to the groups, there would be no way to determine whether the groups were equivalent or whether the results could be a�ributed to praising or some other factor, such as mo�va�on or IQ. Thus, this design s�ll offers limited control over confounding variables and shares similar problems with one-shot case studies and one-group pretest–pos�est designs.

The pos�est-only with nonequivalent groups’ design takes the following form:

Group A: X_____________O

Group B: ______________O

Quasi-Experimental Designs

Quasi-experimental designs fare be�er than pre-experimental designs in terms of both level of control and internal validity. They employ both experimental and control groups with which to make comparisons. However, similar to pre-experimental designs, quasi-experimental designs do not include randomiza�on, either in the selec�on of group par�cipants or in the presenta�on of different interven�ons within the experimental group (Leedy & Ormrod, 2010). Although quasi- experimental designs control more for confounding variables than do pre-experimental designs, they do not control for them completely. These variables must be considered and acknowledged when analyzing and interpre�ng the data, as alterna�ve explana�ons for the results are possible. The following sec�ons discuss the four most common types of quasi-experiments.

Nonequivalent Pretest–Pos�est Control-Group Design

The nonequivalent pretest–pos�est control-group design is probably the most popular quasi-experimental design. In this design, both the experimental group (Group A) and the control group (Group B) receive a pretest observa�on (O) and a pos�est observa�on (O), although only the experimental group receives the interven�on (X). Neither the experimental group nor the control group is randomly assigned. This design takes the following form:

Group A: O_____________X_____________O

Group B: O___________________________O

The advantage of this design is that its pretest and pos�est allow for comparisons. For example, researchers are able to see whether receiving or not receiving the interven�on had an effect on the par�cipants. The disadvantage of this design is that it does not include random assignment to groups. Random assignment allows us to confirm that the two groups are similar in terms of the dependent variable. However, in this design, we cannot be sure that the experimental group and control group are equivalent with respect to age, gender, and other demographic variables. While this design can rule out some alterna�ve explana�ons for the study results, it cannot control for all confounding variables. As a consequence, the results may not be valid.

Single-Group Interrupted Time Series Design

Time series designs consist of making a series of observa�ons over �me, introducing an interven�on at some point during the study, and then making addi�onal observa�ons (Leedy & Armrod, 2010). Single-group �me series designs contain only one experimental group, which is observed over �me to evaluate predictable pa�erns of events. For example, suppose you wanted to examine the effects that a 3-week Republican presiden�al campaign (X) would have on the pro-choice beliefs (favoring legalized abor�on) of a group of independent voters in a par�cular area (Group A). The campaign might include commercial ads, phone calls, or presenta�ons and speeches. Assuming that you had a reliable measure of pro-choice beliefs in these voters prior to the campaign, you could start observa�ons (O) a few months before the campaign and then con�nue observa�ons for a few months a�er the campaign had ended. The design would take the following form:

Group A: O_______O_______O_______O_______O_______X_______O_______O_______O_______O_______O

So did the campaign affect Group A’s pro-choice beliefs? Researchers could determine whether an increase in beliefs actually occurred. However, they could not determine whether the change was solely due to the campaign without including an equivalent control group, a pretest, and a pos�est. Thus, the major disadvantage of this design is that it cannot allow researchers to conclude that the results were caused by the interven�on and not by some other real-world event or confounding variable.

Control-Group Interrupted Time Series Design

Another varia�on of the �me series design is the control-group interrupted �me series design. This design is similar to the single-group �me series design; however, it includes a control group that does not receive the interven�on. Thus, the design takes the following form:

Group A: O_______O_______O_______O_______O_______X_______O_______O_______O_______O_______O

Group B: O_______O_______O_______O_______O________________O_______O_______O_______O_______O

Using the campaign example discussed in the previous design, in this design, the control group would not be exposed to the campaign. Instead, they would simply receive observa�ons throughout the study at the same �me as the experimental group. Providing a control group ensures greater internal validity because it allows researchers to examine whether the campaign had a direct effect on par�cipants’ beliefs. Assuming that the groups were fairly equivalent demographically and that there were no other confounding variables that could have influenced their beliefs, we should see a change only in Group A.

Reversal Time Series Design

Reversal �me series designs use a variable interven�on approach in order to minimize the possibility that outside influences might affect the results. This type of design includes one experimental group (Group A), presence or absence of the interven�on over �me (X and No X, respec�vely) and observa�ons of the dependent variable at regular intervals throughout the study (O). Thus, each par�cipant is either exposed or not exposed to the interven�on at various points during the study. A�er each exposure or nonexposure, observa�ons are made. The design takes the following form:

Group A: X________O________No X________O________X________O________No X________O

For example, let’s say you were interested in studying whether including audiovisual presenta�ons in an online courseroom would help students learn sta�s�cs. On some weeks, you might want to include video presenta�ons in addi�on to PowerPoint slide shows, while on other weeks, you might omit the audiovisual material altogether and include just the PowerPoint slides. This approach would let you examine how effec�vely students learn under both condi�ons and how effec�ve adding audiovisual materials can be for student learning. If no confounding variables influenced student performance, and audiovisual materials did, in fact, promote student learning, you should see posi�ve changes in student performance on the weeks that included audiovisual materials. If confounding variables were present, you might see very different results.

True Experimental Designs

In contrast to pre-experimental and quasi-experimental designs, true experimental designs offer a greater degree of control, randomly select par�cipants and assign them to groups, and randomly assign treatments or interven�ons to these groups. These characteris�cs all increase the internal validity of a study, which, as we learned earlier in this chapter, reflects the degree to which results can be a�ributed to the independent variables. Thus, a higher degree of internal validity ensures that the results are related to the independent variable rather than to other, extraneous variables. And, unlike pre-experimental and quasi-experimental designs, true experiments can determine a clear cause-and-effect rela�onship between independent and dependent variable. With true experiments, there is a limited chance that the results are not directly related to the independent variable.

The following three designs are the most highly recommended true experimental designs in the literature because they have been found to best control for the threats to internal validity (Campbell & Stanley, 1966). These designs also demonstrate the impact that random selec�on and assignment have on the internal validity of a study.

Pretest–Pos�est Control Group Design

The pretest–pos�est control group is a classic true experimental design. It begins with randomly assigning (R) par�cipants into both an experimental and a control group. Both groups are administered a pretest or observa�on (O) at the beginning of the study and a pos�est or observa�on (O) at the end of the study; only the experimental group is exposed to the interven�on (X). Thus, the control group is isolated from the interven�on and is subject only to observa�ons. The design takes the following form:

Group A: R_____________O_____________X_____________O

Group B: R______________O________________________O

Such a design controls for two major threats to internal validity. First, randomly assigning par�cipants to groups allows researchers to assume that the two groups were equal in terms of demographic characteris�cs before the study began. Second, including a pretest and a pos�est for both groups allows researchers to control for confounding factors such as history, matura�on, and tes�ng, since both groups should be affected equally by those. Thus, in this design, one should see a change in the experimental group only, which would indicate a cause-and-effect rela�onship. If a change occurs in the control group, then the researcher would need to consider whether some other confounding variable, such as the presence of a human observer, may have influenced the study.

Solomon Four-Group Design

One weakness of the pretest–pos�est control group design is that the process of observing par�cipants at the beginning of the study may influence how the experimental group responds to the interven�on (Leedy & Omrod, 2010). Thus, the pretest observa�on may mo�vate par�cipants to respond to the interven�on in a more posi�ve and beneficial manner. To address this weakness, Solomon (1949) proposed an extension of the pretest–pos�est control group design.

As we can see in the following depic�on, the Solomon four-group design enrolls the experimental and control groups, included in the pretest–pos�est control group design, along with two addi�onal groups. The two addi�onal groups are also randomly assigned (R); however, they do not receive a pretest observa�on. Instead, one group receives the interven�on (X) and pos�est observa�on (O), and the other group receives only a pos�est observa�on (O). This design allows the researcher to control for the influences of a pretest observa�on.

Group A: R_____________O_____________X_____________O

Group B: R______________O__________________________O

Group C: R___________________________X______________O

Group D: R_________________________________________O

By adding two groups, the Solomon four-group design also increases the external validity of the study. For instance, if Groups C and D differed in the same way that Groups A and B did, one could assume that a pretest is not required and could generalize the findings to other situa�ons where a pretest had not been provided.

Although this design was developed to eliminate the possibility of pretest influences, it is not without limita�ons. One disadvantage is the large sample size needed to ensure randomly assigned groups. Another is the �me demanded for carrying out a more complex design. Finally, Smith and Davis (2007) explained that no available sta�s�cal test can treat all four data sets at the same �me; therefore, conduc�ng sta�s�cal analyses may be challenging.

Pos�est-Only Control Group Design

The pos�est-only control group design resembles the pretest–pos�est control group design, except that the pretest is removed from both the experimental and control groups. It is also iden�cal to the addi�onal two groups’ protocol in the Solomon four-group design.

Pos�est-only group designs randomly assigns par�cipants to an experimental group and a control group. Only the experimental group receives the interven�on (X), whereas both the experimental and control groups receive a pos�est observa�on (O). There are no pretest observa�ons. The design takes the following form:

Group A: R_____________X_____________O

Group B: R___________________________O

Does the lack of a pretest make this design less desirable than the other two true experimental designs? The answer is no. Rather, randomly assigning par�cipants to groups is cri�cal in this design because it ensures that the groups are demographically equivalent at the beginning of the study. Thus, any changes during the pos�est observa�on in the experimental group are likely the result of the interven�on and not differences between the two groups. So random assignment and including a control group both control for threats to internal validity, making this a very powerful design.

As we have observed, true experiments exhibit more control over a study and provide for greater internal validity. They are o�en preferred over pre-experimental and quasi-experimental designs because their results can be generalized to a larger popula�on. Table 5.3 summarizes the characteris�cs of each type of experimental design.

Table 5.3: Characteris�cs of pre-experimental, quasi-experimental, and true experimental designs

Design Characteris�cs Pre-Experimental Design Quasi-Experimental Design True Experimental Design

Inclusion of a control group In some cases, but not always O�en Yes, always

Random selec�on of par�cipants from a popula�on No No Yes, always

Random assignment of par�cipants to groups No No Yes, always

Random assignment of treatments or interven�ons to groups No No Yes, always

Control over extraneous or confounding variables None Some Yes

Designing True Experiments

The process of designing true experiments boils down to deciding what variables to manipulate and how to do so. In this sec�on, we will cover two broad issues related to experimental design: deciding how to structure the levels (different versions) of an independent variable and deciding on the number of independent variables necessary to test the hypotheses. While these decisions may seem tedious, they are at the crux of designing successful experiments, and, therefore, are the key to performing successful tests of research hypotheses.

Levels of the Independent Variable

In designing internally valid experiments, the primary goal is to ensure that the levels of independent variables are equivalent in every way but one. This allows researchers to make causal statements about the effects of that single change. There are two primary ways in which researchers can control what levels of the independent variables the groups will receive: using between-subject designs and within-subject designs.

Between-Subject Designs

In some of the research designs discussed so far (primarily quasi-experimental and true experimental designs), the levels of our independent variables have represented two dis�nct groups—par�cipants are in either the control group or the experimental group. This type of design is referred to as a between-subject design because the levels differ between one subject and the next. Each par�cipant who enrolls in the experiment is exposed to only one level of the independent variable. That is, an individual par�cipant might be in either the experimental or the control group.

Most of the examples so far illustrated between-subject designs: Par�cipants receive either alcohol or placebo; students read an ar�cle designed to prime either their Asian or their female iden�ty; and graduate students train rats that are falsely labeled either bright or dull. This approach is very common and has the advantage of using dis�nct groups to represent each level of the independent variable. In other words, par�cipants who are asked to consume alcohol are completely dis�nct from those asked to consume the placebo drink. However, this is only one op�on for structuring the levels of the independent variable. In this sec�on, we will examine two addi�onal ways to structure these levels.

Carryover effects can be understood through the example of monitoring people's reac�ons to different film clips. How they feel about one image may influence how they react to the next image.

Ryan McVay/Photodisc/Thinkstock

Within-Subject Designs

In some cases, the levels of the independent variable represent the same par�cipants but at different �me periods. This type of design is referred to as a within- subject design because the levels differ within people. Each par�cipant who enrolls in the experiment would be exposed to both or all levels of the independent variable. That is, every par�cipant would be in both the experimental and the control group. Within-subject designs are o�en used to compare changes over �me in response to various s�muli. For example, you might measure anxiety symptoms before and a�er people are locked in a room with a spider or measure depression symptoms before and a�er people undergo drug treatment.

Within-subject designs have two main advantages over between-subject designs. First, because the same people determine both levels of the independent variable, these designs require fewer par�cipants. Let’s say you decided to collect data from 20 par�cipants at each level of your independent variable. In a between-subject design with three levels, you would need 60 people. However, if you ran the same experiment as a within-subject design—exposing the same group of people to three different sets of circumstances—you would need only 20 people. Thus, within-subject designs are o�en a good way to conserve resources.

Second, par�cipants serve as their own control group, allowing the researcher to minimize a big source of error variance. Remember that one key feature of experimental design is the researcher’s power to assign people to groups; this is done to randomly distribute subject differences across the levels of the independent variable. Using a within-subject design solves the problem of subject differences in another way—by examining changes within people. For instance, in the example regarding spiders and anxiety, some par�cipants would likely have higher baseline anxiety than others. By measuring changes in anxiety in the same group of people before and a�er spider exposure, the researcher would be able to minimize the effects of individual differences.

Problems With Within-Subject Designs

Within-subject designs also have two clear disadvantages compared with between-subject designs. First, there is the risk of carryover effects, in which the effects of one level are s�ll present when another level is introduced. Because the same people are exposed to all levels of the independent variable, it can be difficult to separate the effects of one level from the effects of the others. One common paradigm in research on emo�ons is to show par�cipants several film clips that elicit different types of emo�on. People might view one clip showing a puppy playing with a blanket, another showing a child crying, and another showing a surgical amputa�on. Even without seeing these in full color, you can imagine that it would be hard to shake off the disgust triggered by the amputa�on in order to experience the joy triggered by the puppy.

When researchers use a within-subject design, they take steps to minimize carryover effects. In studies of emo�on, for example, researchers typically show a brief, neutral clip, such as waves rolling onto a beach, between emo�onal clips so that par�cipants experience each emo�on a�er viewing a benign image. Another simple technique is to collect data from the control condi�on first whenever possible. In the example study of spiders and anxiety, it would be important to measure baseline anxiety at the start of the experiment before exposing people to spiders. Once people had been surprised by a spider, it would be hard to get them to relax enough to collect control ra�ngs of anxiety.

Second, there is a risk of order effects, meaning that the order in which levels are presented can moderate their effects. Order effects fall into two categories. The prac�ce effect happens when par�cipants’ performance improves over �me simply due to making repeated a�empts. This is a par�cular problem in studies that examine learning. Let’s say you use a within-subject design to compare two techniques for teaching people to solve logic problems. Par�cipants would learn technique A, then take a logic test, then learn technique B, and then take a second logic test. The possible problem that arises is that par�cipants will have had more opportuni�es to prac�ce logic problems by the �me they took the second test. This would make it difficult to separate the effects of prac�ce from the effects of different teaching techniques.

The flipside of prac�ce effects is the phenomenon of the fa�gue effect, which happens when par�cipants’ performance decreases over �me owing to repeated tes�ng. Let’s say you ran a varia�on of the experiment just described, a�emp�ng to teach people to improve their reac�on �me. Par�cipants might learn each technique and have their reac�on �me tested several �mes a�er each one. The problem would be that people gradually started to become �red, and their reac�on �mes slowed down due to fa�gue. Thus, it would be difficult to separate the effects of fa�gue from the effects of the different teaching techniques.

The result of both of these problems is that the order of presenta�on becomes confounded with the level of the independent variable. Fortunately, there is a rela�vely easy way to avoid both carryover and fa�gue effects by using a process called counterbalancing. Counterbalancing involves varying the order of presenta�on to groups of par�cipants. The simplest approach is to divide par�cipants into as many groups as there are combina�ons of levels in the experiment. That

is, the researchers create a group for each possible order, allowing them to iden�fy the effects of encountering the condi�ons in different orders. In the preceding examples, the learning experiments involved two techniques: A and B. To counterbalance these techniques across the study, it would be necessary to divide the par�cipants into two groups. One group would be exposed to A and then B; the other group would be exposed to B and then A. When it came �me to analyze the data, researchers would be able to examine the effects of both presenta�on order and teaching technique. If the order of presenta�on made a difference, then one could conclude that the A/B group differed from the B/A group in some way.

Mixed Designs

The third common way to structure the levels of an independent variable is in a mixed design (discussed in more detail in Sec�on 5.8), which contains at least one between-subject variable and at least one within-subject variable. So, in the example just discussed, par�cipants would be exposed to both teaching techniques (A and B), but in only one of two possible orders of presenta�on. In this case, teaching technique is a within-subject variable because par�cipants experience both

levels. And presenta�on order is a between-subject variable because par�cipants experience only one level. When there is one of each in the overall experiment, this is a mixed design.

One common use of mixed designs is in studies that compare the effects of different drugs. Imagine you wanted to compare three new drugs—Drug X, Drug Y, and a placebo control—to determine which had the strongest effects on reducing depression symptoms. To do this study, you would want to measure depression symptoms on at least three occasions: before star�ng drug treatment, a�er a few months of taking the drug (or placebo), and then again a�er a few months of stopping the drug (to assess relapse rates). So the par�cipants would be given one of three possible drugs and then measured at each of three �me periods. In this mixed design, measurement �me is a within-subject variable because par�cipants experience all three levels, whereas the drug is a between-subject variable because par�cipants experience only one of three possible levels.

The hypothe�cal results of this study are shown in Figure 5.4. You can see that the placebo pill has no effect on depression symptoms; depression scores in this group are the same at all three measurements. Drug X appears to significantly improve depression symptoms; depression scores drop steadily across measurements in this group. And, strangely, Drug Y seems to make depression worse; depression scores increase steadily across measurements in this group. The mixed design allows researchers both to track people over �me and to compare different drugs in one study.

Figure 5.4: Example of a mixed-subjects design

Research: Thinking Critically

Outwalking Depression

By Michael O�o

There is a great deal of evidence that exercise not only improves mood and enhances well-being but also is an effec�ve interven�on for depression. That is, well-conducted clinical trials have repeatedly shown mood benefit from exercise in adults with clinical depression. Indeed, there is evidence that exercise provides benefits at levels similar to that found for an�depressant medica�on.

Now, there is new evidence for the power of exercise when other treatments for depression have not provided adequate help. A recent study, published in the August 2011 issue of the Journal of Psychiatric Research, provides data that exercise can provide benefits when medica�on alone does not (Moto- Pereira et al, 2011). Specifically, researchers in Portugal examined the effects of exercise—in this case programmed episodes of walking for 30–45 minutes five �mes per week—on depressed adults who failed to respond to two previous trials of an�depressant medica�on. In the study, all pa�ents remained on their an�depressant medica�on, and two thirds of the sample received the program of regular (walking) exercise.

The results were drama�c. Of those pa�ents who received only medica�on, no average changes in depression mood ra�ngs were seen over the next 12 weeks. In contrast, clear improvement was seen in those who exercised, with 10 of the 19 pa�ents who exercised showing a response or full remission in symptoms during this �me.

Although this was a small study, the results were consistent with previous exercise studies. These studies underscore the importance of considering combined treatment strategies for pa�ents who do not respond to one type of interven�on—if one treatment does not work, consider using other resources with evidence for success. In the case of treatment of depression, these resources prominently include psychotherapy (with a wealth of studies indica�ng that specific forms of psychotherapy like Cogni�ve Behavior Therapy or Interpersonal Psychotherapy can bring about �mely relief of depression), a range of medica�on treatments, or, with increasing evidence, regular exercise. Within these choices there is not a clear cure-all op�on, but, importantly, there is a range of op�ons to be pursued to try to find the right fit for any par�cular person suffering from depression.

A number of important features of the Portuguese study deserve addi�onal note. First, the study provided addi�onal evidence that high intensi�es of exercise are not required to bring about mood changes. Moderate exercise is adequate for mood benefit. Second, the study used an exercise program that relied on the individual efforts of the pa�ent. Exercise was performed on a treadmill at the treatment center for one out of the five weekly sessions; the rest of the sessions were up to the individual. This individual program of exercise helped show that the mood effects of exercise were not simply due to the social contact from a group of pa�ents exercising together. It also showed that moderate exercise is an accessible op�on even for chronically depressed individuals. Third, the exercise prescrip�on included the use of regular reminders for exercise: keeping walking shoes in a visible loca�on, having support for the walking program from family, or arranging for wri�en or cell phone reminders for walking sessions. Simple reminders like these can have a powerful effect on helping people keep up with any new habit.

O�o, M., & Smits, J. (2011, July 26). Outwalking depression: Consider exercise when the an�depressants have not helped. Psychology Today. Retrieved from h�p://www.psychologytoday.com/blog/exercise-and-mood/201107/outwalking-depression (h�p://www.psychologytoday.com/blog/exercise-and-mood/201107/outwalking-depression)

Think about it:

1. Iden�fy the following essen�al aspects of this experimental design: a. What are the independent and dependent variables in this study? b. How many levels does the independent variable have? c. Is this a between-subject, within-subject, or mixed design? d. Draw a simple table labeling each condi�on.

2. What existent differences between groups should the researchers be sure to take into account? Name as many as you can.

3. How should the researchers assign par�cipants to the condi�ons in order to ensure that existent differences cannot account for the results?

4. How might expectancy effects influence the results of this study? Can you think of any ways to control for this?

5. Briefly state how you would replicate this study in each of the following ways: a. exact replica�on b. conceptual replica�on c. par�cipant replica�on d. construc�ve replica�on

One-Way Versus Factorial Designs

The second big issue in crea�ng experimental designs is to decide on the number of independent variables to manipulate. In some cases, we can test hypotheses by manipula�ng a single independent variable and measuring the outcome, for example, giving people either alcohol or a placebo drink and measuring the inten�on to use condoms. In other cases, hypotheses involve more complex combina�ons of variables. Earlier in the chapter, we discussed research findings that people tend to act more aggressively a�er a peer group has rejected them. But this hypothesis could be extended by asking what happens when people are rejected by members of the same sex versus members of the opposite sex. And one could go a step further and test whether the a�rac�veness of the rejecters ma�ers, for a total of three independent variables. These examples illustrate two broad categories of experimental design, known as one-way and factorial designs.

One-Way Designs

If a study involves giving people either alcohol or a placebo and then measuring outcomes, it has a one-way design, which has only one independent variable with two or more levels. These tend to be the simplest experiments and have the advantage of tes�ng manipula�ons in isola�on. One-way designs are used in the majority of drug studies. These types of study compare the effects on medical outcomes for people randomly assigned, for instance, to take the an�depressant

drug Prozac or a placebo. Note that a one-way design can s�ll have mul�ple levels—in many cases, it is preferable to test several different doses of a drug. So, for example, researchers might test the effects of Prozac by assigning people to take doses of either 5 mg, 10 mg, 20 mg once daily, or a placebo control. If so, the independent variable would be the drug dose, and the dependent variable would be a change in depression symptoms. This design would allow researchers to compare all three drug doses to a placebo control, as well as to compare the effects of each dose. Figure 5.5 shows hypothe�cal results from this study. No�ce that even those receiving the placebo showed a drop in depression symptoms, with the maximum benefit caused by the 10-mg dose of Prozac.

Figure 5.5: Comparing drug doses in a one-way design

Factorial Designs

Despite the appealing simplicity of one-way designs, experiments conducted in the field of psychology with only one independent variable are rela�vely rare. The real world is much more complicated, which means that studies that focus on people’s thoughts, feelings, and behaviors must try to capture the associated complexity. Thus, the rejec�on and aggression example discussed previously is not that far-fetched. If a researcher wanted to manipulate the presence of rejec�on, the sex of the rejecters, and the a�rac�veness of the rejecters in a single study, the experiment would need a factorial design. Factorial designs are those that have two or more independent variables, each of which has two or more levels. When using a factorial design, the purpose is to observe both the effects of individual variables and the combined effects of mul�ple variables.

Factorial designs have their own terminology to reflect the fact that they comprise both individual variables and combina�ons of variables. At the beginning of this chapter, we learned that the versions of an independent variable are referred to as both levels and condi�ons, and that there is a subtle difference between the two. This difference comes into play when discussing factorial designs. Specifically, levels refer to the versions of each independent variable, while condi�ons refer to the groups formed by combina�ons of independent variables. Let’s walk through one varia�on of our rejec�on and aggression example from this perspec�ve: The first independent variable has two levels because par�cipants are either rejected or not rejected. And the second independent variable has two levels because members of the same sex or the opposite sex do the rejec�ng. To determine the number of condi�ons in this study, calculate the number of different experiences that par�cipants can have in the study. This is a simple ma�er of mul�plying the levels of separate variables, so 2 is mul�plied by 2, for a total of four condi�ons.

Researchers also have a way to quickly describe the number of variables in their design: A two-way design has two independent variables; a three-way design has three independent variables; an eight-way design has eight independent variables, and so on. Even more useful, the system of factorial nota�on offers a simple way to describe both the number of variables and the number of levels in experimental designs. For instance, you might describe your design as a 2 × 2 (pronounced “two by two”), which instantly communicates two things: (1) you have two independent variables, indicated by the presence of two separate numbers, and (2) each independent variable has two levels, indicated by the number 2 listed for each one.

The 2 × 2 Design

One of the most common factorial designs also happens to be the simplest one: the 2 × 2 design. These designs have two independent variables, with two levels each, for a total of four experimental condi�ons. The simplicity of these designs makes them a useful way to become more comfortable with some of the basic concepts of experiments. This sec�on will walk you through an example of a 2 × 2 design and analyze it in detail.

Beginning in the late 1960s, social psychologists developed a keen interest in understanding the predictors of helping behavior. This was inspired, in large part, by the tragedy of Ki�y Genovese, who was killed outside her apartment building while none of her neighbors called the police (Gansberg, 1964). (See Research: Making an Impact in Chapter 2, Sec�on 2.1 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.1#sec2.1) .) In one representa�ve study, Princeton psychologists John Darley and Bibb Latané examined people’s likelihood of responding to a staged emergency. Par�cipants were led to believe that they were taking part in a group discussion over an intercom system, but in reality, all the other par�cipants’ voices were prerecorded. The key independent variable was the number of other people supposedly present, ranging from two to six. A few minutes into the conversa�on, one par�cipant appeared to have a seizure. The recording went like this (actual transcript; Darley & Latané, 1968):

I could really-er-use some help so if someone would-er-give me a li�le h-hel-puh-er-er-er c-could somebody er-er-hel-er-uh-uh-uh [choking sounds] . . . I’m gonna die-er-er-I’m . . . gonna die-er-hel-er-er-seizure-er [chokes, then quiet].

What do people do in this situa�on? Do they help? How long does it take? Darley and Latané discovered that two things happen as the group became larger: People were less likely to help at all, and those who did help took considerably longer to do so. One of the primary conclusions to come out of this and other studies is that people are less likely to help when other people are present because the responsibility for helping is diffused among the members of the crowd (Darley & Latané, 1968).

Building on this conclusion, the sociologist Jane Piliavin and her colleagues (Piliavin, Rodin, & Piliavin, 1969; Piliavin, Piliavin, & Rodin, 1975) explored the influence of two addi�onal variables on helping behaviors by staging an emergency on a New York City subway train in which a person who was in on the study appeared to collapse in pain. Specifically, the researchers manipulated two variables in their staged emergency. The first independent variable was whether there was a medical intern nearby, who could be easily iden�fied by wearing blue scrubs. The second independent variable was whether the vic�m had a large disfiguring scar on his face. The combina�on of these variables resulted in four condi�ons, as shown in Table 5.4. The dependent variable in this study was the percentage of people taking ac�on to help the person appearing to collapse in pain.

Table 5.4: 2 × 2 Design of the Piliavin et al. study

No intern Intern

No scar 1 2

Scar 3 4

The authors predicted that bystanders would be less likely to help if there was a perceived medical professional nearby since he or she was considered more qualified to help the vic�m. They also predicted that people would be less likely to help when the confederate had a large scar because previous work had demonstrated convincingly that people avoid contact with those who are disfigured or have other s�gma�zing condi�ons (e.g., Goffman, 1963). As seen in Figure 5.6, these hypotheses were supported by the results. Both the presence of a scar and the presence of a perceived medical professional reduced the percentage of people who came to help. But there’s something else going on in these results: When the confederate is not scarred, having an intern nearby leads to a small decrease in helping (from 88% to 84%). However, when the confederate had a large facial scar, having an intern nearby decreased helping from 72% to 48%! In other words, it seems these variables had a combined effect on helping behavior. We will examine these combined effects more closely in the next sec�on.

Figure 5.6: Sample 2 × 2 Design: Results from Piliavin et al. (1975)

Other Factorial Designs

Experimental designs can o�en be more complex than a simple 2 × 2 format. In another varia�on of the rejec�on and aggression study, par�cipants would be categorized in one of eight cells—they are either rejected or not, by a person of either the same sex or the opposite sex, who happens to be either a�rac�ve or una�rac�ve. This design would have eight condi�ons (2 × 2 × 2), as shown in Table 5.5a, with the separate condi�ons numbered for illustra�on purposes.

Table 5.5a: Levels versus condi�ons in a 2 × 2 × 2 design

Rejected Not Rejected

Rejecter A�rac�ve Una�rac�ve A�rac�ve Una�rac�ve

Same sex 1 2 5 6

Opposite sex 3 4 7 8

This is a 2 × 2 × 2 design for a total of eight condi�ons (numbered 1–8). “Rejec�on” has two levels: rejected and not rejected. “A�rac�veness” has two levels: a�rac�ve and una�rac�ve. “Sex” has two levels: the same sex and the opposite sex rejecter.

A different study might describe a design as 2 × 3, which would tell people two things: There are two independent variables, and one variable has two levels (the 2) while the other variable has three levels (the 3). One possibility for this 2 × 3 design is illustrated in Table 5.5b. For this hypothe�cal study, the researcher was interested in the effects of discomfort on mood. She manipulated both hunger, with three levels, and pain, with two levels. Par�cipants were asked to abstain from ea�ng for 4 hours (“mild hunger”) or 24 hours (“extreme hunger”), or they were allowed to eat before the study (“no hunger”). During the laboratory session, par�cipants were asked either to hold their hand in a bucket of ice water (“pain”) or not (“no pain”). As you can see in Table 5.5b, this resulted in six possible condi�ons for her par�cipants to experience.

Table 5.5b: Levels versus condi�ons in a 2 × 3 design

No Hunger Mild Hunger Extreme Hunger

Pain 1 2 3

No Pain 4 5 6

This is a 2 × 3 design for a total of six condi�ons (numbered 1–6). “Pain” has two levels: presence and absence of pain. “Hunger” has three levels: no hunger, mild hunger, and extreme hunger.

Main Effects and Interactions

When experiments involve only one independent variable (IV), the analyses can be as simple as comparing two group means—as with the example in Chapter 2 (Sec�on 2.4 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.4#sec2.4) , Hypothesis Tes�ng) comparing the happiness levels of couples with and without children. But what about cases where the design has more than one independent variable?

In a factorial design, there are two types of effect: A main effect refers to the effect of each independent variable on the dependent variable, averaging values across the levels of other variables. A 2 × 2 design has two main effects; a 2 × 2 × 2 design has three main effects because there are three independent variables. An interac�on occurs when the variables have a combined effect; that is, the effects of one independent variable are different depending on the levels of the other independent variable. So applying this new terminology to the Piliavin et al. (1969, 1975) “subway emergency” study yields three possible results (possible because researchers eventually use sta�s�cal analyses to verify them):

1. The main effect of scar: Does the presence of a scar affect helping behavior? Yes. More people help in absence of a facial scar. In Figure 5.6, you can see that the bars on the le� (no scar) are, on average, higher than those on the right (scar).

2. The main effect of intern: Does the presence of an intern affect helping behavior? Yes. More people help when there is no medical intern on hand. In Figure 5.6, no�ce that the red bars (no intern) are, on average, higher than the orange bars (intern).

3. The interac�on between scar and intern: Does the effect of one variable depend on the effect of another variable? Yes. If you refer to Figure 5.6, you can see that the presence of a medical intern ma�ers more when the vic�m has a facial scar. In visual terms, the gap between red and orange bars is much larger in the bars on the right. This indicates an interac�on between scar and intern.

Let’s walk through a fic�onal example. Imagine you were interested in people’s percep�ons of actors in different types of movies. You might predict that some actors are be�er suited to comedy and others are be�er suited to ac�on movies. One simple way to test this hypothesis would be an experiment that showed four movies in a 2 × 2 design, using the same two actors in two movies (for a total of four condi�ons). Our first independent variable would be the movie type, with two levels: ac�on and comedy. Our second independent variable would be the actor, with two levels: Will Smith and Arnold Schwarzenegger. Our dependent variable would be the ra�ngs of each movie on a 10-point scale. This gives us three possible results:

1. The main effect of actor: Do people generally prefer Will Smith or Arnold Schwarzenegger, regardless of the movie? 2. The main effect of movie type: Do people generally prefer ac�on or comedy movies, regardless of the actor? 3. The interac�on between actor and movie type: Do people prefer each actor in a different kind of movie? (i.e., are ra�ngs affected by the combina�on of actor and

movie type?)

A�er collec�ng data from a sample of par�cipants, we end up with the following average ra�ngs for each movie, shown in Table 5.6.

Table 5.6: Main effects and marginal means: The actor study

Arnold Will Marginal Mean

Ac�on 6 1.5 3.75

Comedy 5 8 6.5

Marginal Mean 5.5 4.75

These two means let us compare movie ra�ngs, ignoring the actor. People have a slight preference for Comedy (6.5) over Ac�on (3.75). These two means let us compare actor ra�ngs, ignoring the movie type. People have a slight preference for Arnold (5.5) over Will (4.75).

Remember that main effects represent the effects of one independent variable, averaging across the levels of the other independent variable. To average across levels, we calculate the marginal mean, or the combined mean across levels of another factor. In other words, the marginal mean for ac�on movies is calculated by averaging together the ra�ngs of both Arnold Schwarzenegger and Will Smith in ac�on movies. The marginal mean for Arnold Schwarzenegger is calculated by averaging ra�ngs of Arnold Schwarzenegger in both ac�on and comedy movies. Doing this for our 2 × 2 design results in four marginal means, which are presented alongside the par�cipant ra�ngs in Table 5.6. We would need to verify these pa�erns with sta�s�cal analyses, but it appears we have two main effects, based on these marginal means: People have a slight preference for comedy over ac�on movies, as well as a slight preference for Arnold Schwarzenegger’s ac�ng over Will Smith’s ac�ng, regardless of the movie.

What about the interac�on? Our main hypothesis here is that some actors perform best in some genres of movies (e.g., ac�on or comedy) than they do in other genres. This suggests that the actor and the movie type combine to influence people’s ra�ngs of the movies. We can get a sense of this from examining the means in Table 5.6, but it is even easier to appreciate in a graph. Figure 5.7 shows the mean of par�cipants’ ra�ngs across the four condi�ons. Focusing first on the ra�ngs of Arnold Schwarzenegger, no�ce that par�cipants did have a slight preference for him in ac�on (6) versus comedy (5) roles. Then, examining ra�ngs of Will Smith, you can see that par�cipants had a strong preference for him in comedy (8) versus ac�on (1.5) roles. Together, this set of means indicates an interac�on between actor and movie type because the effects of one variable depend on another. Or, in plain English: People’s percep�ons of the actor depend on the type of movie he is in. This pa�ern of results is a nice fit for our hypothesis that certain actors are be�er suited to certain types of movie: Arnold should probably s�ck to ac�on movies, and Will should definitely s�ck to comedies.

Figure 5.7: Interaction in the actor study

Before we move on to analyses, let’s look at one more example from a published experiment. A large body of research in social psychology suggests that stereotypes can nega�vely impact performance on cogni�ve tasks (e.g., tests of math and verbal skills). According to Stanford social psychologist Claude Steele and his colleagues, the fear of confirming nega�ve stereotypes about one’s group acts as a distrac�on. This distrac�on—which they term stereotype threat—makes it hard to concentrate and perform well, and thus leads to lower scores on a cogni�ve test (Steele, 1997). One of the primary implica�ons of this research is that ethnic differences in standardized test scores can be viewed as a situa�onal phenomenon, so if we change the situa�on, the differences should go away. In the first published study of stereotype threat, Claude Steele and Josh Aronson (1995) found that when African American students at Stanford were asked to indicate their race before taking a standardized test, this was enough to remind them of nega�ve stereotypes, and they performed poorly. But when the tes�ng situa�on was changed, and par�cipants were no longer asked their race, these students performed at the same level as Caucasian students. It is worth emphasizing that these were Stanford students and had therefore met admissions standards for one of the best universi�es in the na�on. Even this group of elite students was suscep�ble to situa�onal pressure but performed at their best when the pressure was eliminated.

In a great applica�on of stereotype threat, social psychologist Jeff Stone at the University of Arizona asked both African American and Caucasian college students to try their hand at pu�ng on a golf course (Stone, Lynch, Sjomeling, & Darley, 1999). Pu�ng was described as a test of natural athle�c ability to half of the par�cipants and as a test of sports intelligence to the other half. Thus, there were two independent variables: the race of the par�cipants (African American or Caucasian) and the descrip�on of the task (“athle�c ability” or “sports intelligence”). Note that “race” in this study was technically a quasi-independent variable because it was not manipulated. This resulted in a total of four condi�ons, and the dependent variable was the number of pu�s that par�cipants managed to make. Stone and colleagues hypothesized that describing the task as a test of athle�c ability would lead Caucasian par�cipants to worry about the stereotypes regarding their poor athle�c ability. In contrast, describing the task as a test of sports intelligence would lead African American par�cipants to worry about the stereotypes regarding their lower intelligence.

Consistent with their hypotheses, Stone and colleagues found an interac�on between race and task descrip�on but no main effects. That is, neither race was be�er at the pu�ng task overall, and neither task descrip�on had an overall effect on pu�ng performance. But the combina�on of these variables proved fascina�ng. When the task was described as measuring sports intelligence, the African American par�cipants did poorly for fear of confirming nega�ve stereotypes about their overall intelligence. But when the task was described as measuring natural athle�c ability, the Caucasian par�cipants did poorly for fear of confirming nega�ve stereotypes about their athle�cism. This is a beau�ful illustra�on of an interac�on; the effects of one variable (task descrip�on) depend on the effects of another (race of par�cipants). And this is further confirma�on of the power of the situa�on: Neither group did be�er or worse overall, but both were responsive to a situa�onally induced fear of confirming nega�ve stereotypes.

Figure 5.8: Comparing sources of variance

5.5 Analyzing Experiments

So far, we have been drawing conclusions about our experimental findings in conceptual terms. But naturally, before we actually make a decision about the status of our hypotheses, we have to conduct sta�s�cal analyses. In this sec�on, we will cover the most common sta�s�cal techniques for analyzing experimental data.

Dealing With Multiple Groups

You may find yourself wondering why we need a special technique for experimental designs. A�er all, we learned in Chapter 2 (Sec�on 2.4 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.4#sec2.4) , Hypothesis Tes�ng) that we can compare two pairs of means using a t-test. So why not just use several t-tests to analyze our experimental designs? In our movie ra�ngs study, we could analyze the data using a total of six t-tests to capture every possible pair of means: Arnold Schwarzenegger in a comedy versus Will Smith in a comedy; Arnold Schwarzenegger in an ac�on movie versus Will Smith in an ac�on movie; Arnold Schwarzenegger in a comedy versus an ac�on movie; Will Smith in a comedy versus an ac�on movie; Will Smith in a comedy versus Arnold Schwarzenegger in an ac�on movie; and finally Will Smith in an ac�on movie versus Arnold Schwarzenegger in a comedy.

But there is a problem with this approach: The odds of developing a Type I error (a false posi�ve) are increased with every sta�s�cal test. We typically set our alpha level at .05 for a t-test, meaning that we’re comfortable with a 5% chance of a Type I error. Unfortunately, if we conduct six t-tests, each one has a 5% chance of a Type I error, meaning that we’re now rather likely to draw conclusions about a false posi�ve. The moral of this story is that we need a sta�s�cal approach that reduces the number of comparisons we perform. Fortunately, this can be accomplished using a sta�s�cal technique called the analysis of variance (ANOVA), which tests for differences by comparing the amount of variance explained by the independent variables to the amount of variance explained by error. The following sec�ons explore ANOVA in more detail.

The Logic of ANOVA

The logic behind the analysis of variance is rather straigh�orward. As we have discussed throughout this course, the variability in data can be divided into systema�c and error variance. That is, we can a�ribute some of the variability to the factors we are studying, but there will always be a degree of random error. In our movie ra�ngs study, some of the variability in these ra�ngs can be a�ributed to our independent variables (differences in actors and movie types), while some of the variability is the result of other factors— perhaps some people simply like movies more than other people.

The ANOVA works by comparing the influence of these different sources of variance. We always want to explain as much of the variance as possible through the independent variables. If the independent variables have more influence than random error does, this is good news. But if error variance has more influence than the independent variables, this is bad news for the hypotheses. You can get a sense of this by comparing the three pie charts in Figure 5.8. The propor�on of variance explained by our independent variables is shaded in yellow, while the propor�on explained by error is shaded in red. In the top graph, approximately 80% of the variance can be explained by our independent variables, which can be viewed as a good result. But in the middle graphs, variance is explained equally by the independent variables and by error, and in the bo�om graph, only 30% of the variance is explained by the independent variables. Thus, in the la�er two graphs, our independent variables do no be�er than random error at explaining our results.

One more analogy may be helpful. In the field of engineering, the term signal-to-noise ra�o is used to describe the amount of light, sound, energy, and so forth that is detectable above and beyond background noise. This ra�o is high when the signal comes through clearly and low when it is mixed in with sta�c or other interference. Likewise, when you try to tune in your favorite radio sta�on, the goal is to find a clear signal that is not covered up by sta�c. Believe it or not, the ANOVA sta�s�c is doing the same thing. That is, the analysis tells us whether differences in experimental condi�ons (signal) are detectable above and beyond error variance (noise). In the next sec�on, we’ll take a closer look at how ANOVA makes these dis�nc�ons.

Calculating ANOVA

The ANOVA sta�s�c—abbreviated with the capital le�er F—works by comparing the variance that is explained by group differences with the variance explained by error. This is really just a minor adapta�on, with new terminology, of the formula we have encountered before, describing the components of total variance in the dataset:

Total Variance = Systema�c Variance + Error Variance

In conduc�ng an ANOVA, these two components will have special names, but the idea is the same. Specifically, we refer to systema�c variance as between-group variance, or the variance explained by differences in our independent variables. In the movie ra�ngs study, between-groups variance represents the variability in ra�ngs that can be a�ributed to different actors, different movies, and the combina�ons of these independent variables. In addi�on, we refer to error variance as within-group variance, or the total variance that occurs across each condi�on. In the movie ra�ngs study, within-groups variance would represent the sum total of all other influences on movie ra�ngs (i.e., the error).

In this day and age, ANOVA (F) is calculated using one of the numerous sta�s�cal so�ware programs; these programs take columns of data and churn out significance tests to test hypotheses. But in order to understand what these significance tests mean, it is worth walking through a conceptual overview of the elements in the ANOVA formula. The star�ng point for the F sta�s�c is the calcula�on of several sum of squares, or the sum of the squared devia�ons between individual scores and the overall sample mean. We encountered these sums of squares (abbreviated SS) in Chapter 3 (Sec�on 3.5 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec3.5#sec3.5) ), in our discussion of calcula�ng the standard devia�on.

Using this nota�on, our components of the variance formula can be rewri�en as follows:

SST = SSBG + SSWG SST = sum of squares total

SSBG = sum of squares between groups SSWG = sum of squares within groups

In conceptual terms, SSBG represents the devia�on between each condi�on’s mean and the overall mean of the sample. If you refer to the data from our movie

ra�ngs study in Table 5.6, you’ll see the four condi�on means of 6, 5, 1.5, and 8. The overall mean of movie ra�ngs—across condi�ons—is 5.13 (i.e., [6 + 5 + 1.5 + 8] ÷ 4). To compute the SSBG, we subtract each of the four condi�on means from 5.13, square the difference, and add up these squared devia�on scores.

In conceptual terms, SSWG represents the devia�on between each individual score and the condi�on mean for that individual. To compute SSWG in our movie

ra�ngs study, we subtract each individual score from the relevant condi�on mean, square the difference, and add up the squared devia�on scores.

That’s a lot of math, so let’s take a step back. In essence, these terms are calcula�ng and comparing two sources of varia�on around the overall mean. When we combine all the par�cipants’ movie ra�ngs, and ignore everything about what they were ra�ng, we get an overall mean of 5.13. But, naturally, individual par�cipants deviate from this mean—some people gave higher ra�ngs and some gave lower ra�ngs. The point of calcula�ng SS terms is to understand these devia�ons. Our between-group SS gives us an index of how much each condi�on deviates from the overall mean, while our within-group SS gives us an index of how much individuals differ from the mean of their condi�on. If the former is higher, it means that the groups’ results ma�er; if the la�er is higher, it means that individual quirks mean more than our independent variables.

These sums of squares are the first step toward calcula�ng our F sta�s�c, but they have a flaw that has to be addressed. Namely, the sum of squares is an imprecise measure because it does not take sample size into account. You’ll no�ce that the SS terms are calculated by adding up the squared devia�on scores for each par�cipant (or each condi�on). If our experiment happened to have a large sample, the SSWG would automa�cally be higher because there would be a greater

number of squared devia�on scores. Likewise, if our experiment happened to have several condi�ons, the SSBG would be inflated. Fortunately, this flaw is easily

corrected by calcula�ng a value called the mean square, which corrects the sum of squares for sample size by dividing by the appropriate degrees of freedom. The resul�ng values represent the average devia�on from the sample mean, corrected for the number of scores that went into the calcula�on—much like the final step in our calcula�on of standard devia�ons in Chapter 3 (Sec�on 3.5 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec3.5#sec3.5) ). So, our components of variance formula is altered one more �me:

MST = MSBG + MSWG MST = mean square total

MSBG = mean square between groups MSWG = mean square within groups

We now have all the informa�on we need to calculate the F sta�s�c (ANOVA), using the following formula:

F= between-groups variability within-groups variability = M S BG M S WG

This is the formula sta�s�cal so�ware programs use to calculate an F value. F is expressed as a ra�o of variances, expressing group differences rela�ve to error. Thus, the bigger the F value, the higher the ra�o of group differences to error, and the bigger the influence of independent variables. In other words, we want F values to be as large as possible because this indicates that our experimental manipula�ons make a difference on the outcome variable.

We can also frame the components of this ra�o in terms of the null hypothesis versus the experimental hypothesis. If our experimental hypothesis is true, MSWG represents random varia�on and MSBG represents meaningful differences between groups. Thus, MSBG > MSWG, and F will be a large value. In contrast, if the null

hypothesis is true, both MSWG' and MSBG represent random varia�on because our experimental manipula�ons do not have a meaningful effect. Thus, MSBG =

MSWG' and the value of F is 1. One final possibility here is that our value of F can occasionally be less than 1. This occurs when results are more influenced by

random varia�on than they are by the independent variables; in other words, MSWG > MSBG.

One-Way Versus Factorial ANOVA

Before we move on from the steps in our ANOVA, there is one more level of complexity to explore. The formula we have been discussing so far describes an ANOVA only for a one-way design. When there is one independent variable, MST = MSBG + MSWG. However, a factorial design has mul�ple independent variables,

and the between-group variance term (MSBG) is divided into several elements. As we discussed, a 2 × 2 design has three possible results: a main effect of variable

A; the main effect of B; and the interac�on of A and B. To account for these effects, we split our between-group variance into three new components:

MSBG = MSA + MSB + MSAB MSA = MSB + MSAB MSA = mean square for the main effect of A MSB = mean square for the main effect of B

MSAB = mean square for the interac�on of A and B

This results in another update to our components of variance formula, replacing the MSBG term with these three components:

MST = (MSA + MSB + MSAB) + MSWG

This also results in three F values, expressing each mean square rela�ve to within-group variance. These are illustrated next using our movie ra�ngs study. Our 2 × 2 design in that study yields a main effect of actor, a main effect of movie type, and the actor * movie interac�on:

F ACTOR = M S ACTOR M S WG F MOVIE = M S MOVIE M S WG F ACTOR × MOVIE = M S ACTOR × MOVIE M S WG

For illustra�on purposes, consider the following formula, represen�ng the components of variance in a three-way design (which, incidentally, would yield seven F values!). As our designs become more complex, it becomes easier to appreciate the benefits of sta�s�cal so�ware packages.

MST = (MSA + MSB + MSC + MSAB + MSBC + MSAC + MSABC) + MSWG MSA = mean square for the main effect of A MSB = mean square for the main effect of B

MSAB = mean square for the interac�on of A and B MSBC = mean square for the interac�on of B and C MSAC = mean square for the interac�on of A and C

MSABC = mean square for the three-way interac�on of A, B, and C

Research: Thinking Critically

Love Ballad Leaves Women More Open to a Date

Medical News Today

If you’re having trouble ge�ng a date, French researchers suggest that picking the right soundtrack could improve the odds. Women were more prepared to give their number to an “average” young man a�er listening to roman�c background music, according to research that appears in the journal Psychology of Music, published by SAGE.

There’s plenty of research indica�ng that the media affects our behaviour. Violent video games or music with aggressive lyrics increases the likelihood of aggressive behaviour, thoughts, and feelings—but do roman�c songs have any effect? This ques�on prompted researchers Nicolas Guéguen and Céline Jacob from the Université de Bretagne-Sud along with Lubomir Lamy (2009) from Université de Paris-Sud to test the power of roman�c lyrics on 18–20 year old single females. And it turns out that at least one roman�c love song did make a difference.

Guéguen and Jacob were part of a research team that had already shown how roman�c music played in a flower shop led to male customers spending more money. This �me the researchers used ques�onnaires to pinpoint agreed-upon neutral and roman�c songs. They chose ‘Je l’aime à mourir,’ a well-known love song by French songwriter Francis Cabrel, and the neutral song ‘L’heure du thé,’ by Vincent Delerm. A group of young women separate from the main study rated 12 young male volunteers for a�rac�veness, and the researchers picked the one rated closest to “average” to help with the experiment.

The researchers then set up a scenario where the 87 females each spent �me in a wai�ng room with background music playing, before moving to a different room where the experimenter instructed her to discuss the difference between two food products with the young man. Once the experimenter returned, she asked them to wait for a few moments alone, and this gave the “average” male a chance to use his standard chat-up line: “My name is Antoine, as you know, I think you are very nice and I was wondering if you would give me your phone number. I’ll phone you later and we can have a drink together somewhere next week.”

The love song in the wai�ng room almost doubled Antoine’s chances of ge�ng a woman’s number— 52% of par�cipants responded to his advances under the influence of Francis Cabrel, whereas only 28% of those who had heard the “neutral” song by Vincent Delerm offered their details.

“Our results confirm that the effect of exposure to media content is not limited to violence and could have the poten�al to influence a high spectrum of behaviour,” says Guéguen. “The results are interes�ng for scien�sts who work on the effect of background music on individuals’ behaviour.”

The results also add weight to a general learning model proposed by Buckley and Anderson (2006) to explain the effect of media exposure. Their model states that media exposure in general, and not only aggressive or violent media, affects individuals’ internal states, which explains why prosocial media fosters prosocial outcomes.

Why did the music have this effect? It may be that, as shown in earlier research, the music induced posi�ve affect (in psychological terms, affect is the experience of feeling or emo�on). Posi�ve affect is associated with being more recep�ve to courtship requests. Alterna�vely, the roman�c content of the song may have acted as a prime that then led to displays of behaviour associated with that prime. In either case, further research is needed before the researchers will commit to wider generalisa�ons on the targeted use of love songs. But if you’re a hopeful single, awareness of the background music certainly won’t do any harm.

Epping, J. (2010, June 21). Love ballad leaves women more open to a date. Medical News Today. Retrieved from h�p://www.medicalnewstoday.com/ar�cles/192324.php (h�p://www.medicalnewstoday.com/ar�cles/192324.php)

Think about it:

1. In this experiment, the type of song (love song or neutral song) is confounded with at least one other variable. Try to iden�fy one. Do you think that this confounded variable would make a difference? How would you design a study that would overcome this?

2. Describe how demand characteris�cs might compromise the internal validity of this study. Can you think of any ways around this?

3. Toward the end of the ar�cle, the authors suggest that one explana�on for these results is that the roman�c music put the women into a more posi�ve mood, and that this in turn made them more recep�ve to the man. How could you design a study that tested this hypothesis?

4. Given the nature of the dependent variable in this study, would an ANOVA test be appropriate? What would be the more appropriate sta�s�cal test, and why?

Exploring Your Data

Now that we have F values, what do we do with them? At the end of conduc�ng an ANOVA, we have a yes-or-no answer to the following ques�on: Do our experimental groups have a systema�c effect on the dependent variable? The answer lets us decide whether to reject the null hypothesis, but it does not tell us everything we want to know about the data. In essence, a significant F value tells us that there is a significant difference between the groups, but it does not tell us what the difference is. If we conducted an ANOVA on our movie ra�ngs study, we would see a significant interac�on between actor and movie, but we would need to take addi�onal steps to determine the meaning of this interac�on.

In this sec�on, we will walk through the process of exploring and interpre�ng ANOVA results in order to make sense of the data. Our example is drawn from a published study by Newman, Sellers, and Josephs (2005), which was designed to explore the effects of testosterone on cogni�ve performance. Previous research had suggested that testosterone is involved in two types of complex human behavior. On one hand, people with higher testosterone tend to perform be�er on tests of spa�al skills, such as having to rotate objects mentally, and perform worse on tests of verbal skills, such as lis�ng all the synonyms for a par�cular word.

These pa�erns are thought to reflect the influence of testosterone on developing brain structures. On the other hand, people with higher testosterone are also more concerned with gaining and maintaining high status rela�ve to other people. Testosterone is correlated with a person’s posi�on in the hierarchy and tends to rise and fall when people win and lose compe��ons, respec�vely. In one study by the sociologist Alan Mazur and his colleagues, testosterone levels were measured before, during, and a�er a series of professional chess matches. They found that testosterone rose in both players in an�cipa�on of the compe��on, then rose even further in the winners, but plummeted in the losers (Mazur, Booth, & Dabbs, 1992).

Newman and colleagues (2005) set out to test the combina�on of these variables. Based on previous research, they hypothesized that people with higher testosterone would be uncomfortable when they were placed in a low-status posi�on, leading them to perform worse on cogni�ve tasks. The researchers tested this hypothesis by randomly assigning people to a high-status, low-status, or control condi�on and then administering a spa�al and a verbal test. This resulted in a 2 (testosterone: high or low) × 3 (condi�on: high status, low status, control) between-subject design, for a total of six groups. Note that “testosterone” in this study is a quasi-independent variable because it is measured rather than manipulated by the experimenters.

Once the results were in, the ANOVA revealed an interac�on between testosterone and status but no main effects. The results of the study are shown in Figure 5.9. These bars represent z-scores that combine the spa�al and verbal tests into one number. So what do these numbers mean? How do we make sense out of the pa�erns? This involves a combina�on of comparing means and calcula�ng effect sizes, as discussed next.

Figure 5.9: Exploring your data: Results from Newman et al. (2005)

Comparing Means

The first step in interpre�ng our results is to compare the various pairs of means within the design. This might seem counterintui�ve, since the whole point of the ANOVA is to test for effects without comparing individual means. Our goal, therefore, is to somehow explore differences in condi�ons without infla�ng Type I error rates. There are two strategies for achieving this balance.

Planned comparisons (also called a priori comparisons) involve comparing only the means for which differences were predicted by the hypothesis. In the experiment by Newman et al. (2005), the hypothesis explicitly stated that high-testosterone people should perform be�er in a high-status posi�on than a low-status posi�on. So a planned comparison for this predic�on would involve comparing two means with a t-test: high T, high status (the highest red bar); and high T, low status (the lowest yellow bar). Consistent with the researchers’ hypothesis, high-testosterone people did perform higher on both tests, t(27) = 2.35, p = 0.01, but only in a high-status posi�on. Type I errors are of less concern with planned comparisons because only a small number of theore�cally driven comparisons are being conducted.

If you refer to the graph of these results in Figure 5.9 and compare high- with low-testosterone people, you’ll no�ce another interes�ng pa�ern: In a high-status posi�on, high-testosterone people do be�er than low-testosterone people. But in a low-status posi�on, this pa�ern is reversed, and high-testosterone people do worse. However, the researchers did not predict these mean comparisons, so it would be chea�ng to do planned contrasts. Instead, they would use a second strategy called a post hoc comparison, which controls the overall alpha by taking into account the fact that mul�ple comparisons are being performed. In most cases, post hoc tests are allowed only if the overall F test result is significant.

One popular way to conduct post hoc tests while minimizing the error rate is to use a technique called a Bonferroni correc�on. This technique, named a�er the Italian mathema�cian who developed it (Carlo Emilio Bonferroni, 1892–1960), involves simply adjus�ng the alpha level by the number of comparisons that are performed. For example, imagine you want to conduct 10 follow-up post hoc tests to explore your data. The Bonferroni correc�on would involve dividing the alpha level (.05) by the number of comparisons (10), for a corrected alpha level of .005. Then, rather than using a cutoff of .05 for each test, you use this more conserva�ve Bonferroni-corrected value. Transla�on: Rather than accep�ng a Type I error rate of 5%, we are moving to a more conserva�ve 0.5% cutoff to correct for the number of comparisons that we are performing.

Another popular alterna�ve to the Bonferroni correc�on is called Tukey’s HSD (for Honestly Significant Difference). This test works by calcula�ng a cri�cal value for mean comparisons (the HSD) and then using this cri�cal value to evaluate whether mean comparisons are significantly different. This test manages to avoid infla�ng Type I error because the HSD is calculated based on the sample size, the number of experimental condi�ons, and the MSWG, which essen�ally tests all the

comparisons at once. (Note that ANOVA will tell us only that the means are different but not which of them, so for the purpose of iden�fying the means that are different from others, one can apply Tukey’s test.) In the study by Newman et al. (2005), both of these post hoc tests were significant: Compared with those low in testosterone, high-testosterone people did be�er in a high-status posi�on but worse in a low-status posi�on. This suggests that high testosterone magnifies the effect of tes�ng situa�ons on cogni�ve performance.

Calcula�ng Effect Size

As you’ll remember from our discussion in Chapter 2 (Sec�on 2.4 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.4#sec2.4) ), sta�s�cal significance is only part of the story; researchers also want to know how big the effects of their independent variables are. There are several ways to calculate effect size, but in general, bigger values mean a stronger effect. One of these sta�s�cs, Cohen’s d (named for Jacob Cohen, 1923–1998), is calculated as the difference between two means divided by their pooled standard devia�on. The resul�ng values can therefore be expressed in terms of standard devia�ons; a d of 1 means that the means are one standard devia�on apart. How big should we expect our effects to be? Based on his analyses of typical effect sizes in the social sciences, Cohen suggested the following benchmarks: d = .20 is a small effect; d = .40 is a moderate effect; and d = .60 is a large effect. In addi�on to these qualita�ve categories, effect size values can be interpreted in terms of standard devia�on units. So, a d of 1 is equivalent to a standard devia�on of 1. In other words, a large effect in social and behavioral sciences accounts for a li�le more than half of a standard devia�on.

In interpre�ng the results of their testosterone experiment, Newman and colleagues (2005) computed effect size measurements for two of the key mean comparisons. First, they compared high-testosterone people in the high- and low-status condi�ons; the size of this effect was d = .78. Second, they compared the high- and low-testosterone people in the low-status condi�on; the size of this effect was d = .61. Both of these effects fall into the “large” range based on Cohen’s benchmarks. More important, taken together with the mean comparisons, they help us to understand the way testosterone affects behavior. The authors concluded that cogni�ve performance stems from an interac�on between biology (testosterone) and environment (assigned status) such that high-testosterone people are more responsive to their status in a given situa�on. When they are placed in a high-status posi�on, they relax and perform well. But when placed in a low-status posi�on, they become distracted and perform poorly. This nuanced conclusion is possible only through an explora�on of the data, using mean comparisons and effect size measures.

5.6 Wrap-Up: Avoiding Error

As we wrap up our final chapter, it is worth thinking back to one of the key concepts in Chapter 2: Type I and Type II errors. Regardless of the research ques�on, the hypothesis, or the par�culars of the research design, all studies have the goal of making accurate decisions about the hypotheses. That is, we need to be able to correctly reject the null hypothesis when it is false and to fail to reject the null when it is true. But from �me to �me, and despite our best efforts, we make mistakes when we draw conclusions about our hypotheses, as summarized in Table 5.7. A Type I error, or “false posi�ve,” involves falsely rejec�ng a null hypothesis and ge�ng excited about an effect that is the result of chance. A Type II error, or “false nega�ve,” involves failing to reject the null hypothesis and missing an effect that is real and interes�ng. (If you need more of a refresher on these terms, refer back to Sec�on 2.4 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.4#sec2.4) , Hypothesis Tes�ng.)

Table 5.7: Review of Type I and Type II errors

Researcher’s Decision

Reject Null Fail to Reject Null

Null is FALSE Correct Decision Type II Error

Null is TRUE Type I Error Correct Decision

In this sec�on, we take a problem-solving approach to minimizing both of these errors in an experimental context. It turns out that each error is primarily under the researcher’s control at different stages in the research process, which means there are different strategies for reducing each error.

Avoiding Type I Error

Type I errors occur when results are caused by chance but are mistakenly interpreted as significant. We can generally reduce the odds of this happening by se�ng our alpha level at p < .05, meaning that we will get excited only about results that have less than a 5% chance of Type I error. However, Type I errors can s�ll occur as a result of using either extremely large samples or large numbers of sta�s�cal comparisons. Large samples can make small effects seem highly significant, so it is important to set a more conserva�ve alpha level in large-scale studies. And, as we have been discussing in this chapter, the odds of Type I error are compounded with each sta�s�cal test we conduct.

This means that Type I error is primarily under our control during sta�s�cal analysis—the smarter our sta�s�cs, the lower our odds of Type I error. We have covered several examples of “smart” sta�s�cs in this chapter: Instead of conduc�ng lots of t-tests, we use an ANOVA to simultaneously test for differences across the en�re design. Instead of conduc�ng t-tests to compare means a�er an ANOVA, we use a mix of planned contrasts (for comparisons that we predicted) and post hoc tests (for other comparisons we want to explore). There are also more advanced sta�s�cal techniques that take this a step further. For example, the mul�variate analysis of variance (MANOVA) sta�s�c analyzes sets of dependent variables to further reduce the number of individual tests. This approach is used when dependent variables represent different measures of a related concept, such as using heart rate, blood pressure, and muscle tension to capture the stress response. The MANOVA works, broadly speaking, by compu�ng a weighted sum of these separate dependent variables (called a canonical variable) and using this new variable as the dependent variable. If you are interested in learning more about this and other advanced sta�s�cal techniques, check out the excellent volume by James Stevens (2002), Applied Mul�variate Sta�s�cs for the Social Sciences.

Avoiding Type II Error

Type II errors occur when the results are significant but are mistakenly credited to chance. The primary sources of this error are small samples and bad design. Small samples may fail to capture enough variability and may therefore lead to nonsignificant p values in tes�ng an otherwise significant effect. Both large and small mistakes in experimental designs can add noise to the dataset, making it difficult to detect the real effects of independent variables.

This means that Type II error is primarily under the researcher’s control during the design process—the smarter the research designs, the lower the odds of Type II error. First, as we discussed in Chapter 2, it is rela�vely simple to es�mate the sample size needed for our research using a power calculator. These tools take basic informa�on about the number of condi�ons in the research design and the es�mated size of the effect and then es�mate the number of people needed to detect this effect.

Second, as we have discussed in every chapter, it is the experimenter’s responsibility to take steps to minimize extraneous variables that might interfere with the hypothesis test. Whether conduc�ng an observa�on, a survey study, or an experiment, the overall goal is to ensure that the variables of interest are the main cause of changes in the dependent variable. This is perhaps easiest in an experimental context because these designs are usually conducted in a controlled se�ng where the experimenter has control over the independent variables. Nonetheless, as discussed earlier in this chapter, many factors can threaten the internal validity of an experiment—from confounds to sample bias to expectancy effects. In essence, the more the researcher can control the influence of these extraneous variables, the more confidence he or she can have in the results of the hypothesis test.

Table 5.8 presents a summary of the informa�on in this sec�on, lis�ng the primary sources of Type I and Type II errors, as well as the �me period when these are under experimenter control.

Table 5.8: Summary—Avoiding error

Error Defini�on Main Source When Researcher Can Control

Type I False posi�ve Lots of tests; lots of people Conduc�ng sta�s�cs

Type II False nega�ve Bad measures; not enough people Designing experiments

5.7 Cri�quing a Quan�ta�ve Study

As discussed in Chapter 1 (Sec�on 1.5 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.5#sec1.5) , Searching the Literature) and Chapter 3 (Sec�on 3.3 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec3.3#sec3.3) , Cri�quing a Qualita�ve Study), the methods and guidelines for evalua�ng research studies are fairly detailed for both quan�ta�ve and qualita�ve studies. For any type of study, it is important to read the en�re ar�cle so that you can determine which parts of the study are valid, which parts can be considered acceptable but with cau�on, and which parts have significant limita�ons. As men�oned in Chapter 3, a study does not necessarily have valid and reliable results even if it claims that it does. Not all studies are worthy of being taken seriously, and it is important that you, as a consumer of psychological and other types of research, are able to iden�fy poten�al problems.

There are several factors to consider when evalua�ng the sec�ons of a quan�ta�ve research study or proposal. The following sec�ons discuss how to cri�cally evaluate the literature review, the purpose statement, the sampling methods, the procedures, the instruments, the results, and the discussion.

Evaluating the Literature Review Section

Similar to a qualita�ve study, the literature review sec�on of a quan�ta�ve study is meant to provide an adequate ra�onale for conduc�ng the study and to show what difference the study hopes to make in the field. Literature reviews should be thorough and comprehensive and contain relevant research on the topic being studied. They should also be objec�ve, unbiased in the selec�on of ar�cles being reviewed, and include previous research that relates to the current study. Considering these daun�ng requirements, the following ques�ons, adapted from Houser (2009), can help you to evaluate the literature review:

Do the researchers present an adequate ra�onale for conduc�ng the study? What is the significance of the study? What difference will it make to the field? Is the literature review thorough and comprehensive? Do the researchers demonstrate any poten�al biases in the literature review? Are all important concepts clearly defined by the researchers? Do the researchers clearly describe previous methods that are relevant to understanding the purpose for conduc�ng this study?

Evaluating the Purpose Statement and Hypotheses

As discussed in Chapter 3, the purpose statement does the following: provides the aim or intent of the study; the focus, methodology, popula�on, and geographical loca�on of the study; the specific area of the research and the central concepts or ideas of the study; jus�fica�on for the method of inquiry (qualita�ve, quan�ta�ve, or mixed); and a descrip�on of possible conclusions and their implica�ons. It is generally found in the Introduc�on, as the last paragraph before the Literature Review sec�on. Purpose statements can be wri�en in declara�ve form or in the form of a ques�on or ques�ons. They should include the type of research methodology and design used, as well as a descrip�on of the variables and popula�on studied. When evalua�ng the purpose statement, it is important to examine whether the purpose and research problem are researchable. Purpose statements and research problems are researchable when the variables of interest can be opera�onalized, or defined in a measurable and objec�ve way.

It is important to note that not all researchers use purpose statements in quan�ta�ve studies, but instead generally provide only hypotheses. It is not typical for a researcher to include the null hypothesis, as it can be inferred that every researcher has started out with one. Rather, alterna�ve types of research hypotheses are given. For example, some research will include alterna�ve nondirec�onal hypotheses. These indicate that there is enough evidence from the literature to suggest a difference, but not enough to predict the direc�on of that difference. Another hypothesis is the alterna�ve direc�onal hypothesis, which suggests a difference and the direc�on of that difference.

Hypotheses guide researchers in selec�ng appropriate sta�s�cal analyses, so there should be consistency between the hypotheses and sta�s�cal tests chosen. The following ques�ons, adapted from Houser (2009), are provided to help you to evaluate the purpose statement:

Does the ar�cle clearly state the purpose statement? Is the purpose statement clearly based on the argument developed in the literature review? Are the variables of interest (i.e., independent and dependent) clearly iden�fied in the problem statement? What type of hypothesis is presented? Is there a clear ra�onale for the direc�on of the hypothesis, based on the literature review? Is the hypothesis testable and falsifiable? Is the hypothesis consistent with current theories or research?

Evaluating the Methods Section: Sampling

The Sampling sec�on includes thorough, detailed informa�on about the sample that was chosen and the sampling techniques or methods that were used to select it. Descrip�ons of the sample should include all relevant demographic characteris�cs (e.g., age, ethnicity, gender) as well as the size. Unlike qualita�ve approaches, which do not have hard-and-fast rules regarding sample size, quan�ta�ve approaches o�en require much larger samples. As noted in Chapter 4 (Sec�on 4.3 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec4.3#sec4.3) ), Gay, Mills, and Airasian (2009) provided the following guidelines for determining a sufficient sample size based on the size of the popula�on:

For popula�ons that include 100 or fewer individuals, the en�re popula�on should be sampled. For popula�ons that include 400–600 individuals, 50% of the popula�on should be sampled. For popula�ons that include 1,500 individuals, 20% of the popula�on should be sampled.

For popula�ons that are 5,000 or larger, about 8% of the popula�on should be sampled. (Note, however, that it is possible to have a sample that is so large that it does not jus�fy the increased costs, and may produce significant differences that are too small to be meaningful).

The sampling techniques employed should be discussed, including detailed informa�on about how the sample was selected and what methods were used (e.g., simple random sampling, stra�fied sampling, cluster sampling). The following are a few of the most common sampling procedures used in quan�ta�ve studies. These are discussed in more detail in Chapter 4, Sec�on 4.3 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec4.3#sec4.3) , Sampling From the Popula�on.

Simple Random Sampling: The researcher randomly picks study par�cipants from a list of everyone in the popula�on. Stra�fied Random Sampling: The researcher breaks the sampling frame into subgroups and then samples a random number from each subgroup. Cluster Sampling: The researcher randomly selects intact groups. Systema�c Random Sampling: The researcher selects every nth par�cipant from a finite list of those in the popula�on.

Sampling errors and sampling biases should also be considered. Sampling errors can occur when the sample selected is not representa�ve of the popula�on. This generally occurs when smaller sample sizes are used. Sampling bias occurs when the researcher purposely selects par�cipants in a par�cular way that introduces bias into the sample. Convenience samples typically contain sampling bias because the samples are not usually representa�ve of the en�re popula�on.

The following ques�ons, adapted from Houser (2009), can help you evaluate the sampling methods sec�on:

What type of sampling method is used? Are the sampling procedures consistent with the purpose and hypotheses? Are relevant demographic characteris�cs of the sample clearly iden�fied? Do the methods of sample selec�on provide a good representa�ve sample, based on the popula�on? Are there any apparent biases in the selec�on of the sample? Is the sample large enough for the study proposed? How has size been jus�fied? To whom can study results be generalized?

Evaluating the Methods Section: Procedures

The Procedures sec�on describes in detail everything that was conducted in the study. For quan�ta�ve studies, it discusses the study variables, how informed consent was obtained, how the par�cipants were assigned to groups, and how the researcher controlled for extraneous or confounding variables. This sec�on should clearly describe the independent variable as well as give details about whether a control group existed, and if so, how it was treated. Addi�onally, it should cover how the par�cipants were randomly assigned to the experimental group and control group. When evalua�ng the procedures sec�on, one should assess whether there are any threats to the internal or external validity of the study. Furthermore, it is important to examine whether the research design was appropriate for the study and whether it was consistent with the purpose and research ques�ons. The following ques�ons, adapted from Houser (2009), can assist in the evalua�on of the Procedures sec�on:

What type of research design was used? Was the research design consistent with the purpose and hypotheses? Did the researcher provide a detailed descrip�on of what was conducted? Did the researcher introduce any bias in the procedures used? Was the independent variable clearly defined? Were there any threats to the internal validity of the study? Were there any threats to the external validity of the study?

Evaluating the Methods Section: Instruments

The Instruments sec�on describes in detail the types of instruments and measures that were used to collect the data. In quan�ta�ve research, a variety of common instruments are used: direct observa�on and behavioral measures, self-report inventories, ra�ngs of peoples’ behaviors, ques�onnaires, physiological approaches, interviews, and standardized assessments.

When evalua�ng the Instruments sec�on, consider whether the instruments were appropriate for the study and the sample and whether there were any limita�ons in the types of instruments selected. As noted by Houser (2009), “An important considera�on in assessing the objec�vity and quality of a dependent measure is the extent to which the procedures and scoring methods are standardized” (p. 167). A test is not highly reliable or valid simply because it is standardized. Whenever evalua�ng instruments, it is important to assess the test’s reliability and validity as well as its use in various popula�ons. It is also important to assess whether the tests selected were consistent with the sample selected, the research problem, and the hypotheses. Studies that employ standardized assessments or standardized administra�on and scoring procedures generally obtain more reliable and valid data collec�on and results. The following ques�ons, adapted from Houser (2009), can help guide the evalua�on of the Instruments sec�on:

Is there a clear and adequate descrip�on of the instrument (data collec�on measures) used? What types of instruments were used in the study (self-report inventories, ra�ngs, etc.)? Are the instruments standardized? What are some of the poten�al problems or limita�ons with the types of instruments used? Does the instrument appear to be appropriate for the sample? Is the instrument consistent with the hypotheses? Is there a discussion about the reliability and validity of the instruments used?

Evaluating the Results Section

The Results sec�on describes the findings of the study. Unlike qualita�ve studies, which focus more on descrip�ons about the findings, quan�ta�ve studies are usually presented mathema�cally through sta�s�cal analyses.

Two forms of sta�s�cs are presented in the Results sec�on: descrip�ve and inferen�al. Descrip�ve sta�s�cs summarize the data collected from the sample, such as demographic informa�on. Inferen�al sta�s�cs use sta�s�cal methods or analyses to test hypotheses. When evalua�ng the Results sec�on, consider whether appropriate sta�s�cs were used and whether those sta�s�cs were consistent with the purpose of the study and hypotheses. The following ques�ons, adapted from Houser (2009), will help:

What descrip�ve sta�s�cs were used? Do the descrip�ve sta�s�cs adequately describe the sample? What types of inferen�al sta�s�cs were used? Are the inferen�al sta�s�cs used adequate to answer the hypotheses? Is the research design consistent with the sta�s�cal methods used?

Evaluating the Discussion Section

Whether evalua�ng a quan�ta�ve or qualita�ve study, the Discussion sec�on summarizes the purpose of the research, as well as the findings’ implica�ons for future research and actual prac�ce. Addi�onally, the Discussion sec�on includes alterna�ve explana�ons and poten�al limita�ons of the findings. The following ques�ons, adapted from Houser (2009), can shed light on the Discussion sec�on:

Did the researchers clearly restate the purpose and hypotheses? Did the researchers clearly discuss the implica�ons of the findings and how they relate to theories, other findings, and actual prac�ce? Did the researchers provide alterna�ve explana�ons of the findings? Has there been appropriate generaliza�on of significant findings beyond the sample to the popula�on? Did the researchers iden�fy poten�al limita�ons of the study and the results? Did the researchers iden�fy possible direc�ons for future research?

5.8 Mixed-Methods Research Designs

This chapter and the previous three have discussed the differences between qualita�ve and quan�ta�ve research methods. From your readings, it may appear that these methods are totally exclusive and are never mixed or combined in a study. However, this is not actually the case. In recent years, researchers have started to u�lize both approaches in their designs to provide a more comprehensive understanding of research problems. Mixed methods research is fairly new, and the integra�on and mixing of various qualita�ve and quan�ta�ve approaches can be tricky. As a result, books and ar�cles have been published that provide comprehensive overviews of this type of inquiry.

Mixed-methods research uses the process of triangula�on, or the combina�on of mul�ple research methods to study a research problem. In order to successfully achieve this, several approaches can be employed, including scaling, reliability, and convergent valida�on. Scaling involves conver�ng qualita�ve data into quan�ta�ve or numeric form. This involves crea�ng codes and themes in the qualita�ve data collected and then coun�ng the number of �mes those codes or themes occur (Creswell, 2009). This approach allows researchers to compare the qualita�ve results with the quan�ta�ve ones. Reliability involves the use of similar methods to measure the same thing. For example, researchers might u�lize unstructured (qualita�ve) and structured (quan�ta�ve) interviews to examine the experiences of demen�a caregivers. Finally, convergent valida�on involves u�lizing different but complementary methods (Houser, 2009). For example, a researcher might use a survey to collect data about AIDS preven�on knowledge and then follow up with an unstructured or semistructured interview. In each of these triangula�on methods, the researcher is combining qualita�ve and quan�ta�ve approaches to gain a more thorough picture of the research problem.

Mixed-Methods Strategies

Mixed-methods designs generally u�lize one of six mixed strategies or models: sequen�al explanatory, sequen�al exploratory, sequen�al transforma�ve, concurrent triangula�on, concurrent embedded, or concurrent transforma�ve.

The sequen�al explanatory strategy is one of the most popular mixed methods used, especially among quan�ta�ve researchers. This type of strategy includes two phases: The first phase involves the quan�ta�ve collec�on and analysis of data, and the second phase involves the qualita�ve collec�on and analysis of data that build on the results of the quan�ta�ve data. For example, a researcher might administer a standardized instrument to measure depression symptoms and then follow up with a semistructured interview to explore the par�cipants’ personal experiences with depression. In this way, the interview would supply addi�onal informa�on about why the par�cipants responded certain ways on the standardized instrument. This type of strategy is generally used to further explain and interpret the quan�ta�ve results.

The sequen�al exploratory strategy is similar to the sequen�al explanatory strategy except that the phases are switched. In this type of strategy, the first phase is the qualita�ve collec�on and analysis of data, whereas the second phase is the quan�ta�ve collec�on and analysis of data that build upon the qualita�ve results. Thus, the quan�ta�ve results are used to further explain and interpret the qualita�ve results.

The sequen�al transforma�ve strategy is similar to the other two sequen�al strategies in that it includes two phases; however, the sequence of the phases is decided by the needs of the researcher. Thus, the researcher selects methods that will best support his or her theore�cal perspec�ve.

The concurrent triangula�on strategy involves the researcher collec�ng both qualita�ve and quan�ta�ve data concurrently and then comparing them for similari�es and differences (Creswell, 2009). Concurrent triangula�on is o�en used to support or disconfirm quan�ta�ve results. Similar to concurrent triangula�on, the concurrent embedded strategy includes collec�ng qualita�ve and quan�ta�ve data concurrently; however, a primary method guides the research, and a secondary method provides a suppor�ve role. The secondary method is given less priority and is embedded or “nested” within the primary method. For example, a primary method might include assessing treatment outcomes (a quan�ta�ve approach) while a secondary method explores the processes people experience during the study (a qualita�ve approach) (Creswell, 2009).

The last model is concurrent transforma�ve strategy. It resembles the sequen�al transforma�ve strategy in that the methods it employs are based on the theore�cal perspec�ve of the researcher, but the data are collected concurrently. And, similar to the concurrent embedded strategy, this strategy also u�lizes a primary method and a secondary method that is embedded within the primary method.

The following are examples of mixed-methods designs.

Example 1: Sequen�al Explanatory Design

Celes�ne Pea (2012) conducted research to examine whether environmental factors (e.g., human, sociocultural, design) affect inquiry-based teaching. She employed a mixed methods, sequen�al explanatory design with a mul�case research approach to conduct the study. A survey of middle-school science teachers was followed by case studies of three teachers from the survey pool (two females and one male). Here is an excerpt describing the study’s methods:

Data Collec�on and Analysis

Quan�ta�ve methods: Data collec�on and analysis. An open-ended Likert-style survey was used to collect quan�ta�ve data. The survey included a sec�on for demographics informa�on and individual variables that included gender, years of teaching science, degree, cer�fica�on in subject taught, and semester credits in science. The survey data were prepared for computa�on. Then, descrip�ve sta�s�cs, a paired i-test, and analysis of variance (ANOVA) were used to compute the survey data. The outcomes were analyzed and interpreted to build a cross-case report.

Qualita�ve methods: Data collec�on and analysis. Interviews and observa�ons were used to collect qualita�ve data. Protocols were developed for each of these methods. Each teacher was interviewed before, during, and a�er being observed using a semi-structured approach. An exit interview was also conducted at the end of the observa�on sessions. The interviews lasted 30–60 minutes. (Pea, 2012, pp. 36–37)

No�ce how the data collec�on sec�on is organized into quan�ta�ve methods and qualita�ve methods subsec�ons. This is common prac�ce in sequen�al studies, whose aim is to conduct separate data collec�ons, with the la�er elabora�ng on either the original qualita�ve or quan�ta�ve findings.

Example 2: Concurrent Embedded Design

Brune�o and Farr-Wharton (2008) conducted a study to . . .

compare the impact of the quality of employee enablers (in this case measured by the quality of the communica�on rela�onship between local government employees and supervisors) with the ideal “high quality social capital network” on the quality of outputs (such as employees’ level of ambiguity in rela�on to customers and/or their level of produc�vity). (p. 37)

The following is an excerpt describing the study’s methods.

Methodology

This study uses mixed methods—hence, both qualita�ve and quan�ta�ve research tools are used (Rocco et al., 2003). However, a concurrent nested strategy is used in that the primary source of evidence used for the findings is quan�ta�ve—in both data collec�on and analysis (Cresswell, 2003). Therefore data obtained from qualita�ve research methods is used only to support trends iden�fied using quan�ta�ve analysis and aid triangula�on, which in turn, increases the external validity and achieves greater understanding of the phenomenon (Rocco et al., 2003). Quan�ta�ve data was obtained using a cross-sec�onal, survey-based, self-report strategy. While there is some cri�cism of this method (because of the problems of common method variance and same source bias), it is in this case, the best way of iden�fying the intangible values of the organisa�on by asking public sector local government employees about their percep�ons of the quality of informa�on-sharing. On the other hand, the role of qualita�ve research methods (such as interviews) is to generate data that can explain the reasons for the trends evident from analyzing the quan�ta�ve results (Yin, 1994; Rocco et al., 2003). Once the data was collected, the findings were compared with the findings of previous research analysed in the literature review, that is, using a pa�ern-matching logic (Yin, 1994). (Brune�o & Farr-Wharton, 2008, p. 44 [footnotes omi�ed])

No�ce how the data in this concurrent embedded study were not described in separate quan�ta�ve and qualita�ve sec�ons as in the sequen�al study in Example 1. This is because concurrent studies seek to converge or find similari�es among the data and results.

Regardless of the type of strategy, it is important to remember that u�lizing both quan�ta�ve and qualita�ve methods can be �me-consuming and expensive. However, it also can provide more rigor, validity, and u�lity to a study. For further informa�on on mixed methods designs, review the Handbook of Mixed Methods in the Social and Behavioral Sciences by Tashakkori and Teddlie (2003) or the Journal of Mixed Methods Research.

Summary

In this chapter, we have focused on experimental designs, the last group of specific research designs, and the highest point on the con�nuum of control. The primary goal of experimental designs is to explain behavior in terms of cause and effect. We began with an overview of experimental terminology and the key features of experiments. Experiments can be dis�nguished from other research designs by three key features. First, the researcher manipulates a variable, giving him or her a fair amount of confidence that the independent variable causes changes in the dependent variable. Second, the researcher controls the environment, ensuring that everything about the experimental context is the same for different groups of par�cipants—except for the level of the independent variable. Third, the researcher has the power to assign par�cipants to condi�ons, using random assignment. This process helps to ensure that standing differences among par�cipants (e.g., in mood, mo�va�on, intelligence) are balanced out across the experimental condi�ons.

Next, we covered the concept of experimental validity. In evalua�ng our experiments, we have to take into account both internal validity (the extent to which the independent variable is the cause of changes in the dependent variable) and external validity (the extent to which the results can be generalized beyond the specific laboratory se�ng). Several factors can threaten internal validity, including experimental confounds, selec�on bias, and expectancy effects. The common thread among these threats is that they add noise to the hypothesis test and cast doubt on the direct connec�on between independent and dependent variable. External validity involves two components: the realism of the study and the generalizability of the findings. Psychology experiments are designed to study real-world phenomena, but some�mes compromises have to be made to study these phenomena in the laboratory. The balance is o�en achieved via mundane realism, or replica�ng the psychological condi�ons of the real phenomenon. Finally, we have more confidence in the findings of our study when they can be replicated, or repeated in different se�ngs with different measures.

We also discussed the three types of experimental designs: pre-experiments, quasi-experiments, and true experiments. In true experiments, researchers have the most control over extraneous or confounding variables and therefore are more likely to uncover cause-and-effect rela�onships. In addi�on, true experiments always include random assignment and manipula�on of the independent variable, whereas quasi-experiments and pre-experiments do not include random assignment.

In designing the nuts and bolts of true experiments, we have to make decisions about both the nature and number of independent variables. First, our designs can be described as between-subject, within-subject, or mixed. In a between-subject design, par�cipants are subjected to only one experimental condi�on and receive only one combina�on of the independent variables. In a within-subject design, par�cipants find themselves in all experimental condi�ons and receive all combina�ons of the independent variables. A mixed design combines between- and within-subject variables. Second, our designs can be described as either one- way or factorial. One-way designs consist of only one independent variable with at least two levels; factorial designs consist of at least two independent variables, each having at least two levels. In a factorial design, there are several results to examine: the main effect of each independent variable plus the interac�on, or combina�on, of the independent variables.

We also discussed the process of analyzing experimental data, using the analysis of variance (ANOVA) sta�s�c. This test works by simultaneously comparing sources of variance and, therefore, avoids the risk of inflated Type I error. The ANOVA (or F) is calculated as a ra�o of systema�c variance to error variance, or, more specifically, of between-group variance to within-group variance. The bigger this ra�o, the more experimental manipula�ons contribute to overall variability in scores. However, the F sta�s�c only tells us that differences exist in the design, and further analyses are necessary to explore these differences. We walked through an example from a published study, discussing the process of comparing means and calcula�ng effect sizes. In comparing means, we use a mix of planned contrasts (for comparisons that we predicted) and post hoc tests (for other comparisons we want to explore).

We then covered two recurring concepts: Type I error (false posi�ve) and Type II error (false nega�ve) and reviewed ways to minimize such errors. These errors interfere with our broad goal of making correct decisions about the status of our hypothesis. Type I errors are inflated primarily by large samples and performing lots of sta�s�cal analyses. Consequently, these errors are under the experimenter’s control at the data analysis stage. Type II errors are inflated primarily by small samples and flaws in the experimental design. Consequently, these errors are under the experimenter’s control at the design and planning stage.

An explana�on of how to cri�que a quan�ta�ve study was also provided. Finally, we concluded the chapter with a review on mixed methods designs and the six strategies used to incorporate both qualita�ve and quan�ta�ve techniques. Although u�lizing mixed methods approaches can be �me-consuming and costly, they can also increase the validity and u�lity of a study.

Key Terms

Click on each key term to see the defini�on.

between-group variance (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The variance explained by differences in the independent variables.

between-subject design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Experimental design in which each group of par�cipants is exposed to only one level of the independent variable.

Bonferroni correc�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A post hoc test that involves adjus�ng the alpha level by the number of comparisons to set a more conserva�ve cutoff point.

carryover effect (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

When effects of one level of independent variable are present when another level is introduced, making it difficult to separate the effects of different levels.

conceptual replica�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Tes�ng the rela�onship between conceptual variables using new opera�onal defini�ons.

concurrent embedded strategy (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A mixed methods strategy that includes collec�ng qualita�ve and quan�ta�ve data concurrently; however, a primary method guides the research, and a secondary method plays a suppor�ve role.

concurrent transforma�ve strategy (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A mixed methods strategy that resembles the sequen�al transforma�ve strategy in that the methods are based on the theore�cal perspec�ve of the researcher, but the data is collected concurrently. Also similar to the concurrent embedded strategy, this strategy u�lizes a primary method and a secondary method that is embedded within the primary method.

concurrent triangula�on strategy (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A mixed methods strategy that involves the researcher collec�ng both qualita�ve and quan�ta�ve data concurrently and then comparing the data to see whether there are any similari�es or differences.

condi�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

One of the versions of an independent variable, forming different groups in the experiment; in a factorial design, refers to the groups formed by combina�ons of independent variables.

confounding variable (confound) (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A variable that changes systema�cally with the independent variable; in the case of correla�on studies, known as the third variable.

construc�ve replica�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Re-crea�on of the original experiment that adds elements to the design; usually designed to rule out alterna�ve explana�ons or extend knowledge about the variables under study.

control condi�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Group within the experiment that does not receive the experimental treatment.

convergent valida�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A form of triangula�on that involves u�lizing different but complementary methods to measure the same construct of interest.

counterbalancing (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Varia�on of the order of presenta�on among par�cipants to reduce order effects.

cover story (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A misleading statement to par�cipants about what is being studied to prevent effects of demand characteris�cs.

demand characteris�c (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Cue in the study that leads par�cipants to guess the hypothesis.

differen�al a�ri�on (mortality threat) (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Loss of par�cipants, who drop out of experimental groups for different reasons.

environmental manipula�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Changing some aspect of the experimental se�ng.

exact replica�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Re-crea�on of the original experiment as closely as possible to verify the findings.

experimental condi�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Group within the experiment that receives a treatment designed to test a hypothesis.

experimental design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Design whose primary goal is to explain causes of behavior.

experimenter expectancies (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

When researchers see what they expect to see, leading to subtle bias in favor of their hypotheses; threat to internal validity.

external validity (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A metric that assesses generalizability of results beyond the specific condi�ons of the experiment.

extraneous variable (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Variable that adds noise to a hypothesis test.

factorial design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A design that has two or more independent variables, each with two or more levels.

factorial nota�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A system for describing the number of variables and levels in experimental designs.

fa�gue effect (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Decline of par�cipants’ performance as a result of repeated tes�ng.

generalizability (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The extent to which results extend to other studies, using a wide variety of popula�ons and of opera�onal defini�ons; some�mes referred to as popula�on validity.

history effect (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The influence on research par�cipants by events in the outside world, such as poli�cal upheavals and natural disasters.

instruc�onal manipula�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Changing the way a task is described to change par�cipants’ mind-sets.

interac�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The combined effect of variables in a factorial design; when the effects of one independent variable are different depending on the levels of the other independent variable.

internal validity (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A metric that assesses the degree to which results can be a�ributed to independent variables.

invasive manipula�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Taking measures to change internal, physiological processes; usually conducted in medical se�ngs.

level (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Another way to describe the versions of an independent variable; describes the specific circumstances created by manipula�ng a variable.

main effect (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The effect of each independent variable on the dependent variable, collapsing across the levels of other variables.

marginal mean (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The combined mean of one factor across levels of another factor.

matched random assignment (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A varia�on on random assignments; ensures that an important variable is equally distributed between or among the groups; the experimenter obtains scores on an important matching variable, ranks par�cipants on this variable, and then randomly assigns par�cipants to condi�ons.

mean square (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A component of the F sta�s�c; corrects the sum of squares for sample size by dividing by the appropriate degrees of freedom; resul�ng values represent the average devia�on from the sample mean, corrected for sample size.

mixed design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Experimental design that contains at least one between-subject variable and at least one within-subject variable.

mul�variate analysis of variance (MANOVA) (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A sta�s�c that analyzes sets of dependent variables to reduce the number of individual tests.

mundane realism (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Research that replicates the psychological condi�ons of the real-world phenomenon; criterion for judging external validity. Some�mes referred to as ecological validity.

one-shot case study (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A type of pre-experimental design that involves using one group, introducing an interven�on, and then administering a pos�est observa�on to determine the effects of the interven�on.

one-way design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A design that has only one independent variable, with two or more levels to the variable.

order effect (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Modera�on of the effects because of the order in which levels occur.

par�cipant replica�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Repe��on of the study with a new popula�on of par�cipants; usually driven by a compelling theory as to why the two popula�ons differ.

placebo control (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Group added to a study to separate placebo effects from the actual effects of a drug; mimics the experimental condi�on in every way but one.

placebo effect (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Change resul�ng from the mere expecta�on that change will occur.

planned (a priori) comparison (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Comparison that involves comparing only the means for which differences were predicted by the hypothesis.

post hoc comparison (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Comparison that controls the overall alpha by taking into account that mul�ple comparisons are being performed; usually allowed only if the overall F test result is significant.

prac�ce effect (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Improvement of par�cipants’ performance as a result of repeated tes�ng.

pre-experimental design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Research study design that confers virtually no control and involves studying a single group or unbalanced groups and then introducing an interven�on or treatment.

quasi-experimental design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Research study design that gives researchers some control and may or may not include a control group but in which the par�cipants are not randomly assigned to groups.

quasi-independent variable (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Existent difference used to divide par�cipants in an experimental context; referred to as “quasi” because variables are being measured, not manipulated, by the experimenter.

random assignment (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A technique for assigning par�cipants to condi�ons; before par�cipants arrive, the experimenter makes a random decision about each par�cipant’s placement in a group.

reliability (qualita�ve) (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A form of triangula�on that involves the use of similar methods to measure the same construct of interest.

replica�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Repe��on of research results in different contexts and/or different laboratories.

selec�on bias (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

When groups are different before experimental manipula�on; problema�c because existent differences might be the driving factor behind the results.

sequen�al explanatory strategy (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A mixed methods strategy that includes two phases: the first phase involving the quan�ta�ve collec�on and analysis of data and the second phase involving the qualita�ve collec�on and analysis of data that build on the results of the quan�ta�ve data.

sequen�al exploratory strategy (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A mixed methods strategy similar to the sequen�al explanatory strategy except that the phases are switched. The first phase involves the qualita�ve collec�on and analysis of data, whereas the second phase involves the quan�ta�ve collec�on and analysis of data that build upon the qualita�ve results.

sequen�al transforma�ve strategy (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A mixed methods strategy that includes two phases; however, the sequence of the phases is decided by the needs of the researcher. Thus, the researcher selects methods that will best support his or her theore�cal perspec�ve.

sum of squares (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The sum of the squared devia�ons between individual scores and the overall sample mean.

third variable (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

See confounding variable.

triangula�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The combina�on of mul�ple research methods to study a research problem.

true experimental design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Research study design that gives researchers maximum control; involves randomly assigning par�cipants and manipula�ng the independent variable; includes randomized group selec�on and assignments, a control group, and a large degree of control over confounding variables.

Tukey’s HSD (Honestly Significant Difference) (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A post hoc test that calculates a cri�cal value for mean comparisons (the HSD) and then uses this cri�cal value to evaluate whether mean comparisons are significantly different.

unrelated-experiments technique (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

An experimental technique for preven�ng effects of demand characteris�cs; leading par�cipants to believe that they are comple�ng two experiments during one session; experimenter can use this method to present the independent variable during the first experiment and measure the dependent variable during the second experiment.

within-group variance (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The total variance that occurs across condi�ons.

within-subject design (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Experimental design in which each group of par�cipants is exposed to all levels of the independent variable.

Chapter 5 Flashcards

Research Scenarios: Try It

Scenario 5

Imagine that you're working with a pharmaceu�cal company to determine whether exercise enhances the effect of a new an�-depressant medica�on. Prior research by this company has suggested that the drug appears to work be�er among people who exercise regularly than those who are sedentary. To determine this, the researchers recruited depressed individuals who either had a regular exercise rou�ne or were sedentary, and randomly assigned them to an ac�ve drug condi�on or a placebo control condi�on. A�er six months of treatment, they found that the medica�on reduced depression far more drama�cally among individuals who exercised than in the sedentary individuals.

1. The pharmaceu�cal researchers concluded on this basis that exercise enhances the effect of their drug, but you spot a flaw in their design that compromises its internal validity. What is it?

a. Confounding variables b. Differen�al a�ri�on c. Expectancy effects d. Placebo effect

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The experiment conducted by the pharmaceu�cal company is vulnerable to several poten�al (a) confounding variables. Because the par�cipants were already exercising or not exercising, it is quite possible that the two groups differed in other important ways, such as mo�va�on, that could explain the results. For example, the individuals who already exercised regularly may simply have a be�er ability to s�ck to a rou�ne—and consequently were more conscien�ous about taking their medica�on! Based on the descrip�on above, there is no clear evidence that differen�al a�ri�on, expectancy effects, or placebo effects influenced the results.

2. Suppose you design a construc�ve replica�on of the original study which corrects the problem you found by recrui�ng only individuals without regular exercise programs and randomly assigning them to the exercise and no-exercise groups. Therefore, par�cipants are randomly assigned to one of four condi�ons: (1) Exercise + Drug, (2) No Exercise + Drug, (3) Exercise + Placebo, (4) No Exercise + Placebo. You measure depression before and a�er treatment. What are the independent variable(s) (IV) and dependent variable(s) (DV) of your experiment?

a. 1 DV: depression; 3 IV: exercise, drug, �me b. 1 DV: depression; 2 IV: exercise, drug c. 2 DV: depression, �me; 2 IV: exercise, drug d. 3 DV: exercise, depression, drug; 1 IV: �me

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The redesigned study has (a) one dependent variable (depression) and three independent variables (exercise, drug, �me). In other words, depression depends on three factors: whether the par�cipant exercised or not, whether the par�cipant took the ac�ve drug or not, and whether the depression is measured before or a�er treatment. This is a mixed factorial design, because exercise and drug are between-subjects factors, and �me is a within subjects factor (all par�cipants are measured twice, before and a�er treatment).

3. To ensure that pa�ents in the Exercise condi�ons actually work out regularly, you require them to keep an exercise journal and come into the lab three �mes a week to use a treadmill for 45 minutes under medical supervision. What should you instruct the No Exercise condi�on do?

the variance

Click card to see term 👆

a. Not to begin exercise b. Record all physical ac�vity c. Come into the lab three �mes a week for 45 minutes of quiet si�ng under medical supervision d. All of the above

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

To provide a proper control for your exercise condi�on, it would be ideal to have the No Exercise condi�on do (d) all three of the above op�ons. It would be best to not only instruct them specifically not to begin exercising, but also to have par�cipants keep a physical ac�vity log and come into the lab as o�en as the other condi�on. These elements ensure that your par�cipants not only are not exercising, but also get as much medical care and a�en�on as the exercise condi�on.

4. Suppose you complete your study and begin to analyze your data. To simplify the analysis, you analyze the change in depression levels from before to a�er treatment. You obtain the following results for your change in depression scores (more nega�ve numbers mean greater drops in depression levels):

Drug Placebo Excercise -20 -10 No Excercise -10 -10

Assuming that your sta�s�cal tests are significant, what are the main effects of exercise and drug condi�on in this study?

a. Main effect of drug, no main effect of exercise b. Main effect of exercise, no main effect of drug c. No main effect of either drug or exercise d. Main effects of both drug and exercise

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

To evaluate main effects, calculate the marginal means:

The marginal mean for the exercise group is calculated by averaging together the levels in the drug and placebo group, (-20) + (-10)/2 = -15. Complete the same calcula�on for the no exercise group, (-10) + (-10)/2 = -10. The marginal mean for the drug group can be calculated by averaging together the levels of the exercise and no exercise group, (-20) + (-10)/2= -15. Complete the same calcula�on for the placebo group, (-10) + (-10)/2 = -10.

The results above suggest that there is (d) a main effect of both drug and exercise. Using the marginal means, if we ignore the effect of drug, those in the Exercise condi�on dropped 15 points on the depression scale, but the No Exercise group only dropped 10 points. Similarly, if we ignore the effect of exercise, those in the Drug condi�on dropped 15 points while the Placebo condi�on dropped 10 points.

5. Looking at the results more closely, you realize that the two main effects are not the whole story. Select the op�on that best states the nature of the interac�on:

a. the effect of the drug depends on whether the par�cipants exercised or not b. the effect of exercise depends on whether the par�cipants had the ac�ve drug or placebo c. both (a) and (b) are true d. the effect of the drug is independent of the effect of exercise

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The correct answer is (c). The interac�on effect can actually be stated two ways, because the effect of one variable depends on the level of the other variable. Therefore, among those who exercised, the drug worked be�er than the placebo (20 versus 10 points), whereas among those who didn't exercise, the drug did not help at all (both dropped 10 points). Conversely, among those who took the ac�ve drug, the exercisers became less depressed than did the sedentary group (20 versus 10 points), whereas among those who took the placebo, exercise did not make a difference (both dropped 10 points).

Apply Your Knowledge

1. When ANOVA is used to analyze data from experiments with two independent variables (A and B), the total variability (MSTOTAL) is composed of four parts, or

components. What are they? (Wri�ng the equa�on is fine.)

2. List and briefly describe the three dis�nguishing features of an experiment. a. b. c.

3. List the three types of expectancy effect that can affect experimental results, and name one way to avoid each type. a. b. c.

4. The following designs are described using factorial nota�on. For each one, state (a) the number of variables in the design, (b) the number of levels each variable has, and (c) the total number of experimental condi�ons.

3 × 3 × 3

a. b. c.

2 × 3 × 4

a. b. c.

4 × 4

a. b. c.

2 × 2 × 2 × 2

a. b. c.

5. Forty students were asked to rate two authors according to their knowledge of certain topic areas. Each student was given two passages to read. In one passage (“Brain”), the author discussed the roles of various brain structures in perceptual–motor coordina�on. In the second passage (“Mo�va�on”), the author described ways to enhance mo�va�on in preschool children. For half the students, both passages were wri�en by male authors. For the other half of the students, both passages were wri�en by a female author. A�er reading the passages, students rated the authors’ knowledge of their topic areas on a scale ranging from 1 (Displays very li�le knowledge) to 10 (Displays a thorough knowledge).

Male Author Female Author Brain 9 4 Mo�va�on 6 7

a. Iden�fy the following informa�on about the design: (1). Describe the design using factorial nota�on (e.g., 4 × 3). (2). Iden�fy the total number of condi�ons. (3). Iden�fy the design (circle one):    between-subject  within-subject  mixed

b. The data from this study are analyzed and the results indicate the following F values: (1). Topic Area: F = .91 (p = .87). (2). Author’s Gender: F = 3.14 (p < .05). (3). Topic Area × Author’s Gender: F = 12.46 (p < .01).

c. Interpret these results in terms of the three possible effects. That is, for each effect, state whether it is significant and then interpret the sta�s�cs in English. (1). Is there a main effect of Topic Area? (circle one)  YES  NO   Interpreta�on in English: (2). Is there a main effect of Author Gender? (circle one)  YES  NO   Interpreta�on in English: (3). Is there an interac�on between Topic Area and Author Gender?   YES  NO   Interpreta�on in English:

6. For each of the following scenarios, iden�fy what a Type I error and a Type II error would look like. Then, determine which type would be a bigger problem for that scenario.

a. A large interna�onal airport has received a bomb threat. In response, the airport police have �ghtened security and now check every piece of luggage manually. (1). Type I: (2). Type II: (3). Bigger problem:

b. Your friend purchases a pregnancy test. (1). Type I:

(2). Type II: (3). Bigger problem:

Cri�cal Thinking & Discussion Ques�ons

1. Explain the advantages and disadvantages of a within-subject design. 2. Compare and contrast pre-experiments, quasi-experiments, and true experiments. What is the primary difference between true experiments and both pre-

experiments and quasi-experiments? 3. Compare and contrast the following terms. Your answers should demonstrate that you understand each term. Be sure to give some kind of context (e.g., “both are

types of . . .”) or provide an example, and state how they are different. a. internal versus external validity b. between-subject versus within-subject design c. level versus condi�on

4. Explain the difference between Type I and Type II errors. How can each type of error be minimized? 5. Describe a research study that would require a mixed methods design.