Questionaire & Sampling

Chapter 4 Survey Research—Describing and Predic�ng Behavior

Kim Steele/Photodisc/Ge�y Images

Chapter Contents

Introduc�on to Survey Research (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.1#sec1.1) Designing Ques�onnaires (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.2#sec1.2) Sampling From the Popula�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.3#sec1.3) Analyzing Survey Data (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.4#sec1.4) Ethical Issues in Survey Research (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.5#sec1.5)

In a highly influen�al book published in the 1960s, the sociologist Erving Goffman (1963) defined s�gma as an unusual characteris�c that triggers a nega�ve evalua�on. In his words, "The s�gma�zed person is one who is reduced in our minds from a whole and usual person to a tainted, discounted one" (1963, p. 3). People's beliefs about s�gma�zed characteris�cs exist largely in the eye of the beholder but have substan�al influence on social interac�ons with the s�gma�zed (see Snyder, Tanke, & Berscheid, 1977). A large research tradi�on in psychology has been devoted to understanding both the origins of s�gma and the consequences of being s�gma�zed. According to Goffman and others, the characteris�cs associated with the greatest degree of s�gma have three features in common: They are highly visible, they are perceived as controllable, and they are misunderstood by the public.

Recently, researchers have taken considerable interest in people's a�tudes toward members of the gay and lesbian community. Although these a�tudes have become more posi�ve over �me, this group s�ll encounters harassment and other forms of discrimina�on on a regular basis (see Na�onal Gay Task Force, 1984). One of the top recognized experts on this subject is Gregory Herek, professor of psychology at the University of California at Davis (h�p://psychology.ucdavis.edu/herek/ (h�p://psychology.ucdavis.edu/herek/) ). In a 1988 ar�cle, Herek conducted a survey of heterosexuals' a�tudes toward both lesbians and gay men, with the goal of understanding the predictors of nega�ve a�tudes. Herek approached this research ques�on by construc�ng a scale to measure a�tudes toward these groups. In three studies, par�cipants were asked to complete this a�tude measure, along with other exis�ng scales assessing a�tudes about gender roles, religion, and tradi�onal ideologies.

Herek's (1988) research revealed that, as hypothesized, heterosexual males tended to hold more nega�ve a�tudes about gay men and lesbians than heterosexual females. However, the same psychological mechanisms seemed to explain the prejudice in both genders. That is, nega�ve a�tudes were associated with increased religiosity, more tradi�onal beliefs about family and gender, and fewer experiences actually interac�ng with gay men and lesbians. These associa�ons meant that Herek could predict people's a�tudes toward gay men and lesbians based on knowing their views about family, gender, and religion, as well as their past interac�ons with the s�gma�zed group. Herek's primary contribu�on to the literature in this paper was the insight that reducing s�gma toward gay men and lesbians "may require confron�ng deeply held, socially reinforced values" (1988, p. 473). And this insight was possible only because people were asked to report these values directly.

4.1 Introduc�on to Survey Research

Whether you are aware of it or not, you have been encountering survey research for most of your life. Every �me your telephone rings during dinner�me, and the person on the other end of the line insists on knowing your household income and favorite brand of laundry detergent, he or she is helping to conduct survey research. When news programs try to predict the winner of an elec�on two weeks early, these reports are based on survey research of eligible voters. In both cases, the researcher is trying to make predic�ons about the products people buy or the candidates they will elect based on people's reports of their own a�tudes, feelings, and behaviors.

Surveys can be used in a variety of contexts and are most appropriate for ques�ons that involve people describing their a�tudes, their behaviors, or a combina�on of the two. For example, if you wanted to examine the predictors of a�tudes toward the death penalty, you could ask people their opinions on this topic and also ask them about their poli�cal party affilia�on. Based on these responses, you could test whether poli�cal affilia�on predicted a�tudes toward the death penalty. Or, imagine you wanted to know whether students who spent more �me studying were more likely to do well on their exams. This ques�on could be answered using a survey that asked students about their study habits and then tracked their exam grades. We will return to this example near the end of the chapter as we discuss the process of analyzing survey data to test our hypotheses about predic�ons.

The common thread running through these two examples is that they require people to report either their thoughts (e.g., opinions about the death penalty) or their behaviors (e.g., the hours they spend studying). Thus, in deciding whether a survey is the best fit for your research ques�on, the key is to consider whether people will be both able and willing to report these things accurately. We will expand on both of these issues in the next sec�on.

In this chapter, we con�nue our journey along the con�nuum of control, moving on to survey research, in which the primary goal is either describing or predic�ng a�tudes and behavior. For our purposes, survey research refers to any method that relies on people's reports of their own a�tudes, feelings, and behaviors. So, for example, in Herek's (1988) study, the par�cipants reported their a�tudes toward lesbians and gay men, rather than these a�tudes being somehow directly observed by the researchers. Compared with the qualita�ve and descrip�ve designs for observing behavior we discussed in Chapter 3, survey research tends to yield more control over both data collec�on and ques�on content. Thus, survey research falls somewhere between quan�ta�ve descrip�ve research (Chapter 3) and the explanatory research involved in experimental designs (Chapter 5). This chapter provides an overview of survey research from conceptualiza�on through analysis. We will cover the types of research ques�ons that are best suited to survey research and provide an overview of the decisions to consider in designing and conduc�ng a survey study. We will then cover the process of data collec�on, with a focus on selec�ng the people who will complete your survey. Finally, we will cover the three most common approaches for analyzing survey data, bringing us back full circle to addressing our research ques�ons.

Distinguishing Features of Surveys

Survey research designs have three dis�nguishing features that set them apart from other designs. First, all survey research relies on either wri�en or verbal self- reports of people's a�tudes, feelings, and behaviors. This means that researchers will ask par�cipants a series of ques�ons and record their responses. This approach has several advantages, including being rela�vely straigh�orward and allowing access to psychological processes (e.g., "Why do you support candidate X?"). However, researchers are also cau�ous in their interpreta�on of self-reported data because par�cipants' responses can reflect a combina�on of their true a�tude and their concern over how this a�tude will be perceived. Scien�sts refer to this as social desirability, which means that people may be reluctant to report unpopular a�tudes. So if you were to ask people their a�tudes about different racial groups, their answers might reflect both their true a�tude and their desire not to appear racist. We return to the issue of social desirability and discuss some tricks for designing ques�ons that can help to sidestep these concerns and capture respondents' true a�tudes.

The second dis�nguishing feature of survey research is that it has the ability to access internal states that cannot be measured through direct observa�on. In our discussion of observa�onal designs in Chapter 3, we learned that one of the limita�ons of these designs was a lack of insight into why people do what they do. Survey research is able to address this limita�on directly: By asking people what they think, how they feel, and why they behave in certain ways, researchers come closer to capturing the underlying psychological processes.

However, people's reports of their internal states should be taken with a grain of salt, for three reasons. First, as men�oned, these reports may be biased by social desirability concerns, par�cularly when unpopular a�tudes are involved. Second, there is a large literature in social psychology sugges�ng that people may not be very accurate at understanding the true reasons for their behavior. In a highly cited review paper, psychologists Richard Nisbe� and Tim Wilson (1977) argued that we make poor guesses about why we do things, and those guesses are based more on our assump�ons than on any real introspec�on. Thus, survey ques�ons can provide access to internal states, but these should always be interpreted with cau�on. Third, on a more prac�cal note, survey research allows us to collect large amounts of data with rela�vely li�le effort and few resources. However, their actual efficiency depends on the decisions made during the design process. In reality, efficiency is o�en in a delicate balance with the accuracy and completeness of the data.

Broadly speaking, survey research can be conducted using either verbal or wri�en self-reports (or a combina�on of the two). Before we dive into the details of wri�ng and forma�ng a survey, it is important to understand the pros and cons of administering your survey as an interview (i.e., an oral survey) or a ques�onnaire (i.e., a wri�en survey).

Interviews

An interview involves an oral ques�on-and-answer exchange between the researcher and the par�cipant. This exchange can take place either face-to-face or over the phone. So our telemarketer example from earlier in the chapter represents an interview because the ques�ons are asked orally, via phone. Likewise, if you are approached in a shopping mall and asked to answer ques�ons about your favorite products, you are experiencing a survey in interview form because the ques�ons are administered out loud. And, if you have ever taken part in a focus group, in which a group of people gives their reac�ons to a new product, the researchers are

Conduc�ng interviews may allow a researcher to gather more detailed and richer responses.

Alina Solovyova-Vincent/E+/Ge�y Images

essen�ally conduc�ng an interview with the group. (For a more in-depth discussion of focus groups and other interview techniques, see Chapter 3, Sec�on 3.2 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec3.2#sec3.2) , Qualita�ve Research Interviews.)

Interview Schedules

Regardless of how the interview is administered, the interviewer (i.e., the researcher) has a predetermined plan, or script, for how the interview should go. This plan, or script, for the progress of the interview is known as an interview schedule. When conduc�ng an interview— including those telemarke�ng calls—the researcher/interviewer has a detailed plan for the order of ques�ons to be asked, along with follow-up ques�ons depending on the par�cipant's responses.

Broadly speaking, there are two types of interview schedules. A linear (also called "structured") schedule will ask the same ques�ons in the same order for all par�cipants. In contrast, a branching schedule unfolds more like a flowchart, with the next ques�on dependent on par�cipants' answers. A branching schedule is typically used in cases with follow-up ques�ons that make sense only for some of the par�cipants. For example, you might first ask people whether they have children; if they answer "yes," you could then follow up by asking how many.

One danger in using a branching schedule is that it is based partly on your assump�ons about the rela�onships between variables. Granted, it is fairly uncontroversial to ask only people with children to indicate how many children they have. But imagine the following scenario in which you first ask par�cipants for their household income, and then ask about their poli�cal dona�ons:

"How much money do you make? $18,000? Okay, how likely are you to donate money to the Democra�c Party?" "How much money do you make? $250,000? Okay, how likely are you to donate money to the Republican Party?"

The assump�on implicit in the way these ques�ons branch is that wealthier people are more likely to be Republicans and less wealthy people to be Democrats. This might be supported by the data or it might not. But by planning the follow-up ques�ons in this way, you are unable to capture cases that do not fit your stereotypes (i.e., the wealthy Democrats and the poor Republicans). The lesson here is to be careful about le�ng your biases shape the data collec�on process, as this can create invalid or inaccurate findings.

Advantages and Disadvantages of Interviews

Spoken interviews have a number of advantages over wri�en surveys. For one, people are o�en more mo�vated to talk than they are to write. Let's say that an undergraduate research assistant is dispatched to a local shopping mall to interview people about their experiences in roman�c rela�onships. The researcher may have no trouble at all recrui�ng par�cipants, many of whom will be eager to divulge the personal details about their recent rela�onships. But for be�er or for worse, these experiences will be more difficult for the researcher to capture in wri�ng. Related to this observa�on, people's oral responses are typically richer and more detailed than their wri�en responses. Think of the difference between asking someone to "Describe your views on gun control" versus "Indicate on a scale of 1 to 7 the degree to which you support gun control." The former is more likely to capture the richness and subtlety involved in people's a�tudes about guns.

On a prac�cal note, using an interview format also allows you to ensure that respondents understand the ques�ons. If wri�en ques�onnaire items are poorly worded, people are forced to guess at your meaning, and these guesses introduce a big source of error variance (variance from random sources that are irrelevant to the trait or ability the ques�onnaire is purpor�ng to measure). But if an interview ques�on is poorly asked, people find it much easier to ask the interviewer to clarify. Finally, using an interview format allows you to reach a broader cross-sec�on of people and to include those who are unable to read and write—or, perhaps, unable to read and write the language of your survey.

Interviews also have three clear disadvantages compared with wri�en surveys. First, interviews are more costly in terms of both �me and money. It certainly used more of my �me to go to a shopping mall than it would have taken to mail out packets of surveys (but no more money—these research assistant gigs tend to be unpaid!). Second, the interview format allows many opportuni�es to glean personal bias from the interview. These biases are unlikely to be deliberate, but par�cipants can o�en pick up on body language and subtle facial expressions when the interviewer disagrees with their answers. These cues may lead them to shape their responses in order to make the interviewer

happier (the influence of social desirability again). Third, interviews can be difficult to score and interpret, especially with open-ended ques�ons. Although administering them may be easy, scoring them is rela�vely more complicated, o�en involving subjec�vity or bias in the interpreta�on. Because the researcher o�en has to make judgments based on personal beliefs about the quality of the response, mul�ple raters are generally used to score the responses in order to minimize bias.

The best way to understand the pros and cons of interviewing is that both are a consequence of personal interac�ons. The interac�on between interviewer and interviewee allows for richer responses but also the poten�al for these responses to be biased. As a researcher, you have to weigh these pros and cons and decide which method is the best fit for your survey. In the next sec�on, we turn our a�en�on to the process of administering surveys in wri�ng.

Questionnaires

A ques�onnaire is a survey that involves a wri�en ques�on-and-answer exchange between the researcher and the par�cipant. The ques�onnaire can be in open- ended format (e.g., the par�cipant writes in his or her answer) or forced-choice response format (e.g., the par�cipant selects from a set of responses, such as with mul�ple choice ques�ons, ra�ng scales, or true/false ques�ons), which will be discussed later in this chapter. The exchange is a bit different from what we saw with

interview formats. In wri�en surveys, the ques�ons are designed ahead of �me and then given to par�cipants, who write their responses and return the ques�onnaire to the researcher. In the next sec�on, we will discuss details for designing these ques�ons. But before we get there, let's take a quick look at the process of administering wri�en surveys.

Distribu�on Methods

Ques�onnaires can be distributed in three primary ways, each with its own pa�ern of advantages and disadvantages.

Distribu�ng by Mail

Un�l recently, one common way to distribute surveys was to send paper copies through the mail to a group of par�cipants (see Sec�on 4.3, Sampling From the Popula�on, for more discussion on how this group is selected). Mailing surveys is rela�vely inexpensive and rela�vely easy to do, but is unfortunately one of the worst methods when it comes to response rates. People tend to ignore ques�onnaires they receive in the mail, dismissing them as one more piece of junk. There are a few tricks available to researchers to increase response rates, including providing incen�ves, making the survey interes�ng, and making it as easy as possible to return the results (e.g., with a postage-paid envelope). However, even using all of these tricks, researchers consider themselves extremely lucky to get a 30% response rate from a mail survey. That is, if you mail 1,000 surveys, you will be doing well to receive 300 back. Because of this low return on investment, researchers have begun using other methods for their wri�en surveys.

Distribu�ng in Person

Another op�on is to distribute a wri�en survey in person, simply handing out copies and asking par�cipants to fill them out on the spot. This method is certainly more �me-consuming, as a researcher has to be sta�oned for long periods of �me in order to collect data. In addi�on, people are less likely to answer the ques�ons honestly because the presence of a researcher makes them worry about social desirability. Last, the sample for this method is limited to people who are in the physical area at the �me that ques�onnaires are being handed out. As we will discuss later, this might lead to problems in the composi�on of the sample. On the plus side, however, this method tends to result in higher compliance rates because it is harder to say no to someone face-to-face than it is to ignore a piece of mail.

Distribu�ng Online

Over the past 20 years, Internet, or Web-based, surveys have become increasingly common. In Web-based survey research, the ques�onnaire is designed and posted on a Web page, to which par�cipants are directed in order to complete the ques�onnaire. The advantages of online distribu�on are clear: This method is easiest for both researchers and par�cipants and may give people a greater sense of anonymity, thereby encouraging more honest responses. In addi�on, response �mes are faster and the data are easier to analyze because they are already in digital format. The disadvantages include the following: Specific groups being underrepresented because they do not have access to the Internet, the researcher has li�le to no control over sample selec�on, and the researcher receives responses only from those who are interested in the topic—so-called self-selec�on bias. All these limita�ons could raise ques�ons about the validity and reliability of the data collected. In addi�on, several ethical issues might arise regarding informed consent and the privacy of par�cipants. So when considering conduc�ng Web-based surveys, researchers should evaluate all the advantages and disadvantages, as well as any ethical or legal implica�ons.

For readers interested in more informa�on on designing and conduc�ng Internet research, Sam Gosling and John Johnson's 2010 book Advanced Methods for Conduc�ng Online Behavioral Research is an excellent resource. In addi�on, several groups of psychological researchers have been a�emp�ng to understand the psychology of Internet users. (You can read about recent studies on this website (h�p://www.spring.org.uk/2010/10/internet-psychology.php) .)

Advantages and Disadvantages of Ques�onnaires

Just as with interview methods, wri�en ques�onnaires have their own set of advantages and disadvantages. Wri�en surveys allow researchers to collect large amounts of data with li�le cost or effort, and they can offer a greater degree of anonymity than interviews. Anonymity can be a par�cular advantage in dealing with sensi�ve or poten�ally embarrassing topics. That is, people may be more willing to answer a ques�onnaire about their alcohol use or their sexual history than they would be to discuss these things face-to-face with an interviewer. On the downside, wri�en surveys miss out on the advantages of interviews because no one is available to clarify confusing ques�ons or to gather more informa�on as needed. Fortunately, there is one rela�vely easy way to minimize this problem: Write ques�ons (and response choices, if using mul�ple choice or forced choice formats) that are as clear as possible. In the next sec�on, we turn our a�en�on to the process of designing ques�onnaires.

Simple language is one characteris�c of an effec�ve ques�onnaire.

iStockphoto/Thinkstock

4.2 Designing Ques�onnaires

One of the most important elements when conduc�ng survey research is deciding how to construct and assemble the ques�onnaire items. In some cases, you will be able to use ques�onnaires that other researchers have developed in order to answer your research ques�ons. For example, many psychology researchers use standard scales that measure behavior or personality traits, such as self-esteem, prejudice, depression, or stress levels. The advantage of these ready-made measures is that other people have already gone to the trouble of making sure they are valid and reliable. So if you are interested in the rela�onship between stress and depression, you could distribute the Perceived Stress Scale (Cohen, Kamarck, & Mermelstein, 1983) and the Beck Depression Inventory (Beck, Steer, Ball, & Ranieri, 1996) to a group of par�cipants and move on to the fun part of data analyses. For further discussion, see Chapter 2, Sec�on 2.2 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.2#sec2.2) , Reliability and Validity.

However, in many cases there is no perfect measure for your research ques�on—either because no one has studied the topic before or because the current measures do not accurately assess the construct of interest you are inves�ga�ng. When this happens, you will need to go through the process of designing your own ques�onnaire. In this sec�on, we discuss strategies for wri�ng ques�ons and choosing the most appropriate response format.

Five Rules for Designing Better Questionnaires

Each of the following rules is designed to make your ques�ons as clear and easy to understand as possible in order to minimize the poten�al for error variance. We discuss each one and illustrate them with contras�ng pairs of items, consis�ng of "bad" items that do not follow the rule, and then "be�er" items that do.

1. Use simple language. One of the simplest and most important rules to keep in mind is that people have to be able to understand your ques�ons. This means you should avoid jargon and specialized language whenever possible.

BAD: BETTER: BAD: BETTER:

"Have you ever had an STD?" "Have you ever had a sexually transmi�ed disease?" "What is your opinion of the S-CHIP?" "What is your opinion of the State Children's Health Insurance Program?"

It is also a good idea to simplify the language as much as possible so that people spend �me answering the ques�on rather than trying to decode your meaning. For example, words like assist and consider can be replaced with words like help and think. This may seem odd—or perhaps even condescending to your par�cipants—but it is always be�er to err on the side of simplicity. Remember, if people are forced to guess at the meaning of your ques�ons, these guesses add error variance to their answers.

In addi�on, when developing and administering surveys to various popula�ons, it is important to remember to use language and examples that are not culturally or linguis�cally biased. For example, par�cipants from various cultures may not be familiar with ques�ons involving the historical development of the United States or the nature of gender roles in the United States. Thus, with the changing U.S. popula�on and the different languages being spoken in the home, it is important to use language that does not discriminate against par�cular popula�ons. Also, it is advisable to avoid slang at all costs.

2. Be precise. Another way to ensure that people understand the ques�on is to be as precise as possible in your wording. Ques�ons that are ambiguous in their wording will introduce an extra source of error variance into your data because people may interpret these ques�ons in varying ways.

BAD: BETTER: BAD: BETTER:

"What kind of drugs do you take?" (Legal drugs? Illegal drugs? Now? In college?) "What kind of prescrip�on drugs are you currently taking?" "Do you like sports?" (Playing? Watching? Which sports?) "How much do you like watching basketball on television?"

3. Use neutral language. It is important that your ques�ons be designed to measure your par�cipants' a�tudes, feelings, or behaviors rather than to manipulate their responses. That is, you should avoid leading ques�ons, which are wri�en in such a way that they suggest an answer.

BAD: BETTER: BAD: BETTER:

"Do you beat your children?" (Clearly, bea�ng isn't good; who would say yes?) "Is it acceptable to use physical forms of discipline?" "Do you agree that the president is an idiot?" (Hmmm . . . I wonder what the researcher thinks. . . .) "How would you rate the president's job performance?"

This guideline can also be used to sidestep social desirability concerns. If you suspect that people may be reluctant to report holding an a�tude such as using corporal punishment with their children, it helps to phrase the ques�on in a nonthreatening way—"using physical forms of discipline" versus "bea�ng your children." Many current measures of prejudice adopt this technique. For example, McConahay's Modern Racism Scale contains items such as "Discrimina�on against Blacks is no longer a problem in the United States" (McConahay, 1986). People who hold prejudicial a�tudes are more likely to agree with statements like this one than with more blunt ones like "I hate people from Group X."

4. Ask one ques�on at a �me. One remarkably common error that people make in designing ques�ons is to include a double-barreled ques�on, which fires off more than one ques�on at a �me. When you fill out a new pa�ent ques�onnaire at your doctor's office, these forms o�en ask whether you suffer from "headaches and nausea." What if you suffer from only one of these? Or what if you have a lot of nausea and only an occasional headache? The be�er approach is to ask about each of these symptoms separately.

BAD: BETTER: BAD: BETTER:

"Do you suffer from pain and numbness?" "How o�en do you suffer from pain?" "How o�en do you suffer from numbness?" "Do you like watching football and boxing?" "How much do you enjoy watching professional football on TV?" "How much do you enjoy watching boxing on TV?"

5. Avoid nega�ons. One final and simple way to clarify your ques�ons is to avoid ques�ons with nega�ve statements because these can o�en be difficult to understand. In the following examples, the first is admi�edly a li�le silly, but the second comes from a real survey of voter opinions.

BAD: BETTER: BAD: BETTER:

"Do you never not cheat on your exams?" (Wait—what? Do I cheat? Do I not cheat? What is this asking?) "Have you ever cheated on an exam?" "Are you against rejec�ng the ban on pes�cides?" (Wait—so am I for the ban? Against the ban? What is this asking?) "Do you support the current ban on pes�cides?"

Participant Response Options

In this sec�on, we turn our a�en�on to the issue of deciding how par�cipants should respond to your ques�ons. The decisions you make at this stage affect the type of data you ul�mately collect, so it is important to choose carefully. We also review the primary decisions you will need to make about response op�ons, as well as the pros and cons of each one.

One of the first choices you have to make is whether to collect open-ended or fixed-format responses. As the names imply, fixed-format responses require par�cipants to choose from a list of op�ons (e.g., "Pick your favorite color"), whereas open-ended responses ask par�cipants to provide unstructured responses to a ques�on or statement (e.g., "How do you feel about legalizing marijuana?"). Open-ended responses tend to be richer and more flexible but harder to translate into quan�fiable data—analogous to the trade-off we discussed in comparing wri�en versus oral survey methods. To put it another way, some concepts are difficult to reduce to a seven-point fixed-format scale, but these scales are easier to analyze than a paragraph of free-flowing text.

Another reason to think carefully about this decision is that fixed-format responses will, by defini�on, restrict people's op�ons in answering the ques�on. In some cases, these restric�ons can even act as leading ques�ons. In a study of people's percep�ons of history, James Pennebaker and his colleagues (2006) asked respondents to indicate the "most significant event over the last 50 years." When this was asked in an open-ended way (i.e., "list the most significant event"), 2% of par�cipants listed the inven�on of computers. In another version of the survey, this ques�on was asked in a fixed-format way (i.e., "choose the most significant event"). When asked to select from a list of four op�ons (World War II, Inven�on of Computers, Tiananmen Square, or Man on the Moon), 30% chose the inven�on of computers! In exchange for having easily coded data, the researchers accidentally forced par�cipants into a smaller number of op�ons and ended up with a skewed sense of the importance of computers in people's percep�ons of history.

Fixed-Format Op�ons

Although fixed-format responses can some�mes constrain or skew par�cipants' answers, the reality is that researchers tend to use them more o�en than not. This decision is largely prac�cal; fixed-format responses allow for more efficient data collec�on from a much larger sample. (Imagine the chore of having to hand-code 2,000 essays!) But once you have decided on this op�on for your ques�onnaire, the decision process is far from over. In this sec�on, we discuss three possibili�es you can use to construct a fixed-format response scale.

True/False

One op�on is to ask ques�ons using a true/false format, which asks par�cipants to indicate whether they endorse a statement. For example:

"I a�ended church last Sunday" "I am a U.S. ci�zen" "I am in favor of abor�on"

True True True

False False False

This last example may strike you as odd, and in fact illustrates an important point about the use of true/false formats: They are best used for statements of fact rather than a�tudes. It is rela�vely straigh�orward to indicate whether you a�ended church or whether you are a U.S. ci�zen. However, people's a�tudes toward abor�on are o�en complicated—one can be "pro-choice" but s�ll support some common-sense restric�ons, or "pro-life" but support excep�ons (e.g., in cases of rape). The point is that a true/false ques�on cannot even come close to capturing this complexity. However, for survey items that involve simple statements of fact, the true/false format can be a good op�on.

Mul�ple Choice

A second op�on is to use a mul�ple-choice format, which asks par�cipants to select from a set of predetermined responses.

"Which of the following is your favorite fast-food restaurant?"

a. McDonald's b. Burger King c. Wendy's

d. Taco Bell

"Who did you vote for in the 2008 presiden�al elec�on?"

a. John McCain b. Barack Obama

"How do you travel to work on most days? (Select all that apply.)"

a. drive alone b. carpool c. take public transporta�on

As you can see in these examples, mul�ple-choice ques�ons give you quite a bit of freedom in both the content and the response scaling of your ques�ons. You can ask par�cipants either to select one answer or, as in the last example, to select all applicable answers. You can cover everything from preferences (e.g., favorite fast-food restaurant) to behaviors (e.g., how you travel to work).

In addi�on to assessing preferences or behaviors, as in the preceding examples, mul�ple-choice ques�ons are o�en used to examine knowledge and abili�es. For example, intelligence and achievement tests u�lize mul�ple-choice ques�ons to assess the cogni�ve and academic abili�es of individuals. In these cases, there is only one correct response choice and three to four incorrect or "distracter" response choices. Most research indicates that there should be at least three but no more than four incorrect response choices. Providing more than four incorrect response choices can make the ques�on too difficult. It is also important that the correct response choice not obviously differ from the incorrect choices. Thus, all response choices, both correct and incorrect, should be approximately the same length and be plausible choices. If the incorrect response choices are obviously wrong, this will make the item too easy, which will affect its validity. In addi�on, including obviously wrong answers allows those individuals who do not know the answer to deduce the correct response.

Regardless of whether you are developing mul�ple-choice ques�ons to assess behaviors, knowledge, or abili�es, all ques�ons should be clear, unambiguous, and brief so that they can be easily understood. In addi�on, as with all types of ques�on formats, it is best to avoid nega�ve ques�ons such as, "Which of the following is not correct?" because they can be confusing to test-takers and cause the ques�on to be invalid (i.e., not measure what it is supposed to be measuring). Finally, it is best prac�ce to avoid response choices that include "All of the above" or "None of the above." Most individuals know that when such response choices are provided, they are usually the correct answer.

You may have already spo�ed a downside to mul�ple-choice formats. Whenever you provide a set of responses, you are restric�ng par�cipants to those choices. This is the problem that Pennebaker and colleagues encountered when asking people about the most significant events of the last century. In each of the preceding examples, the categories fail to capture all possible responses. What if your favorite restaurant is In-and-Out Burger? What if you voted for Ralph Nader? What if you telecommute or ride your bicycle to work? There are two rela�vely easy ways to avoid (or at least minimize) this problem. First, plan carefully when choosing the response op�ons. During the design process, it helps to brainstorm with other people to ensure you are capturing the most likely responses. However, in many cases, it is almost impossible to provide every op�on that people might think of. The second solu�on is to provide an "other" response to your mul�ple-choice ques�on. This allows people to write in an op�on that you neglected to include. For example, our last ques�on about traveling to work could be rewri�en as follows:

"How do you travel to work on most days? (Select all that apply.)"

a. drive alone b. carpool c. take public transporta�on d. other (please specify):__________________

This way, people who telecommute, bicycle, or even ride their trained pony to work will have a way to respond rather than skipping the ques�on. And, if you start to no�ce a pa�ern in these write-in responses (e.g., 20% of people adding "bicycle"), then you will have gained valuable knowledge to improve the next itera�on of the survey.

Ra�ng Scales

Last, but certainly not least, another op�on is to use a ra�ng scale format, which asks par�cipants to respond on a scale that represents a con�nuum.

"Some�mes it is necessary to sacrifice liberty in the name of security."

1 2 3 4 5

strongly agree agree neither agree nor disagree disagree strongly disagree

"I would vote for a candidate who supported the death penalty."

1 2 3 4 5

Always o�en about half of the �me seldom never

"The poli�cal party in power right now has messed things up."

1 2 3 4 5

strongly agree agree neither agree nor disagree disagree strongly disagree

This format is well suited to capturing a�tudes and opinions and, in fact, is one of the most common approaches to a�tude research. Ra�ng scales are easy to score, and they give par�cipants some flexibility in indica�ng their agreement with or endorsement of the ques�ons. As a researcher, you have two cri�cal decisions to make about the construc�on of ra�ng scale items. Both have implica�ons for how you will analyze and interpret your results.

First, you'll need to decide on the anchors, or labels, for your response scale. Ra�ng scales offer a good deal of flexibility in these anchors, as you can see in the preceding examples. You can frame ques�ons in terms of "agreement" with a statement or "likelihood" of a behavior; alterna�vely, you can customize the anchors to match your ques�on (e.g., "not at all necessary"). Scales that use anchors of strongly agree and strongly disagree are also referred to as Likert scales. At a fairly simple level, the choice of labels affects the interpreta�on of the results. For example, if you asked the "poli�cal party" ques�on, you would have to be aware that the anchors were phrased in terms of agreement with the statement. In discussing these results, you would be able to discuss how much people agreed with the statement, on average, and whether agreement correlated with other factors. If this seems like an obvious point, you would be amazed at how o�en researchers (or the media) will take an item like this and spin the results to talk about the "likelihood of vo�ng" for the party in power—confusing an a�tude with a behavior! So, in short, make sure you are being honest when presen�ng and interpre�ng research data.

At a more conceptual level, you need to decide whether the anchors for your ra�ng scale make use of a bipolar scale, which has polar opposites at its endpoints and a neutral point in the middle, or a unipolar scale, which assesses the presence or absence of a single construct. The difference between these scales is best illustrated by an example:

Bipolar: How would you rate your current mood?

1 2 3 4 5 6 7

very happy happy slightly happy neither happy nor sad slightly sad sad very sad

Unipolar: How would you rate your current mood?

1 2 3 4 5

not at all sad slightly sad moderately sad very sad completely sad

1 2 3 4 5

not at all happy slightly happy moderately happy very happy completely happy

With the bipolar op�on, par�cipants are asked to place themselves on a con�nuous scale somewhere between sad and happy, which are polar opposites. The assump�on in using a bipolar scale is that the endpoints represent the only two op�ons—par�cipants can be sad, happy, or somewhere in between. In contrast, with the unipolar op�on, par�cipants are asked to rate themselves on a con�nuous scale, indica�ng their level of either sadness or happiness. The assump�on in using a pair of unipolar scales is that it is possible to experience varying degrees of each item: For example, par�cipants can be moderately happy but also a li�le bit sad. The decision to use a bipolar or a unipolar scale comes down to the context. What is the most logical way to think about these constructs? What have previous researchers done?

In the 1970s, Sandra Lipsitz Bem revolu�onized the way researchers thought about gender roles by arguing against a bipolar approach. Previously, gender role iden�fica�on had been measured on a bipolar scale from "masculine" to "feminine," the assump�on being that a person could be one or the other. Bem (1974) argued instead that people could easily have varying degrees of masculine and feminine traits. Her scale, the Bem Sex Role Inventory, asks respondents to rate themselves on a set of 60 unipolar traits. Someone with mostly feminine and hardly any masculine traits would be described as "feminine." Someone with high ra�ngs on both masculine and feminine traits would be described as "androgynous." And, someone with low ra�ngs on both masculine and feminine traits would be described as "undifferen�ated." You can view and complete Bem's scale online at this website (h�ps://openpsychometrics.org/tests/OSRI/) .

The second cri�cal decision in construc�ng a ra�ng scale item is to decide on the number of points in the response scale. You may have no�ced that all the examples in this sec�on have an odd number of points (e.g., five or seven). This is usually preferable for ra�ng scale items because the middle of the scale (e.g., "3" or "4") allows respondents to give a neutral, middle-of-the-road answer. That is, on a scale from strongly disagree to strongly agree, the midpoint can be used to indicate "neither" or "I'm not sure." However, in some cases, you may not want to allow a neutral op�on in your scale. By using an even number of points (e.g., four or six), you can essen�ally force people to either agree or disagree with the statement; this type of scaling is referred to as forced choice.

So how many points should your scale have? As a general rule, more points will translate into more variability in responses—the more choice people have, the more likely they are to distribute their responses among those choices. From a researcher's perspec�ve, the big ques�on is whether this variability is meaningful. For example, if you wanted to assess college students' a�tudes about a student fee increase, opinions will likely vary depending on the size of the fee and the ways in which it will be used. Thus, a five- or seven-point scale would be preferable to a two-point (yes or no) scale. However, past a certain point, increasing the scale range ceases to be linked to meaningful varia�on in a�tudes. In other words, the difference between a 5 and a 6 on a seven-point scale is fairly intui�ve for your par�cipants to grasp. But what is the real difference between an 80 and an 81 on a 100-point scale? When scales become too large, you risk introducing another source of error variance as par�cipants impose their interpreta�ons on the scaling. In sum, more points do not always translate into a be�er scale.

Back to our ques�on: How many points should you have? The ideal compromise supported by many sta�s�cians is to use a seven-point scale for bipolar scales. The reason has to do with the differences between scales of measurement. As you'll remember from our discussion in Chapter 2, the way variables are measured has implica�ons for data analyses. For the most popular sta�s�cal tests to be legi�mate, variables need to lie on either an interval scale (i.e., with equal intervals between points) or a ra�o scale (i.e., with a true zero point). Based on mathema�cal modeling research, sta�s�cians have concluded that the variability generated by a seven-point scale is most likely to mimic an interval scale (see, e.g., Nunnally, 1978). So a seven-point scale is o�en preferable because it allows us the most flexibility in data analyses. Note, however, that whereas seven-point scales produce reliable results with bipolar scales, unipolar scales tend to perform the best with five-point scales.

Finalizing the Questionnaire

Once you have finished construc�ng the ques�onnaire items, one last important step remains before beginning to collect data. This sec�on discusses a few guidelines for assembling the items into a coherent ques�onnaire. The main issues at this stage are to think carefully about the order of the individual items; how many items to include; whether to include open-ended, fixed-format, or mul�ple choice ques�ons; and wri�ng the instruc�ons in clear and concise language.

First, keep in mind that the first few ques�ons will set the tone for the rest of the ques�onnaire. It is best to start with ques�ons that are both interes�ng and nonthreatening to help ensure that respondents complete the ques�onnaire with open minds. For example:

BAD OPENING: BETTER OPENING: BAD OPENING: BETTER OPENING:

"Do you agree that your child's teacher is incompetent?" (threatening and also a leading ques�on) "How would you rate the performance of your child's teacher?" "Would you support a 1% sales tax increase?" (boring) "How do you feel about raising taxes to help fund educa�on?"

Second, strive whenever possible to have con�nuity in the different sec�ons of your ques�onnaire. Imagine you are construc�ng a survey to give to college freshmen—you might have ques�ons on family background, stress levels, future plans, campus engagement, and so on. It is best to have the ques�ons grouped by topic on the survey. So, for instance, students would fill out a set of ques�ons about future plans on one page and then a set of ques�ons about campus engagement on another page. This approach makes it easier for par�cipants to progress through the ques�ons without having to mentally switch between topics.

Third, remember that individual ques�ons are always read in context. This means that if you start your college student survey with a ques�on about plans for the future and then ask about stress, respondents will likely have their future plans in mind when they think about their level of stress. Another example is a graduate school that would administer a gigan�c survey packet to every student enrolled in its Introductory Psychology course. One year, a faculty member included a measure of iden�ty, asking par�cipants to complete the statements "I am ______" and "I am not ______." As the students started to analyze data from this survey, they found that an astonishing 60% of students had filled in the blank with "I am not a homosexual." This response seemed pre�y unusual un�l they realized that the ques�onnaire immediately preceding this one in the packet was a measure of prejudice toward gay and lesbian individuals. So, as these students completed the iden�ty measure, they had homosexuality on their minds and felt compelled to point out that they were not homosexual. This proves once again that results can be skewed by the context.

Finally, once you have assembled a dra� version of your ques�onnaire, do a test run. This test run, called pilot tes�ng, involves giving the ques�onnaire to a small sample of people, ge�ng their feedback, and making any necessary changes. One of the best ways to pilot test is to find a pa�ent group of friends to complete your ques�onnaire because this group will presumably be willing to give more extensive feedback. Another effec�ve way to pilot test a ques�onnaire is to administer it to individuals in the target group. In solici�ng their feedback, you should ask ques�ons like the following:

Was anything confusing or unclear?

Was anything offensive or threatening?

How long did the ques�onnaire take you to complete?

Did it get repe��ve or boring? Did it seem too long?

Were there par�cular ques�ons that you liked or disliked? Why?

The answers to these ques�ons will give you valuable informa�on to revise and clarify your ques�onnaire before devo�ng the resources for a full round of data collec�on. In the next sec�on, we turn our a�en�on to the ques�on of how to find and select par�cipants for this stage of the research.

Research: Thinking Critically

"Beautiful People Convey Personality Traits Better During First Impressions"

Medical News Today

A new University of Bri�sh Columbia study has found that people iden�fy the personality traits of people who are physically a�rac�ve more accurately than others during short encounters.

The study, published in the December [2010] edi�on of Psychological Science, suggests people pay closer a�en�on to people they find a�rac�ve, and is the latest scien�fic evidence of the advantages of perceived beauty. Previous research has shown that individuals tend to find a�rac�ve people more intelligent, friendly, and competent than others.

The goal of the study was to determine whether a person's a�rac�veness impacts others' ability to discern their personality traits, says Prof. Jeremy Biesanz, UBC Dept. of Psychology, who coauthored the study with PhD student Lauren Human and undergraduate student Genevieve Lorenzo.

For the study, researchers placed more than 75 male and female par�cipants into groups of five to 11 people for three-minute, one-on-one conversa�ons. A�er each interac�on, study par�cipants rated partners on physical a�rac�veness and five major personality traits: openness, conscien�ousness, extraversion, agreeableness, and neuro�cism. Each person also rated his or her own personality.

Researchers were able to determine the accuracy of people's percep�ons by comparing par�cipants' ra�ngs of others' personality traits with how individuals rated their own traits, says Biesanz, adding that steps were taken to control for the posi�ve bias that can occur in self-repor�ng.

Despite an overall posi�ve bias towards people they found a�rac�ve (as expected from previous research), study par�cipants iden�fied the "rela�ve ordering" of personality traits of a�rac�ve par�cipants more accurately than others, researchers found.

"If people think Jane is beau�ful, and she is very organized and somewhat generous, people will see her as more organized and generous than she actually is," says Biesanz. "Despite this bias, our study shows that people will also correctly discern the rela�ve ordering of Jane's personality traits—that she is more organized than generous—be�er than others they find less a�rac�ve."

The researchers say this is because people are mo�vated to pay closer a�en�on to beau�ful people for many reasons, including curiosity, roman�c interest, or a desire for friendship or social status. "Not only do we judge books by their covers, we read the ones with beau�ful covers much closer than others," says Biesanz, no�ng the study focused on first impressions of personality in social situa�ons, like cocktail par�es.

Although par�cipants largely agreed on group members' a�rac�veness, the study reaffirms that beauty is in the eye of the beholder. Par�cipants were best at iden�fying the personali�es of people they found a�rac�ve, regardless of whether others found them a�rac�ve.

According to Biesanz, scien�sts spent considerable efforts a half-century ago seeking to determine what types of people perceive personality best, to largely mixed results. With this study, the team chose to inves�gate this long-standing ques�on from another direc�on, he says, focusing not on who judges personality best, but rather whether some people's personali�es are be�er perceived.

Think about it:

1. Suppose the following ques�ons were part of the ques�onnaire given a�er the 3-minute one-on-one conversa�ons in this study. Based on the goals of the study and the rules discussed in this chapter, iden�fy the problem with each of the following ques�ons and suggest a be�er item.

a. Jane is very neat.

1 2 3 4 5

strongly agree

agree neither agree nor disagree disagree strongly disagree

main problem: be�er item:

b. Jane is generous and organized.

1 2 3 4 5

strongly agree agree neither agree nor disagree

disagree strongly disagree

main problem: be�er item:

c. Jane is extremely a�rac�ve    TRUE    FALSE

main problem: be�er item:

2. What are the strengths and weaknesses of using a fixed-format ques�onnaire in this study versus open-ended responses?

3. The researchers state that they took steps to control for the "posi�ve bias that can occur in selfrepor�ng." How might social desirability influence the outcome of this par�cular study? What might the researchers have done to reduce the effect of social desirability?

George, R. (2010, December 31). Beau�ful people convey personality traits be�er during first impressions. Medical News Today. Retrieved from h�p://www.medicalnewstoday.com/ar�cles/212245.php (h�p://www.medicalnewstoday.com/ar�cles/212245.php)

4.3 Sampling From the Popula�on

By now, you should have a good feel for how to construct survey items. Once you have finalized your measures, the next step is to find a group of people to fill out the survey. But where do you find this group? And how many of them do you need? On the one hand, you want as many people as possible in order to capture the full range of a�tudes and experiences. On the other hand, researchers have to conserve �me and other resources, which o�en means choosing a smaller sample of people. In this sec�on, we will examine the strategies researchers can use in selec�ng samples for their studies.

Researchers refer to the en�re collec�on of people who could possibly be relevant for a study as the popula�on. For example, if you were interested in the effects of prison overcrowding in this country, you would want to study the popula�on of prisoners in the United States. If you wanted to study vo�ng behavior in the next U.S. presiden�al elec�on, your popula�on would be United States residents eligible to vote. And if you wanted to know how well college students cope with the transi�on from high school, your popula�on would include every college student who was graduated from high school and is now enrolled in any college in the country.

You may have spo�ed an obvious prac�cal complica�on with these popula�ons. How on earth are you going to get every college student, much less every prisoner, in the country to fill out your ques�onnaire? You can't; instead, researchers will collect data from a sample, a subset of the popula�on. Instead of trying to reach all prisoners, you might sample inmates from a handful of state prisons. Rather than a�empt to survey all college students in the country, researchers might restrict their studies to a collec�on of students at one university.

The goal in choosing a sample for quan�ta�ve research is to make it as representa�ve as possible of the larger popula�on. This is the goal, though it is not always prac�cal. That is, if you choose students at one university, they need to be reasonably similar to college students elsewhere in the country. If the phrase "reasonably similar" sounds vague, this is because the basis for evalua�ng a sample varies depending on the hypothesis and the key variables of your study. For example, if you wanted to study the rela�onship between family income and stress levels, you would need to make sure that your sample mirrored the popula�on in the distribu�on of income levels. Thus, a sample of students from a state university might be a be�er choice than students from, say, Harvard (which costs about $50,000 per year). On the other hand, if your research ques�on dealt with the pressures faced by students in selec�ve private schools, then Harvard students could be a representa�ve sample for your study.

Figure 4.1 is a conceptual illustra�on of both a representa�ve and nonrepresenta�ve sample, drawn from a larger popula�on. The popula�on in this case consists of 144 individuals, split evenly between Xs and Os. Thus, we would want our sample to come as close as possible to capturing this 50/50 split. The sample of 20 individuals on the le� is representa�ve of the sample because it is split evenly between Xs and Os. But the sample of 20 individuals on the right is nonrepresenta�ve because it contains 75% Xs. Because there are far fewer Os than we might expect in the right-hand popula�on, this sample does not accurately represent the popula�on. This failure of the sample to represent the popula�on is also referred to as sampling bias.

Figure 4.1: Representative and nonrepresentative samples of a population

Stra�fied random sampling allows researchers to include all subgroups of a popula�on in a study.

iStockphoto/Thinkstock

So, where do these samples come from? As a researcher, you have two broad categories of sampling strategies at your disposal: probability sampling and nonprobability sampling.

Probability Sampling

Probability sampling is used when each person in the popula�on has a known chance of being in the sample. This is possible only in cases where you know the exact size of the popula�on. For instance, the 2010 popula�on of the United States was 308,745,538 (United States Census Bureau, 2010). If you were to have selected a U.S. resident at random, each resident would have had a one in 308,745,538 chance of being selected. Whenever you have informa�on about total popula�on, probability-sampling strategies are the most powerful approach because they greatly increase the odds of ge�ng a representa�ve sample. Within this broad category of probability sampling are three specific strategies: simple random sampling, stra�fied random sampling, and cluster sampling.

Simple Random Sampling

Simple random sampling, the most straigh�orward approach, involves randomly picking study par�cipants from a list of everyone in the popula�on. The term for this list is a sampling frame (e.g., imagine a list of every resident of the United States). To have a truly representa�ve random sample, several criteria need to be met: You must have a sampling frame, you must choose from it randomly, and you must have a 100% response rate from those you select. (As we discussed in Chapter 2, it can threaten the validity of your hypothesis test if people drop out of your study.)

Stra�fied Random Sampling

Stra�fied random sampling, a varia�on of simple random sampling, is used when subgroups of the popula�on might be le� out of a purely random sampling process. Imagine a city with a popula�on that is 80% Caucasian, 10% Hispanic, 5% African American, and 5% Asian. If you were to choose 100 residents at random, the chances are very good that your en�re sample would consist of Caucasian residents. As a result, you would inadvertently ignore the perspec�ve of all ethnic minority residents. To prevent this problem, researchers use stra�fied random sampling—breaking the sampling frame into subgroups and then sampling a random number from each subgroup. In the preceding city example, you could divide your list of residents into four ethnic groups and then pick a random 25 from each of these groups. The result would be a sample of 100 people who captured opinions from each ethnic group in the popula�on.

Cluster Sampling

Cluster sampling, another varia�on of random sampling, is used when you do not have access to a full sampling frame (i.e., a full list of everyone in the popula�on). Imagine that you wanted to do a study of how cancer pa�ents in the United States cope with their illness. Because there is not a list of every cancer pa�ent in the country, you have to get a li�le crea�ve with your sampling. The best way to think about cluster sampling is as "samples within samples." Just as with stra�fied sampling, you divide the overall popula�on into groups; however, cluster sampling is different in that you are dividing into groups based on more than one level of analysis. In the cancer example, you could start by dividing the country into regions, then randomly selec�ng ci�es from within each region, and then randomly selec�ng hospitals from within each city, and finally randomly selec�ng cancer pa�ents from each hospital. The result would be a random sample of cancer pa�ents from, say, Phoenix, Miami, Dallas, Cleveland, Albany, and Sea�le; taken together, these pa�ents would cons�tute a fairly representa�ve sample of cancer pa�ents around the country.

Nonprobability Sampling

The other broad category of sampling strategies is known as nonprobability sampling. These strategies are used in the (remarkably common) case in which you do not know the odds of any given individual being in the sample. This is an obvious shortcoming—if you do not know the exact size of the popula�on and do not have a list of everyone in it, there is no way to know that your sample is representa�ve! But despite this limita�on, researchers use nonprobability sampling on a regular basis.

Nonprobability sampling is used in qualita�ve, quan�ta�ve, and mixed methods research whose focus is on selec�ng rela�vely small samples in a purposeful manner rather than selec�ng samples that are representa�ve of the en�re popula�on. Unlike probability sampling, nonprobability sampling does not include randomly selected, stra�fied samples that provide for generaliza�on. Rather, the procedures used to select samples are purposeful; that is, the samples are selected to obtain informa�on-rich cases that yield in-depth insights. The sampling techniques used for quan�ta�ve and qualita�ve research probably make up one of the biggest differences between the two methods. The following sec�ons will discuss some of the most common nonprobability strategies. All of the nonprobability strategies described (except convenience sampling) are considered categories of purposive sampling, or sampling with a purpose. Convenience sampling is considered a form of so-called accidental or haphazard (serendipitous) sampling, since the cases are selected based on ready availability. Even so, convenience sampling can be purposeful, in that researchers do their best to recruit in ways that are convenient and yet a�ract individuals that meet meaningful criteria.

In many cases, it is not possible to obtain a sampling frame. When researchers study rare or hard-to-reach popula�ons, or inves�gate poten�ally s�gma�zing condi�ons, they o�en recruit by word of mouth. The term for this is snowball sampling—imagine a snowball rolling down a hill, picking up more snow (or par�cipants) as it goes. If you wanted to study how o�en homeless people took advantage of social services, you would be hard-pressed to find a sampling frame that listed the homeless popula�on. Instead, you could recruit a small group of homeless people and ask each of them to pass the word along to others, and so on. The resul�ng sample is unlikely to be representa�ve, but researchers o�en have to compromise for the sake of obtaining access to a popula�on.

One of the most popular nonprobability strategies is known as convenience sampling, or simply enrolling people who show up for the study. Any �me you see results of a viewer poll on your favorite 24-hour news sta�on, the results are likely based on a convenience sample. CNN and Fox News do not randomly select from a list of their viewers; they post a ques�on on-screen or online, and people who are mo�vated enough to respond will do so. For that ma�er, a large majority of psychology research studies are based on convenience samples of undergraduate college students. O�en, experimenters in psychology departments adver�se their studies on a bulle�n board or website, and students sign up for studies for extra cash or to fulfill a research requirement for a course. Students o�en pick a par�cular study based on whether it fits their busy schedules or whether the adver�sement sounds interes�ng. Another example of convenience sampling is a researcher seeking to collect informa�on from human resources managers to study the issue of bullying in organiza�ons. The researcher might use a database of poten�al par�cipants who belong to one or more chapters of the Society of Human Resource Management (SHRM), an organiza�on for human resources professionals. In this example, the convenience component of the sample is using a body of exis�ng data; although these data are not representa�ve of all human resources managers or all organiza�ons, they are a useful proxy for a broad sample of human resource managers across industries.

Other popular nonprobability strategies include extreme or deviant case sampling, typical case sampling, heterogeneous sampling, expert sampling, criterion sampling, and theory-based sampling. Whatever strategy you decide on, the goal here is to make you mindful that all the decisions that you make as a researcher have inherent strengths and weaknesses.

Extreme or deviant case sampling is used when we want informa�on-rich data on unusual or special cases. For example, if we are interested in studying university diversity plans, it would be important to examine plans that work excep�onally well, as well as plans that have high expecta�ons but are not working for some reason or another. The focus is on examining outstanding successes and failures to learn lessons about those condi�ons.

On the other hand, some�mes researchers are not interested in learning about unusual cases but rather want to learn about typical cases. Typical case sampling involves sampling the most frequent or "normal" case. It is o�en used to describe typical cases to people unfamiliar with a se�ng, program, or process. For example, a typical case sample may be used to study the prac�cum experiences of psychology students from universi�es that are rated average. One of the biggest drawbacks of this method involves the difficulty of knowing how to iden�fy or define a typical or normal case.

Heterogeneous sampling, which is also called maximum varia�on sampling, may include both extreme and typical cases. It is used to select a wide variety of cases in rela�on to the phenomenon being inves�gated. The idea here is to obtain a sample that includes diverse characteris�cs from mul�ple dimensions. For example, if the researchers are interested in examining the experiences of community health clinic pa�ents, they may wish to conduct a focus group and then select 10 different groups that represent various demographics from several health clinics in the area. Heterogeneous sampling is beneficial when it is desirable to view pa�erns across a diverse set of individuals. It is also valuable in describing experiences that may be central or core to most individuals. Heterogeneous sampling can be problema�c in small samples, though, because it is difficult to obtain a wide variety of cases using this method.

Expert sampling involves sampling a panel of individuals who have known exper�se (knowledge and training) in a par�cular area. While expert sampling can be used in both qualita�ve and quan�ta�ve research, it is used more o�en in qualita�ve research. Expert sampling involves a process of iden�fying individuals with known exper�se in an area of interest, obtaining informed consent from each expert, and then collec�ng informa�on from them either individually or as a group.

The goal of criterion sampling is to select cases "that meet some predetermined criterion of importance" (Pa�on, 2002, p. 238). A criterion could be a par�cular illness or experience that is being inves�gated, or even a program or a situa�on. For example, criterion sampling could be used to inves�gate a range of topics: the experiences of individuals who have a�empted suicide, why students are exceedingly absent from online course rooms, or cases that have exceeded the standard wai�ng �me at a doctor's office. The idea with this method is that the par�cipants meet a par�cular criterion of interest.

Theory-based sampling is a version of criterion sampling that focuses on obtaining cases that represent theore�cal constructs. As Pa�on (2002) discussed, theory- based sampling involves the researcher obtaining "sampling incidents, slices of life, �me periods, or people on the basis of their poten�al manifesta�on or representa�on of important theore�cal constructs" (p. 238). This type of sampling arose from grounded theory research and follows a more deduc�ve or theory- tes�ng approach. As Glaser (1978) described, theory-based sampling is "the process of data collec�on for genera�ng theory whereby the analyst jointly collects, codes, and analyzes his data and decides which data to collect next and where to find them" (p. 36). Thus, data collec�on is driven by the emerging theory, and par�cipants are selected based on their knowledge of the topic.

Choosing a Sampling Strategy

Although quan�ta�ve researchers strive for representa�ve samples, there is no such thing as a perfectly representa�ve one. There is always some degree of sampling error, defined as the degree to which the characteris�cs of the sample differ from the characteris�cs of the popula�on. Instead of aiming for perfec�on, then, researchers aim for an es�mate of how far from perfec�on their samples are. These es�mates are known as the error of es�ma�on, or the degree to which the data from the sample are expected to deviate from the popula�on as a whole.

One of the main advantages of a probability sample is that we are able to calculate these errors of es�ma�on (or margins of error). In fact, you have likely encountered errors of es�ma�on every �me you see the results of an opinion poll. For example, CNN may report that "Candidate A is leading the race with 60% of the vote, ± 3%." This means Candidate A's percentage in the sample is 60%, but based on sta�s�cal calcula�ons, her real percentage is between 57% and 63%. The smaller the error (3% in this example), the more closely the results from the sample match the popula�on. Naturally, researchers conduc�ng these opinion polls want the error of es�ma�on to be as small as possible; imagine how nonpersuasive it would be to learn that "Candidate A has a 10-point lead, ± 20 points." In general, these errors are minimized when three condi�ons are met: The overall popula�on is smaller; the sample itself is larger; and there is less variability in the sample data. When samples are created using a probability method, all of this informa�on is available because these methods require knowing the popula�on.

If probability sampling is so powerful, why are nonprobability strategies so popular? One reason is that convenience samples are more prac�cal; they are cheaper, easier, and almost always possible to enroll with rela�vely few resources because you can avoid the costs of large-scale sampling. A second reason is that convenience is o�en a good star�ng point for a new line of research. For example, if you wanted to study the predictors of rela�onship sa�sfac�on, you could start

by tes�ng hypotheses in a controlled se�ng using college student par�cipants, and then you could extend your research to the study of adult married couples. Finally, and relatedly, in many types of qualita�ve research, it is acceptable to have a nonrepresenta�ve sample because you do not need to generalize your results. If you want to study the prevalence of alcohol use in college students, it may be perfectly acceptable to use a convenience sample of college students. Although, even in this case, you would have to keep in mind that you were studying drinking behaviors among students who volunteered to complete a study on drinking behaviors.

There are also cases, however, where it is cri�cal to use probability sampling despite the extra effort it requires. Specifically, researchers use probability samples any �me it is important to generalize and any �me it is important to predict behavior of a popula�on. The best example for understanding these criteria is to think of poli�cal polls. In the lead-up to an elec�on, each campaign is invested in knowing exactly what the vo�ng public thinks of its candidate. In contrast to a CNN poll, which is based on a convenience sample of viewers, polls conducted by a campaign will be based on randomly selected households from a list of registered voters. The resul�ng sample is much more likely to be representa�ve, much more likely to tell the campaign how the en�re popula�on views its candidate, and therefore, much more likely to be useful.

Determining a Sufficient Sample Size

Several factors come into play when determining what a sufficient sample size is for a research study. Probably the most important factor is whether the study is a quan�ta�ve or a qualita�ve one. In quan�ta�ve research, there is one basic rule for sample size: The larger, the be�er. This is because a larger sample will be more representa�ve of the popula�on being studied. Having said that, even small samples using quan�ta�ve data can yield meaningful correla�ons if the researcher makes efforts to achieve a representa�ve sample or a meaningful convenience sample.

How does one make a prac�cal decision, though, regarding the exact sample size a par�cular research situa�on requires? Gay, Mills, and Airasian (2009) provided the following guidelines for determining sufficient sample size based on the size of the popula�on:

For popula�ons that include 100 or fewer individuals, the en�re popula�on should be sampled. For popula�ons that include 400–600 individuals, 50% of the popula�on should be sampled. For popula�ons that include 1,500 individuals, 20% of the popula�on should be sampled. For popula�ons larger than 5,000, about 8% of the popula�on should be sampled.

Thus, as we can see, the larger the popula�on size, the smaller the percentage required to obtain a representa�ve sample. This does not mean that the sample shrinks as the popula�on increases, but rather the opposite: The sample size required actually increases when studying larger popula�ons.

Although larger sample sizes are generally preferred in quan�ta�ve studies, the size of an adequate sample also depends on the similari�es and dissimilari�es among members of the popula�on. If the popula�on is fairly diverse for the construct being measured, then a larger sample will likely be required in order to obtain a representa�ve sample. Popula�ons whose members have similar characteris�cs will require smaller samples. Researchers have developed fairly sophis�cated methods for determining sufficient sample sizes through the use of sta�s�cal power analysis; however, such methods are beyond the scope of this book. Gay et al.'s (2009) guidelines are sufficient for most types of research.

As discussed in Chapter 3, recommended samples for qualita�ve research are typically much smaller than for quan�ta�ve research, which strives to be representa�ve of larger popula�ons. In qualita�ve research, sample sizes are not based on numbers but rather on how well the variables of interest are represented (Houser, 2009). Unlike quan�ta�ve approaches, which require large samples, qualita�ve techniques do not have any rules regarding sample size. Thus, sample size depends more on what the researcher wants to know, the purpose of the inquiry, what the findings will be useful for, how credible they will be, and what can be done with available �me and resources. Qualita�ve research can be very costly and �me-consuming, so choosing informa�on-rich cases will yield the greatest return on investment. As noted by Pa�on (2002), "The validity, meaningfulness, and insights generated from qualita�ve inquiry have more to do with the informa�on-richness of the cases selected and the observa�onal/analy�cal capabili�es of the researcher than with sample size" (p. 245).

Nonresponse Bias in Survey Research

Some�mes par�cipants do not submit a survey or do not fill it out completely. When this occurs, it is considered nonresponse bias, which can affect the size and characteris�cs of the sample, as well as the external validity of the study. External validity refers to how well the results obtained from a sample can be extended to make predic�ons about the en�re popula�on, which will be further discussed in Chapter 5. Nonresponses are par�cularly problema�c if a large number of par�cipants or a specific group of par�cipants fails to respond to par�cular ques�ons. For example, perhaps all females skip a certain ques�on, or several par�cipants skip certain ques�ons because they are too personal. This omission creates bias not only in the characteris�cs of the sample (e.g., the sample may no longer be representa�ve) but also in the size of the sample that is required for the study.

Nonresponses can occur for many reasons, including the survey being too long, the ques�ons being worded awkwardly, the survey topic being uninteres�ng, or, in the case with Web-based surveys, the par�cipants not knowing how to access or log into the website to complete the survey.

There are a few ways to minimize the threat of nonresponse bias. These include increasing the sample size to account for the possibility of nonresponses, making sure that survey direc�ons and ques�ons are worded clearly, making sure the survey is not too long, providing rewards or incen�ves for comple�ng the survey, sending out reminders to complete the survey, and providing a cover le�er that describes the exact reasons for conduc�ng the survey.

4.4 Analyzing Survey Data

Once you have designed a survey, chosen an appropriate sample, and collected some data, now comes the fun part. As with the quan�ta�ve descrip�ve designs covered in Chapter 3, the goal of analyzing survey data is to subject your hypotheses to a sta�s�cal test. Surveys can be used both to describe and predict thoughts, feelings, and behaviors. However, since we have already covered the basics of descrip�ve analysis in Chapter 3, this sec�on will focus on predic�ve analyses, which are designed to assess the associa�ons between and among variables.

Researchers typically use three approaches to test predic�ve hypotheses: correla�onal analyses, chi-square analyses, and regression analyses. Each one has its advantages and disadvantages, and each is most appropriate for a different kind of data. Correla�onal analysis allows one to examine the strength, direc�on, and sta�s�cal significance of a rela�onship; chi-square analysis determines whether two nominal variables are independent from or related to one another; and simple linear regression is the method used to "predict" scores for one variable based on another. In this sec�on, we will walk through the basics of each analysis.

Correlational Analysis

In the beginning of this chapter, we encountered an example of a survey research ques�on: What is the rela�onship between the number of hours that students spend studying and their grades in the class? In this case, the hypothesis claims that we can predict something about a student's grades by knowing how many hours he or she spends studying.

Imagine we collected a small amount of data to test this hypothesis, shown in Table 4.1. (Of course, if we really wanted a good test of this hypothesis, we would need more than 10 people in the sample, but this will do as an illustra�on.)

Table 4.1: Data for quiz grade/hours studied example

Par�cipant Hours Studied Quiz Grade

1 1 2

2 1 3

3 2 4

4 3 5

5 3 6

6 3 6

7 4 7

8 4 8

9 4 9

10 5 9

The Logic of Correla�on

The important ques�on here is whether and to what extent we can predict grades based on study �me. One common sta�s�c for tes�ng these kinds of hypotheses is a correla�on, which assesses the linear rela�onship between two variables. A stronger correla�on between two variables translates into a stronger associa�on between them. Or, to put it differently, the stronger the correla�on between study �me and quiz grade, the more accurately you can predict grades based on knowing how long the student spends studying.

Before we calculate the correla�on between these variables, it is always a good idea to visualize the data on a graph. The sca�erplot in Figure 4.2 displays our sample data from the studying/quiz grade study.

Figure 4.2: Scatterplot for quiz grade/hours studied example

Figure 4.3: Curvilinear relationship between arousal and performance

Each point on the graph represents one par�cipant. For example, the point in the top right corner represents a student who studied for 5 hours and earned a 9 on the quiz. The two points in the bo�om right represent students who studied for only 1 hour and earned a 2 and a 3 on the quiz.

There are two reasons to graph data before conduc�ng sta�s�cal tests. First, a graph conveys a general sense of the pa�ern—in this case, students who study less appear to do worse on the quiz. As a result, we will be be�er informed going into our sta�s�cal calcula�ons. Second, the graph ensures that there is a linear rela�onship between the variables. This is a very important point about correla�ons: The math is based on how well the data points fit a straight line, which means nonlinear rela�onships might be overlooked. Figure 4.3 demonstrates a robust nonlinear finding in psychology regarding the rela�onship between task performance and physiological arousal. As this graph shows, people tend to perform their best on just about any task when they have a moderate level of arousal.

When arousal is too high, it is difficult to calm down and concentrate; when arousal is too low, it is difficult to care about the task at all. If we simply ran a correla�on with data on performance and arousal, the correla�on would be zero because the points do not fit a straight line. Thus, it is cri�cal to visualize the data before jumping ahead to the sta�s�cs. Otherwise, you risk overlooking an important finding in the data.

Interpre�ng Coefficients

Once we are sa�sfied that our data look linear, it is �me to calculate our sta�s�cs. This is typically done using a computer so�ware program, such as SPSS (IBM), SAS/STAT (SAS), or Microso� Excel. The number used to quan�fy our correla�on is called the correla�on coefficient. This number ranges from –1 to +1 and contains two important pieces of informa�on:

The size of our rela�onship is based on the absolute value of our correla�on coefficient. The farther our coefficient is from zero in either direc�on, the stronger the rela�onship between variables. For example, both a + .80 and a – .80 indicate strong rela�onships. The direc�on of the rela�onship is based on the sign of our correla�on coefficient. A + .80 would indicate a posi�ve correla�on, meaning that as one variable increases, so does the other variable. A – .80 would indicate a nega�ve correla�on, meaning that as one variable increases, the other variable decreases. (Refer back to Sec�on 2.1, Overview of Research Designs, for a review of these two terms.)

So, for example, a + .20 is a weak posi�ve rela�onship and a – .70 is a strong nega�ve rela�onship.

When we calculate the correla�on for our quiz grade study, we get a coefficient of .96, indica�ng a strong posi�ve rela�onship between studying and quiz grade. What does this mean in plain English? Students who spend more hours studying tend to score higher on the quiz.

How do we know whether to get excited about a correla�on of .96? As with all of our sta�s�cal analyses, we look up this value in a cri�cal value table, or, more commonly, let the computer so�ware do this for us. This gives us a p value represen�ng the odds that our correla�on is the result of random chance. In this case, the p value is less than .001. This means that the chance of our correla�on being a random fluke is less than 1 in 1,000, so we can feel pre�y confident in our results. Note, however, that because the p value associated with r is dependent on sample size, even a �ny, unimportant correla�on can be sta�s�cally significant when drawing conclusions from a large popula�on.

We now have all the informa�on we need to report this correla�on in a research paper. The standard way of repor�ng a correla�on coefficient includes informa�on about the sample size and p value, as well as the coefficient itself. Our quiz grade study would be reported as shown in Figure 4.4.

Figure 4.4: Correlation coefficient diagram

So where does this leave our hypothesis? We started by predic�ng that students who spend more �me studying would perform be�er on their quizzes than those who spend less �me studying. We then designed a study to test this hypothesis by collec�ng data on study habits and quiz grades. Finally, we analyzed these data and found a significant, strong, posi�ve correla�on between hours studied and quiz grade. Based on this study, our hypothesis has been supported—students who study more have higher quiz grades! Of course, because this is a correla�onal study, we are unable to make causal statements. It could be that studying more for an exam helps you to learn more. Or, it could be the case that previous low quiz grades make students give up and study less. Or, the third variable of mo�va�on could cause students to both study more and perform be�er on the quizzes. To tease these explana�ons apart and determine causality, we will need an experimental type of research design, which we will cover in

Chapter 5.

Regression Analysis

Correla�ons are the best tool for tes�ng the linear rela�onship between pairs of quan�ta�ve variables. However, in many cases, we are interested in comparing the influence of several variables at once. Imagine that you wanted to expand the inves�ga�on about hours studying and quiz grade by looking at other variables that might predict students' quiz grades. We have already learned that the hours students spend studying posi�vely correlate with their grades. But what about SAT scores? We might predict that students with higher standardized test scores will do be�er in all of their college classes. Or what about the number of classes that students have previously taken in the subject area? We might predict that increased familiarity with the subject would be associated with higher scores. In order to compare the influence of all three variables, we will use a slightly different analy�c approach. Mul�ple regression analysis is a varia�on on correla�onal analysis; in it, more than one predictor variable is used to foresee a single outcome variable. In this example, we would be a�emp�ng to predict the outcome variable of quiz scores based on three predictor variables: SAT scores, number of previous classes, and hours studied.

Mul�ple regression analysis requires an extensive set of calcula�ons; consequently, it is always done using computer so�ware. A detailed look at these calcula�ons is beyond the scope of this book, but a conceptual overview will help you understand the unique advantages of this form of analysis. Essen�ally, the calcula�ons for mul�ple regression are based on the correla�on coefficients between each of our predictor variables, and between each of these variables and the outcome variable. These correla�ons for our revised quiz grade study are shown in Table 4.2. If we scan the top row, we can see the correla�ons between quiz grade and the three predictor variables: SAT score (r = .14), previous classes (r = .24), and hours studied (r = .25). The remainder of the table shows correla�ons between the various predictor variables; for example, hours studied and previous classes correlate at r = .24. When we conduct mul�ple regression analysis using computer so�ware, the so�ware will use all of these correla�ons in performing its calcula�ons.

Table 4.2: Correla�ons for a mul�ple regression analysis

Quiz Grade SAT Score Previous Classes Hours Studied

Quiz Grade — .14 .24* .25*

SAT Score — .02 –.02

Previous Classes — .24*

Hours Studied —

The advantage of mul�ple regression is that it considers both the individual and the combined influence of the predictor variables. Figure 4.5 is a visual diagram of the individual predictors of quiz grades. The numbers along each line are known as regression coefficients, or beta weights. These are standardized coefficients that allow comparison across predictors. Their values are very similar to correla�on coefficients but differ in an important way: They represent the effects of each predictor variable while controlling for the effects of all the other predictors. That is, the value of b = .21 linking hours studied with quiz grades is the independent contribu�on of hours studied, controlling for SAT scores and previous classes. If we compared the size of these regression coefficients, we would see that, in fact, hours spent studying were s�ll the largest predictor of quiz grades (b = .21) compared with both SAT scores (b = .14) and previous classes (b = .19).

Figure 4.5: Predictors of quiz grades

Even if individual variables have only a small influence, they can add up to a larger combined influence. So, if we were to analyze the predictors of quiz grades in this study, we would find a combined mul�ple correla�on coefficient of r = .34. The mul�ple correla�on coefficient represents the combined associa�on between the outcome variable and the full set of predictor variables. Note that in this case, the combined r of .34 is larger than any of the individual correla�ons (r) in Table 4.2, which ranged from .14 to .25. These numbers mean that we are be�er able to predict quiz grades from examining all three variables than we are from examining any single variable. Or, as the saying goes, the whole is greater than the sum of its parts!

Mul�ple regression analysis is an incredibly useful and powerful analy�c approach, but it can also be a tough concept to grasp. Before we move on, let's revisit the concept in the form of an analogy. Imagine you've just eaten the most delicious hamburger of your life and are determined to understand what made it so good. Lots of factors will contribute to the taste of your hamburger: the quality of the meat, the type and amount of cheese, the freshness of the bun, perhaps the smoked chili peppers layered on top. If you were to approach this inves�ga�on using mul�ple regression analysis, you would be able to separate out the influence of each variable (e.g., How important is the cheese compared with the smoked peppers?) as well as take into account the full set of ingredients (e.g., Does the freshness of the bun really ma�er when the other elements taste so good?). Ul�mately, you would be armed with the knowledge of which elements are most important in cra�ing the perfect hamburger. And you would understand more about the perfect hamburger than if you had examined each ingredient in isola�on.

Chi-Square Analyses

Both correla�ons and regressions are well suited to tes�ng hypotheses about predic�on, as long as it is possible to demonstrate a linear rela�onship between two variables. But linear rela�onships require that variables be measured on one of the quan�ta�ve scales—that is, ordinal, interval, or ra�o scales (see Sec�on 2.3 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec2.3#sec2.3) , Scales and Types of Measurement, for a review). What if we wanted to test the associa�on between nominal, or categorical, variables? In these cases, we would need an alterna�ve sta�s�c called the chi-square sta�s�c, which determines whether two

nominal variables are independent from or related to one another. Chi-square is o�en abbreviated with the symbol X2, which shows the Greek le�er chi with the superscript 2 for squared. (This sta�s�c is also referred to as the chi-square test for independence—a slightly longer but more descrip�ve synonym.)

The idea behind this test is similar to that of the correla�on coefficient. If two variables are independent, then knowing the value of one variable does not tell you anything about the value of the other. As we will see in the following examples, a larger chi-square reflects a larger devia�on from what we would expect by chance and is thus an index of sta�s�cal significance.

The Logic of Chi-Square

To determine whether two variables are associated, the chi-square works by comparing the observed frequencies (collected data) with the expected frequencies if the variables were unrelated. If the results significantly deviate from these expected frequencies, then we conclude that our variables are related. And, consequently, we are able to predict one variable based on knowing the values of the other. Let's look at a couple of examples to make this more concrete.

First, let's say we wanted to know whether gender is related to poli�cal party affilia�on. We might randomly select 100 men and 100 women and ask them whether they iden�fied as Republican or Democrat. Because both of these variables are nominal—that is, they iden�fy only categories, not quan�ta�ve measures— chi-square will be our best choice to test the associa�on between them. The first step in conduc�ng this analysis is to arrange our data in a con�ngency table, which displays the number of individuals in each of the combina�ons of our nominal variables. We encountered these tables before in our examples of observa�onal studies in Chapter 3 (Sec�on 3.1 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec3.1#sec3.1) ) but stopped short of conduc�ng the sta�s�cal analyses. So imagine we get the results shown in Table 4.3a from our survey of gender and party affilia�on.

Table 4.3a: Gender and party affilia�on

Male Female

Democrat 60 60

Republican 40 40

In this case, there is no associa�on between sex and party affilia�on. It does not ma�er that the sample consists of 60% Democrats and 40% Republicans. What ma�ers for our hypothesis test is that the pa�ern for males is the same as the pa�ern for females: Our sample consists of 1.5 �mes the number of Democrats for both sexes. In other words, knowing a person's sex does not tell us anything about their poli�cal affilia�on.

For illustra�on purposes, imagine we tested the same hypothesis again but could recruit only 50 women, compared with 100 men. Now, if we found the same 60%/40% split among men again, and assuming that the variables were s�ll not related, here's the ques�on: What would we expect the split to look like among women? If the ra�o of Democrats to Republicans remains at 1.5 to 1, we would expect to see the women divided into 30 Democrats and 20 Republicans (i.e., the same ra�o; shown in Table 4.3b). This concept is referred to as the expected frequency, or the frequency you would expect to see if the variables were not related. In this example, we have a 60/40 split among men. If gender were unrelated to party affilia�on, we would expect to see the same pa�ern among women.

Table 4.3b: Gender and party affilia�on with unequal ns

Male Female

Democrat 60 30

Republican 40 20

The chi-square sta�s�c is calculated by comparing our observed data with these expected frequencies. In our gender and party affilia�on example, the observed data match the expected frequencies, meaning that the variables are not related. Let's walk through another example and see how these calcula�ons work.

Calcula�ng Chi-Square

In this second example, imagine we wanted to know whether people in rural or urban areas were more likely to support a sales tax increase. It would be easy to speculate why either group might be more likely to do so—perhaps people living in ci�es are more poli�cally liberal or perhaps people living in small towns are be�er able to see the benefits of higher local taxes. So, once again, imagine we surveyed a sample of 100 people, asking them to indicate both their party affilia�on and their support for a sales tax proposal. We get the following con�ngency table of results (in Table 4.4a). No�ce that we have more urban than rural residents, reflec�ng the higher popula�on density in ci�es. But, as with our preceding gender and poli�cal affilia�on example, the raw numbers are less important than the ra�os within each group.

Table 4.4: Chi-square example: Support for a sales tax increase

4.4a: Observed data

Rural Urban Total

Support 10 45 55

Don't support 30 15 45

Total 40 60 100

4.4b: Expected frequencies

Rural Urban Total

Support 10 (24.75) 45 (33) 55

Don't support 30 (18) 15 (27) 45

Total 40 60 100

4.4c: Calcula�ng devia�ons between observed and expected values

Rural Urban Total

Support (10 – 24.75)2/24.75 (55 – 33)2/33 55

Don't support (30 – 18)2/18 (15 – 27)2/27 45

Total 40 60 100

4.4d: Devia�ons between observed and expected values

Rural Urban Total

Support 8.79 14.67 65

Don't support 8 5.33 45

Total 40 60 100

The first stage in calcula�ng our chi-square is to determine the expected frequencies. We begin by calcula�ng the sums across each row in column, as shown in Table 4.4a. This gives us a sense of the overall pa�erns in the data. Overall 40% of the sample consisted of rural residents, compared with 60% urban residents. And, overall, 55% supported the sales tax increase, while 45% did not. But, as in our previous example, these descrip�ve sta�s�cs do not tell us anything about the rela�onship between the two variables. If the variables are independent, then the 55/45 split in support for the sales tax will not differ based on where people live.

So we need to determine how much these observed data differ from what would be expected under independence. That is, what would these cells look like if there were no rela�onship? These expected frequencies are calculated using the following formula:

Expected Frequency= R × C Total N

For each of the four cells, we mul�ply the row total (R) by the column total (C), and then divide by the total N in the sample. For example, in the rural resident/support cell, we would mul�ply the row total (55) by the column total (40), and then divide by the total sample size (100); (55 × 40) ÷ 100 = 22. Conceptually, this makes perfect sense: If the two variables are unrelated, we can guess the value of each cell using the overall totals. Table 4.4b shows expected frequencies for each cell in parentheses.

The second stage in calcula�ng chi-square is to determine the extent to which our observed data deviate from these expected frequencies. We will need to calculate this devia�on in each cell and then add them up for a total chi-square. So, for each cell we (1) subtract the expected from the observed value; (2) square the difference to remove any nega�ve numbers; and (3) divide by the expected frequency in order to standardize the devia�on. The final chi-square value is obtained by adding up all four of these devia�on scores, which translates into the following formula:

x 2 = ∑ ( observed - expected ) 2 expected

For example, in our rural resident/support cell, we calculated an expected frequency of 2, represen�ng the number we would expect under independence. But in our sample, there were 10 people in this cell. To calculate how much this deviates from what is expected, we (1) subtract 22 from 10 (= –12), (2) square this difference to remove the nega�ve number (=144), and (3) divide this by the expected frequency to standardize the devia�on (144 ÷ 22 = 6.55). Tables 4.4c and 4.4d illustrate the steps for obtaining these devia�on scores in each of the four cells.

Finally, we add up all four of our devia�on scores (one for each cell) to get the total chi-square value:

x 2 = ∑ ( observed-expected ) 2 expected =6.55 + 14.67 + 8 + 5.33 = 34.55

Our final chi-square value, 34.55, represents the sum of our devia�ons from the expected value. The larger this number, the more our observed data differ from the expected frequencies. Remember that these expected frequencies represent our null hypothesis—we would expect these frequencies only if the variables were unrelated. So the greater our chi-square value, the more our variables are related to one another. In the present example, this means we can predict a person's support for a sales tax increase based on where he or she lives, which is consistent with our ini�al hypothesis.

But how do we know whether our value of 34.55 is meaningful? As with the other sta�s�cal tests we have discussed, this requires looking up our result in a cri�cal value table to determine whether the calculated value is above threshold. In this case, the cri�cal value for a chi-square with a 2 × 2 table = 3.84, so we can feel confident in our value of 34.55—almost 10 �mes higher than the threshold value!

However, unlike correla�on and regression coefficients, our chi-square results cannot tell us anything about the direc�on or magnitude of the rela�onship. A larger chi-square reflects a larger devia�on from what we would expect by chance and is thus an index of sta�s�cal significance. In order to interpret the pa�erns of our data, we need to visually inspect the numbers in our data table. Be�er yet, we can create a bar graph like we did in Chapter 3 to visually display these frequencies.

As Figure 4.6 shows, the cell frequencies suggest a fairly clear interpreta�on: People who live in urban se�ngs are much more likely than people who live in rural se�ngs to support a sales tax increase. In fact, urban residents support the increase by a 3-to-1 margin, while rural residents oppose the increase by a 3-to-1 margin.

Figure 4.6: Graph of chi-square results

4.5 Ethical Issues in Survey Research

Like all research, surveys should be carried out in ways that protect and avoid harm to the par�cipants. Informed consent, as we discussed in Chapter 1 (Sec�on 1.7 (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/sec1.7#sec1.7) , Ethics in Research), is a requirement for survey research and should include a clear descrip�on of the survey content and the purpose for conduc�ng the research. Whether the survey is completed in person or online, a cover le�er should be included that contains this informa�on. Although not always required for surveys (as there is minimal risk of harm to par�cipants taking ques�onnaires and surveys), some researchers also like to obtain signed consent forms from the par�cipants. This is especially true for ins�tu�onal review boards (IRBs), who want to ensure that par�cipants were fully informed of any sensi�ve informa�on that may be collected, any poten�al limits to the confiden�ality of the data, or access to private records (such as medical records) that are being sought in addi�on to the survey. In any case, a researcher should never administer a survey if the par�cipant has not provided verbal or wri�en consent and should never use the data other than for reasons for which the par�cipant provided consent.

Assurances of par�cipant anonymity and confiden�ality are also important, especially with respect to response rates to sensi�ve ques�ons and response rates in general. Par�cipants are likely to feel more comfortable responding to sensi�ve subjects when they know that they are par�cipa�ng anonymously. Anonymity ensures that there is no way for their responses to be linked back to them. Conduc�ng survey research in a completely anonymous manner is much easier with online surveys; however, researchers can take steps to protect the anonymity of par�cipants in individual or group administra�ons as well. For example, anyone who has access to the surveys and completed data must commit in wri�ng to preserving their confiden�ality. Any links between the answers and the par�cipants' personal iden�fying informa�on, such as names, email addresses, and phone numbers, should be minimized by removing the la�er from the data collec�on and coding it (e.g., numbering each par�cipant with an ID code rather than using his or her name). If personal informa�on must be kept, it should be separated from the survey responses and destroyed as soon as the study is over. Any person who can iden�fy the par�cipant by looking at the pa�ern of responses, such as a supervisor, should not be permi�ed to view the survey responses. It is also important to be careful when repor�ng results for a small subpopula�on of the sample, whose personal informa�on may be iden�fiable. Finally, when the study is completed, researchers must ensure that they either destroy all survey responses and personal informa�on or store them securely.

Another important considera�on for researchers is to be aware of their own biases and the impact that those might have on the tes�ng process. This is especially important in face-to-face inquiries. For example, if a researcher has a strong opinion or bias toward a par�cular topic, the ques�ons might be worded in a way that persuades the par�cipants to answer in a specific manner. Addi�onally, if the researcher has a strong belief about the topic, he or she may make this evident during the tes�ng process, which may encourage the par�cipant to respond in a way that is consistent with what the researcher believes. As we discussed in Chapter 3 (Sec�on 3.1), par�cipant-expectancy bias can occur during observa�ons and interviews, as well as during survey research, and needs to be considered when analyzing and interpre�ng data.

Finally, when using standardized ques�onnaires or surveys that have been developed by other researchers, professional training and competence in the tests being used are essen�al. It is unethical for any researchers to administer, score, and interpret a test that they have not been trained on. In addi�on, it is unethical for researchers to select tests based on limited knowledge and experience and assume that these tests are reliable and valid for the purpose for which they are using them.

Summary

This chapter has covered the process of survey research from conceptualiza�on through analysis. We first discussed the types of research ques�ons that are best suited to survey research—essen�ally, those that can be answered based on people's observa�ons of their own behavior and characteris�cs. Survey research can involve either verbal reports (i.e., interviews) or wri�en reports (i.e., ques�onnaires). In both cases, surveys are dis�nguished by their reliance on people's self- reports of their a�tudes, feelings, and behaviors.

This chapter covered several key points for wri�ng survey items. The take-home point to our Five Rules for Designing Be�er Ques�onnaires is that your ques�ons should be wri�en as clearly and unambiguously as possible. This helps to minimize the error variance that might result from par�cipants imposing their own guesses and interpreta�ons on the material. In designing survey items, you also have a broad choice between open-ended and fixed-format responses. The former provide richer and more extensive data but are harder to score and code; the la�er are easier to code but can constrain people's responses to your choice of categories. If and when you se�le on a fixed-format response, you have another set of decisions to make regarding the response scaling, labels, and general format.

Once you have constructed the scale, it is �me to begin collec�ng the data. This chapter discussed the concept of sampling, or choosing a por�on of the popula�on to use for your study. Broadly speaking, sampling can be either "probability" or "nonprobability," depending on whether you have a known popula�on size from which you sample randomly. Probability sampling is more likely to result in a representa�ve sample, but this approach is not possible in all studies. In fact, a significant propor�on of psychology research studies uses a form of nonprobability sampling called convenience sampling, meaning that the sample consists of those who show up for the study.

This chapter also covered three approaches to analyzing survey data and tes�ng hypotheses about predic�on. The first, correla�onal analysis, is a very popular way to analyze survey data. The correla�on is a sta�s�cal test that gives an assessment of the linear rela�onship between two variables. The stronger the correla�on between variables, the more we can accurately predict one based on knowing the other. Second, regression analyses allow us to expand our inves�ga�ons into mul�ple predictors. The advantage of mul�ple regression analysis is that it considers both the individual and the combined influence of the predictor variables. However, both correla�on and regression require the variables to be quan�ta�ve—that is, measured on an ordinal, interval, or ra�o scale. In cases where our survey produces nominal or categorical data, we use an alterna�ve called the chi-square sta�s�c, which determines whether two nominal variables are independent or related. The chi-square works by examining the extent to which our observed data deviate from the pa�ern we would expect if the variables were unrelated— that is, the null hypothesis. The common thread running through these analyses is that they measure the associa�on between variables and do not tell us anything about the causal rela�onship between them. To make causal statements, we have to conduct experiments, which we will cover in the next chapter.

Finally, this chapter discussed the ethical issues that arise in survey research. In addi�on to the concepts discussed in Chapter 1 regarding informed consent and confiden�ality, it is important that researchers conduc�ng survey research ensure anonymity so that par�cipants feel safe responding to sensi�ve topics and comfortable par�cipa�ng in the overall study. Par�cipant response rates tend to be higher when par�cipants know that there is no way to link them to their responses. Addi�onally, the researcher should be aware of his or her own biases and the impact that these might have on the collec�on, analysis, and interpreta�on of the data, as well as on how par�cipants may respond. Also, when using standardized tests or tests that have been developed by other researchers, it is impera�ve that researchers be trained and competent in administering and scoring the results, as well as in interpre�ng those results.

Key Terms

Click on each key term to see the defini�on.

anchors (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Labels, or endpoints, for a ra�ng scale.

beta weights (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

See regression coefficients.

bipolar scale (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Ra�ng scale that has polar opposites as its anchors.

branching schedule (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

An interview format in which ques�ons take different direc�ons depending on par�cipants' answers.

chi-square sta�s�c (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A sta�s�cal test similar to the correla�on coefficient; determines whether two nominal variables are independent or related.

cluster sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A varia�on of simple random sampling that involves dividing the sample into groups based on more than one level of analysis.

con�ngency table (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A data summary table that shows the number of individuals in each combina�on of the nominal variables; used as the first step in calcula�ng chi-square.

convenience sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A nonprobability sampling strategy that involves simply enrolling people who show up for the study.

correla�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Sta�s�cal test that assesses the linear rela�onship between two variables; the stronger the correla�on between variables, the more accurately the predic�on about one based on knowing the other.

correla�on coefficient (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The number used to quan�fy a correla�on; this coefficient (r) ranges from –1 to +1 and contains informa�on about both the size and direc�on of the correla�on.

criterion sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A qualita�ve sampling method used to select cases that meet a predetermined criterion of importance.

double-barreled ques�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A flawed survey item that asks more than one ques�on at a �me.

error of es�ma�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The degree to which the data from the sample are expected to deviate from the popula�on as a whole.

error variance (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Variance from random sources that are irrelevant to the trait or ability that a ques�onnaire is purpor�ng to measure.

expected frequency (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The frequency one would expect to see if the variables were not related.

expert sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Sampling of a panel of individuals who have known exper�se (knowledge and training) in a par�cular area.

extreme or deviant case sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A form of sample selec�on used when the researcher wants informa�on-rich data on unusual or special cases; the focus is on examining both ends of the spectrum of outcomes (e.g., outstanding successes and failures of the phenomenon being studied).

fixed-format response (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Answer to a limi�ng ques�on or statement, involving choosing from a list of op�ons; on a survey, fixed-format responses are easier to code but can constrain the data into narrow categories.

forced choice (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A ra�ng scale that requires respondents to agree or disagree with a statement, usually through the use of an even number of scale points.

heterogeneous (maximum varia�on) sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A qualita�ve sampling method that may include both extreme and typical cases; used to select a wide variety of cases in rela�on to the phenomenon being inves�gated.

interview (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A verbally administered survey.

interview schedule (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A plan, or script, for the progress of an interview, describing the list of ques�ons and the order in which they should be asked.

leading ques�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A flawed survey item worded in a way that suggests an answer.

Likert scale (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Format that uses anchors of "strongly agree" and "strongly disagree" to rate responses to a survey ques�on.

linear schedule (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

An interview format that asks the same ques�ons in the same order for all par�cipants.

mul�ple-choice format (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A fixed-formatresponse survey format that asks par�cipants to select from a set of predetermined responses.

mul�ple correla�on coefficient (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A number that represents the combined associa�on between the outcome variable and the full set of predictor variables.

mul�ple regression analysis (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A varia�on on correla�onal analysis in which more than one predictor variable is used to predict a single outcome variable.

nonprobability sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A group of sampling strategies used when the odds of any given individual's being in the sample are unknown.

nonresponse bias (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Bias introduced when certain ques�ons or en�re surveys are not completed by par�cipants, affec�ng the characteris�cs and size of the sample.

open-ended response (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Unstructured answer to a ques�on or statement; on a survey, open-ended responses provide rich data but are difficult to code.

pilot tes�ng (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A "test run" of a survey that involves giving the ques�onnaire to a small sample of people, ge�ng their feedback, and making any necessary changes.

popula�on (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The en�re collec�on of people who could possibly be relevant for a study.

probability sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A group of data-collec�on strategies used when each person in the popula�on has a known chance of being in the sample.

purposive sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A qualita�ve sampling method that includes selec�ng rela�vely small samples in a purposeful manner.

ques�onnaire (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A survey that is administered in wri�ng.

ra�ng scale (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A fixed-format response that asks par�cipants to place responses on a con�nuum.

regression coefficients (beta weights) (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Values that represent the effects of each predictor variable while controlling for the effects of all the other predictors.

sampling bias (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The failure of the sample to represent the underlying distribu�on in the popula�on.

sampling error (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The degree to which the characteris�cs of the sample differ from the characteris�cs of the popula�on.

sampling frame (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A list of all members of a par�cular popula�on (e.g., a list of every resident of the United States) and a necessary requirement for probability sampling strategies.

self-reports (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Par�cipants' reports of their own a�tudes, feelings, and behaviors.

self-selec�on bias (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Bias introduced into a survey study when the researcher receives responses only from those who are interested in the topic.

simple random sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A probability sampling strategy that involves randomly picking par�cipants from a list of everyone in the popula�on.

snowball sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A nonprobability sampling strategy that involves recrui�ng by word-of-mouth referrals.

social desirability (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Par�cipants' reluctance to give unpopular answers to survey ques�ons; concern over how their a�tude will be perceived.

stra�fied random sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A varia�on of simple random sampling, used when subgroups of the popula�on might be le� out of a purely random sampling process; breaking the sampling frame into subgroups and then sampling a random number from each subgroup.

survey research (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Any method that relies on people's observa�ons of their own behavior.

theory-based sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A qualita�ve sampling method (a version of criterion sampling) that focuses on obtaining cases that represent theore�cal constructs.

true/false format (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

A fixed-format survey response that asks par�cipants to indicate whether they endorse a statement.

typical case sampling (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Research method that involves sampling the most frequent or "normal" case from a popula�on.

unipolar scale (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Ra�ng scale that assesses a single construct.

Chapter 4 Flashcards

Research Scenarios: Try It

Scenario 4

Mindfulness can be defined as "awareness of the present moment without judgment". Recent research has found that a tendency to be mindful in everyday life is associated with reduced risk for emo�onal disorders such as depression and anxiety.

Suppose you are interested in understanding the role that mindfulness may play in reducing nega�ve reac�ons to stressful events. You decide to inves�gate this rela�onship in a sample of college students who undergo a significant stressor at the same �me: final exams. You hypothesize that college students who are less mindful will be more stressed during final exams than those who are more mindful.

1. What psychological construct(s) would you have to measure in order to test this hypothesis?

a. Mindfulness b. Mindfulness and final exams c. Mindfulness and well-being d. Mindfulness and stress

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

To test the hypothesis that mindfulness predicts nega�ve reac�ons to the stress of finals, you will need to measure (d) both mindfulness and stress. Because all students will have to take final exams, exams are not a variable in the study and do not need to be measured. Although the rela�onship between well-being and mindfulness may be interes�ng and relevant to future studies in this general area, your par�cular hypothesis specifically concerns the rela�onship between mindfulness and stress.

labels, or endpoints, for a rating scale

Click card to see term 👆

Choose a Study ModeView this study set

2. Suppose you decide that a ques�onnaire survey would be the most appropriate method to measure mindfulness and stress. While you can use exis�ng ques�onnaire measures of stress, you decide to create your own survey measure of mindfulness. What problem might be a par�cular concern in construc�ng a ques�onnaire asking about the tendency to be mindful?

a. Social desirability b. Inefficient data collec�on c. Double-barreled ques�ons d. May be too �me consuming

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

In construc�ng a ques�onnaire survey about mindfulness, the researcher should be aware of how (a) social desirability might influence responses. The ability to be aware of your surroundings and remain focused on the present are valued characteris�cs in American culture, so students may be reluctant to admit how unmindful they really are. It is important, therefore, to construct ques�ons with neutral wording as much as possible to reduce this source of bias. Although double-barreled ques�ons (c) are always something to avoid when construc�ng ques�onnaires, there is nothing in our topic that makes this a par�cular concern here. In contrast to more involved methods, a simple ques�onnaire survey actually has the advantages that it is (b) efficient and (d) fast way to gather informa�on.

3. Suppose you want to create a ques�onnaire item with the following base: "When I have distressing thoughts or images, I just no�ce them and let them go." Which of the following response op�ons would be both easy to score and appropriate for this ques�on?

a. True/false b. Mul�ple choice c. Likert-style ra�ng scale d. Open-ended

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The best choice for an easily-scored response op�on for this ques�on is (c) a Likert-style ra�ng scale. For example, par�cipants might rate how much the statement describes them on a scale of 1 to 7. Behavioral traits such as mindfulness are similar to a�tudes and opinions, in that we tend to think of them on a con�nuum rather than a yes/no category. In other words, people are not likely to respond that they always or never behave certain way—so the true/false response op�on is not appropriate. Similarly, other types of mul�ple choice response op�ons would not be as well suited as a Likert-style scale. Although an open-ended ques�on would also allow a nuanced response, it would not be easy to score.

4. The design of your study requires that par�cipants be students who undergo a final exam. Suppose you run this study at a local college, and adver�se by pos�ng flyers around school invi�ng all students to par�cipate in the survey. There are no eligibility requirements other then being a student and having a final exam at the end of the quarter. What type of sampling does this represent?

a. Snowball sampling b. Convenience sampling c. Cluster sampling d. Stra�fied random sampling

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

Your college-student study is a common example of (b) convenience sampling. This sampling strategy is inexpensive and less resource-intensive than other sampling methods, which a�empt to create a sample that is representa�ve of the broader popula�on of interest. Ul�mately, we are interested in generalizing our findings beyond college students and the stress of finals. However, obtaining a convenience sample is an excellent way to begin researching this ques�on. If the results are promising, it is likely worth inves�ng the resources to create a more representa�ve sample of individuals and stressors.

5. Suppose you administer your mindfulness ques�onnaire at the beginning of the semester, and administer your stress ques�onnaire to the same students during finals. Which of the following correla�ons between mindfulness and stress would provide support for your original hypothesis?

a. r = .65, p < .05 b. r = −.65, p < .05 c. r = −. 20, p > .10 d. r = .001, p > .1

Click here for the answer. (h�p://content.thuzelearning.com/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover/books/Malec.5743.13.1/sec�ons/cover

The correct answer is (b)—observing a correla�on of r = −.65 (p < .05) would support your original hypothesis that mindfulness predicts stress in response to finals. Because we expect that higher levels of mindfulness will correspond to lower levels of stress, the expected correla�on is nega�ve, rather than posi�ve. Because the p value is below 0.05, we can be assured that our results are very unlikely due to chance alone.

Apply Your Knowledge

1. For each of the following poorly wri�en ques�onnaire items, iden�fy the major problem and then rewrite it so that the problem is resolved. a. How much do you like cats and ponies?

main problem: be�er item:

b. Do you think that John McCain's complete lack of personality proved that he would have been a terrible president? main problem: be�er item:

c. Do you dislike not playing basketball? main problem: be�er item:

d. Do you support SB 1070? main problem: be�er item:

e. How o�en do you take drugs? main problem: be�er item:

2. Dr. Truxillo is interested in Arizona residents' thoughts and feelings about global warming. For each of the following examples, state the sampling method used by her research assistants.

a. Reese sets up a table in the mall and hands a survey to people who approach her. b. Catherine randomly chooses 5 ci�es, then chooses 3 neighborhoods in each, then randomly samples 5,000 households for a phone survey. c. Jason starts with a list of the en�re popula�on of Arizona and selects par�cipants by dialing random phone numbers. d. Anna gets the master list from Jason and divides the popula�on according to educa�on level. She then randomly chooses 500 high school dropouts, 500 college

graduates, and 500 people with some postgraduate educa�on.

3. Based on each of the following study descrip�ons, choose whether the best analysis would be a correla�on, a mul�ple regression, or a chi-square. a. Jim is interested in the rela�onship between annual income and self-reported happiness. b. Shelia is interested in whether some ethnic groups are more likely to use counseling services (a yes-or-no ques�on). c. Angela is interested in knowing the best predictors of recovery from depression, comparing the influence of drugs, therapy, and family resources. d. Adam is interested in whether high school dropouts or college graduates are more likely to vaccinate their children. e. Nicole is interested in understanding the best predictors of weight loss. f. Samantha is interested in the rela�onship between self-esteem and prejudice.

Cri�cal Thinking & Discussion Ques�ons

1. In survey research, explain the trade-off between the "richness" of people's responses and the ease of analyzing their responses.

2. When doing interviews, the researcher has a personal interac�on with the subject. Why is this both good and bad?