Discussion 1

Research Methods

Shamus Khan, Princeton University

Gwen Sharp, Nevada State College

Research Methods (Fall 2021 Edition)

2

Research Methods

S H A M U S K H A N , P R I N C E T O N U N I V E R S I T Y

G W E N S H A R P , N E V A D A S T A T E C O L L E G E

INTRODUCTION

The importance of being wrong

Research ethics

TYPES OF RESEARCH METHODS

Five common sociological methods

Choosing a method

DESIGNING A RESEARCH METHOD

From topic to question

Variables

Independent and dependent variables

From research question to hypothesis

Selecting a sample

CORRELATION & CAUSATION

Validity & Reliability

CONCLUSION

Research Methods (Fall 2021 Edition)

3

INTRODUCTION

 How do we “know” things about the social world?

 What principles guide ethical research on people?

For decades, scholars knew that people who had served time in prison are much less

likely to have a job than other people are, but we didn’t exactly know why. The answer may

seem obvious, but as it turns out, there are lots of possible answers. One is discrimination:

perhaps employers just don’t trust people who were incarcerated and don’t hire them. Or

maybe people with criminal records are somehow different than other job applicants—

perhaps they aren’t very interested in working, so they don’t search very hard for jobs or quit

more quickly if they don’t like their coworkers. Maybe they missed out on getting important

training and skills while they were in prison, so they aren’t as qualified as other job applicants.

Or they might have trouble following rules, so they get fired.

Which explanation is correct? Are several of them accurate? How would we know?

Devah Pager studied this question as

a graduate student. She conducted an audit

study to look for an answer.1 She sent young

people to apply for jobs to see who was

most likely to get an interview; two people

applied for each position. She created fake

resumés for them to use with fake

qualifications that were similar, with one

exception: whether or not they had a (fake)

criminal record for a non-violent drug offense

(she also used Black and White applicants, to

see whether race mattered; you’ll learn

more about that in another chapter).

The advantage of an audit study is that if everything about the applicants is carefully

matched except one characteristic, then any differences you see must be explained by the

one thing that was different—in this case, whether applicants said they had a criminal history.

And Pager found that it mattered: having a criminal record affected the applicants’ chances

of getting an interview. Even though their qualifications were the same, applicants who

revealed their criminal record were less likely to be called back for an interview.

When Pager decided to use an audit study, she was following a particular method—a

study design that allows us to systematically investigate the world and be relatively certain

that we arrive at accurate conclusions. Sociology is a social science, and a critical aspect of

any science is that there are agreed-upon ways to generate knowledge. This sets science

apart from other ways of explaining the world, such as common sense or religious faith. At the

(Source)

Research Methods (Fall 2021 Edition)

4

core of scientific methods is a particular research attitude: skepticism. No matter who makes a

claim, and even if it seems to make sense, the job of scientists is to be skeptical of the claim

and to try to find problems with it.

All scientific studies of the social world share a key feature: scholars work hard to find

evidence that our conclusions are wrong. This may seem confusing – don’t we want to show

that our conclusions are right? But this is how scientific knowledge advances: it’s not enough

to provide evidence that a claim is right; you must search for evidence that it’s wrong. We’re

never absolutely certain that our claims about the social world are correct, but the more times

we try to show that our claim is wrong and can’t do it, the more comfortable we can be that

our explanation is correct. Whether we’re testing subjects in a lab or wandering the hallways

of a school observing how students and teachers interact, the basic approach is the same: we

look for other potential explanations for what we observe, or any evidence that our claim isn’t

accurate.

Remaining skeptical and considering other explanations can help us avoid confirmation

bias, the tendency we all have to look for and accept information that reinforces what we

already believe.2 Confirmation bias is a basic part of our psychology. We don’t do it on

purpose, and usually we aren’t aware it’s happening. But confirmation bias can lead us to

quickly accept information that matches our existing theories or beliefs, while we remain

doubtful about, or fail to notice, evidence that contradicts what we already think. The

scientific emphasis on searching for evidence that a claim is wrong can help us address this

bias in our thinking as we try to explain the social world around us.

Research ethics

The most essential consideration of any research project should be ensuring the project

is done safely and ethically. Research ethics are important for all research, but they are

especially crucial when you are conducting research on people, or human subjects.3

Unfortunately, scientists haven’t always agreed on what makes research ethical, and

they don’t always design ethical research projects. The most infamous cases involve medical

research. For instance, during World War II, German researchers (mostly doctors) conducted

painful and often deadly experiments on people imprisoned in Nazi concentration camps;4

the prisoners were forced to take part, and the experiments left them with burns, wounds, and

other injuries. Aside from the horrific suffering and death they caused, many of these

experiments had little or no scientific value; they didn’t help scientists cure diseases or

otherwise benefit humanity.

After the war ended, many of these researchers were criminally charged and

convicted. The international outrage at what the Nazi experimenters had done led to the

establishment of the Nuremberg Code in 1948, which outlined basic ethical principles for

research on people.5 The first, and perhaps most important, principle is that people who take

part in research must voluntarily consent to do so; they cannot be forced. The Code also

established other key ethical rules, including the following:

Research Methods (Fall 2021 Edition)

5

 Researchers should avoid all unnecessary physical and mental suffering and injury to

subjects;

 The degree of risk to subjects has to be justified by the likely benefit to humanity of the

knowledge gained from the research;

 Subjects must be free to stop participating at any time;

 If researchers discover their project poses serious risks to human subjects, they must end

the project immediately.

Despite these clear

principles, researchers

sometimes ignored the

guidelines. The Tuskegee Syphilis

Experiment, conducted in

Alabama from 1932 to 1972,

looked at how the symptoms of

syphilis developed over time if

left untreated.6 Researchers from

the U.S. Public Health Service

used hundreds of poor Black

men in rural Alabama as their

subjects. They never told the

men that they had syphilis—they

said they had “bad blood.”

Worst of all, after 1947 there was

a treatment for syphilis: penicillin

could completely cure it in the early stages. Even after the establishment of the Nuremberg

Code in 1948 and its acceptance by the U.S. scientific community, the Tuskegee study

researchers didn’t tell their subjects about the cure or offer them penicillin; they let the men’s

syphilis progress so they could see what happened. Many of the men died when they could

have been cured. Others gave the disease to their female partners, who transferred syphilis to

their children during pregnancy, leading to lifelong complications including seizures and

blindness. The study finally ended in 1972 when a whistleblower reported the project.

The Tuskegee experiment’s lingering impacts came up as a major concern during the

COVID-19 outbreak as public health experts tried to convince people to get tested and, later,

vaccinated. Doctors and others working in Black communities worried that the legacy of the

Tuskegee experiment would make it harder to convince Black Americans to now trust the

medical establishment on the best way to address COVID-19.7 The harm of unethical research,

they argued, isn’t just in the suffering of those directly affected by the study, but in the anger

at and lack of trust in scientists and medical experts that may last for decades. The

understandable mistrust Black communities may feel as a result of past unethical research

could make it harder to effectively treat health issues today. Public health officials worried that

Doctor drawing blood from a patient as part of the Tuskegee Syphilis

Study. (Source: National Archives, Atlanta, GA.)

Research Methods (Fall 2021 Edition)

6

this would lead to more outbreaks of COVID-19 among African Americans, which could then

lead to higher numbers of hospitalizations and deaths among them than in other racial

groups.

However, other researchers found that African Americans’ concerns about the vaccine

were driven by many of the same factors causing other groups to be hesitant—a concern

about its safety or a broader mistrust of how it had been so quickly developed under President

Trump’s administration—and that we should be careful about assuming that African

Americans’ mistrust or hesitancy about medical issues is only rooted in unethical research that

happened in the past.8 Doing so can allow us to see research ethics as part of history, rather

than confronting more recent problematic research as well as unequal treatment in the

medical system that may affect how different racial groups feel about, and how much they

trust, doctors and other healthcare providers today.

There are many other examples of unethical research.9 As a result of such ethical

failures, today federal guidelines attempt to protect research subjects.10 Though most of these

guidelines were established primarily to cover medical research, regulations also cover social

science research. A key requirement is informed consent. This means that all human subjects

must be informed about the research project, including any likely risks, before they agree to

participate. For a participant to give informed consent, they have to fully understand the risks

(and possible benefits) of the research.

While the problems with unethical medical research can appear obvious, it can be

harder to imagine how social scientists could hurt participants. But social scientists often

collect sensitive information about people, and it could be harmful if that information is

released. For instance, imagine you were interviewing married subjects about whether they

had ever had an affair. That information could be very harmful if you released it in a way that

allowed readers to figure out the identities of your participants. It could potentially affect their

reputations in the community or end their marriage, and could also be very embarrassing and

upsetting for their spouse, who wasn’t even a participant in your study. For sociologists,

protecting the privacy and identities of participants is essential; we must make sure that the

research findings we publish do not put participants at risk by releasing private information

that could hurt them.

Research Methods (Fall 2021 Edition)

7

TYPES OF RESEARCH METHODS

 What are the benefits of experiments, surveys, participant observation, historical analysis,

and content analysis?

 What are the weaknesses of each of these methods?

 How do we choose a particular method?

As you plan your research project, you will decide how to collect your data and what

types of data you’ll collect. Data generally fall into two categories: quantitative and

qualitative. Quantitative data come in the form of numbers and reflect quantities or amounts.

Qualitative data aren’t numbers; they usually reflect general themes and might include

transcripts from interviews, survey questions that ask people to explain something in their own

words, or detailed notes from visiting a particular place to observe it. Each of the methods we

review below can produce both quantitative and qualitative data. While some researchers

prefer one or the other, in reality many use a mixture of both.

Five common sociological methods

At the beginning of this chapter, we described Devah Pager’s audit study. Audit studies

are one type of experiment, a research method in which characteristics or behaviors are

carefully controlled. By controlling the environment, researchers can isolate the impacts of the

one characteristic that changes. Perhaps we want to know whether people feel more anxious

after looking at their friends’ social media accounts. We might bring people into a lab and

give them a short survey to measure how anxious they are. We could then have them scroll

through their friends’ social media accounts for 15 minutes and give them the anxiety survey

again afterward. Since nothing else happened during the study, if we find they’re more

anxious after looking at social media than they were before, we can presume that viewing

their friends’ posts increased their anxiety.

Experiments can be extremely useful because they allow us to carefully study the

impact of one thing at a time. Because we can control what happens to subjects, we can

make sure that the only thing that changes is the item we’re interested in. But there are

downsides to experiments, too. Especially for those that take place in a laboratory

environment, researchers may wonder whether the situation was realistic. Would we see the

same effect in the “real world” outside of the carefully-controlled lab? It’s possible that a

relationship that appears in an experimental setting wouldn’t work the same way in our

everyday lives, where we’re never affected by just one factor at a time. Finally, because

experiments give researchers so much control over subjects, it’s especially important to think

about ethical issues when designing them.

Research Methods (Fall 2021 Edition)

8

You may never have taken part in an experiment. But there’s a very good chance

you’ve participated in surveys, or sets of questions that subjects answer. They may be

conducted in person or sent through the mail, but increasingly surveys are completed over the

phone or online. During the 2020 U.S. presidential campaign, you may have received phone

calls asking you to rate how concerned you were about different issues or how likely you were

to vote for a particular candidate. Or maybe you’ve been asked to complete a satisfaction

survey after contacting a customer service office, rating your feelings from “very satisfied” to

“very unsatisfied.” Because so many groups use surveys today—including social scientists,

marketers, political campaigns, companies, and more—you’re likely to encounter them

frequently.

Surveys are a very common

method because they’re a relatively

cheap and quick way to get lots of

information from large groups of

people. That can give us a good

idea of widespread patterns, as well

as differences between groups (for

instance, we might get different

survey responses from men and

women). But surveys can have

problems, too. A common issue is

low response rates; that is, only a

small proportion of people you try to

contact complete the survey

(perhaps because they’re frustrated

from receiving so many requests to complete surveys!). Another problem is wording issues.11

The way you write questions can affect the answers you get. For instance, one group of

political scientists found that people responded differently when asked about “gay or lesbian”

rights than when asked about “homosexual” rights;12 because people tend to feel more

negatively about the word “homosexual,” using it can change how they respond on surveys.

As you read other chapters in this text, you’ll encounter several descriptions of

participant observation.13 In this method, the researcher spends time among a group, directly

observing and participating in that social world. This can mean moving to another country to

live among a different culture, but you can also do participant observation closer to home. For

instance, as she describes in the book Class Acts, sociologist Rachel Sherman worked at the

front desk of two expensive hotels in the U.S. to study how the hotels ensure that their wealthy

guests feel pampered.14

The benefit of participant observation is that it allows researchers to collect a lot of

extremely detailed information about social life in a particular group; we can learn what

people do, how they interact, and what they think about those interactions. Sherman learned

Researchers may visit public places and collect survey responses on

the spot. (Source)

Research Methods (Fall 2021 Edition)

9

about the tactics hotel employees engaged in to create a “luxury” experience. For instance,

room service waiters took notes on how hotel guests like their food served and gift store clerks

kept track of any special requests from guests. This information was entered into a computer

database, allowing one guest to receive her papaya cut exactly the way she wanted without

having to ask each time and another to have his favorite cigarettes waiting in his room on

future visits, though the hotel didn’t normally stock that brand. Observing and actively

participating in life at the hotel allowed Sherman to understand the intricate ways hotel

employees attended to the needs and preferences of their wealthy guests, making the guests

feel valued and effortlessly pampered.

However, participant observation can be

time-consuming and expensive (especially if you

have to move somewhere specifically to do your

research). It may take years to earn the trust of a

group and feel confident that you truly

understand the social world you’re studying

(especially if there are language barriers). And

you’ll only gather data on a small number of

people; you can’t realistically get to know and

talk to thousands of people. This can lead to

questions about whether your findings apply

outside of that small group.15 Finally, two related

methods are historical analysis and content

analysis.16 These methods involve analyzing

existing sources (such as historical records, media stories, or episodes of TV shows) to find key

themes. Sociologists Erin Hatton and Mary Nell Trautner completed a content analysis of Rolling

Stone cover photos, looking at how men and women were sexually objectified by the

magazine.17 Analyzing nudity, poses, and the focus of the photography, they found that

sexualization of both men and women has increased over time, but that women are still

sexualized more often, and to a greater degree, than men. In his study of suicide, Émile

Durkheim used historical death records from towns across France to see how frequently suicide

occurred.18 Content analysis can help us identify recurring themes that are hard to see when

we look at just one instance (for example, we can see patterns in objectification of women by

looking at magazine covers over many years that might not be evident if we looked at just

one example). A weakness of both methods is that you’re stuck with the data that exists,

whether or not it includes all the information you’d like. Maybe you want to look at differences

among racial groups, but you’re using historical documents; if those documents don’t indicate

the person’s race, then you can’t study that topic, no matter how interesting it might be.

Participant observation involves taking detailed

notes about every aspect of the environment.

(Source)

Research Methods (Fall 2021 Edition)

10

Choosing a method

So which method is right for your research project? There’s no simple answer. Any topic

can be studied with any of these methods (and with others; we’ve only covered the most

common here), and every method has strengths and weaknesses.

If you want to understand how thousands of people think about an issue, or what

behaviors they engage in (say, whether cigarette taxes have reduced the number of teens

who smoke19), a survey is likely the best method for your project. On the other hand, maybe

you want to study smoking, but you’re interested in how teens view anti-smoking campaigns

and how interactions with friends and peers affect their decisions to smoke. Then you might

conduct a participant observation in a high school;20 a survey probably won’t get you the

detailed information you need to fully capture how teens navigate the sometimes conflicting

signals from friends, parents, and teachers about smoking. Participant observation might

provide richer, more informative data. Another researcher might want to know how smoking is

portrayed in movies; a content analysis of how often women are shown smoking, particularly

in films aimed at young audiences, would provide insights into how smoking is represented in

pop culture.21 Finally, if you want to see whether those representations in pop culture affect

attitudes about smoking, you could conduct an experiment where you show a scene with a

famous actor smoking and then ask subjects whether they would date someone who smokes.

Each of these studies could provide you with valuable information about smoking.

None of them are automatically better than the others. You have to consider what question

you want to answer, what research skills you’ve developed, and what resources you have

access to. If you don’t have the time or resources to spend months or even years getting to

know people and hanging out with them to observe their interactions, the participant

observation study won’t be realistic for you. If you don’t enjoy using statistics to analyze

quantitative data, or haven’t developed that skill yet, then collecting a large amount of

survey data won’t help you find meaningful patterns.

Every sociological study you read about was designed based on the skills, resources,

and limitations that the researchers faced, as well as what method they thought would best

get at their question. Instead of thinking of a study on its own, it’s helpful to think of it as one

piece in a bigger puzzle, each contributing a small piece to completing the puzzle.

Research Methods (Fall 2021 Edition)

11

REVIEW SHEET: TYPES OF RESEARCH METHODS

CLICK THE LINK FOR:

LEARNING OBJECTIVES KEY QUESTIONS

AUDIO KEY POINTS

PRACTICE QUIZ KEY PEOPLE

VOCABULARY CROSSWORD PUZZLES KEY TERMS

DESIGNING A RESEARCH PROJECT

 What kinds of data can we collect to study the social world?

 What elements do we include when stating a hypothesis?

 What are the benefits of different types of sampling?

While the exact steps of a research project may vary somewhat, in general you can think

of a research project as following several steps: 1) choose a research question, 2) state your

hypothesis, 3) gather data, 4) analyze your data, and 5) use the results of your analysis to

come to conclusions about what you found. We have already discussed methods you might

use to gather data; in this section, we explain other key elements of research design. However,

we won’t discuss the analysis stage in detail; you will learn more about if you take a research

methods or social statistics course.

From topic to question

Once you’ve identified a research topic, you’re ready to turn that topic into a research

question. Reading previous studies about the topic you’re interested in will let you see what we

already know and what you might add with your own research.

Your research question must really be a question. “I want to show that people from

different cultures have different ideas about ‘the family’” isn’t a question. Who would disagree

with you? Most people would probably agree that ideas about family life probably differ

across cultures. A research question has to have more than one possible answer or outcome;

the point of your study is to identify the answer that seems most accurate.

Research Methods (Fall 2021 Edition)

12

There’s another problem with this example: “I want to show” is the wrong attitude for

research. It sets up the project to find an answer you already have in your mind rather than a

true question. Your goal isn’t to have a point you want to show; your goal is to have a question

you want to answer. And remember the problems with confirmation bias. The logic of science

is to try to find evidence that your claim is wrong, not to show that what you already believed

about the world was right.

Variables

Once you have a question, you have to decide what you actually want to observe—

your unit of analysis. Sometimes we’re interested in individual people, but not always. We may

ask questions about groups of people, or larger units like organizations, companies, or nations.

For example, we might ask how people’s incomes are influenced by their education22 (our unit

of analysis is individual people) or how democratic nations tax their citizens compared to

those ruled by royalty (our unit of analysis is the nation). There is no “correct” unit of analysis;

the appropriate unit depends on what question you want to answer. Once you identify your

unit of analysis, you can determine what types of data to collect and which research methods

are more or less appropriate for your project.

The thing you will observe is called a variable, a factor or characteristic that has more

than one possible value.

Independent and dependent variables

The goal of research is to identify co-variation, or relationships between variables. Let’s

say we suggest a relationship between two variables: that a person’s education influences

their income. In this case, education is the independent variable (usually represented as X),

meaning it affects the variable you’re trying to explain. The other variable—income—is the

dependent variable (usually represented as Y), the one you’re trying to explain; its value

depends on the independent variable.

Sometimes when we look for a relationship, we don’t observe any co-variation. Perhaps

there just isn’t any relationship between variables. To take a silly example, we might ask if the

length of your thumb influences your income. We could observe the lengths of many people’s

thumbs (we have variation), and see how this characteristic is related to their income (again

we have variation). But it’s unlikely that we have any meaningful co-variation; our two

variables aren’t related to one another. And that’s good to know, too! Finding out that

characteristics are not related can be as important as finding out that they are, especially if

people previously thought they were related.

Research Methods (Fall 2021 Edition)

13

From research question to hypothesis

Now that you have a sense of some of the basic building blocks of research, we’re

ready to make our question a little more specific by turning it into a hypothesis, a statement

about how variables relate to one another.

To create a hypothesis, you need to define the population you’re interested in studying

and the variables you think are important. The general form of a hypothesis looks something

like this:

For Population (P), Independent Variable (X) is related to Dependent Variable (Y)

Are you interested in people from the United States, or just people from Texas? If it’s

Texans, then there’s no point in gathering information about people from California. We rarely

want to know about the entire world; we usually want to know about a very small part of it. So

we have to define who we want to know things about: our population.

Say we’re interested in the relationship between education and income in the entire

United States. Now we’ve got a much more specific hypothesis:

For Americans (P), their education (X) explains how much income they make (Y)

These decisions about how to measure our variables are referred to as

operationalization. This is how we convert an idea into something concrete that we can

measure. In this example, operationalizing our variables was fairly simple. But other variables

can be trickier. Imagine you wanted to study the effect that stress at work has on a person’s

satisfaction with their marriage. How would you operationalize marital satisfaction? Would you

ask spouses to fill out a survey about how satisfied they are, from “very satisfied” to “very

dissatisfied”? Would you have them count how often they fight over a two-week period?

Whether they have had an affair in the past year? And what about operationalizing work

stress so we could measure it? We could do physical tests of the level of stress hormones in

their bloodstream, or ask how often they experience behaviors associated with stress (such as

difficulty sleeping). We could also ask them to rate their stress level, from “very high” to “very

low.”

Whenever you do research, it’s likely there are multiple ways you could choose to

operationalize your variables. It’s essential that you are clear about what your variables are

and how you will measure them. Although social research aims to answer big questions about

social life, research projects typically focus on narrow questions. When we’re developing a

research question, we have to narrow it to a question we can actually answer. But being more

specific has its benefits: by asking a question we can actually answer, we’ll know more about

the world when we complete our research project than when we started.

The key lesson here is that before beginning any research project, you must be able to

answer the following questions: what are the relationships I’m interested in studying? How do I

decide who counts as part of my population of interest? What items do I want to study? And

Research Methods (Fall 2021 Edition)

14

how will I observe those items? Whether you do participant observation, content analysis, a

survey, or an experiment, these are important questions you must be able to answer.

Selecting a sample

Once you have an operationalized hypothesis, it’s time to figure out who or what you’ll

observe to test it. It’s very rare that we can study everyone we’re interested in (our

population). Instead, we study a smaller group of people who represent that population.

Sampling is how social

scientists select representatives of

their population.23 Sampling

occurs in both quantitative and

qualitative work. For example,

sociologist Mitch Duneier was

interested in homelessness. He

couldn’t study all homeless

people in the country, or even in

New York City. Instead, he

conducted an ethnography—an

in-depth qualitative study of a

social group and the group’s

culture—of a neighborhood in

lower Manhattan where homeless people (mostly men) sold used books and magazines they

retrieved from recycling bins out on the sidewalk.24 He discussed how the homeless community

informally managed their sidewalk markets and how they interacted with the wealthier

residents of the area. Duneier wasn’t studying all homeless people; he studied a sample of

them (within a particular neighborhood), with the hope that what he learned from his sample

might reveal themes that applied elsewhere.

When sampling, we have to decide how to select a sample that truly represents the

larger population we want to understand. This step involves creating a sampling frame. The

sampling frame is how you determine who will be contacted to be part of your sample.

Examples include randomly selecting from a telephone book, voter list, or a mailing list, or

randomly dialing phone numbers.

Every sampling frame comes with challenges. If you use phone listings, you won’t be

able to access people who have unlisted phone numbers, people who don’t have phones, or

people who only have cell phones. If you use voter lists, you’ll only get people who are

registered to vote. With home addresses, you miss people who have moved since your mailing

list was created. You will also miss those living in institutions (such as nursing homes or prisons)

and people who don’t have homes. Selecting a sampling frame means considering issues

such as cost, time, what it is you want to know, and from whom. If you want to know what

young people think about an issue, for example, using a telephone directory as your sampling

A homeless person’s belongings in Rijeka, Croatia. (Source)

Research Methods (Fall 2021 Edition)

15

frame may not be wise since many, if not most, young people only have cell phones that

won’t be included in the directory.

When you draw conclusions from your study, strictly speaking, you can’t draw

conclusions about the entire population. You can only draw them about the group of people

represented by your sampling frame. For example, if we’re interested in the attitude of

Americans about civic engagement and we decide to use a telephone directory as our

sampling frame, we can only make claims such as, “For people listed in the telephone book,

their attitudes about civic engagement are...” It’s important to pay attention to the limits of

findings based upon the sampling frame.

Once we’ve defined a sampling frame, we draw a sample. This can be done randomly

or non-randomly. Many scholars, particularly researchers involved with large surveys, use

random samples. For a sample to be random, each member of the population must (1) be

known and (2) have some chance of being selected. If some elements of the population

can’t be selected (they have no chance of selection), then the sample isn’t random. An

example would be if you excluded people who were in the sampling frame (say, a mailing list)

because they live too far away and it would be too expensive to travel to talk to them. The

goal of a random sample is to get a sample that is truly representative of the larger

population. That allows you to generalize your conclusions, or apply them to a larger

population outside of the group you studied.

If we draw a non-random sample, where some members of the population don’t have

any chance of being selected, we’re very restricted in the claims we can make. However,

many social scientists do use non-random samples and still make claims beyond the particular

people studied, generalizing to a larger group. In these cases, scholars argue that even

though their sample is non-random, it still represents general trends. These types of samples are

common in qualitative work like interviews and ethnographies, but they also appear in

experiments and surveys.

When selecting a sample, a serious concern is nonresponse bias. If people don’t

respond to your attempts to include them in your research, you have to figure out if there is a

systematic reason why they aren’t participating. Is there anything unusual about the people

who aren’t responding? In other words, are particular types of people participating at lower

rates, and, if so, why? And does that mean you’re missing out on an important group, making

your sample unrepresentative of the population? Or are the people who do respond unusual?

Maybe they care a lot more about the topic than most people and that’s why they agree to

participate when others don’t. If there’s a systematic reason why some people don’t respond

and others do, you run the risk of drawing incorrect conclusions based on a sample that is

biased in some way.25

Say you’re asking people their attitudes about sexual behavior. You construct a sample

that is representative of the American population. And based on their responses, it looks like

people have very accepting attitudes about sexual activity among teenagers. However, you

see that a lot of people chose not to respond to your survey. What if those people also

Research Methods (Fall 2021 Edition)

16

happen to have more conservative attitudes about teens having sex? Perhaps people who

are likely to be more comfortable with teenagers being sexually active are also more likely to

answer your questions, while people with conservative attitudes decline to answer. Because of

the nonresponse bias—the patterns in who didn’t respond to your survey—you can’t be

confident in claiming that your findings represent the larger population.

We end with a final word on sampling, particularly related to qualitative work. As we

noted, qualitative work often uses non-random samples. So what can we learn from this work?

Keep in mind that different methods have different aims. Quantitative methods seek to

establish associations between variables. They answer questions like, “what is the association

between education and income?” Qualitative methods also look at associations, but they

often address how and why questions. What is going on inside schools or with students that

their education helps them earn more? Or we might explore how people use their educations

to earn more money; how do they get access to the types of internship experiences that lead

to job offers? Showing these processes at work often requires digging down to specifics

through ethnographic observation or interviews. Because of the ways these methods are

conducted, representativeness is much harder to achieve, and sometimes it’s impossible.

Qualitative researchers are sensitive to biases that might make their data unique and

not generalizable.26 But the potential weaknesses are often balanced by the benefits: they

can provide insights into the rich texture of how social processes work that large-scale

representative studies can’t. Research doesn’t happen in isolation. As researchers develop

ideas about how the world works, these ideas can be tested and evaluated in other settings,

by other researchers. Some qualitative research may be limited in its generalizability, but it can

provide ideas that are critically evaluated by quantitative work that is generalizable beyond

the sample. In other words, both types of methods have strengths and weaknesses, but they

can complement one another.

REVIEW SHEET: DESIGNING A RESEARCH PROJECT

CLICK THE LINK FOR:

LEARNING OBJECTIVES KEY QUESTIONS

AUDIO KEY POINTS

PRACTICE QUIZ KEY PEOPLE

VOCABULARY CROSSWORD PUZZLES KEY TERMS

Research Methods (Fall 2021 Edition)

17

CORRELATION AND CAUSATION

 How is correlation different from causation?

 How can we demonstrate causation?

 Why are spurious variables a challenge for social science research?

After we’ve designed our study,

chosen our sample, and collected data,

we can analyze what we’ve found.

Imagine we collect data and find a

relationship between how much time

fathers spend with their children and how

healthy their children are; the more time

fathers spend with their kids, the healthier

the children are, on average. What can

we say about that relationship? Did our

independent variable (X – time fathers

spend with their kids) cause our

dependent variable (Y – kids’ health) to

change? Maybe—but we don’t know for

sure yet. We’ve demonstrated a

correlation between the variables—that they are related in some way. But that doesn’t

necessarily mean we’ve found causation, or evidence that the independent variable caused

the change in the dependent variable.27

First, we may not have identified the correct direction of the relationship (which

variable affects the other). We may think that X causes Y, but maybe it’s the reverse: Y could

be causing X. In our example, we might think that children are healthier because their fathers

spend time with them. This explanation seems to makes sense. But we could have the direction

of the relationship completely wrong. Perhaps the health of children affects how much time

fathers spend with them; maybe it’s stressful to spend time with unhealthy children, so fathers

don’t engage with them as much as with healthy children. Or maybe unhealthy children have

high medical expenses, so their fathers work more to pay for the treatments, leaving them with

less time to spend with their child.

Establishing that we’ve found a causal relationship (one where causation exists)

requires considerably more work than demonstrating a correlation. One way we can prove

causality is through research design—for example, by using experiments.28 As we explained

earlier, experiments carefully control the environment to isolate the effects of the independent

variable. If we then see a change in our dependent variable, we can be more confident that

This research presentation suggests a correlation between

types of vehicles (sedans vs. trucks) and voting patterns.

(Source)

Research Methods (Fall 2021 Edition)

18

it was caused by the independent variable, since that’s the only thing that changed during

the experiment.

We can also identify the direction of a relationship if one variable clearly happens first,

or precedes the other one; the variable that changes later can’t possibly affect the variable

that changed before it. To test our example, we might look for cases where children get sick

and see what happens. Do fathers decrease their parenting time after their child gets sick? Or

we could look in the other direction: If fathers begin to spend less time with their children, does

their kids’ health suffer? If we can figure out which variable comes first, we have a solid case

for arguing that we know the direction of the relationship.

But even if we figure out the direction of the relationship, it’s possible there isn’t a true

causal relationship between our variables. A spurious relationship exists when it looks like

there’s a connection between two variables, but in reality some other variable we haven’t

taken into account is affecting both our independent and dependent variables.

Let’s look at the impact of education on income. Researchers observe a strong

relationship between these two variables; people with more education make more money.

Education precedes (it comes before) income, so we can be fairly certain of the direction:

education causes higher earnings. So we have a situation that looks something like this:

Higher education Higher earnings

However, we still have to worry about whether we’ve found a spurious relationship.

What if some other variable affects both level of education and earnings?

Higher education Higher earnings

Something else?

Perhaps the “something else” we didn’t take into account is parents’ income. Maybe

children of wealthier parents are likely to complete more schooling. And children of wealthier

parents are also more likely to earn higher incomes. Parents’ income might explain both their

kids’ education and earnings. In that case, the correlation between these two variables

exists—they are related—but education wouldn’t explain or cause earnings as we initially

thought. The relationship between education and earnings would be a spurious relationship,

since family background (in this case, how much parents earned) affects how much

education their children get and their children’s future earnings (perhaps because wealthier

parents are able to connect their children to hard-to-get internships that lead to future jobs,

for instance).

Spuriousness is a challenge for most social science methods except experiments.

Experiments isolate the effects of a single variable, so there are fewer worries about spurious

results. But for all other methods, an unobserved spurious variable is always a concern. As we

Research Methods (Fall 2021 Edition)

19

design research projects, scholars try to gather information on the most likely spurious variables

so we can rule them out as possible explanations.

Validity and reliability

A key question for all research projects is whether we are measuring what we think we

are measuring – that is, do our findings have validity?29 This is an important consideration.

Random sampling and complex statistical analysis are pointless if it turns out that you weren’t

actually measuring what you meant to be.

Say we studied attitudes toward different racial groups. We ask people, “Do you have

racist attitudes toward other groups?” The problem we run into here is social desirability bias—

the tendency for subjects to give answers that they think are socially acceptable.30 In the U.S.,

most people are aware that it’s generally unacceptable to be racist. This means that even if

people hold extremely negative views of certain racial or ethnic groups, they are very

reluctant to identify as racist.31 So our question probably won’t be a valid measure of racial

attitudes. A better approach would be to avoid the loaded term “racist” and instead ask a

series of questions about specific interactions or beliefs (such as how comfortable they would

be with members of other races as neighbors, coworkers, or in-laws).

We can encounter validity problems even when social desirability bias isn’t a factor.

Sometimes questions simply don’t get at what we meant to measure. Maybe we’re studying

how satisfied spouses are with their married life, and we ask participants, “How likely are you to

get divorced?” as a measure of their satisfaction. But probably only the most dissatisfied

people would say they are likely to get divorced, so you may miss a lot of dissatisfaction that

exists but isn’t severe enough to cause people to consider divorce. Or people might be

extremely unhappy with their marriages, but unlikely to get divorced; perhaps they have

children that affect their decision, are members of a religious group that discourages divorce,

or simply can’t afford to set up their own independent household. There are lots of reasons

that someone’s prediction of whether they will get divorced might not be a good indicator of

how satisfied they are with their marriage. Whenever social scientists design studies, we have

to carefully consider what questions to ask to get at the characteristic we’re hoping to learn

about.

In addition to asking how valid our research is, we must ask about the reliability of our

observations, or the consistency of the measurements. Challenges to reliability can come from

problems with the instrument used to collect the data, such as when survey questions are too

vague and open to interpretation. For instance, psychologists often administer surveys to

identify someone’s personality type; you may have taken one of these surveys yourself at

some point. Since personality is seen as a stable characteristic—while your mood might shift

quickly, someone’s personality should be relatively unchanged—then a person who takes a

personality test two years apart should get the same results. If a person gets different results on

a personality test, there’s a good chance the test isn’t reliable—it doesn’t consistently measure

the same thing in the same way each time.

Research Methods (Fall 2021 Edition)

20

REVIEW SHEET: CORRELATION AND CAUSATION

CLICK THE LINK FOR:

LEARNING OBJECTIVES KEY QUESTIONS

AUDIO KEY POINTS

PRACTICE QUIZ KEY PEOPLE

VOCABULARY CROSSWORD PUZZLES KEY TERMS

CONCLUSION

We have introduced you to some of the key elements of research design and

interpretation. The main points we hope you take away from this chapter are that studying

social life is messy and difficult, but that careful research design can help us investigate it

scientifically, giving us confidence in our findings. Nonetheless, whenever you encounter

research claims, it’s always good to maintain some skepticism, especially when the findings

reflect what you already want to believe. Social science is an ongoing project, where studies

build on those that have already been completed. Later studies, with different research

designs, may alter what we think we know—or may confirm previous findings. As we slowly

add to sociological research on a topic, we collectively come to a better understanding of

the complex and fascinating social world around us.

Research Methods (Fall 2021 Edition)

21

REFERENCES

Pager, Devah. 2003. “The Mark of a Criminal Record.” American Journal of Sociology 108(5): 937-975. 2 Klayman, Joshua. 1995. “Varieties of Confirmation Bias.” Psychology of Learning and Motivation 32: 385-418. 3 National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. 1978. The

Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. Bethesda, MD. 4 Mellanby, Kenneth. 1947. “Medical Experiments on Human Beings in Concentration Camps in Nazi Germany.”

British Medical Journal 1: 148-150. 5 Weindling, Paul. “The Origins of Informed Consent: The International Scientific Commission on Medical War Crimes,

and the Nuremberg Code.” Bulletin of the History of Medicine 75(1): 37-71. 6 Brandt, Allan M. 1978. “Racism and Research: The Case of the Tuskegee Syphilis Study.” The Hastings Center

Report 8(6): 21-29. 7 Elliott, Debbie. 2021 (February 16). “In Tuskegee, Painful History Shadows Efforts to Vaccinate African Americans.”

NPR. Retrieved at https://www.npr.org/2021/02/16/967011614/in-tuskegee-painful-history-shadows-efforts-to-

vaccinate-african-americans 8 Dembosky, April. 2021 (February 25). “No, the Tuskegee Study Is Not the Top Reason Some Black Americans

Question the COVID-19 Vaccine.” KQED. Retrieved from https://www.kqed.org/news/11861810/no-the-tuskegee-

study-is-not-the-top-reason-some-blac8k-americans-question-the-covid-19-vaccine 9 Babbie, Earl. 2004. “Laud Humphreys and Research Ethics.” International Journal of Sociology and Social Policy

24(3): 12-19. 10 Protection of Human Subjects. 45 CFR part 46. 2017. Retrieved from https://www.ecfr.gov/cgi-

bin/retrieveECFR?gp=&SID=83cd09e1c0f5c6937cd9d7513160fc3f&pitd=20180719&n=pt45.1.46&r=PART&ty=HTML#se

45.1.46_1101 11 Schuman, Howard and Stanley Presser. 1996. Questions and Answers in Attitude Surveys: Experiments on Question

Form, Wording, and Context. Thousand Oaks, CA: Sage Publications. 12 Smith, Brianna A., Zein Murib, Matthew Motta, Timothy H. Callaghan, and Marissa Theys. 2017. “‘Gay’ or

‘Homosexual’? The Implications of Social Category Labels for the Structure of Mass Attitudes.” American Politics

Research 46(2): 336-372. 13 Jorgensen, Danny L. 2015. “Participant observation.” Emerging Trends in the Social and Behavioral Sciences: An

Interdisciplinary, Searchable, and Linkable Resource. 1-15. Retrieved at

https://doi.org/10.1002/9781118900772.etrds0247 14 Sherman, Rachel. 2007. Class Acts: Service and Inequality in Luxury Hotels. Berkeley: University of California Press. 15 Becker, Howard S. 1958. “Problems of Inference and Proof in Participant Observation.” American Sociological

Review 23(6): 652-660. 16 Neuendorf, Kimberly A. 2017. The Content Analysis Guidebook, 2nd edition. Los Angeles: Sage Publications. 17 Hatton, Erin and Mary Nell Trautner. 2011. “Equal Opportunity Objectification? The Sexualization of Men and

Women on the Cover of Rolling Stone.” Society & Culture 15: 256-278. 18 Durkheim, Émile. 1951[1897]. Suicide: A Study in Sociology. London: Routledge. 19 Carpenter, Christopher and Philip J. Cook. 2008. “Cigarette Taxes and Youth Smoking: New Evidence from

National, State, and Local Youth Risk Behavior Surveys.” Journal of Health Economics 27(2): 287-299. 20 Eckert, Penelope. 1983. “Beyond the Statistics of Adolescent Smoking.” American Journal of Public Health 73(4):

439-441. 21 Escamilla, Gina, Angie L. Cradock, and Ichiro Kawachi. 2000. “Women and Smoking in Hollywood Movies: A

Content Analysis.” American Journal of Public Health 90(3): 412-414. 22 Day, Jennifer Cheeseman and Eric C. Newburger. 2002. The Big Payoff: Educational Attainment and Synthetic

Estimates of Work-Life Earnings. Washington, D.C.: U.S. Census Bureau. 23 Cochran, William G. 2007. Sampling Techniques. New York: John Wiley & Sons. 24 Duneier, Mitchell. 1999. Sidewalk. New York: Farrar, Straus, and Giroux. 25 Groves, Robert M. and Emilia Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-

Analysis.” Public Opinion Quarterly 72(2): 167-189. 26 Gobo, Giampeitro. 2004. “Sampling, Representativeness, and Generalizability.” Pp. 435-456 in Clive Seale,

Giampietro Gobo, Jaber F. Gubrium, and David Silverman (Eds.), Qualitative Research Practice. Thousand Oaks:

Sage. 27 Goldthorpe, John H. 2001. “Causation, Statistics, and Sociology.” European Sociological Review 17(1): 1-20. 28 Shadish, William R., Thomas D. Cook, and Donald Thomas Campbell. 2002. Experimental and Quasi-Experimental

Designs for Generalized Causal Inference. Boston: Houghton Mifflin. 29 Drost, Ellen A. 2011. “Validity and Reliability in Social Science Research.” Educational Research and Perspectives

38(1): 105-123.

Research Methods (Fall 2021 Edition)

22

30 Krumpal, Ivar. 2011. “Determinants of Social Desirability Bias in Sensitive Surveys: A Literature Review.” Quality &

Quantity 47(4): 2025-2047. 31 Bonilla-Silva, Eduardo. 2003. Racism without Racists: Color-Blind Racism and the Persistence of Racial Inequality in

the United States. Lanham, MD: Rowman & Littlefield. Cover Photo: Source; Creative Commons License