DBA 701 5.2
CHAPTER 7
ANALYSIS
As a sophomore in college, I shared an apartment with three friends I met in the dorms. One of them—“Jeremy”—had grown up in an extremely wealthy family. Jeremy was a humorous and affable guy, but after a few weeks of living together, I noticed that he tended to leave his dirty dishes scattered around the apartment. Mostly eaten bowls of cereal were abandoned in front of the television or on the kitchen table—as if Jeremy’s meal had been interrupted by some emergency (“Gotta go-go-go!”). Loading the dishwasher or scrubbing a pan seemed completely foreign to him.
“Jeremy never does the dishes!” I complained to my other two apartment mates.
“Yeah, what’s up with that?”
“Probably he was spoiled as a kid. I bet his family had a full-time maid!”
Then my two roommates and I devised a plan to give Jeremy some gentle reminders, which produced mixed results, but that’s a long story.
This mundane example may sound remote from social science, but there are some familiar and researchable questions in play: Who’s doing what? How frequently is a phenomenon occurring, and why?
In everyday life, we raise these sorts of questions more often than we may realize. We routinely collect data, analyze it, and present our findings to others. In the example above, my roommates and I had engaged in some casual observations of Jeremy’s behavior and formulated an explanation. Was it accurate to assert that Jeremy “never” cleaned any dishes? Probably not. Were there other factors that might explain his neglect of the dishes? Probably so. Our data collection, analysis, and findings were pretty shoddy.
My roommates and I could have behaved more “sciency” if we had carefully observed Jeremy’s behavior over the course of several weeks, measured the number of dishes he cleaned versus the number he left dirty, calculated a precise frequency (e.g., “Jeremy cleans up after himself 37 percent of the time”), and investigated a wider range of variables 1 that might explain his behavior—such as his busy work, school, and fraternity schedule, relationship dysfunctions within the apartment, different standards of cleanliness, cultural notions of dishwashing as “feminine,” and so on.
Needless to say, we did not do any of that. It would seem odd—if not morally repugnant—to subject a roommate to such scrutiny. Yet, laypersons’ reticence to conduct rigorous inquiries into everyday life is not shared by social scientists. Researchers want to know—as best they can—what’s happening, how often, and why. That strong drive to know helps make social science better than ordinary human inquiry.
THE ANALYTICAL STRENGTHS OF SOCIAL RESEARCH
As with conceptualization, literature reviews, measurement, and sampling, researchers’ efforts are superior to laypersons’ when it comes to analyzing and presenting data. Rather than jumping to conclusions based on a few haphazardly collected pieces of information, social scientists tend to search for more data, and then they systematically search through that data to find subtle patterns. I guess there’s a reason why they’re called re-searchers. Social scientists look, and look again, for information that might confirm or contradict their theories and expectations.
Let’s briefly discuss researchers’ analytical strengths before we turn to weaknesses.
1) Social scientists carefully scrutinize their data using sophisticated analytical techniques, and they report their findings precisely and cautiously.
Social scientists have developed a wide range of impressive analytical techniques, both quantitative and qualitative. On the quantitative side, statistical software packages (such as SPSS) enable researchers to analyze thousands of pieces of data simultaneously. Qualitative researchers, too, can use computer software to code hundreds or thousands of pages of interview transcripts and field notes. And, even when no sophisticated technology is employed, researchers bring a great deal of expertise, determination, and concentration to the data they analyze.
As quantitative articles are the main focus of this book, let’s elaborate on that form of analysis. Researchers who use statistical software can examine thousands of pieces of data—more than any one person could possibly hold in his or her head at once (and far more than the five or ten observations that my friends and I made of Jeremy’s behavior). Quantitative researchers can look for correlations among numerous variables in ways that far surpass ordinary human capabilities.
Two variables are positively correlated when an increase in one variable is associated with as increase in the other; for example, perhaps the variable of dishwashing goes up the more people feel connected to their apartment mates. A negative correlation is when an increase in one variable leads to a decrease in another variable; perhaps the more hours a person is employed outside the home, the less time they spend doing dishes.
Social scientists can search for positive and negative correlations across hundreds or thousands of cases—and they can do so for many variables simultaneously, not just two variables. They usually report these correlations, along with other important statistics, in clearly marked tables. Compare this to everyday life, where people (1) routinely use hyperbolic phrases (e.g., “You always . . . ” or “He never . . .”) and (2) fixate on one causal variable (e.g., childhood upbringing). In journal articles, researchers tend to be more cautious, thorough, and precise in reporting their findings. They specify that exact degree to which one variable is correlated with other variables, and they use tentative phrases such as “Childhood experiences with housework appear to be an important factor” rather than speaking in terms of absolutes.
In sections on results, discussion, and conclusions, scholars draw connections between their findings and the results of earlier studies, noting any consistencies or contradictions. Authors invite future researchers to conduct additional studies that might corroborate or challenge their results. Though an article may take many months or even years to write, scholars usually remain humble about their findings and admit the limitations of their studies.
I would argue that this reporting style is far superior to confidence with which laypersons declare impromptu “truths” based on quick and haphazard data collection and analysis. Don’t take my word for it. Do Exercise 7.1, and see for yourself.
2) Social scientists base their analyses on theoretical arguments about the constraints that shape human behavior.
In everyday life, we are free to make confident arguments based on the first theory that comes to mind. If a roommate isn’t doing the dishes, then maybe it’s due to his wealthy childhood—sure, good enough. Anything that sounds plausible may suffice, and there is no requirement that we read or think deeply about other possible factors.
In contrast, and as we discussed in Chapter 3, social scientists perform extensive literature reviews in conjunction with their research. From this reading, they are exposed to a wide range of theoretical perspectives and causal arguments. Moreover, editors and peer reviewers often ask scholars to consider factors that they may have left out of the first draft of the articles they submit for possible publication. As a result of this thinking, social scientists tend to have more nuanced, complicated, and wide-ranging explanations for human behavior.
Theoretical explanations vary somewhat by discipline (e.g., brain chemistry is more relevant in psychology than sociology) and also within disciplines (as there many different kinds of psychologists, many of whom don’t study the brain at all). Yet, despite the theoretical diversity, there are overlapping concerns. In journal articles—especially quantitative ones—the emphasis is usually on constraints, or factors that shape behavior. Let’s pause to explore the notion of constraints by relating it to a simple example: the food you eat.
Some scholars examine the biological constraints that shape conduct—such as the ways our bodies and brains may be “hardwired” to seek out fats, sugar, and salt or how some people may have a predisposition to alcoholism. Other scholars focus on geographical constraints—such as the kinds of crops that can be grown, given the local climate and soil. Probably the majority of social scientists, however, tend to emphasize either external social constraints or internal social constraints (see Berger, 1963).
External social constraints are those social forces that surround us and guide our behavior. Picture a maze or a prison—there may be directions you want to go, but there are obstacles in your way. With respect to the food you eat, we might say that your choices are guided by many external factors:
What kinds of food are offered in stores or restaurants in your area? If they don’t sell it (e.g., red bananas, okra, sushi, etc.), then you probably won’t eat it.
How expensive is the food that surrounds you? If fresh vegetables are more expensive than processed foods, that might shape your choices.
What are the general food norms in your particular region? In some countries (but not others), it is acceptable to eat cows, dogs, worms, deer penises, bull testicles, and duck embryos. Will you be encouraged or rewarded for eating certain foods, or will you be criticized or ostracized?
Do the food norms vary by age, gender, race, class, and other variables? A small jar of baby food might be tasty and convenient, but an American teenager who ate it for lunch would risk being ridiculed. Similarly, sometimes men may be disparaged for drinking “feminine” drinks (e.g., appletinis, wine coolers), while women may be discouraged from ordering “manly” drinks (e.g., whiskey, dark beer); as a result, they might avoid such beverages even if they like the taste.
Internal social constraints are those forces that guide our conduct because we internalize them. Picture a robot that is programmed to think and do certain things. People are taught certain beliefs, perspectives, and identities. We often learn to want that which our culture or subculture has encouraged us to want. For example:
You may be taught to believe that cows are sacred and off-limits (rather than a routine source of hamburgers) or that lobster is a delicacy (rather than a large, underwater bug). Hence, what you want to make for dinner will be shaped by these beliefs, even if you are eating alone with nobody watching.
Your college friends may persuade you that eating meat is immoral and unhealthy. These beliefs might then shape your conduct and your emotions whenever your parents serve up a big Thanksgiving turkey or holiday ham.
Through exposure to messages in mass media, you may be encouraged to acquire a positive identity—such as “I am a slender person, not fat” or “I am muscular and strong, not weak”—which may shape how many calories you consume or how much protein you eat.
In my discussion of four types of constraints, I have avoided technical terms and complicated theories in order to make a simple point: Social scientists consider and explore a range of factors when they study human behavior. They are aware that anything we do can be shaped by numerous forces from many different directions. They read the existing literature to determine what those forces may be and what effect previous studies have found them to have. They use this theoretical knowledge before, as, and after they analyze their data.
When I read empirical journal articles, I almost always come away with the feeling that the researchers have attempted to bring more theoretical awareness and sophistication than laypersons would bring to the same topic. Don’t take my word for it, though. Do Exercise 7.1—repeatedly, if you can—and see for yourself.
EXERCISE 7.1
Search for an example of “data analysis” in your everyday life. You might focus on a discussion of why some person is engaging in a particular behavior (besides dishwashing, as I covered that topic). Listen to what you and your friends say in ordinary conversations, or to what politicians and pundits say on television, or to what your coaches, parents, priests, coworkers, or employers say.
Can you identify the weaknesses of this everyday analysis like I did with the case of Jeremy? You might describe a better way of analyzing data and reporting findings about the topic—one that is more scientific. Can you think of some factors or social constraints that were neglected in the discussion?
If you are ambitious, try to find a journal article on a topic similar to the discussion you examined in #1. Search the methods and analysis sections of the journal article. Find and (as best you can) describe the type of analysis that the researchers used and the variables or factors they included in their study. Identify examples of precision and cautiousness in the text or in the tables the authors provide. Can you explain why the researchers’ efforts are more impressive than the discussion you heard in the everyday example you found in #1 above?
FINDING ANALYTICAL WEAKNESSES IN SOCIAL RESEARCH
Social scientists’ analytical skills far surpass those that are usually employed in ordinary human inquiry. Nevertheless, researchers are far from perfect. In this section, I will discuss two simple strategies for critiquing the kinds of analyses that regularly appear in standard journal articles (SJAs).
1) Whenever researchers perform an analysis, they must be selective about the factors or variables they include.
Researchers do the best they can to take into account as many of the factors that existing theories and previous research suggest are important. However, no scholar can focus on everything. Very few journal articles consider external constraints, internal constraints, biological constraints, and geographical constraints all at the same time. And even if a scholar focuses on a particular form of constraint—such as external factors—they can’t include everything that falls under that category. Instead, scholars tend to have pet theories and pet variables—forms of explanation that they prefer and hope to support. Rather than fully exploring every possible perspective and every possible factor that relates to their topic, they tend to craft research projects that revolve around particular orientations and issues. If they find evidence that supports their preferred theory and causal factors—or discredits a competing orientation—then so much the better.
As a result of these predilections, the literature that scholars read and summarize tends to be somewhat delimited rather than expansive and inclusive. Researchers do not have time, nor do journals offer the space, for truly comprehensive literature reviews—as we saw in Chapter 4.
Moreover, a further winnowing down of scholars’ analytical focus occurs when authors move from their literature reviews to their methodology sections. Researchers face limitations in the data they can collect. Some information may simply not be accessible or affordable, which complicates the tasks of sampling and measurement. Consequently, most researchers cannot truly test every theoretical notion they might want to; their data can only speak to a portion of the theoretical ideas they would like to apply to their topic.
Recall Sun et al.’s (2003) study of binge drinking by college students. A wide range of factors might influence whether a person tends to drink excessively. Although Sun et al. conducted admirable research, they cannot account for every potentially important factor. In their analysis, the authors included only 13 variables. The selection of these was shaped by the authors’ reading of the literature as well as by the questionnaire that they adopted—a document that was created by prior researchers. Let’s look through these variables and, as you read them, imagine whether each one might have some influence on whether or not you engaged in binge drinking:
1) Gender of respondent
2) Age of respondent
3) Ethnic origin of respondent
4) Marital status of respondent
5) Respondent’s living arrangements—e.g., living with parents, living with spouse, living with roommates, living alone
6) Family history of alcohol or drug use (e.g., by a parent or sibling)
7) Precollege history of alcohol use by respondent
8) Whether respondent’s close friends disapprove of binge drinking
9) Whether respondent believes that alcohol has positive effects (e.g., allows people to have more fun)
10) Whether respondent believes that binge drinking brings great risk, some risk, or no risk
11) Whether respondent has experienced peer pressure to binge drink
12) Whether respondent believes that drinking is central part of social life on campus
13) What respondent believes about alcohol use by “the average student” on campus
There are some important and logical variables being studied here. It seems clear that the decision to binge drink could (in some cases) be shaped by each factor. Sun et al. (2003) should be applauded for attempting to determine if there was a positive or negative correlation between these variables and the practice of binge drinking.
Nevertheless, this list of 13 variables is far from exhaustive. Despite the time and effort that Sun et al. (2003) put into their research, they ended up neglecting a wide range of potentially relevant factors. I’ll list some below:
1) Whether respondent has a physiological proclivity to binge drink
2) Respondent’s religious beliefs
3) Respondent’s socioeconomic status (e.g., personal income, parents’ household income)
4) Whether respondent works full time, part time, or not at all
5) Whether respondent is a member of a fraternity or sorority
6) Respondent’s major
7) Whether respondent attends a university that is officially dry (i.e., no alcohol allowed)
8) The number and proximity of local stores that sell alcohol
9) Whether respondent is living in a location where taxes on liquor are much higher or lower than the national average
10) Respondent’s exposure to advertisements, television shows, and movies that promote alcohol consumption
Arguably, all ten of these factors could have some influence on whether a person tends to drink excessively. Some of these issues—#3 and #6—were included on the questionnaire that Sun et al. (2003) used, yet (for potentially good reasons) the researchers chose not to incorporate them into their analysis. Some of these issues are admittedly more difficult to measure—such as #10. Moreover, some issues seem beyond the scope of what a social scientist might be expected to study—#1 is arguably a topic for biologists (e.g., Herman, Philbeck, Vasilopoulos, & Depetrillo, 2003). But, none of that matters. If you can imagine a potentially important factor that might increase or decrease a person’s likelihood of binge drinking, then you can legitimately argue that Sun et al.’s (2003) work is imperfect for ignoring that factor.
Perfection is a high standard. Most authors wouldn’t take offense at the critique of selectivity because they recognize (more than laypersons do) just how complicated social life is. As long as readers recognize that a study is better than ordinary human inquiry and admit that the authors’ work is useful (or is at least a serious attempt to conduct a careful study), then the charge of selective analysis is welcomed by most researchers. Such discussions can even be enjoyable and can provoke ideas for future research. In their articles’ conclusions, authors sometimes call attention to (some of the) factors or forces that they neglected in order to prompt future researchers to address those limitations in follow-up studies.
Test your ability to identify selectivity in the analyses that appear in journal articles by doing Exercise 7.2.
EXERCISE 7.2
Read a journal article that interests you—perhaps a paper that you used for the exercises in Chapters 3 through 6. Look in the methods section and find the variables that the authors include in their analysis. Then, think of at least two variables that the researchers could have studied but didn’t. Try to explain why these factors are potentially important. If possible, discuss at least one internal social constraint and one external social constraint.
2) Causal order
A second way to critique a journal article is to challenge authors’ assumptions about the causal relationships that exist between the variables they study. For example, in Sun et al.’s (2003) analysis, the dependent variable was whether or not respondents engaged in binge drinking. There were 13 independent variables in the study, including “Whether respondent’s close friends disapprove of binge drinking.” Sun et al. crunched the numbers and found that this independent variable (“close friends disapproval”) had a strong negative correlation with the dependent variable (“binge drinking”). As disapproval increased, binge drinking tended to decrease.2
This makes intuitive sense: It is easy to imagine that our close friends might exert a strong influence over our drinking behavior. Let’s refer to this causal influence as X→Y.
On the other hand, it is also easy to imagine an alternative scenario: Perhaps students who are already predisposed to binge drinking tend to seek out companions who engage in that practice. If I want to “party hard,” maybe I will tend to form close friendships with people who are likeminded. In this scenario, Y→X. An inclination to binge drink shapes the kinds of friends we have.
So, which is it? Does X→Y, or does Y→X, or both? It makes a difference. If researchers want to understand why a phenomenon is occurring, then they need to unravel the complexities of causal order. (I’m tempted to call this the chicken–egg dilemma.) Otherwise, we get some funny-looking sentences like this: The independent variable may depend on the dependent variable.
One way researchers attempt to deal with this dilemma is to conduct longitudinal research. By collecting data over time, social scientists can sometimes show that changes in one variable precede changes in another variable. So, we might notice patterns where students’ drinking practices change after they form close friendships with people who approve or disapprove of excessive alcohol consumption.
Unfortunately, longitudinal research is comparatively rare. It is more expensive and difficult to conduct and still provides no guarantee that a researcher will be able to unravel causal order dilemmas. Most research is cross-sectional—collecting data at one point in time—so researchers must make larger assumptions and inferences about causal order.
Thus, an effective strategy for critiquing many (but not all) journal articles is to look for problems with causal order, as we did with the example of binge drinking. My undergraduate students have successfully used this strategy to highlight imperfections in articles on a wide range of topics—even research that employs statistically sophisticated forms of analysis. Below are some examples. I’ll include (X) and (Y) in order to help you find the causal order dilemmas:
Wong (1997) studied some of the factors that shaped whether Canadian youth of Chinese decent would engage in delinquent acts. He hypothesized that adherence to Chinese culture (X) may lower delinquency (Y) due to that culture’s emphasis on family commitment and moral values. However, one can ask whether engaging in delinquency (Y) may lead to a rejection of Chinese culture (X) due to the resocialization that can occur during interactions with fellow delinquents.
Peyrot and Sweeney (2000, p. 219) examined whether parishioners’ satisfaction with their pastor’s performance (X) shaped their overall satisfaction with their local church (Y). But, the authors admit that parishioners’ global sense of satisfaction (Y) may shape the satisfaction they feel with specific dimensions of their church, including the pastor’s performance (X). Perhaps you’ve heard of the halo effect?
Brandl, Stroshine, and Frank (2001, p. 527) noted that police officers who make more arrests tend to receive more complaints about excessive force. However, they could not say unequivocally if arrests (X) led to complaints about excessive force (Y) or if excessive force was used (Y) and then arrests were made in an attempt to justify it or cover it up (X).
Klendauer and Deller (2009) investigated whether perceptions of unfairness (X) led people to feel less committed to the companies they work for (Y). However, they admitted that people who feel less committed (Y) may be more inclined to perceive situations as unfair (X) compared to people who feel more committed.
Simpson et al. (2011) set out to study whether “feeling powerful” (X) improves people’s memories of social networks (Y). But, one might ask whether having a good memory (Y) may be helpful in obtaining power or feeling powerful (X).
These examples are not meant to be representative of all social science; they are merely some simple examples intended to demonstrate how to make a causal order argument. Causal order may or may not be an issue in the journal articles you read. Try Exercise 7.3, with a few articles on different topics, and see what you find.
EXERCISE 7.3
Skim through a quantitative journal article that interests you. Pay extra attention to the description of the variables in the methods section, so you can quickly identify the authors’ independent and dependent variables. Don’t stop there, though. The literature review and any discussion or limitations sections may prove helpful.
Can you explain and then critique the authors’ assumptions about the causal order of their independent and dependent variables? Can you argue that it is just as likely—or at least somewhat likely—that the dependent variable could have a causal effect on one of the independent variables?
Unlike Exercise 7.2, I cannot guarantee that this line of questioning will always generate results. Many articles exhibit trouble with causal order, but some don’t. You may need your instructor to help you find a workable article in order to complete this exercise.
CONCLUSION
In everyday life, a few casual observations provide enough data to declare that a trend is taking place and to speculate about its supposed causes. After a few unfortunate incidents, a journalist or pundit may state, “There is an ‘epidemic’ of school shootings—is bullying to blame?” Or, college roommates may decide, “Jeremy never does the dishes. He must have been spoiled as a child.”
In contrast, social scientists usually collect more data more carefully, and they look for trends and patterns using rigorous analytical techniques. Research often involves hundreds or thousands of pieces of data. Information can be compiled for many different variables, and computer programs can be used to model the relationships between them. From their reading of prior studies, social scientists bring a deeper theoretical understanding of the many constraints that shape human behavior, which informs their sampling, measurements, and analyses. Then, as they compose their final reports, researchers describe their findings with precision and caution rather than speaking in hyperbole. All of this is, arguably, far superior to ordinary human inquiry.
Yet, research is far from perfect. Researchers can’t help but neglect potentially relevant factors as they collect and analyze their data. Virtually any study can be criticized for what it left out—due to the researchers’ theoretical preferences, due to methodological constraints, or due to simple human error. When researchers investigate what is going on and why, they unavoidably ignore some important causal forces as they focus on what is most interesting, and study-able, for them.
Moreover, researchers may speak of independent and dependent variables (X→Y), but trouble with causal order is often present, with the latter variable potentially influencing the former (Y→X). Studies can often be criticized for treating as an independent variable a phenomenon that may actually be shaped by (or dependent on) the dependent variable. If the goal of the research is to illuminate the intricacies of what causes what, then an article can be reasonably criticized for any problematic causal order assumptions that haunt the analysis. Admirably, social scientists often admit when such dilemmas exist, which makes the job of critical thinking—finding imperfections in research while appreciating its strengths—a bit easier.
___________
1 In journal articles, the term variable is often used instead of concept in order to highlight the fact that a phenomenon varies—it can be present or not present, or (in some cases) there can be more or less of it. In the previous section of this chapter, I mentioned the issues of temperature, binge drinking, and intelligence—these could be considered variables in a study. They can be measured and correlated: Do more people engage in binge drinking when the temperature is higher? Is binge drinking associated with a lower level of intelligence?