Basic of Research

Warner, R. M. (2021). Applied sta s cs I: Basic bivariate techniques (3rd ed.). Thousand Oaks, CA: Sage Publica ons. ISBN: 978-1-5063-5280-0.

CHAPTER 1 EVALUATING NUMERICAL INFORMATION 1.1 INTRODUCTION In everyday use, sta s cs can refer to specific pieces of numerical informa on, such as average income for all employed persons in the United States. In science and technical fields, the term sta s cs more o en describes techniques for analyzing and interpre ng numerical informa on. Readers should not assume that all published numerical informa on is correct. Numeracy skills are needed to understand and evaluate how numerical informa on is collected, analyzed, and presented. 1.2 GUIDELINES FOR NUMERACY A report published by the American Sta s cal Associa on’s Commi ee on Guidelines for Assessment and Instruc on in Sta s cs Educa on (GAISE College Report ASA Revision Commi ee, 2016) described numeracy skills as follows: Students should become cri cal consumers of sta s cally-based results reported in popular media, recognizing whether reported results reasonably follow from the study and analysis conducted. To be a cri cal consumer of sta s cally-based results, it is necessary to understand the components that produced them: the design of the inves ga on, the data, its analysis, and its interpreta on. Iden fying the variables in a study, which includes considera on of the measurement units, is a necessary step to inform judgments or comparisons. Iden fying the subjects (cases, observa onal units) of a study and the popula on to which the results of an analysis can be generalized helps the consumer to recognize whether the reported results can reasonably support the conclusions claimed for an analysis. Being able to interpret displays of data (tables, graphs, and visualiza ons) and sta s cal analyses also informs the consumer about the reasonableness of the claims being presented. (Italics added) Italicized terms in the preceding quota on iden fy components of the research and data analysis process; these are discussed further in Chapter 2 and research methods courses. This chapter briefly considers other fundamental issues in the communica on of numerical informa on: (a) sources (or communicators), (b) types of evidence, (c) ques ons about generalizability and causal inference, (d) quality control mechanisms, (e) ethical responsibili es, and (f) degrees of belief. 1.3 SOURCE CREDIBILITY 1.3.1 Self-Interest or Bias Communicators can be mo vated by self-interest or bias. Self-interest is o en clear in mass media; messages are o en intended to influence audience beliefs or behaviors (such as vo ng or product purchases). Science communicators can also be mo vated by self-interest; for instance, some researchers receive funding from alcohol or pharmaceu cal companies, and

their future funding may depend on research results. Many science journals require authors to declare poten al conflicts of interest. Self-interest of informa on providers is not always obvious. Many webpages offer “sponsored content”: paid messages from adver sers that look like news ar cles but in fact promote the interests of adver sers. For instance, a new diet pill might be presented as “news” when in fact the ar cle is an adver sement. Communicator self-interest raises concerns about credibility of messages. 1.3.2 Bias and “Cherry-Picking” Communicators generally cannot (or do not) present all available informa on. Selec on of informa on by communicators can be influenced by confirma on bias, a preference for informa on that confirms preexis ng beliefs or ideas. Biased selec on of evidence can be informally called cherry-picking. Informa on and ideas that are excluded may be as important as informa on that is included. As an example of cherry-picking, suppose 20 studies show no associa on between consuming meat and cancer risk, and 3 studies do show an associa on. A journalist might report only the 3 studies that showed an associa on or might report only the single most recent study. Whether the bias was inten onal or not, the ar cle will not provide an accurate summary of research results. When scien sts write literature reviews (reviews of past research), they are expected to discuss all past relevant research.1 Literature reviews are included in the introduc ons to most primary source research reports; literature reviews can also be stand-alone papers or books. 1.3.3 Primary, Secondary, and Third-Party Sources An old game called “telephone” illustrates the problem of distance from a source. People form a line; the first person whispers a message to the second person, the second person whispers it to the third, and so forth. When the final message is compared with the original message, there are changes and distor ons. Transmission of informa on can introduce errors because of each person’s biases or misunderstandings. In science, a primary source is a research report wri en by a researcher who has firsthand knowledge of behaviors and events in a study. Primary source reports (some mes called ar cles or papers) are published in science journals.2 Primary source data may also appear in books wri en for science audiences. A secondary source is a descrip on or summary of past research, created by someone who did not experience the reported data collec on or observa ons firsthand. In many disciplines, secondary sources are scholarly books. Some journal ar cles are also secondary sources because they only review past research and do not present new data about which their authors have firsthand knowledge. Literature reviews in the introduc ons to science journal ar cles are

secondhand discussions of past studies. (In the sciences, literature refers to past published research.) Unfortunately, primary source reports are usually long and difficult to read (par cularly for readers unfamiliar with sta s cs and technical terms). Language in research reports is some mes unnecessarily obscure. Some full-length science research reports are published on the web as open-access materials; anyone can view these. However, many publishers require fees or subscrip ons for access. The consequence is that many people can’t easily understand most primary source informa on in science and some mes cannot even gain access to it. Much content on websites for news organiza ons is third-party content. This is content wri en by someone who may have examined only secondary sources or other thirdhand content, such as news reports or press releases. O en, third-party content is authored by someone who has no technical knowledge of the research field and sta s cal methods. Examples include ar cles published by news organiza ons. These ar cles usually don’t provide complete or accurate informa on about research results. In the past, editors of pres gious newspapers required reporters to fact-check claims carefully. Increasingly, news reports on the web are paraphrases of, or uncri cal repos ng of, third-party content from other news sources. Some mass media news sources specifically disclaim responsibility for accuracy. Here is an example; many other news organiza ons post similar disclaimers: CNN is a distributor (and not a publisher or creator) of content supplied by third par es and users. … Neither CNN nor any third-party provider of informa on guarantees the accuracy, completeness, or usefulness of any content. … (CNN, 2018) Communicators can provide be er quality informa on when they are closer to original sources of informa on, and they are likely to provide be er quality informa on when they assume responsibility for accuracy. In everyday life, most of us rely on thirdhand informa on most of the me. Because so much of what we think we know is based on thirdhand informa on, we should not be overly confident about things we think we know. 1.3.4 Communicator Creden als and Skills Communicators are more believable when they have training and background related to informa on in the message. Researchers generally have creden als that provide evidence of this training and background, including advanced degrees such as a PhD or MD, affilia ons with respected organiza ons such as universi es, and publica ons in high-quality science journals. Some journalists have strong creden als in science, but many do not. People who do not have training in sta s cs can easily misunderstand studies that use sta s cal terms such as logis c regression and odds ra os.

Celebrity status is not a meaningful creden al. Famous media personali es, such as Dr. Oz3 and other self-appointed lifestyle or health experts, may base recommenda ons on incomplete or incorrect informa on. Scien fic research reports include source informa on (authors, university affilia ons, and so forth). News reports and websites some mes do not include source informa on; they provide no basis to evaluate self-interest, distance from informa on source, and creden als. Guidelines for evalua on of websites are provided by Kiely and Robertson (2016) and Montecino (1998). 1.3.5 Track Record for Truth-Telling There are independent, nonpar san organiza ons that evaluate communicator track records for truth-telling in journalism, for example, the Pulitzer Prize–winning site www.poli fact.com. Poli Fact rates statements as true, mostly true, half true, mostly false, false, and “pants on fire” (extremely false). Other respected fact-checking sites are www.snopes.com and www.factcheck.org. These fact-checkers do the work that informa on consumers usually don’t have the me to do. Informa on published in scien fic journals can be incorrect because of fraud; fraud in science is rare, but it has occurred. A notorious example was a claim by Andrew Wakefield that vaccines cause au sm (discussed by Godlee, Smith, & Marcovitch, 2011). There are severe penal es for fraud or plagiarism in science, including forced retrac on of publica ons, withdrawal of research funds, loss of reputa on, and job dismissal. Rare instances of fraud in science can be iden fied by a web search for the researcher name and terms such as fraud. Informa on consumers should be skep cal of informa on from sources with poor records for truth-telling. 1.4 MESSAGE CONTENT 1.4.1 Anecdotal Versus Numerical Informa on Anecdote means “story,” o en about an individual person or situa on. First-person accounts are o en called tes monials. Audiences may find narra ve stories or anecdotes more persuasive and memorable than numerical informa on. There are many poten al problems with anecdotes (anecdotal evidence). Some mes individual situa ons are not reported accurately (for example, adver sements for weight loss products o en include falsified before and a er photos). Even when anecdotal evidence is accurate, it is difficult to know whether the experience shown is generalizable: Has this experience happened to many other people, or was this a unique situa on? Diet product adver sers are required to acknowledge this and typically do so in a ny footnote: “Individual results may vary.” In science, a detailed report of an individual person or situa on is called a case study. The study of unique cases, such as the brain damage suffered by railway worker Phineas Gage (Kihlstrom, 2010; Twomey, 2010) can be valuable. However, generalizability concerns are s ll relevant. Anecdotal evidence can drama ze genuine problems. However, anecdotal evidence can also drama ze and promote incorrect beliefs. It is obviously easy to cherry-pick anecdotes.

Suppor ng evidence in the form of systema c numerical informa on can provide a more accurate overview of evidence than anecdotal reports. 1.4.2 Cita on of Suppor ng Evidence In science, iden fica on of outside sources of evidence is done by cita on. Author names and years of publica on are included in the text (to iden fy sources of ideas and evidence), and complete informa on to locate each source is included in a reference list. Cita on has two purposes. First, it gives credit to others for their ideas and evidence; this avoids plagiarism, which occurs if authors present ideas or contribu ons of other people as if they were the authors’ own new contribu ons. Second, it shows how the present study builds upon an exis ng body of evidence. A message is more believable when it includes or refers to specific suppor ng evidence. In science, the most complete and detailed suppor ng evidence appears in primary source research reports in science journals. Documenta on of informa on sources is typically less detailed and systema c in journalism and mass media. (The best science journalists provide references or links to primary source research reports.) It is possible for a writer or an adver ser to claim a spurious air of authority by ci ng numerous sources. However, a long list of references does not guarantee accuracy. On closer examina on, readers may find that communicators have cherry-picked, misinterpreted, or misrepresented evidence; cited sources that are not relevant to the topic; or referred only to opinion pieces that do not actually contain evidence. To evaluate the quality of evidence, we need to know how it was collected. Collec on of evidence in science is systema c; that is, there are rules and procedures that specify what researchers should do to gather evidence and limit the kinds of interpreta ons they are permi ed to make. Rules for sta s cal analysis are an important part of this. 1.5 EVALUATING GENERALIZABILITY Researchers and journalists usually want to generalize about their findings. In other words, instead of just saying: “45% of the respondents I talked to said they plan to vote for candidate X,” they want to say something like “45% of all registered voters plan to vote for candidate X.” Generalizability of results is the degree to which a researcher can claim that results obtained in a specific sample would be the same for a popula on of interest. Results from a sample can be generalized to an actual popula on of interest if the sample is representa ve of the popula on; representa veness can o en be obtained using random or systema c methods to select the sample. Results from an accidental or a convenience sample may be generalizable to a hypothe cal popula on if the sample resembles that hypothe cal popula on. Results from a biased sample are not generalizable. In experiments, generalizability also depends on similarity of type and dosages of experimental treatment to real-world experiences with the treatment variable, se ng, and other factors.

Polling organiza ons, such as Gallup, collect public opinion informa on in ways that provide a good basis for generaliza on. They use large samples (usually at least 1,000 individuals) and obtain these samples using combina ons of random and systema c selec on so that the people who responded to the survey resemble the larger popula on (such as all registered voters) in terms of age, income, and so forth (Gallup, n.d.). When journalists report informa on from polls and demographic studies, they are (once again) in a posi on to cherry-pick. Because of differences in procedures and types of people contacted, various polling organiza ons may report different predic ons about presiden al candidate preference. A journalist who wants to make a case to support Candidate X may report only the poll in which Candidate X had the highest approval ra ngs. In behavioral and social science, the problem of generalizability can have a different form. A researcher may want to know whether cogni ve behavioral therapy (CBT) reduces depression. Typically, studies examine small to moderate numbers of cases, for instance, 35 pa ents who receive CBT and 35 who do not. To generalize results about effects of CBT to a large hypothe cal popula on of “all depressed persons,” ideally, we would want a random sample drawn from that popula on. However, par cipants are o en convenience samples, that is, people who were easy to recruit. It is important to know what kinds of people were (and were not) included in a study. For example, if a drug study finds evidence that a new medica on is effec ve and safe for healthy young men, that does not necessarily mean that the drug is also effec ve and safe for women, elders, children, and other kinds of people not included in the study. Be careful not to overgeneralize results, par cularly when there is li le informa on about the types and numbers of people (or cases) included. It makes sense to generalize informa on from a small group to some larger popula on only when people in the group resemble the popula on of interest. This is discussed further in Chapter 2 in sec ons about samples and popula ons. In science communica on, authors are expected to discuss limita ons that must be considered before drawing any conclusions. Limita ons include the number and kinds of people (or cases) included in a study. Science wri ng should make limita ons of evidence clear; media repor ng o en does not. 1.6 MAKING CAUSAL CLAIMS In everyday life, and in science, we o en want to know about causal connec ons. Consider a ques on raised by Wootson (2017). Do diet (ar ficially sweetened) so drinks cause weight gain? If you are concerned about weight gain, and if ar ficially sweetened so drinks cause weight gain, then you might consider avoiding diet so drinks to avoid weight gain. However, it is possible that the associa on reported in some studies did not arise because of any direct causal impact of diet so drinks on weight. Perhaps when people drink diet so drinks, they feel

free to indulge in other high-calorie foods, and perhaps it is those other high-calorie foods, not the so drinks in and of themselves, that cause weight gain. If that is the correct explana on, then what you need to do to avoid weight gain is to avoid consuming high-calorie foods (rather than reduce diet soda consump on). Causal explana ons are a rac ve because they e events together in meaningful ways. This is useful in science as well as everyday life. Some mes when a cause–effect rela onship is known, it suggests what we can do to change outcomes. Demonstra ng that two events are causally connected can be difficult, because there are o en rival possible explana ons. Well-controlled experiments can rule out many rival explana ons. In everyday life, people some mes jump to conclusions about causality on the basis of insufficient evidence. 1.6.1 The “Post Hoc, Ergo Propter Hoc” Fallacy News commentators frequently offer causal explana ons for events (e.g., the stock market went down because of a blizzard the previous day). This explana on is o en just an opinion of the news commentator. The stock market might have gone down for other reasons (including random varia ons). This is an example of a common logical fallacy called “post hoc, ergo propter hoc.” This La n phrase means “a er this, therefore, because of this.” This (incorrect) reasoning goes like this: If Event A happens, and then Event B happens, then A must have caused B. Before we can conclude that Event A caused Event B, addi onal condi ons are required. Here is another example. If you have a cold, take a large dose of vitamin C, and then the cold goes away, you might conclude that vitamin C cured the cold. However, the cold might have gone away on its own, whether you took vitamin C or not. Post hoc, ergo propter hoc reasoning uses one instance of co-occurrence (vitamin C, end of cold) to draw a causal conclusion. That is poor-quality reasoning that o en leads to mistaken beliefs in causality. To conclude that vitamin C cures colds, you would need an experiment to evaluate whether the dura on of colds was less in a group that took vitamin C than in a group that did not (controlling for other factors, such as placebo effects). 1.6.2 Correla on (by Itself) Does Not Imply Causa on You may have frequently heard the warning that correla on does not imply causa on. This warning should be stated more precisely. It is more accurate to say, Existence of a sta s cal rela onship, such as a correla on, between variables X and Y, is needed to make claims that X causes Y. However, the mere existence of a sta s cal rela onship does not prove that X causes Y. Alterna ve explana ons for the sta s cal rela onship between X and Y must be ruled out before we can believe that X causes Y. Let’s examine this idea carefully. The word correla on has two meanings. First, some mes people use the term correla on to refer to a specific sta s c: the Pearson product–moment correla on, also called Pearson’s r. Second, the term correla on can be used in a broader sense; we can say that variables are

correlated if they are sta s cally related using some sta s cal analysis. The sta s cal analysis can be something other than Pearson’s r. For example, if we compare average height for male and female groups and find that men are taller than women, we can say that sex (X) is sta s cally related to height (Y) or that sex is correlated with height. We cannot claim that an X variable “causes” a Y variable if there is no sta s cal rela onship of any kind between X and Y. In other words, the existence of a sta s cal rela onship between X and Y is a necessary condi on before we can consider causal inference. However, existence of a sta s cal associa on is not enough evidence by itself to prove causality. Some mes variables are sta s cally related (correlated) just by chance, or because the X and Y variables are related to some third variable Z, and Z may be the real “cause.” Consider this example: If we measure ice cream sales (X) and number of homicides (Y) once a month for a year, there is a correla on between them. Months that have the most ice cream sales also have the largest number of homicides (Peters, 2013). Does ea ng ice cream cause people to commit homicide? That idea is obviously silly. A more plausible explana on is that temperature is related to both ice cream consump on and homicide. During ho er months, people may buy more ice cream; homicide rates are higher in ho er months (perhaps because people hang around outside more, or perhaps heat makes people more irritable). Correla on (sta s cal associa on) is a necessary but not sufficient condi on for making causal inference. Sta s cal associa on is necessary because we can’t conclude that X causes Y unless X and Y go together or co-occur. Sta s cal associa on is not sufficient by itself to prove causa on because, even if X and Y covary, this co-occurrence may be due to the influence of one or more other variables; one of those other variables might be the real cause of X, or of Y, or both. In this example, heat or temperature might cause (or at least predict) ice cream purchase and homicide. The effects of rival explanatory variables can be reduced or eliminated in well-controlled experiments and reduced by sta s cal controls. Mere co-occurrence is not enough evidence to make a causal inference. Some mes the need to look for a different explana on is obvious (as in the ice cream/homicide example). It would be absurd to argue that ice cream causes homicide. However, the need to consider rival explana ons also arises in situa ons that are not so obviously silly. In the diet so drink/weight gain example, it is conceivable that ar ficial sweeteners have causal effects on appe te or metabolism that really do lead to weight gain, even though the ar ficial sweeteners contain zero (or negligible) calories. However, the other explana on (that drinking diet beverages leads people to indulge in other high-calorie foods) is also plausible. (It is also conceivable that both these explana ons are partly correct.) Both experimental and nonexperimental studies, with humans and nonhuman animals, would be helpful in sor ng out the rela ons among variables and whether any of the associa ons are causal.

1.6.3 Perfect Correla on Versus Imperfect Correla on Perfect co-occurrence (perfect correla on or sta s cal associa on) is rare. Consider the gene c muta on for hemophilia (Table 1.1). If a male child inherits this gene c muta on, he will have hemophilia. Most other heritable diseases do not show this perfect associa on. (For female children, effects of the hemophilia gene are ruled out by informa on on the other X chromosome.) Table 1.1 Example of Perfect Co-occurrence or Perfect Correla on (Between Gene and Disease)

If a male child does not inherit the gene for hemophilia, he will not have hemophilia. In logical terms, the mutated gene is both necessary and sufficient for the disease. The mutated gene is necessary for hemophilia because a person can’t get hemophilia without it. The mutated gene is sufficient for hemophilia, because if a person has it, he always has hemophilia. In other words, hemophilia always occurs when the mutated gene is present and never occurs when the mutated gene is absent. Most associa ons in behavioral and social sciences and medicine are not perfect. Consider this hypothe cal example for a behavior (washing or not washing hands) and a disease outcome (ge ng sick). Table 1.2 shows an imperfect associa on. Only 25% of regular hand washers got sick, while 67% of the those who don’t regularly wash their hands got sick. While most people who washed their hands did not get sick, hand washing did not guarantee that they could avoid ge ng sick. The associa on between lung cancer and smoking is also not perfect. The risk for ge ng lung cancer is much higher for smokers than for nonsmokers. However, a few nonsmokers do get lung cancer, and many smokers do not get lung cancer. In situa ons where associa ons are not perfect, it is likely that other variables are involved. Behaviors or condi ons that some mes (but not always) precede disease are o en usually called “risk factors” rather than causes. Smoking is a risk factor for lung cancer. Some diseases have numerous risk factors (for example, risk for heart disease is related to smoking, body weight, sex, age, high blood pressure, and other factors).

We call behaviors that reduce risk for a nega ve outcome “protec ve factors.” For example, hand washing is a protec ve factor against ge ng sick. 1.6.4 “Individual Results Vary” Unless there is a perfect correla on (as in the hemophilia example), sta s cal associa ons or correla ons between variables do not predict exact outcomes for all individuals. Consider the results of a study by Judge and Cable (2004), informally reported in Di man (July/ August 2014). They reported that taller persons tend to earn more money (that is, height is correlated with salary). This is not a perfect correla on. If you are short, that does not necessarily mean that you will earn very li le. Mark Zuckerberg (the founder of Facebook) is reported to be 5’7”, but that did not prevent him from becoming one of the wealthiest men in the world. If you think about the implica ons correla ons might have for your own outcomes, realize that individual outcomes differ when correla ons are not perfect. 1.6.5 Requirements for Evidence of Causal Inference Training in research methods and sta s cs provides the skills scien sts need to think carefully about the evidence needed to support causal claims. Mass media journalists o en rely on secondary sources or third-party content. By the me informa on filters through mul ple communica on links, details about the nature of the evidence and concerns about limita ons that affect the ability to generalize and make causal inferences are o en lost. Third-party content o en does not provide accurate informa on about generalizability and poten al causality. 1.7 QUALITY CONTROL MECHANISMS IN SCIENCE 1.7.1 Peer Review The science research process has mechanisms for informa on quality control. The most important mechanism is peer review. Researchers submit research reports to science journals (also called academic journals) for considera on (see note 2). The editor sends papers to peer reviewers (peers are expert researchers in the same field). Reviewers provide detailed cri cism of studies, including evalua on of their research methods. On the basis of reviews, editors decide whether to reject a paper as inadequate, ask authors to revise the paper to correct errors or deficiencies, or (very rarely) accept the paper with only minor correc ons. Papers are rarely accepted in their ini ally submi ed form. Rejec on rates for some journals are 80% or higher. Peer review is fallible. Reviewers can also be subject to confirma on bias (they are more likely to favor conclusions consistent with their own beliefs). Reviewers may not no ce all of the problems in a research report. However, peer review weeds out much poorly conducted research and improves the quality of published papers. The community of scien sts in effect systema cally polices the work of all individual scien sts. 1.7.2 Replica on and Accumula on of Evidence A second important mechanism for data quality control in academic research is replica on. Replica on means repea ng or redoing a study. This can be an exact replica on (keeping all

methods the same) or a conceptual replica on (changing elements of the study, such as loca on, measures, or type of par cipants, to evaluate whether the same results occur in different situa ons). We should not treat findings from any one study as a conclusive answer to a research ques on. Any single study may have unique problems or flaws. In an ideal world, before we accept a research claim, we should have a substan al body of good-quality and consistent evidence to back up that claim; this can be obtained from replica ons. Peer review and replica on in science are fallible. However, they provide the best ongoing quality control checks we have. In contrast to science, there are few quality control mechanisms for most mass media communica on. 1.7.3 Open Science and Study Preregistra on There are recent ini a ves to improve the reproducibility and quality of research results in biomedicine, psychology, and other fields (Begley & Ioannidis, 2015; Open Science Collabora on, 2015). The Open Science model includes components such as preregistra on of research plans and sharing details of data and methods. For further discussion, see Cumming and Calin-Jageman (2016). 1.8 BIASES OF INFORMATION CONSUMERS 1.8.1 Confirma on Bias (Again) Informa on consumers or receivers also tend to select evidence consistent with their preexis ng beliefs. Media consumers need to be aware that they can systema cally miss kinds of informa on (which may be of high or low quality) when they select news sources they like. Ra ngs of many web news sources on a con nuum from le /liberal to right/conserva ve, along with assessment of accuracy, are provided at h ps://mediabiasfactcheck.com/poli fact/. News sources that are extremely far le or far right tend to be less accurate. Because of confirma on bias, people can get stuck: They con nue to believe “facts” that aren’t true, and ideas that are wrong, because they never expose themselves to informa on that might prompt them to consider different possibili es. Consumers of mass media usually avoid evidence that challenges their beliefs. Philosopher of science Karl Popper argued that scien sts also need to examine evidence that might falsify their beliefs. Scien sts and people in general should consider evidence that challenges their beliefs. 1.8.2 Social Influence and Consensus Should we believe something simply because many people, par cularly those whom we know and respect, believe it? Not necessarily. Some incorrect beliefs are widely reported in mass media and held by millions of people. My personal favorite conspiracy theory is that alien rep les control U.S. poli cs. Bump (2013) reported that more than 12 million people, or 4%, of the U.S. popula on said that they believed this theory in 2012–2013. To be clear, I strongly disbelieve that we are ruled by alien rep les. (I am also not sure whether to believe Bump’s report that 12 million people really believe this; surveys are not always accurate.)

Consensus among science researchers can enhance the believability of a claim. However, even in science, consensus does not always guarantee accuracy. Experts can turn out to be wrong. For example, there was a consensus among nutri on researchers that eggs are bad for health because of their cholesterol content. Some recent research suggests that this widely held belief may be incorrect4 (Gray & Griffin, 2009), but the issue con nues to be controversial. A belief shared by millions of people is not necessarily wrong. However, consensus is neither necessary nor sufficient evidence that informa on is correct. 1.9 ETHICAL ISSUES IN DATA COLLECTION AND ANALYSIS 1.9.1 Ethical Guidelines for Researchers: Data Collec on Ethical issues arise when collec ng data about people and nonhuman animals. For psychologists, the American Psychological Associa on has codes of ethics that protect the well- being of subjects (Campbell, Vasquez, Behnke, & Kinscherff, 2009). Research that involves human par cipants is evaluated by an ins tu onal review board; research that involves nonhuman animals is evaluated by an ins tu onal animal care and use commi ee. Ethical codes govern research in other areas such as biomedicine. Data collec on cannot begin un l ethics board approval of procedures has been obtained. Adherence to those rules is an ethical obliga on for researchers. We should not harm the people or en es we study. As an example of poten al harm to a research par cipant, suppose that a study reveals that a person has a history of addic on. If that informa on gets into the hands of poten al landlords or employers, it could have an impact on that person’s search for housing and jobs. Researchers must keep such records confiden al. Researchers also have an ethical responsibility to think about the poten al impact of their research (both posi ve and nega ve) on public policy and the behavior of organiza ons and individuals. 1.9.2 Ethical Guidelines for Sta s cians: Data Analysis and Repor ng The GAISE report states, “Students should demonstrate an awareness of ethical issues associated with sound sta s cal prac ce” (GAISE College Report ASA RevisionCommi ee, 2016). A separate document (American Sta s cal Associa on, 2015) discusses ethical issues in detail. Here is a list of ethical prac ces for data analysts, paraphrased from the American Sta s cal Associa on’s ethics document. You will be reminded about these issues as you con nue through the book. Ensure that numbers are accurate. Fully disclose data handling procedures (such as dele on of cases or replacement of missing values) that could alter conclusions. Make the limita ons of the type of sta s cal analysis clear. (As each new analysis is introduced, you will learn about its limita ons.) Avoid behaviors that can lead to errors (including, but not limited to, cherry-picking a few results).

Avoid misleading presenta ons (such as “lying graphs”; see Sec on 1.10). Avoid language that obscures results. Do not overgeneralize. Do not make strong claims about characteris cs of a popula on when your sample does not resemble that popula on. Real-world problems in applica ons of data analysis are o en not clear in introductory courses; students learn to do one analysis at a me using one small set of numbers. In actual prac ce, data analysts o en work with large sets of messy data. Data analysts need to make many choices that involve difficult judgment calls. This book points out differences between the ideal use of sta s cs in ar ficially simplified situa ons and the actual applica on of sta s cs to real- world data. Some mes decisions about “best prac ce” are difficult. As Harris (2001) said, “Sta s cs is a form of social control over the professional behavior of researchers. The ul mate jus fica on for any sta s cal procedure lies in the kinds of research behavior it encourages or discourages.” Science has rules and standards about good prac ce in collec on, analysis, and presenta on of evidence. These are discussed throughout this book. Researchers should be aware that press releases from universi es some mes overhype research findings (Resnick, 2019). This book discusses good prac ces in applied sta s cs that can poten ally improve the clarity and honesty of research reports. When communicators present informa on in misleading, unclear, or dishonest ways, they risk loss of credibility, trust, and respect, not just for themselves but for the professions of sta s cs and science. When informa on consumers rely on incorrect informa on, they may make poor decisions. 1.10 LYING WITH GRAPHS AND STATISTICS The most extreme form of lying with sta s cs is fabrica on or falsifica on of data; this is rare. However, some common research prac ces slant informa on presenta on in ways that can be called “lying with sta s cs.” The classic book How to Lie With Sta s cs (Huff, 1954) presented numerous examples. Decep ve bar graphs are among the most common ways informa on communicators mislead informa on consumers. If you will be an informa on producer, you need to know how to set up “honest” bar graphs. When you are an informa on consumer, you need to know how to examine graphs to make sure that they are not misleading. Chapter 5 provides examples of clear versus misleading graphs and guidelines for evalua on of graphs. 1.11 DEGREES OF BELIEF People rarely have me to collect all necessary informa on. Even for ques ons in science, we o en do not have enough informa on to be confident about conclusions. Uncertainty is more common than people realize, even in areas such as medicine. There are many ques ons in medicine (such as what causes autoimmune disorders) for which medical research does not have good answers (Fox, 2003).

It is useful to think about scien fic knowledge in terms of degree of belief instead of certainty. The philosopher David Hume said that “a wise [person] … propor ons his [or her] belief to the evidence” (Schmidt, 2004). Degree of belief should be based on the quan ty of consistent and good-quality, systema cally collected suppor ng evidence. When there is li le evidence (for example, results from only one study), people should not have strong belief in a claim. As addi onal good-quality evidence accumulates, degree of belief can increase. People should revise degree of belief upward or downward as new (good-quality) evidence becomes available. This ra ng scale illustrates the concept of degree of belief. The use of a five-point scale and the exact verbal descrip ons for each numerical ra ng are arbitrary.

1.12 SUMMARY Here are some ques ons to keep in mind when evalua ng numerical (and other) informa on. Is there evidence of communicator bias or self-interest? Is evidence cherry-picked to fit the communicator’s argument? Is the communicator far from the informa on source or not well qualified to evaluate the informa on? Does the communicator have a good record for truth-telling? What types of evidence are included. Anecdotes? Cita ons of specific, credible sources? Have you considered your own possible biases as an informa on consumer? Do you accept informa on uncri cally because it confirms when you already believe? Are you influenced by what other people believe? Do data come from people (or cases) who resemble the popula on of interest? Are results generalizable? Are causal inferences drawn when there is not enough informa on to prove a causal associa on? Remember that imperfect correla on or co-occurrence does not indicate causa on. Has informa on been subjected to quality control? (In science, this includes peer review and replica on.) Is the presenta on of informa on decep ve (e.g., lying graphs)? What ethical issues are at stake in the conduct and applica on of the research? Is your degree of belief propor onal to the quan ty of good quality and consistent evidence? (You should never believe a claim on the basis of just one scien fic study or one journalism report.)

Some mes the best answer to ques ons such as “Are eggs harmful to cardiovascular health?” is that we don’t have enough evidence yet to answer the ques on. Unfortunately, lack of evidence does not prevent some communicators from making premature claims. When claims are made on the basis of limited evidence, contradic on and confusion o en arise. It is be er to reserve judgment un l a large quan ty of good-quality evidence is available. One single media report, or one single science report, is not “proof.” Even if you do not plan to be a researcher, you can benefit from thinking like a scien st and sta s cian about numerical evidence you encounter in everyday life. Some decisions have high stakes. For example, you may need to decide whether to undertake a risky but poten ally beneficial medical treatment. Ideally, you should have accurate informa on about poten al outcomes. The higher the stakes, the more you need to know how to obtain trustworthy informa on. The take-home message from this chapter is: We all know a lot less than we think we do, because most of us rely heavily on third-party content that has li le or no informa on quality control. All of us (scien sts, journalists, and informa on consumers) should be cau ous about degree of belief. Some mes the best answer to a ques on is: We don’t have enough good quality evidence. Courses in sta s cs and research methods teach you good prac ce in evalua on and presenta on of evidence. COMPREHENSION QUESTIONS What is cherry-picking of evidence, and why is it decep ve? (Can you think of a book or media report that seems to present cherry-picked evidence?) Give examples of self-interest that might make a communicator less believable. Why is distance to original source of informa on an important factor when you evaluate message credibility? What does it mean to say that a correla on (or associa on) between variables is imperfect? Give an example of a risk factor, and a protec ve factor, not discussed in the chapter. Why is the existence of a correla on (existence of co-occurrence or associa on) between X and Y not enough evidence for us to say that X causes Y? What is the post hoc, ergo propter hoc fallacy? (Give an example you have seen, different from the one in this chapter.) What is confirma on bias? What quality control mechanisms are used in science? What is peer review? How can it improve the credibility of science repor ng? What is research replica on? How can this improve the quality of evidence in science? How do exact replica on and conceptual replica on differ? A researcher might say “the results of this one study prove” something. Is this jus fied? What (approximate) degree of belief should you have on the basis of only one study? NOTES 1 Scien sts are expected to be objec ve when they select informa on to report. However, scien sts tend to focus selec vely on informa on consistent with the most widely accepted exis ng theories; Kuhn and Hacking (2012) called this “selec on of significant fact.”

2 Numerous predatory, for-profit online journal publishers have emerged in recent years. It has become more difficult to determine whether online publica ons are credible. Research reports published in predatory journals are not valued by professional colleagues and universi es. Beall’s List of Predatory Journals and Publishers names many publishers that are almost certainly predatory (h ps://beallslist.weebly.com). Addi onal warning signs that a publisher may be predatory: The journal invites you to submit your undergraduate or graduate thesis for publica on (par cularly if the journal tle is not in your discipline or field). The journal offers to publish your paper without peer review. The journal asks you to pay for publica on. (However, many legi mate publishers charge author fees to make journal ar cles open access on the web; therefore, a request for payment is not always an indica on that a journal is predatory.) If you are not sure whether a journal or publisher is predatory, search <journal name> or <publisher name> along with the term predatory. You can also ask mentors, advisers, or colleagues. 3 About half of Dr. Oz’s medical advice is not supported by medical research (Belluz, 2014). Dr. Oz was inves gated in a congressional hearing and paid large se lements in lawsuits for false adver sing (Cohen, 2015). 4 This video about an imaginary me-traveling die cian makes fun of changes in dietary recommenda ons across the decades: h ps://www.youtube.com/watch?v=5Ua-WVg1SsA. DIGITAL RESOURCES Find free study tools to support your learning, including eFlashcards, data sets, and web resources, on the accompanying website at edge.sagepub.com/warner3e. CHAPTER 2 BASIC RESEARCH CONCEPTS 2.1 INTRODUCTION Basic understanding of research methods is needed to understand and interpret sta s cal results. This chapter is a brief, nontechnical introduc on to selected research methods terms men oned in the GAISE (GAISE College Report ASA Revision Commi ee, 2016) numeracy guidelines in Chapter 1. The design of an inves ga on refers primarily to the dis nc on between designs in which inves gators have a high degree of control over the research situa on (such as experiments) and situa ons in which researchers have li le or no control (nonexperimental studies). Experimental methods of control include techniques such as random assignment of par cipants to groups and holding variables other than the treatment variable constant. Sta s cal methods of control are included in some types of analysis. Other design issues are discussed in greater detail in research methods textbooks (e.g., Cozby & Bates, 2017).

Data (or data set) refers to informa on, usually in numerical form in a computer file, about mul ple cases and/or mul ple variables. Analysis refers to sta s cal techniques. A variable is a characteris c that differs or varies across subjects or cases. Examples of variables for human research par cipants include sex, height, heart rate, and salary. Subjects or cases are the en es or observa onal units studied. In psychological research, cases are usually individual persons or nonhuman animals. In other disciplines, cases can be different kinds of en es; for example, in sociology, a case can be a group or an organiza on; in poli cal science, a case may be a na on; in forestry, a case may be a geographic loca on. The terms sample and popula on are o en used differently in ideal textbook situa ons than in many real-life research situa ons, as discussed in Sec on 2.11. For now, it is sufficient to say that a sample is a subset of a popula on; that is, a sample consists of cases selected from a popula on. A generaliza on is a statement that results obtained for people and situa ons included in a study are applicable to other people and situa ons not included in the study. Ability to generalize results from a sample to a popula on depends on similarity of the sample to the popula on of interest. Examples of errors in interpreta on include (a) generalizing results more widely than can be jus fied, (b) arguing that one variable causes another variable when there is not enough evidence to support that claim, and (c) misunderstanding the limits of research methods and sta s cal analyses. Other types of error are possible. 2.2 TYPES OF VARIABLES 2.2.1 Overview It is useful to dis nguish between categorical variables and quan ta ve variables (Jaccard & Becker, 2009). Scores for categorical variables tell us which group or category each case belongs to (e.g., whether a person is male or female). Scores for quan ta ve variables provide informa on about the amount of something (for example, height). Some psychologists make further dis nc ons among levels of measurement; see Appendix 2A for discussion. Two addi onal types of variables are discussed in this sec on: ra ng scales and ordinal (also called rank). 2.2.2 Categorical Variables Categorical variables iden fy group (or category) membership for each case. They are also called nominal variables because numbers serve only as names or labels for groups. This is a common type of variable. Examples of categorical variables include sex (for example, with group membership coded 1 = male, 2 = female) and marital status (with values coded 1 = never

married, 2 = divorced, 3 = currently married). Addi onal categories could be included; for example, marital status could include categories such as engaged, cohabi ng, separated, and remarried. Numerical values used for categorical variables are arbitrary; we could code divorced as 3 instead of 2, and this change in group numbering will make no difference in results of analyses. When numbers are only labels for group membership, it is not meaningful to compare these numbers in terms of “greater than” or “less than.” A person whose marital status is represented by the number 2 (divorced) is not greater than or be er than a person whose marital status is represented by 1 (never married). We can say only that these individuals differ in marital status. It makes no sense to apply arithme c opera ons (+, –, ×, ÷) to numbers when they are used only as labels for group membership. It makes no sense to calculate sta s cs such as sample means for scores on categorical variables; for example, it would be nonsense to compute a mean marital status. O en the number of different score values for a categorical variable is small. However, it is possible for categorical variables to have many different score values. For example, a categorical variable to iden fy choice of future career could include dozens of different possible careers. 2.2.3 Quan ta ve Variables Quan ta ve variables indicate “how much” of some characteris c or behavior each case or person has. For example, we can measure height or blood pressure for each person. When numerical scores for these variables are compared, it makes sense to describe them in terms of “more than” and “less than.” A person who is 70 inches tall is taller than a person who is 65 inches tall. It is reasonable to apply arithme c opera ons to numerical values for quan ta ve variables; we can add, subtract, mul ply, and divide scores. Thus, it is reasonable to compute a mean for variables such as height. Quan ta ve variables are common in behavioral and social science research. 2.2.4 Ordinal Variables Some mes researchers rank subjects instead of measuring amount. For example, we could tag the runners in a race as 1, 2, 3, … last (the order of crossing the finish line). Alterna vely, we could measure running me in seconds. Variables with scores that correspond to ranks are called ordinal variables. Later you will see that there are specific analyses for scores that are collected in the form of ranks or are converted to ranks to get rid of problems such as outliers. Ranks are not widely used in data collec on in behavioral and social sciences; measurements of quan ty are generally preferred. 2.2.5 Variable Type and Choice of Analysis Categorical and quan ta ve variables require different types of descrip ve sta s cs (Chapter 4), graphs (Chapter 5), and other sta s cal analyses. It is necessary to dis nguish between categorical and quan ta ve variables to choose appropriate sta s cal techniques. For some variables the decision is easy. Clearly, height and age are quan ta ve; sex and marital status are

categorical. However, there are examples of variables that can be handled as either categorical or quan ta ve, as noted in the next sec on. 2.2.6 Ra ng Scale Variables A Likert scale is a common response format in survey and personality research. A typical Likert scale ques on consists of a statement (worded so that it expresses a clearly posi ve or nega ve view about an issue) followed by a choice among degree of agreement ra ngs, as in the following example; each person chooses the number that best represents his or her degree of agreement. Originally Likert scales included five degrees of agreement, but mul ple-point ra ng scales o en have different numbers of responses (such as seven). Example: “I believe the president is doing a great job.”

If five-point ra ngs are evaluated according to the formal levels of measurement standards proposed by Stevens (1946, 1951; see Appendix 2A), they lie somewhere between the ordinal (rank) and interval levels of measurement. Ra ng scores provide at least rank-order informa on (e.g., 4 represents stronger agreement than 3). However, the differences between scores probably don’t represent equal intervals; for example, the difference in degree of agreement represented by 4 versus 5 may not be the same as the difference between 3 and 4. Five-point ra ng scale scores fall into a gray area: probably more informa ve than ranks, but probably less informa ve than measurements that assume equal intervals. That leads to disagreement as to whether it makes sense to compute means and other sta s cs for variables rated on five-point scales. Authori es cited in Appendix 2B argue that is acceptable to treat ra ng scale variables as quan ta ve variables in some circumstances. In prac ce, ra ngs on five-point scales can o en be treated as either categorical or quan ta ve variables, whichever makes more sense in a specific research situa on. Scores for the ques on above could be used to divide people into five groups that have different degrees of agreement (i.e., used as a categorical variable). It would also be reasonable to compute a mean for ra ngs. 2.2.7 Scores That Represent Counts Consider this survey ques on: “How many children do you want to have in the future?” Possible responses include none, one, and so forth. This is a quan ta ve variable; three children are more than two children. Unlike many other quan ta ve variables, scores for this variable have a limited number of possible values; it is rare in the United States to encounter persons who want more than four children. In a small sample, a researcher might find that the only responses to this ques on are zero, one, and two. In some analyses it may be convenient and informa ve to treat these scores as labels for group membership (e.g., Group 1 does not want any children, Group 2 wants only one child, and Group 3 wants two children). However, it is also reasonable to compute the mean number of children. For variables that consist of counts (e.g., number of children) and variables that represent ra ngs on degree of agreement or behavior frequency (as

in Sec on 2.2.3), it some mes makes more sense to handle them as categorical, and it some mes makes more sense to treat them as quan ta ve. 2.3 INDEPENDENT AND DEPENDENT VARIABLES The first sta s cal techniques you will learn are ways to describe scores for just one variable. However, real-world research usually begins with ques ons about the way two or more variables are related. It o en makes sense to iden fy one of the variables as the independent or predictor variable (X) and the other as a dependent or an outcome variable (Y). The decision about which variable to iden fy as independent depends on the nature of the research ques on about the variables. 2.4 TYPICAL RESEARCH QUESTIONS This sec on describes three types of research ques ons about the rela onship between two variables. When we dis nguish between independent and dependent variables, the independent variable is o en denoted X and the dependent variable Y. 2.4.1 Are X and Y Correlated? A researcher can simply ask whether scores on two variables (X and Y) tend to co-occur or go together (without assuming any causal connec on between them). There are alterna ve ways to word this ques on, such as: Are scores on X and Y correlated? Do scores for X and Y tend to co-occur? Are high scores on X associated with high scores on Y? Are X and Y associated? I prefer this wording: Are scores on X and Y sta s cally related? For this research ques on, it is not necessary to iden fy one variable as independent and the other variable as dependent. The term correlated can refer specifically to the results of a Pearson r correla on analysis. However, researchers some mes use the word correlated in a much broader sense, to refer to any sta s cal rela onship between variables, even when informa on about the rela onship comes from some sta s c other than a correla on coefficient (for example, from an independent-samples t test). We can evaluate whether X and Y are sta s cally related by doing whatever sta s cal analysis is appropriate for the types of variables (categorical vs. quan ta ve). The bivariate sta s cs described in later chapters provide different ways to evaluate the extent to which scores on two variables are sta s cally related. The specific sta s c that is most appropriate for a pair of X and Y variables depends on the types of variables (categorical or quan ta ve) and other issues; see Sec on 2.10. We can evaluate whether X and Y are sta s cally related on the basis of the outcome of any of these bivariate sta s cal analyses. 2.4.2 Does X Predict Y?

In this ques on, X is iden fied by the researcher as the predictor or independent variable; Y is the outcome or dependent variable. To predict means to an cipate or guess something that will happen in the future. A predictor should occur before the outcome (or at least not a er the outcome). This is called temporal precedence. If X happens before Y, X has temporal precedence. Consider these examples. Does height at age 10 years (X) predict height at age 21 years (Y)? Do high school grades (X) predict college grades (Y)? When temporal precedence is clear, it does not make sense to reverse these variables, that is, to ask whether height at age 21 predicts height at age 10 or whether college grades predict high school grades. 2.4.3 Does X Cause Y? This ques on can be worded in several similar ways; we can replace the word cause with words such as change, determine, increase, decrease, or influence. Here are examples of ques ons about cause: Does the death of a spouse (X) cause depression (Y)? Does study me (X) increase exam score (Y)? Does social stress (X) influence blood pressure (Y)? Does cigare e smoking (X) increase the risk for lung cancer (Y)? Is cigare e smoking a risk factor for lung cancer? (If a variable is called a risk factor, this usually implies that there may be other risk factors or causes.) Note that the word order in ques ons can vary, for example, Is exam score (Y) increased by amount of study me (X)? In this ques on, study me is s ll the independent variable (presumed cause), and exam score is the dependent variable. We need stronger evidence for claims that X causes or influences Y than for claims that X merely predicts Y or that X co-occurs with Y. Keep in mind that no ma er what results we obtain in one study of X and Y, we should not view those results as a final answer to any of these ques ons. 2.5 CONDITIONS FOR CAUSAL INFERENCE When researchers select variables to include in a study, the first considera on is: 1. There should be a plausible theory that explains why X and Y might be related (cf. Brannon, Feist, & Updegraff, 2017). It does not make sense to choose an X variable and a Y variable at random. Variables are selected because past research or theories suggest that they may be related in meaningful ways. Three addi onal condi ons should be considered when interpre ng research results as poten al evidence for causa on (Cozby & Bates, 2017).

2. We can say that X and Y are associated only if we find that X and Y are related when we do an appropriate sta s cal analysis. To evaluate whether X and Y are sta s cally related (or correlated), you will use the sta s cal analyses that you will learn in later chapters, such as the independent-samples t test and Pearson correla on. The next condi on is required for ques ons about predic on and causa on. 3. We can say that X predicts Y only if X happens earlier in me than Y (or at least not later than Y) and, in addi on, X is sta s cally related to Y. Ques ons about causal rela onships (does X cause or influence Y) require all these preceding types of evidence as well as this fourth addi onal type of evidence: 4. We can infer that X causes Y only if no other variables are plausible rival explana ons for changes in Y. In other words, X must be the only possible explana on for changes in Y. This condi on can be very difficult to sa sfy, because rival explanatory variables are common in many research situa ons. Rival explanatory variables (also called confounds or confounded variables) arise in situa ons where many variables (other than X) might cause or influence Y. Suppose a researcher wants to know whether social stress (X) causes higher blood pressure (Y). Many other variables, in addi on to social stress, can influence blood pressure, including but not limited to cardiovascular fitness, body weight, use of caffeine, alcohol, and other drugs, smoking, and family history of high blood pressure. Smoking and use of alcohol might well be correlated with or confounded with anxiety. We can evaluate whether anxiety influences high blood pressure only if we control for other explanatory variables (or take them into account in sta s cal analysis). In experiments, we take other rival explanatory variables into account by using experimental controls, such as holding the variables constant. For example, a study may include only people who do not use any drugs that may influence blood pressure. In nonexperimental studies, we use sta s cal control to try to rule out effects of rival explanatory variables. Techniques to do this are not covered in this volume; they involve more advanced forms of analysis. When a more sophis cated type of analysis is performed, the correct answer to the ques on “Does stress cause high blood pressure?” may be that stress is one among many variables that predict, and may possibly influence, blood pressure. Whether experimental or sta s cal control is used, readers need to know what variables have and have not been controlled in some way. When scores for two poten al causal or independent variables co-occur, we say that they are confounded. If people who experience a lot of social stress in their everyday lives tend to smoke a lot, then social stress and smoking are confounded, and it may be difficult to separate their effects. If people who report high levels of social stress have high blood pressure, the real reason for this (or at least a par al explana on for this) may be that people with high levels of stress also smoke or drink heavily.

The next sec on describes the extent to which various research designs (including nonexperimental, experimental, and quasi-experimental) can provide the evidence needed to sa sfy Condi ons 2, 3, and 4. 2.6 EXPERIMENTAL RESEARCH DESIGN A typical experimental research design includes two or more groups of cases; each group is exposed to a different type of treatment or different amount of treatment (such as a drug). Experiments require comparisons. If a researcher wants to evaluate the effects of caffeine (X) on heart rate (Y), the researcher needs to examine situa ons in which people do, and do not, receive caffeine (or situa ons in which people receive varying amounts of caffeine). In many studies, a control group that receives no treatment is included. Figure 2.1 is a schema c outline of a simple experiment. Read from le to right: the researcher has a group of available par cipants. Par cipants are divided into groups using a method that should ensure that similar people are included in Groups 1 and 2. O en random assignment to treatment groups is used to do this. In this example, Group 1 receives a beverage that does not contain caffeine; Group 2 receives a beverage that does contain caffeine. The outcome variable, heart rate, is measured a er par cipants drink the beverage. Sta s cal analysis compares mean heart rate to see if people who consumed caffeine (Group 2) have a higher average heart rate than people who did not consume caffeine (Group 1). (A placebo control group could be added.) The independent-samples t test is one example of a sta s c that provides informa on about the differences for means of Y across groups.

Figure 2.1 Schema c Outline of Simple Experimental Design In behavioral and social sciences, experimental design typically includes several kinds of experimental control. One form of experimental control is that a researcher controls assignment

of par cipants to groups. In many experiments, cases are assigned to groups randomly. The intended purpose of random assignment is to avoid a confound of preexis ng subject characteris cs with type of treatment. (Note that random sampling of par cipants from a popula on is not the same as random assignment of those par cipants to treatment groups.) Here is an example of a poten al confound of par cipant characteris cs with type of treatment. Suppose that a researcher arbitrarily assigns people to groups. Suppose that people in Group 1 (who do not consume caffeine) have low anxiety levels; people in Group 2 (who consume caffeine) have high anxiety levels. If average heart rate is higher in Group 2, it will not be clear whether this is due to caffeine or to preexis ng anxiety (or both). There is a confound between the independent variable X (whether caffeine is present, no or yes) and a personal characteris c (preexis ng anxiety). Preexis ng anxiety is a plausible rival explanatory variable; we cannot conclude that caffeine caused a higher heart rate unless we can control for, rule out, or get rid of the differences in anxiety between groups. A common way to try to prevent confound of treatment with par cipant characteris cs is random assignment of par cipants to groups or condi ons. Random assignment means that each subject or case has an equal chance of being placed in either group. An example of a method of random assignment is tossing a coin for each person and assigning the person to the no-caffeine group for heads and to the caffeine group for tails. This should result in a mixture of high and low anxiety scores within each of the two groups, with the same average anxiety score in Group 1 as in Group 2. This should also make the groups similar on other par cipant characteris cs, such as age, past experience with caffeine, and body weight. When it works well, random assignment of par cipants to condi ons prevents confounds of most par cipant characteris cs with type of treatment. The researcher has control over the type and amount of treatment. In this example, the researcher controls whether each par cipant receives caffeine and the amount of caffeine. The researcher can control other variables and tries to keep them the same across par cipants both between groups and within groups. This is called standardiza on and experimental control over other situa onal factors or extraneous variables. Variables that are not included in the research ques on are extraneous (not of interest) in the present study. Many things other than the caffeine administered by the researcher could influence heart rate (for example, me of day, whether the research assistant is calm or upset, and whether par cipants know that they are consuming caffeine). To achieve standardiza on, ideally, all par cipants would be tested at the same me of day; the behavior of the research assistant would be made consistent, perhaps by training or even the use of a script; and neither par cipants nor research assistants would know which drinks contain caffeine. Researchers need background knowledge about their variables to understand what kinds of confounds they need to an cipate and avoid. For example, if heart rate is the dependent variable, the researcher needs to know what other factors (apart from the manipulated variable, caffeine) might influence heart rate.

Some mes experimental control does not work as well as hoped. Random assignment of par cipants to groups can result in “unlucky randomiza on,” that is, groups that are not similar on one or more par cipant characteris cs. In implementa on of a treatment, variables may be uninten onally confounded with type of treatment. Consider the hypothe cal flawed study in Figure 2.2. Figure 2.2 illustrates two possible confounds. First, Groups 1 and 2 include different types of students (high vs. low academic ability). Second, Groups 1 and 2 had different instructors (Dr. Feelgood vs. Dr. Deadly). Any differences we find between final exam scores in these groups might be due to one or more of the following rival explanatory variables: classroom versus online se ng, academic ability levels of students, and behaviors of the different instructors. We cannot conclude that classroom and online instruc on cause different results on final exams unless we can rule out or get rid of the effects of the two confounded, rival explanatory variables (student ability and teacher iden ty). In many experimental situa ons, there are large numbers of poten al confounds. See research methods textbooks (such as Cozby & Bates, 2017) for further discussion of experimental control.

Figure 2.2 Flawed Study to Compare Mean Exam Scores (Y) Between Classroom and Online Instruc on (X) When poten al confounds and extraneous variables can be ruled out by these forms of experimental control, an experiment can provide good-quality evidence that may be consistent with a researcher hypothesis about causal inference. (The results of a single study should not be considered proof of causal influence.) Nonexperimental designs lack all these types of experimental control. Quasi-experimental studies typically have some, but not all, of these forms of control.

2.7 NONEXPERIMENTAL RESEARCH DESIGN In a typical nonexperimental research design (also called a correla onal study), a researcher measures two or more variables that are believed to be meaningfully related, and the researcher does not introduce a treatment or interven on. Consider this example. Suppose that X is a measurement of amount of (naturally occurring) physical exercise, and Y is a score for depression. Both variables might be measured using self- report survey ques ons. A researcher may suspect that there is a causal associa on (ge ng more exercise reduces depression). See Figure 2.3. Suppose that there is a strong correla on: People who report that they choose to exercise more tend to report lower levels of depression; people who report that they choose to exercise less tend to report higher levels of depression. That outcome cannot be interpreted as evidence that exercise causes a reduc on in depression, because the data do not come from an experiment. One requirement for causal inference is that the variable thought to be the cause should happen earlier in me than the variable thought to be the outcome. A nonexperimental study can (partly) sa sfy that requirement by measuring exercise first and depression at a later point in me. Another op on is to measure exercise and depression at mul ple points in me.

Figure 2.3 Diagram of Nonexperimental Design With Two Variables A more serious problem is that exercise is confounded with other variables, and those other variables might influence depression. For example, a person who experiences chronic stress may not feel like exercising, and chronic stress might cause depression. It is also possible that depression causes people to exercise less.

Advanced courses in sta s cs include methods for sta s cal control that can help separate the influences of rival explanatory variables (for example, using mul ple regression). However, if all you have is a sta s cal rela onship between amount of exercise and depression, and amount of exercise has not been manipulated, that is not sufficient evidence to conclude that lack of exercise causes depression. It may occur to you that you could do an experiment in which you randomly assign people to high-exercise and no-exercise groups and measure later depression. That is possible, although it would be a challenge to create a good experiment for these variables. Results from nonexperimental studies can sa sfy the first two requirements in the list of condi ons for causal inference. Variables X and Y can be chosen so that there is some logical or theore cal connec on between them. Some mes, but not always, there is clear temporal precedence, so that one variable can be iden fied as predictor and the other as outcome. Nonexperimental research can be sufficient to answer the ques on, Do X and Y co-occur? If a strong argument can be made for temporal precedence, data from nonexperimental studies can also be used to ask, Does X predict Y? Researchers o en iden fy variables in nonexperimental studies as independent and others as dependent, on the basis of theories about possible causal connec ons. However, dis nc ons between independent and dependent variables in nonexperimental studies are some mes arbitrary (and even ques onable). Consider a survey that measures self-esteem (X) and grades (Y) at the same point in me for a group of schoolchildren. If the analysis shows that higher self- esteem tends to co-occur with higher grades, and if the theory says that self-esteem causes be er performance in school, a researcher may be tempted to phrase the interpreta on in ways that suggest that the study proved a causal connec on; the researcher might say, “High self- esteem leads to higher grades” (leads to is one of many synonyms for causes). It is plausible to theorize that grades increase self-esteem, but it is also possible that self-esteem increases grades or that both grades and self-esteem are influenced by other variables, such as intelligence. In a situa on like this, I would say that neither self-esteem nor grades are clearly “the” independent variable or dependent variable. When there is no temporal precedence and no ability to rule out rival explanatory variables, it is preferable to say that X and Y are correlated variables (instead of calling one independent and the other dependent). 2.8 QUASI-EXPERIMENTAL RESEARCH DESIGNS Studies that compare group outcomes but lack the full set of controls in true experiments (such as researcher control over assignment of par cipants to groups, researcher administra on of treatments, and researcher control over other situa onal variables) are called quasi- experiments. Quasi-experimental research designs fall between experimental and nonexperimental designs in their ability to rule out rival explanatory variables. Quasi- experiments o en arise when programs are evaluated in field se ngs. Occasionally, true experiments are run in field se ngs, but it is generally easier for researchers to have control over variables when they are in laboratory se ngs.

The simplest types of quasi-experiments involve comparison of two or more groups that receive different treatments (Figure 2.4) using preexis ng groups instead of groups formed by a researcher. For example, each of two classrooms or schools may be used as a group. When preexis ng groups are compared, the members of groups are likely to differ in many preexis ng characteris cs. This is called a nonequivalent control group design. Consider poten al problems in the group comparison design in Figure 2.4. Because the researcher cannot control the assignment of subjects to groups, the groups that do versus do not experience the program o en include different kinds of par cipants (i.e., there may be a confound between par cipant characteris cs and type of treatment).

Figure 2.4 Quasi-Experimental Nonequivalent Control Group Design In addi on, when data are collected in field se ngs such as schools over long periods of me, other events that might influence the outcome variable may occur. As an example, consider a hypothe cal study to evaluate a drug educa on program (students in School 1 do not receive it; students in School 2 do receive it). The outcome measure could be self-reported inten on to use drugs. To what extent does the drug educa on program have an impact on this? It is possible that School 1 and School 2 differ in ways that would influence drug use inten on, for example, family religious backgrounds. It is possible that things happen in School 1 that did not happen in School 2 over the course of the study; for example, a popular student in School 1 dies from a drug overdose, which does not happen in School 2. These confounds would make it impossible to tell whether the drug educa on program causes any observed difference between groups for inten on to use drugs.

A second simple quasi-experimental design compares scores for one group a er the interven on with scores for the same group before the interven on (Figure 2.5). At first glance this may seem to be less problema c than the nonequivalent control group design, but this simple design is quite problema c. Many events, in addi on to the interven on, may occur between Times 1 and 2, and any of these events might influence the outcome. A student may die in an alcohol-related car accident, and that event is a rival explanatory variable. If the study takes place over 3 years, there is me for matura on to occur (students are 3 years older at Time 2 than at Time 1, and changes in scores might be related to age). Shadish, Cook, and Campbell (2001) provided extensive informa on about the design and analysis of quasi- experimental studies. 2.9 OTHER ISSUES IN DESIGN AND ANALYSIS Beginning students some mes ask ques ons such as “Which is be er, an experiment or a nonexperimental study?” It is more informa ve to ask, What are the poten al advantages and disadvantages of experimental versus nonexperimental studies?

The three types of design just reviewed (experimental, nonexperimental, and quasi- experimental) differ in the amount of control a researcher has over assignment to groups and ability to rule out rival explanatory variables. Some mes situa ons in which a researcher has a substan al amount of control are in laboratory se ngs. Laboratory se ngs and experiments may be ar ficial or contrived situa ons (in other words, different from real-world situa ons). Consider one highly contrived research situa on in psychology: the Skinner box. A rat or pigeon is placed in a glass box. No other animals are present. Lights or tones act as signals for the performance of a specific behavior, such as lever pressing for the availability of a reward. Food, water, or other rewards drop into the box when a lever is pressed. The schedule for the availability of rewards is completely under researcher control. All other variables, for all prac cal purposes, are held constant: temperature, ligh ng condi ons, the age and health of the rat, and so forth. Interac ons of the human researcher with the animal may be minimal.

This situa on is ideal if the goal is to make causal inferences: How does the schedule of reinforcement or reward influence the frequency of lever-pressing behavior? There are few or no rival explanatory variables. However, this situa on is not ideal if we want to know about learning or food foraging in natural environments, where different factors may be important, or learning in species other than rats and pigeons. In psychology the terms internal validity and external validity are used to describe two different aspects of research situa ons. A study has high internal validity when control of rival explanatory variables is so thorough that there are no rival explanatory variables to worry about when making a causal inference. Experiments in lab se ngs can poten ally have high internal validity. Nonexperimental studies typically have low internal validity, because the ability to rule out rival explanatory variables is limited. External validity refers to the similarity of the situa on in the study to real-world situa ons we would like to be able to talk about. A study has high external validity if the situa ons resemble real-world situa ons of interest and low external validity if the situa ons are so ar ficial and contrived that they don’t resemble any real-world situa ons of interest. O en nonexperimental research has higher external validity than experimental research, because researchers observe or ask about naturally occurring behaviors, some mes in real-world se ngs. There tends to be a trade-off between internal and external validity. O en, we have the best internal validity in experimental situa ons that are highly controlled and ar ficial, but these situa ons may have poor external validity. O en, we have the best external validity in uncontrolled nonexperimental studies, but these studies usually have poor internal validity. There are things researchers can do to improve external validity in lab experiments; the goal is to make the situa on as lifelike and believable as possible. There are things researchers can do to improve internal validity in nonexperimental studies; o en this involves the use of sta s cal control to compensate for the lack of experimental control. We can build the strongest possible cause for a claim (for example, that crowding increases hos lity) when we can show that the evidence for this claim is consistent across many different contexts: lab versus field se ng, experimental versus nonexperimental design, animal and human subjects, different ways of measuring hos lity, and so forth. Another issue to consider in thinking about possible designs for a study is whether the groups in a design are between-S, as in Figures 2.1, 2.2, and 2.4, or within-S or repeated measures, as in Figure 2.5. In a typical between-S (also called independent-groups) study, each par cipant is assigned to just one group and contributes one score for the outcome variable. In a within-S or repeated-measures study, each case or par cipant receives mul ple treatments or is evaluated at mul ple points in me, or both. It is usually easy to tell whether a study is within-S or repeated measures because terms and phrases such as “each par cipant received all

treatments,” repeated measures, longitudinal, prospec ve, or pretest–pos est are included in descrip ons of within-S studies. The examples provided so far are extremely simple. However, group comparison designs can have more than two groups. In addi on, research designs can include both within- and between-S factors (for example, pretest and pos est measures could be added to the study in Figure 2.4). Correla onal or nonexperimental studies (as in Figure 2.3) usually include large numbers of variables. You will learn sta s cal techniques for each of these situa ons one at a me. Later educa on in sta s cs shows ways to combine these simple research designs into more complex designs and analyses. 2.10 CHOICE OF STATISTICAL ANALYSIS (PREVIEW) Chapters 9 through 17 in this book describe sta s cs used to assess whether two variables are related. There are four possible combina ons of types of independent and dependent variables. (To select a sta s cal analysis, it may be necessary to iden fy one of your variables as an independent variable even if you do not have a causal hypothesis.) As a brief preview, here are some (not all) of the commonly used sta s cs for each combina on of variables. X is categorical, Y is categorical: χ2 analysis of con ngency table X is categorical, Y is quan ta ve: t test or analysis of variance (ANOVA) X is quan ta ve, Y is quan ta ve: Pearson r, bivariate regression What do each of these analyses tell us? A χ2 (chi-squared) test evaluates whether membership in one type of group is sta s cally related to membership in another type of group. Consider sex (a group membership variable) and poli cal party (a second group membership variable). A χ2 analysis and examina on of percentages can answer ques ons such as, Are women more likely to be Democrats, and are men more likely to be Republicans? The χ2 test is more o en used in nonexperimental research; however, it can be used in experiments when the outcome variable is categorical. An independent-samples t test or analysis of variance compares mean scores on a dependent variable across two or more groups. O en this is done in an experiment in which a researcher has divided people into groups and then given a different type of treatment to each group. For example, a study might compare mean anxiety scores between Group 1 (which received psychotherapy) and Group 2 (which did not receive psychotherapy) to see if people who received psychotherapy had lower anxiety on average. This analysis can also be used to compare means between naturally occurring groups, such as mean height between male and female groups. A Pearson correla on (denoted r) is used to examine scores for two quan ta ve variables (such as X, height, and Y, salary). Pearson correla on is an appropriate analysis only when there is a linear associa on between X and Y, as discussed in a later chapter. Chapters 9 through 17 do not cover analyses for the situa on in which X is quan ta ve and Y is categorical. (Logis c regression can be used in this situa on.)

2.11 POPULATIONS AND SAMPLES: IDEAL VERSUS ACTUAL SITUATIONS 2.11.1 Ideal Defini on of Popula on and Sample Sta s cal techniques were developed on the basis of ideal, imaginary situa ons. The development of sta s cal techniques began by specifying a popula on of interest. In an industrial quality control study, for example, the popula on could be all the widgets that are produced by a machine in a month. Let’s assume that the variable of interest is the diameter of the widgets. If it is possible and not too expensive to measure the diameter of every single widget in the popula on, it makes sense to do that. However, it is o en too costly or difficult to obtain informa on for every case in a popula on. Sta s cians knew that it would be useful to develop techniques that can use informa on from a sample to make inferences (es mates) about popula on characteris cs. A sample can be defined as a subset of the cases in a popula on, as in the following example. All members of the sample are members of the popula on. However, some members of the popula on are not included in the sample. Popula on (of 7): [72, 81, 98, 67, 101, 78, 79] Sample (subset) of size N = 3: [98, 72, 78] To develop the techniques you will learn, sta s cians made assump ons that simplified the problem. They assumed that all members of the popula on can be iden fied and can poten ally be included in a sample. For the development of some sta s cs, they assumed that scores for the variable are normally distributed in the popula on. They assumed that the sample would be randomly selected from the popula on, in a way that gave every member of the popula on an equal chance of being included in the sample. Here’s an example of a simple random selec on method to obtain a sample that includes 50% of the popula on: Toss a coin for each case and include that case in the sample if the result is heads. 2.11.2 Two Real-World Research Situa ons Similar to the Ideal Popula on and Sample Situa on Industrial quality control involves a situa on similar to the one imagined by sta s cians. Returning to the widget example, the popula on of interest, all widgets produced by a machine in a month, can be iden fied. It is possible to select a sample of widgets randomly from this popula on of interest. A second situa on that is somewhat comparable with the ideal situa on arises in poli cal polling. Polling organiza ons such as Gallup define the popula on of interest in terms such as “all registered U.S. voters.” It is more difficult to iden fy all members of this popula on than in the widget example, and there are cases in this popula on that cannot be contacted and included in a sample. Organiza ons such as Gallup use complex sampling methods that include both random and systema c selec on to obtain samples that should be representa ve of the popula on. A representa ve sample of a popula on is created if the cases in the sample have characteris cs similar to those of the popula on. For example, if 51% of a Gallup sample are women, 10% are Hispanic, and 20% are older than 65, and the popula on of all registered voters includes 51% women, 10% Hispanic voters, and 20% voters older than 65, then the sample is representa ve of that popula on for those three variables. On the other hand, if the

sample has 23% women, but the popula on has 51% women, then the sample is not representa ve of the popula on in sex distribu on. This book does not deal with complex sampling issues and technical tools such as case weigh ng; for a comprehensive discussion, see Thompson (2012). 2.11.3 Actual Research Situa ons That Are Not Similar to Ideal Situa ons In many behavioral and social science studies and in medicine, researchers o en begin not with a well-specified popula on but with a convenience sample (some mes called an accidental sample). Convenience samples consist of cases that are easy for the researcher to get. However, researchers almost always want to say something about cases not included in the study. Most textbooks don’t address this ques on: What popula on can researchers talk about in this situa on? Trochim (2006) suggests that researchers rely on a proximal similarity model to generalize from convenience samples. The proximal similarity model says that it is reasonable to generalize results from a sample to some broader hypothe cal or imaginary popula on if the members of the sample have par cipant characteris cs like those of the popula on of interest (i.e., if the sample is representa ve of the popula on of interest). For example, a psychologist might run a study with a small sample of moderately depressed pa ents to evaluate whether cogni ve behavioral therapy (CBT) (X) improves life sa sfac on (Y). The psychologist can see whether pa ents in the study who received CBT have higher life sa sfac on scores than pa ents in the study who did not. However, the psychologist does not want to be limited to saying, “CBT increased life sa sfac on for the 30 pa ents in my study.” The psychologist hopes to be able to say something like “CBT probably increases life sa sfac on for many other depressed pa ents” (i.e., a broader hypothe cal popula on of other depression pa ents). How far can researchers go when making such generaliza ons? They should limit themselves to generaliza ons about popula ons similar to members of the study. If a CBT study finds that CBT increases life sa sfac on for women ages 20 to 50 with moderate levels of depression, the researcher should not assume that CBT would have similar effects for men, older and younger persons, and persons with mild or severe depression. When a sample is selected randomly from an actual well-specified popula on, cases in the sample should be like the popula on. In this situa on we can jus fy generaliza ons beyond the sample to the popula on from which that sample was selected on the basis of the sampling model (Trochim, 2006). In behavioral, social, and medical laboratory research situa ons, it is common for researchers to generalize from convenience samples to broader hypothe cal popula ons (relying implicitly on the proximal similarity model). Research situa ons such as industrial quality control and poli cal polling, where samples are obtained by random sampling from a popula on, can jus fy generaliza ons on the basis of the sampling model. In either case, generaliza ons from sample to popula on should be made cau ously. Even random selec on of cases from a clearly defined popula on can some mes yield a nonrepresenta ve sample.

2.12 COMMON PROBLEMS IN INTERPRETATION OF RESULTS Authors of research reports some mes interpret findings incorrectly or use language that is misleading or inconsistent. Here are three major types of errors in interpreta on: Describing an associa on between variables as causal when the researcher does not have the evidence needed to rule out rival explanatory variables. Overgeneralizing, that is, claiming that results should be true for popula ons and situa ons not similar to those included in the study. Misunderstanding or minimizing the limita ons of research design and sta s cal analysis. I urge you to avoid oversta ng claims. Avoid language that suggests high levels of certainty about causality (such as “X causes Y”). Avoid misleading statements about generalizability. If the sample in a pain drug study includes only healthy adult men ages 21 to 30, we do not have enough informa on to make inferences about the effec veness or safety of the drug for women, children, frail elders, and other kinds of people not included in the sample. Behavioral and social science studies have tended to overrepresent White college students and underrepresent people of color, people of other ages, and people who do not a end college (Guthrie, 2004; Henry, 2008; Sears, 1986). Animal studies tend to focus on species that are small, inexpensive, and easy to handle. The narrow range of par cipant characteris cs limits the poten al generalizability of results. Researchers should be careful not to generalize from ar ficial laboratory situa ons to real-life situa ons different from those in the laboratory. For example, it would be misleading to argue that the effects of consump on of an ar ficial sweetener by people in small doses would be iden cal to the effects of consump on of an ar ficial sweetener in very large doses by rats isolated in laboratory cages. When mass media talk about research results that they believe will interest the public, they some mes make extremely inflated claims about the strength of evidence. As you con nue to study sta s cs, you will learn procedures such as sta s cal significance tests (reports of sta s cal significance o en include statements such as “p < .05”). Misunderstandings of significance test results are common; researchers and readers some mes think that p < .05 provides us with a much greater degree of certainty about results of a study than it really does. Misunderstandings about the limited nature of informa on we obtain from p values are another common source of error in interpreta on and repor ng of research results. In all three areas (causal inference, generaliza ons from study results to other popula ons, and interpreta ons of significance tests), authors and readers need to beware of false confidence. As noted in Chapter 1, a single study is never sufficient evidence to draw confident conclusions. Every individual study has limita ons in what it can include, and some studies have flaws that compromise the kinds of conclusions that can legi mately be drawn. APPENDIX 2A: MORE ABOUT LEVELS OF MEASUREMENT

Some sta s cs textbooks (par cularly those for psychologists) discuss the four classic levels of measurement defined by Stevens (1946, 1951): nominal, ordinal, interval, and ra o (Table 2.1). At the nominal (also called qualita ve or categorical) level of measurement, each number code serves only as a label for group membership. For example, the nominal variable sex might be coded 1 = male, 2 = female; the nominal variable religion could be coded 1 = Buddhist, 2 = Catholic, 3 = Hindu, 4 = Islamic, 5 = Jewish, 6 = Protestant, 7 = other. The values of the numbers associated with groups do not imply any rank ordering among groups. Because these numbers serve only as labels, Stevens argued that the only opera ons that could appropriately be applied to the scores are = and ≠. That is, persons with scores of 2 and 3 on religion could be labeled as “the same” or “not the same” on religion. It would be nonsense to add up the religion scores in a sample and divide by the number of cases in the sample to obtain an “average” religion. We can count the number of persons who iden fy themselves as members of each group and obtain percentages. Table 2.1 Stevens’s Levels of Measurement

At the ordinal (also called rank) level of measurement, numbers represent ranks, but the differences between scores do not necessarily correspond to equal intervals with respect to any underlying characteris c. The runners in a race can be ranked in terms of speed (runners are tagged 1, 2, and 3 as they cross the finish line, with 1 represen ng the fastest me). These scores supply informa on about rank (1 is faster than 2), but the numbers do not necessarily represent equal intervals. The difference in speed between Runners 1 and 2 (i.e., 2 – 1) might be much larger or smaller than the difference in speed between Runners 2 and 3 (i.e., 3 – 2), despite the difference in scores in both cases being one unit. For ordinal scores, the opera ons > and < would be meaningful (in addi on to = and ≠). However, according to Stevens, addi on or subtrac on would not produce meaningful results with ordinal measures (because a one-unit difference does not correspond to the same “amount of speed” for all pairs of scores). Scores that have interval level of measurement quali es supply ordinal informa on and, in addi on, represent equally spaced intervals. That is, no ma er which pair of scores is considered (such as 3 – 2 or 7 – 6), a one-unit difference in scores should correspond to the same amount of the thing that is being measured. The interval level of measurement does not necessarily have a true zero point. The Fahrenheit temperature scale is a good example of the interval level of measurement: The 10-point difference between 40°F and 50°F is equivalent to the 10-point difference between 50°F and 60°F (in each case, 10 represents the same number of degrees of change in temperature). However, because 0°F does not correspond to a complete absence of any heat, it does not make sense to look at a ra o of two temperatures. For

example, it would be incorrect to say that 40°F is “twice as hot” as 20°F. On the basis of this reasoning, it makes sense to apply the plus and minus opera ons to interval scores (as well as the equality and inequality operators). However, on the basis of this reasoning, it would not make sense to mul ply and divide numbers that do not have a true zero point. Ra o-level measurements are interval-level scores that also have a true zero point. A clear example of a ra o-level measurement is height. It is meaningful to say that a person who is 6 feet tall is twice as tall as someone who is 3 feet tall because there is a true zero point for height measurements. The strictest interpreta on of this reasoning would suggest that ra o level is the only type of measurement for which mul plica on and division would yield meaningful results. APPENDIX 2B: JUSTIFICATION FOR THE USE OF LIKERT AND OTHER RATING SCALES AS QUANTITATIVE VARIABLES (IN SOME SITUATIONS) When people report degree of agreement on a five-point Likert ra ng scale, does the one-point difference between 5 = strongly agree and 4 = agree correspond to the same change in amount of agreement as the difference between 4 = agree and 3 = neutral? Probably not. These scores probably fall into a gray area: They provide informa on that may be a li le be er than ordinal but falls short of providing true equal interval informa on. People in some disciplines (par cularly psychology) some mes argue that widely used analyses such as correla ons and t tests should not be applied to ra ng scale data. Vogt (1999) noted considerable controversy about this. He stated that “as with cons tu onal law, there are in sta s cs strict and loose construc onists in the interpreta on of adherence to assump ons” (p. 158). Similarly, Howell (1992) concluded that the underlying level of measurement is not crucial in the choice of a sta s c: The validity of statements about the objects or events that we think we are measuring hinges primarily on our knowledge of those objects or events, not on the measurement scale. We do our best to ensure that our measures relate as closely as possible to what we want to measure, but our results are ul mately only the numbers we obtain and our faith in the rela onship between those numbers and the underlying objects or events … the underlying measurement scale is not crucial in our choice of sta s cal techniques … a certain amount of common sense is required in interpre ng the results of these sta s cal manipula ons. (pp. 8–9) Harris (2001) wrote, I do not accept Stevens’s posi on on the rela onship between strength [level] of measurement and “permissible” sta s cal procedures…the most fundamental reason for [my] willingness to apply mul variate sta s cal techniques to such data, despite the warnings of Stevens and his associates, is the fact that the validity of sta s cal conclusions depends only on whether the numbers to which they are applied meet the distribu onal assump ons … used to derive them, and not on the scaling procedures used to obtain the numbers. (pp. 444–445)

Gaito (1980) reviewed these issues and concluded that “scale proper es do not enter into any of the mathema cal requirements” for various sta s cal procedures, such as ANOVA. When scores are obtained by summing responses across many ques ons, these summary scores are o en nearly normally distributed; Carifio and Perla (2008) reviewed evidence that the applica on of parametric sta s cs to these scale scores produces meaningful results. Zumbo and Zimmerman (1993) used computer simula ons to demonstrate that varying the level of measurement for an underlying empirical structure (between ordinal and interval) did not lead to problems when several widely used sta s cs were applied. Tabachnick and Fidell (2018) also addressed this issue: “The property of variables that is crucial to applica on of mul variate procedures is not type of measurement so much as the shape of the distribu on” (p. 6). They concluded that it is more important to consider distribu on shapes for scores on quan ta ve variables (rather than their levels of measurement). These arguments suggest that it is reasonable to apply the parametric sta s cs covered in this textbook (such as the sample mean, Pearson’s r, t tests, and ANOVA) to quan ta ve scores even if they do not sa sfy the strict requirements for the interval level of measurement. Some teachers and journal reviewers con nue to prefer the more conserva ve sta s cal prac ces advocated by Stevens; they may advise you to avoid computa on of means, variances, and Pearson correla ons for data that aren’t clearly at the interval or ra o level of measurement. However, this statement should be qualified. In Chapter 5 you will see histograms in which the distribu ons of data are bimodal (with modes at the lowest and highest scores). In such situa ons, the sample mean is not a good way to describe the data. In this situa on, it would be preferable to treat ra ng scores as categorical (e.g., Group 1 = response of strongly disagree, Group 2 = response of disagree, etc.). COMPREHENSION QUESTIONS How do the meanings of score values differ for categorical and quan ta ve variables? Is each of the following variables categorical or quan ta ve? Type of pet owned: 1 = none, 2 = dog, 3 = cat, 4 = other animal IQ score Personality type (Type A, heart a ack prone; Type B, not heart a ack prone) Body weight Salary Geographic region (1 = northeastern United States, 2 = south central United States, etc.) If you have scores on a categorical variable (e.g., religion coded 1 = Catholic, 2 = Buddhist, 3 = Protestant, 4 = Islamic, 5 = Jewish, and 6 = other religion), would it make sense to use these numbers to compute a mean? Give reasons for your answer. In each of these hypothe cal research ques ons, iden fy the independent and dependent variables. (For some of these ques ons, the answer is that there is no dis nc on between the independent and dependent variables.) Is height predic ve of salary?

Is blood pressure influenced by stress? (Note that this could be worded “Does stress influence blood pressure?”) Is depression associated with insomnia? What types of control in experiments help us meet the condi ons for causal inference? How do quasi-experiments differ from true experiments? What two common designs are used in quasi-experimental studies? Explain the difference between these types of samples: Random samples (selected randomly from a clearly defined popula on) Accidental or convenience samples Which type of sample (a or b) is more commonly used in laboratory studies in fields such as psychology? Which type of sample (a or b) is more likely to be representa ve of a clearly defined popula on? When a researcher has an accidental or a convenience sample, what kind of popula on can he or she try to make inferences about? What does it mean to say that a sample is representa ve of a popula on? How can you evaluate whether a sample is representa ve of a popula on? What are the sampling model and the proximal similarity model? How do they differ, and in what kinds of situa ons are they used? For each of the following hypothe cal situa ons: Is this best described as an experiment, a nonexperimental study, or a quasi-experiment? Do you think this study has high internal validity? Why or why not? A researcher looks at two groups of nursing home residents, those who live at Garden Meadows and those who live at Dreary Acres. People at Garden Meadows receive weekly visits from volunteers who do things to cheer them up; people at Dreary Acres do not. Measures of depression are taken at the end of the study. A researcher randomly assigns college students to Group 1, which is required to par cipate in a 1-hour workshop designed to make them have more favorable a tudes toward medita on, versus Group 2, which does not go to a workshop. Other variables are controlled through standardiza on of procedures. A tudes toward medita on are measured. A researcher goes through a parking lot and records the following informa on: the make and model of each car (later, the researcher looks up car prices) and whether the car does or does not have bumper s ckers. If bumper s ckers are present, do they suggest poli cally liberal or conserva ve views or neither? For each of these studies, do you think it is high or low on external validity? Why or why not? A researcher demonstrates that it is possible to increase the frequency of quacking by giving human par cipants $20 each me they quack. (They are coached ahead of me to make a variety of animal sounds.)

A researcher conducts observa ons in restaurants and codes whether the server smiles or not and then looks at the p le on the table. A researcher wants to know whether eye color (blue vs. brown) is related to a quan ta ve measure of introversion (on the basis of a self-report measure). Does this sound like an experimental or a nonexperimental study? What sta s cal analysis do you think would be used? A researcher wants to know whether the frequency of marijuana use (a self-reported quan ta ve measure) is linearly related to life sa sfac on (a self-reported quan ta ve measure). Does this sound like an experimental or a nonexperimental study? What sta s cal analysis do you think would be used? A researcher measures vocabulary test scores for one group of students as they grow up, with measurements at ages 3, 5, 7, 9, and 11. Do you think this is a between-S or a within-S (repeated-measures) design? DIGITAL RESOURCES Find free study tools to support your learning, including eFlashcards, data sets, and web resources, on the accompanying website at edge.sagepub.com/warner3e. Descrip ons of Images and Figures Back to Figure A flowchart shows an outline of how a simple experimental design scheme func ons. A Group is formed by a researcher, perhaps by random assignment of people to groups. Two types of treatment are administered by the researcher in similar situa ons and the mean HR of both groups is compared to determine which group has the higher HR. Group 1 is no caffeine or control condi on. Here the HR equals Y1. Group 2 is caffeine. Here the HR equals Y2. Note: HR refers to heart rate. Ques ons at the bo om of the figure read as follows: How was total sample obtained? Types of people ingroups should be similar (equivalent) Has researcher done anything else different to these two groups that would create a rival explanatory variable or confound? Measure the dependent variable a er treatment Sta s cal analysis Back to Figure In the flowchart: Group 1 with high academic ability gets classroom instruc on run by Doctor Feelgood. Result is the Group 1 exams scores. Group 2 with low academic ability gets online course run by Doctor Deadly. Result is the Group 2 exams scores.

The ques ons posed by the researcher are at the right of the flow chart and read as follows: Researcher wants to know: Is a difference in exam scores due to classroom vs. online se ng? Difference could be due to academic ability difference between groups before the study. A confound of par cipant characteris cs with type of treatment. Difference could be because Dr. Feelgood is be er liked or more capable than Dr. Deadly. Confound of another aspect of procedure with type of instruc on. Cannot be certain that final exam score differences are due to type of instruc on. A note at the bo om reads: Here are two confounds with X: par cipant characteris cs and teacher iden ty. Back to Figure In the flow diagram X variable measured and Y variable measured flow into the analysis. The analysis is to see whether X and Y are sta s cally related. The type of analysis depends on whether X is categorical (naturally occurring groups) or quan ta ve. The researcher may have some control over a situa on in which X and Y are measured, but has no control over any group memberships or poten al confounded variables. Back to Figure In the flow chart there are 2 groups; Group 1 is not a part of the program and the condi ons are not controlled by the researcher. The result is Y1. Group 2 is a part of the program, but is not controlled by the researcher. The result is Y2. A note on the right reads: Analysis to compare group means for Y. At the bo om of the figure, there are text boxes that read: The groups are not formed by researcher and probably are not equivalent. The program has not been developed by researcher, and the researcher cannot control poten al confounds from other events.