11

Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=upcp20

Political Communication

ISSN: 1058-4609 (Print) 1091-7675 (Online) Journal homepage: https://www.tandfonline.com/loi/upcp20

Fact-Checking: A Meta-Analysis of What Works and for Whom

Nathan Walter, Jonathan Cohen, R. Lance Holbert & Yasmin Morag

To cite this article: Nathan Walter, Jonathan Cohen, R. Lance Holbert & Yasmin Morag (2020) Fact-Checking: A Meta-Analysis of What Works and for Whom, Political Communication, 37:3, 350-375, DOI: 10.1080/10584609.2019.1668894

To link to this article: https://doi.org/10.1080/10584609.2019.1668894

Published online: 24 Oct 2019.

Submit your article to this journal

Article views: 8010

View related articles

View Crossmark data

Citing articles: 98 View citing articles

Fact-Checking: A Meta-Analysis of What Works and for Whom

NATHAN WALTER, JONATHAN COHEN, R. LANCE HOLBERT, and YASMIN MORAG

Despite its growing prominence in news coverage and public discourse, there is still considerable ambiguity regarding when and how fact-checking affects beliefs. Informed by theories of motivated reasoning and message design, a meta-analytic review was undertaken to examine the effectiveness of fact-checking in correcting political misinformation (k = 30,N = 20,963). Fact-checking has a significantly positive overall influence on political beliefs (d = 0.29), but the effects gradually weaken when using “truth scales,” refuting only parts of a claim, and fact-checking campaign-related statements. Likewise, the ability to correct political misinformation with fact-checking is substantially attenuated by participants’ preexisting beliefs, ideology, and knowledge. The study concludes with a discussion of the fact-checking literature in light of current gaps and future opportunities.

Keywords meta-analysis, fact-checking, misinformation, correction, motivated reasoning

Political misperceptions are prevalent across contexts and nation-states and have proven themselves to be shockingly resilient to correction (Bode & Vraga, 2015; Garrett, Nisbet, & Lynch, 2013). There are clear normative concerns associated with persistent beliefs grounded in factual inaccuracies (see Althaus, 2012), and perceptions of this kind which appear to strengthen after exposure to correction are especially worrisome (Thorson, 2016). Misinformation, and attempts to correct it, are as old as democracy itself (Grant, 1995), and a wide range of journalistic (e.g. Associated Press) and non-journalistic (e.g., FactCheck.org) organizations currently engage a diverse set of practices and procedures

Nathan Walter (PhD, University of Southern California, 2018) is an assistant professor at the Department of Communication Studies, Northwestern University. His research focuses on cogni- tive, metacognitive, and emotional processes at the heart of decision-making and persuasion. Jonathan Cohen (PhD, University of Southern California, 1995) is a professor of communication at the University of Haifa in Israel. His recent research investigates processes of narrative influences as well as media effects more generally. R. Lance Holbert (PhD, University of Wisconsin-Madison, 2000) is a faculty member within the Department of Communication and Social Influence, Klein College of Media and Communication, Temple University. He studies persuasion-based processes of influence within the context of media and politics. He currently serves as Editor-in-Chief of Journal of Communication. Yasmin Morag (BA University of Haifa, 2017) is a research assistant in the Department of Communication at the University of Haifa, where she is currently completing her Master’s degree. Her thesis explores the ambivalence and resistance in women's experience with sexual texts.

Address correspondence to Nathan Walter, Department of Communication Studies, Northwestern University, 710 N. Lake Shore Drive, Chicago, IL 60611. E-mail: [email protected]

Political Communication, 37:350–375, 2020 Copyright © 2019 Taylor & Francis Group, LLC ISSN: 1058-4609 print / 1091-7675 online DOI: https://doi.org/10.1080/10584609.2019.1668894

350

to correct misinformation that is perceived to soil the marketplace of ideas (see Young, Jamieson, Poulsen, & Goldring, 2018).

One of the most popular innovations designed to address the prevalence of misinforma- tion has been fact-checking. Fact checking is the practice of systematically publishing assessments of the validity of claims made by public officials and institutions with an explicit attempt to identify whether a claim is factual. News consumers are encountering this practice often across media with an increase of more than 900% in its use since 2001 for newspapers and more than 2,000% in broadcast media (Amazeen, 2013). Often referred to as the successor of ad watches (e.g., Graves, Nyhan, & Reifler, 2016), fact-checkers evaluate the accuracy of claims made by political entities. Among the most recognizable fact-checking organizations are FactCheck.org and PolitiFact. A strong empirical indicator of the perva- siveness of fact-checking by news organizations is that the practice has been embraced by standard-bearers like The New York Times and organizations dedicated to fact-checking have won major journalism awards, including a Pulitzer Prize (Graves et al., 2016). This profes- sional routinization has, to some extent, made fact-checking a necessary component of high- quality, comprehensive political coverage (Schumacher-Matos, 2012).

The growing prominence of this element of reporting, however, is not an indication of its ability to correct misinformation. Given the rise of fact-checkers and their success- ful integration into traditional reporting, one might expect to find mounting evidence pointing to a reduction in the effects of misinformation. However, the empirical evidence regarding the effects of fact-checking is highly divided, with some studies finding that exposure to fact-checking can reduce misinformation (e.g., Fridkin, Kenney, & Wintersieck, 2015), but other works recording null findings (e.g., Garrett & Weeks, 2013), or even boomerang effects (e.g., Nyhan & Reifler, 2010). To address this gap, the present study (1) reports findings from a meta-analysis that examines the overall effectiveness of fact-checking; (2) determines the consistency of the field’s empirical findings; (3) tests the role of several theory-driven moderators that are commonly used to explain the contingencies of fact-checking; and (4) identifies areas where additional research is needed.

On Strengths and Limitations of Fact-Checking

The diversity of political topics about which misinformation is published is matched only by the myriad of communication strategies employed in seeking to alter political mis- perceptions. From simple retractions (e.g., Ecker, Lewandowsky, Swire, & Chang, 2011) and ad watches (e.g., Pfau & Louden, 1994) to full-blown correction campaigns (e.g., Dyer & Kuehl, 1978), different techniques are used to monitor the veracity and correct- ness of political discourse. The newest members in the family of potential antidotes to misinformation are fact-checking organizations best described as “nonpartisan, nonprofit consumer advocate[s] for voters that aim[s] to reduce the level of deception and confu- sion in U.S. politics” (FactCheck.org, 2018). Beyond their nonpartisan nature, transpar- ency, and reliance on independent research, these enterprises stand apart from other organizations due to their exclusive focus on factual statements made by major political players, as part of debates, speeches, interviews, and news releases, limiting the topics to claims that can be definitively proven or disproven (Amazeen, 2016). It is not surprising, therefore, that different organizations such as PolitiFact, FactCheck.org, and The Fact Checker generally agree with each other and reach very similar verdicts (Amazeen, 2016; but see Marietta, Barker, & Bowser, 2015).

Fact-Checking 351

Though fact-checkers have undoubtedly acquired professional prestige and a large audience (Graves et al., 2016), it is unclear whether they shift people’s beliefs and correct misinformation. After all, fact-checkers are outspoken about their primary objective being to inform the public (Amazeen, 2013). To the extent that citizens should desire to have correct information on which to base their beliefs and political attitudes, being told that a claim is incorrect should make them believe it less. This is based on the simple notion that beliefs and attitudes are based on information received from one’s environment and that information should affect attitudes (see Wolfsfeld, 2011). The credibility and non- partisan nature of fact-checkers should make their judgment more influential and their pronouncements more effective, all the more so in cases where much conflicting infor- mation exists, making it difficult to know who to believe. Despite this expectation of a strong effect of fact-checkers, Zaller (1992) argues that not all information is used to shape beliefs and that people combine portions of available information with preexisting opinions when forming their political beliefs. Indeed, it would be somewhat naïve to suggest that the mere exposure to accurate information can shape beliefs in highly polarized political environments, where the notion of objectivity and impartiality is constantly called into question. But it is also reasonable to expect that accurate informa- tion should help shape more accurate beliefs.

When attempting to estimate the efficacy of fact-checking with individual studies, a host of complexities arise. Namely, a close inspection of individual studies reveals contradictory findings, with some researchers summarizing their results as “the failure of fact-checking” (Jarman, 2014, p. 1), whereas others show that “fact-checks influence people’s assessments of the accuracy, usefulness, and tone of negative political ads” (Fridkin et al., 2015). This ambiguity is further confounded by the idiosyncratic features of studies that tend to differ along critical dimensions such as individual-differences (e.g., political interest, political ideology) and contextual factors (e.g., political issue, proximity to Election Day). A preferred approach to disentangle such inconsistencies and synthesize research findings is meta-analysis (O’Keefe, 2015). Thus, the first aim of the present study is to synthesize research findings and identify the average effect of fact-checking:

RQ1: What is the average effect of fact-checking on message-consistent beliefs?

With that in mind, a meta-analysis is not necessary to understand that there is no simple main effect of fact-checking. Therefore, the current study also attempts to under- stand when and how fact-checking efforts succeed or fail. As has been argued of late in the political communication literature, there is a need to remain steadfast in the study of conditional media effects (see Bennett & Pfestch, 2018; Neuman, 2016). Hence, the following sections introduce theory-driven moderators that attempt to identify the condi- tions that either augment or attenuate fact-checking efforts.

Motivated Reasoning

Though traditional theories in education and persuasion identified source characteristics such as credibility (Hovland & Weiss, 1951), expertise (Holvand, Janis, & Kelley, 1953), compe- tence and objectivity (Whitehead, 1968) as prime determinants of message effectiveness, more recent findings have consistently shown that people do not approach messages even- handedly and that preexisting beliefs play a major role in determining the way information is

352 Nathan Walter et al.

processed even in the face of concrete evidence and mounting facts (Flynn, Nyhan, & Reifler, 2017). Motivated reasoning is a dominant theory used to explain the formation and resiliency of political misperceptions (Druckman, 2012; Kahne & Bowyer, 2017). This perspective argues that citizens are more accepting of (mis)information that match their pre-existing worldview. Furthermore, the ease with which misinformation can be placed within a pre- existing worldview makes it all the more difficult to correct (Lewandowsky, Ecker, Seifert, Schwarz, & Cook, 2012). Often traced back to Maslow’s hierarchy of needs, the theory argues that people are guided by different goals when they process information, including directional goals (i.e., confirming a preexisting belief) and accuracy goals (i.e., trying to be correct and precise; Kunda, 1990; Nir, 2011).

Unsurprisingly, in the political context, directional goals appear to be a very common way to process information (Taber, Cann, & Kucsova, 2009). For instance, voters are likely to feel dissonance if they supported a culturally-conservative candidate and then later read a fact-checking report accusing the candidate of infidelity. Given the psycho- logical discomfort that will arise from trusting the revealing news report, individuals will be motivated to reduce their ambivalence by carefully scrutinizing the new information in an attempt to discredit the report in a manner that will allow for continued support for the candidate. In such situations, people are likely to point to a political bias of the fact- checking organization, the reliance of the report on anonymous sources, or a secret agenda to damage the popularity of the candidate.

Notably, though the frameworks of motivated reasoning and cognitive dissonance have gained prominence in the fact-checking literature, it is important to note that other theories can be used to explain people’s tendency to stick to their preexisting beliefs even in light of reliable corrections. For instance, the mental models approach argues that individuals construct coherent representations of unfolding events (Johnson & Seifert, 1994). If a fact-checking message dismisses key information in the established mental model, individuals will be reluctant to accept the new information because it will result in an incomplete or incoherent representation (Lewandowsky et al., 2012). In line with this logic, individuals with a mental model that identifies the free market system as a way to reach the highest standard of living will be reluctant to accept a fact-checking message that speaks to the benefits of regulating the U.S. healthcare system not merely because it contradicts their political ideology (i.e., motivated reasoning) but also due to its ability to create gaps in an otherwise coherent mental model. In sum, the literature proposes several cognitive mechanisms to explain how individuals can prioritize favorable or coherent information even if it is inaccurate or unreliable. Based on this, it was hypothesized that:

H1: The effect of fact-checking on beliefs is stronger for pro-attitudinal corrections compared to counter-attitudinal corrections.

Studies have demonstrated that motivated reasoning affects some news consumers more than others. For example, politically-sophisticated individuals – who also tend to be more partisan – are likely to scrutinize (i.e., counter-argue) fact-checking more than their less knowledgeable counterparts (Young et al., 2018) and thus be less likely to accept corrections of misinformation. For instance, while studies found that cognitive reflection (i.e., the propensity to engage in analytical reasoning) is negatively correlated with adoption of fake news and positively correlated with the ability to discern fake news from real news (Pennycook & Rand, 2017), Nyhan, Reifler, and Ubel (2013) demonstrated that attempts to correct misinformation backfired among knowledgeable conservatives. Similar evidence of

Fact-Checking 353

motivated reasoning among political sophisticates demonstrated that climate change knowl- edge among conservatives decreased with greater attention to political news (Nisbet, Cooper, & Ellithorpe, 2015). It is, therefore, hypothesized that:

H2: The more sophisticated the individual, the less their beliefs will be affected by exposure to fact-checking.

Further, though motivated reasoning is evident among both Republicans and Democrats, there is some evidence to suggest that the former partisan group might be especially susceptible to its bias in relation to current fact-checking practices. First, Republicans consistently report more negative perceptions of mainstream media than Democrats (Swift, 2016). Second, Democrats fare much better than Republicans in recent fact-checking, with statements made by Republicans more likely to be classified as false (Amazeen, 2013; Wintersieck, 2015). Third, popular conservative media outlets such as Fox News and Breitbart have been outspoken in their disdain of fact-checking organiza- tions, referring to them as “leftists” and “partisan” (Bokhari, 2018). Fourth, studies have demonstrated that conservatives are more prone to closed-mindedness (Levitan & Visser, 2008) and less likely to seek-out attitude-dissonant information (Barberá, Jost, Nagler, Tucker, & Bonneau, 2015). Finally, while Democratic presidential candidates have embraced fact-checking organizations, their Republican counterparts - in the 2012 and the 2016 presidential elections – vocally condemned fact-checkers as biased (Shin & Thorson, 2017). Based on these collective insights, the following is posited:

H3a: The effect of fact-checking on beliefs is stronger for Democrats/liberals compared to Republicans/conservatives.

H3b: There is an interaction effect such that Republicans/conservatives are more likely to accept pro-attitudinal fact-checking and less likely to accept counter-attitudinal fact- checking, compared with Democrats/liberals.

The level of motivated reasoning may also vary by the context in which fact- checking takes place. It is not surprising that people are more likely to be in-tune with political matters and more likely to be exposed to misinformation during an election season (Lewandowsky et al., 2012). Hence, the confluence of opportunities for, and challenges to, fact-checking is at its peak during election events (Coddington, Molyneux, & Lawrence, 2014). Though there is an obvious opportunity to educate a larger and more attentive audience during election season, motivated reasoning is especially prevalent during these times. As argued by Miller and Conover (2015), “average partisans in contemporary U.S. politics view elections as group competitions in which partisan identities are at stake” (2015, p. 225). Hence, as elections near, the processing of political information tends to rely less on facts and more on political calculations and the interest of the political party (Achen & Bartels, 2017; Kunda, 1990). In line with these findings, it is hypothesized that:

H4: The effect of fact-checking on beliefs is weaker when it concerns election candi- dates/campaign statements than routine fact-checking.

354 Nathan Walter et al.

(In)effective Presentation

Though motivated reasoning is the dominant theoretical approach to explaining the effectiveness of fact-checking, another body of knowledge links the success and failure to correct misinformation to a range of message variables. Accordingly, fact-checking may fail not merely because it deals with value-laden beliefs but simply because it is presented ineffectively. One of the prominent features of fact-checking that differentiates it from other approaches to correction of misinformation is its deployment of graphical meters or “truth scales.” From Politifact’s Truth-O-Meter to The Washington Post’s counting illustrations of pinocchios, fact-checkers proved to be remarkably innovative in supplementing their text with visual representations. Given the complex nature of political information, visual rating scales that provide a simplified summary of the fact- checking message (e.g., mostly true, completely false) may facilitate the correction of misinformation, especially among less engaged individuals (Amazeen, Thorson, Muddiman, & Graves, 2018). Put differently, presenting corrective information in gra- phical forms can serve as a heuristic cue and increase its acceptance compared with messages that rely solely on text (Nyhan & Reifler, 2012). Hence, we pose the following hypothesis:

H5: The effect of fact-checking on beliefs is stronger when it integrates a “truth scale”.

Relatedly, message length can serve as an additional presentation factor that poten- tially influences fact-checking. In line with dual-processing models (e.g., Petty & Cacioppo, 1986), message length is considered a heuristic cue that enables a quick judgment regarding the veracity of the argument - longer messages are often believed to be more persuading because they, ostensibly, contain more arguments. As indicated in a meta-analysis of involvement and persuasion (Johnson & Eagly, 1989), the length of the message increases persuasion for less-involved individuals. Similarly, Shen, Sheer, and Li (2015) found that message length is significantly correlated with narrative persuasion, such that, on average, longer narratives tend to be more persuasive. By the same token, longer fact-checking messaging may also enjoy this heuristic benefit. Therefore, the following hypothesis is proposed:

H6: The longer the fact-checking message, the more effective it is in influencing message-related beliefs.

Another message characteristic that can play a substantial role as a heuristic or a systematic cue is lexical complexity (Chaiken & Eagly, 1976). As with other message features, fact-checking organizations are expected to navigate a tricky terrain, discussing complex and often esoteric issues while simultaneously attempting to effectively translate the information to the general public. Thus, it is not surprising that FactCheck.org is often criticized for using convoluted language and lacking simplicity (Media Bias/Fact Check, 2016). Complex language can be perceived as elitist potentially alienating audiences. Yet, though simple language can make fact-checking more accessible, it can also compromise its perceived accuracy and objectivity for politically-sophisticated audiences (Shulman & Sweitzer, 2018). Therefore, the following hypotheses are proposed:

H7a: The more complex the fact-checking message, the less likely it is to be effective in influencing message-related beliefs.

Fact-Checking 355

H7b: There is an interaction effect such that, lexical complexity carries less influence for politically-sophisticated individuals.

Finally, a more recent line of research suggests that fact-checking messages can backfire due to an “implied truth effect” (Pennycook & Rand, 2017). The logic that underlies this concern is simple: (1) not all political statements are factually verifiable, as a large part of political talk focuses on opinions (Amazeen, 2016); (2) in general, people are bad at distinguishing between statements that can be factually verified and statements that represent personal views (Pew Research Center, 2018); (3) even if a fact-checker’s verdict is that a particular statement was false, readers may infer that if the remaining information was not explicitly tagged as false, it must be true. According to this naïve theory, the implication of the absence of a disconfirmation is ambiguous: does it imply that the information is verified (leading to the unintended “implied truth effect”) or does it mean that the information was not fact-checked (Pennycook & Rand, 2017). Simply put, in an environment where some statements are flagged as inaccurate and others are not, individuals may assume that unflagged messages are trustworthy (Pennycook & Rand, 2017). To this end, compared to messages that receive an ambiguous judgment (e.g., mostly false), messages that are flagged as completely false are more likely to be corrected, since equivocal verdicts may encourage individuals to wrongfully trust por- tions of the message that were not explicitly flagged as false. Thus, the final hypothesis is posed:

H8: The effect of fact-checking on beliefs is stronger for fact-checking that refutes the entire message rather than fact-checking that refutes only portions of a message.

Method

Selection Criteria

Literature Search. Five different strategies were used to generate effect sizes. First, electronic databases (e.g., Google Scholar, All Academic, JSTOR, Medline, ProQuest, PubMed, Communication and Mass Media Complete, Educational Resources Information Center, PsycINFO) were searched using relevant key terms (and their derivations), including “fact-check,” “misinformation,” “disinformation,” “correction,” and “continued influence.” Those key terms were often coupled with research outcomes such as “beliefs,” “attitudes,” “perceptions,” “intent,” and “behavior.” This initial search yielded potentially relevant journal articles, books, dissertations, and conference papers that were screened for retrieval on the basis of their title or abstract. Second, reference lists were examined and additional relevant studies (albeit, with many duplicates) were retrieved. Third, recent programs of leading conferences in communication and political science (i.e., APSA, ICA, NCA) were systematically searched using the same key terms. Fourth, we shared a brief explanation of this project along with its inclusion criteria on two large list servers to solicit further relevant studies. Finally, based on personal acquaintance and the list of authors from our dataset, 17 leading scholars in the field of misinformation, journalism, and political communication were contacted and asked to review our corpus (both included and excluded studies) and identify omissions. In total, 11 individuals replied, providing us with four additional studies, which were later added do the dataset.

356 Nathan Walter et al.

Inclusion Criteria. Studies were retained based on five criteria: (1) research reports had to include an experimental comparison between exposure to fact-checking and a completely equivalent condition that was not exposed to fact-checking (e.g., Amazeen et al., 2018) or a pretest-posttest experiment where outcomes were assessed before and after exposure to fact-checking (e.g., Jarman, 2016); thus, isolating the effect of exposure to a fact-checker;1 (2) fact-checking message had to discredit an original statement made by a political entity;2 (3) study material had to expressly name either a real or a fictitious fact-checking organization; (4) research reports had to provide a quantitative estimate for the effect of fact-checking on message-relevant beliefs/atti- tudes; (5) studies had to report on relevant statistics (e.g., means, SDs, t-values, exact-p, Cohen’s d, odds ratio, frequencies, zero-order correlations). When appropriate statistical data was missing (k = 4), we attempted to obtain the information from the corresponding authors (two relevant studies were excluded because the relevant information could not be obtained). As illustrated in Figure 1, the final sample included 30 individual studies (14, 46.67% unpublished), from 20 research reports, with a total sample size of 20,963 (M = 698.77, Med = 465.5, SD = 594.39).

Coding of Variables

Outcomes. Single effect size of exposure to fact-checking on message-consistent beliefs was calculated per sample. Though there is no single consistent and accepted definition for attitudes and beliefs across different disciplines, following Rokeach’s (1968) explica- tion, we identified beliefs as a specific judgment of true/false, good/bad, or desirable/ undesirable toward an object or a situation; whereas attitudes were conceptualized as an organization of several favorable or unfavorable beliefs. Hence, keeping in mind the overlap with which beliefs and attitudes are used in the literature, there is good rationale to combine them into a single index, which is the way the results will be presented.

It can be argued that different types of beliefs should not be synthesized and aggregated since issue favorability is not the same as issue accuracy, beliefs about immigration are different from beliefs about healthcare, and fact-checking of a politician is not equivalent to rating of information originating from a news outlet. Despite such potential contradictions, the current analysis synthesizes different belief domains for several reasons. First, this practice is in line with other meta-analyses in the area of misinformation (i.e., Chan et al., 2018; Walter & Murphy, 2018), as well as political communication more broadly (e.g., Benoit, Hansen, & Verser, 2003; Boulianne, 2009; Paul, Salwen, & Dupagne, 2000) that aggregated and averaged outcomes across different topics. In fact, compared with other studies of misinformation that averaged beliefs across diverse contexts such as politics, health, marketing, and science, with a wide range of correction methods, including fact-checking, retractions, and forewarn- ings (i.e., Chan et al., 2017; Walter & Murphy, 2018), the current study seems to be on safer grounds as it has a narrower focus on fact-checking of political statements. Second, from an empirical point of view, potential variations among different types of beliefs made no significant difference in terms of their effect size. Of note, the average effect of fact-checking on beliefs did not vary as a function of question type (Q(1) = 1.75, p = .19),3

policy domain (Q(3) = 5.69, p = .14),4 issue type (Q(1) = 0.21, p = .65),5 and source of misinformation (Q(2) = 3.24, p = .20).6

The vast majority of studies reported a single effect of fact-checking on beliefs; yet, in five cases, two or more different estimates were provided (e.g., Weeks, 2015; belief in claims

Fact-Checking 357

Figure 1. Search strategy. Note. Only completely irrelevant entries were excluded on the basis of the title/abstract, including patents and newspaper articles. All academic papers that were considered to be even remotely relevant were further examined.

358 Nathan Walter et al.

about immigration/beliefs in claims about the death penalty). When more than one effect size was reported, estimates were averaged to represent a single effect size of change in the direction advocated by the fact-checking message. Further, due to concerns over general- izability, when studies included different conditions of exposure to fact-checking messages, only those that corresponded with real-world practices were retrieved for the meta-analysis. For example, we focused on traditional fact-checking as opposed to a humorous corrective video (e.g., Young et al., 2018), fact-checking messages that integrate partisan cues (e.g., Wintersieck, 2015), and an aggregated summary of fact-checking data (e.g., Agadjanian et al., 2019). Likewise, though we included studies that used real-world fact-checkers (e.g., FactCheck.org) and studies that used fictitious fact-checkers (e.g., GetTheFacts.org), the inclusion criteria required studies to frame the source of the corrective message as a fact- checker. Hence, studies that used corrective information that was not explicitly framed as coming from a fact-checker were excluded (e.g., Bode&Vraga, 2015;Wood& Porter, 2019). Considering the fact that only a small fraction of themisinformation literature deals with fact- checking, by including all types of corrections, it would have been impossible to discern findings that are relevant to fact-checking from findings that speak more broadly to mis- information and the continued influence effect (Chan, Jones, Hall Jamieson, & Albarracín, 2017; Walter & Murphy, 2018).

Moderators. Ten theoretically-driven variables were tested. Broadly speaking, these moderators reflect two frameworks that explain the contingencies of fact-checking - motivated reasoning and message presentation. To protect against violations of indepen- dence of effect sizes, most moderators were coded at the level of the study, treating a study as the unit of analysis (for an explanation see Borenstein, Hedges, Higgins, & Rothstein, 2009; Hunter & Schmidt, 2004). When moderators were either manipulated or tested within studies (i.e., pro/counter-attitudinal fact-checking, participants’ political affiliation, and integration of visual “truth scales”), subsamples were treated as the level of analysis.7 In cases where the original study did not provide the complete version of the fact-checking message (k = 5), information was obtained from corresponding authors.

Seventeen effect sizes were associated with pro-attitudinal fact-checking (e.g., fact- checking of false negative claims about Obamacare among Democrats) and 18 were associated with counter-attitudinal fact-checking (e.g., fact-checking of false negative claims about the Keystone Pipeline among Republicans). Political sophistication was measured as a continuous variable in the six studies that directly assessed participants’ political knowledge and then standardized from 0 to 1 (M = .56, SD = .16). Overall, 20 studies attempted to use fact-checking to correct campaign-related misinformation com- pared with 10 studies that focused on routine fact-checking. With respect to visual components of fact-checkers, eight studies used fact-checks that integrated truth scales or visual indicators and 23 samples were exposed only to text. The average word count for a fact-checking message was 307.22 (SD = 365.19). A validated lexical complexity analyzer (LCA) was used to gauge the complexity of fact-checking messages. Though there are numerous ways to assess lexical complexity, it is commonly treated as a multidimensional construct, consisting of lexical sophistication, variation, density, and vocabulary errors (Read, 2000). Keeping in mind that the fact-checking messages used in the primary studies were relatively short and screened for grammatical errors, textual density and vocabulary errors were considered less relevant. To this end, lexical complex- ity was gauged with four different indices, including lexical sophistication (M = .37,

Fact-Checking 359

SD = .11), lexical word variation (M = .71, SD = .13), verb variation (M = .18, SD = .12), and noun variation (M = .66, SD = .13) (for more information see Ai & Lu, 2010; Lu, 2012).8 Then, the four different indicators were averaged to create a single composite of lexical complexity (M = .47, SD = .09, α = .82). Finally, 15 fact-checkers refuted an entire original statement and 15 refuted portions of a statement. Table 1 provides a summary of the studies included in the meta-analysis.

Inter-Coder Reliability. The first author trained an independent coder on a subsample of related studies that were not included in the final sample. Then, reliability was calculated on 41.9% of the total database using Krippendorff’s alpha. The overall reliability was satisfactory, with the agreement ranging from .81 to 1.00. Disagreements were resolved by discussion and further guidelines were provided.

Data Analysis

All analyses were conducted using the statistical package Comprehensive Meta-Analysis (CMA) (version.3; Borenstein, Hedges, Higgins, & Rothstein, 2005). The effect size estimate employed was Cohen’s d, allowing easy interpretation of both directionality and strength. The results are based on uncorrected estimates of random-effects models (Hedges & Vevea, 1998). As argued by Card (2012), meta-analyses using fixed-effects models are only justified in drawing conclusions about the specific set of studies included in the sample, whereas random-effects models allow to generalize beyond the particular set of studies to a broader population of potential research. In CMA, random-effects models are computed by assigning more weight to the studies that carry more informa- tion, based on the inverse of a study’s variance (Borenstein et al., 2005).

After assessing the average effect of fact-checking on message-consistent beliefs, heterogeneity was assessed using the Q statistics. In such models, a significant Q coefficient represents data heterogeneity beyond that expected by sampling error alone (Higgins & Thompson, 2002). Then, moderator analyses were conducted to explore potential causes of heterogeneity (Ashford, Edmunds, & French, 2010). Specifically, for categorical variables (i.e., H1, H3a, H4, H5, H8), moderation analyses focused on Q statistics, by comparing the mean variability in effect size estimates across different values of the moderator. For continuous variables (i.e., H2, H6a, H7a) and interactions (i.e., H3b, H6b, H7b), a meta-regression was employed, allowing to test different moderators as potential predictors of effect sizes. Due to concerns over statistical power, all moderation analyses included, at least, five cases for each value (for a discussion of statistical power in meta-analysis see Jackson & Turner, 2017).

Results

Effects of Fact-Checking

The first research question dealt with the average effect of fact-checking on beliefs. Across 30 studies, the mean effect size of fact-checking was positive and significant (d = 0.29, 95% CI [.23, .36], p = .005), with significant heterogeneity in effect sizes, Q (29) = 207.83, I2 = 86.05%, p = .0005 (see Figure 2 for a forest plot of effect sizes by study). The heterogeneity of effect sizes provides further evidence that the influence of fact-checking is potentially contingent on various moderating variables.

360 Nathan Walter et al.

Fact-Checking and Motivated Reasoning

As predicted by H1, there was a significant difference between counter-attitudinal fact- checking and pro-attitudinal fact-checking (Q(1) = 3.41, p = .04). In particular, pro- attitudinal fact-checking (i.e., debunking the opposing ideology) resulted in stronger effects (d = 0.43, 95% CI [.29, .53], p = .001, k = 17) compared with counter- attitudinal fact-checking (i.e., debunking personal ideology) (d = 0.28, 95% CI [.19, .38], p = .001, k = 18). Additionally, as predicted by H2, the efficacy of fact-checking tended to weaken with increased political sophistication; b = −1.18, SE = .53, (Q (1) = 5.04, p = .02). The analysis also tested a potential curvilinear relationship between political sophistication and fact-checking. Yet, the data failed to find any support for this assumption; b = 1.30, SE = 2.89, p = .65 (Q(2) = 5.48, p = .06).

Showing no support for H3a, the analysis did not find a significant difference (Q (1) = 0.69, p = .41) between samples who self-identified as Democrats/liberals (d = 0.31, 95% CI [.20, .42], p = .001, k = 17) and samples who self-identified as Republicans/ conservatives (d = 0.27, 95% CI [.16, .37], p = .001, k = 17). To further probe the role

Figure 2. Forest plot for the effect (Cohen’s d) of fact-checking on beliefs compared with a neutral control condition. Note. The effects presented in the forest plot concern the full study samples.

Fact-Checking 361

T ab

le 1

L is t of

st ud

ie s in cl ud

ed in

m et a- an al ys is

S tu dy

Y ea r

S tu dy

N um

be r

N L ex ic al

C om

pl ex it yq

P ol it ic al

S op

hi st ic at io nr

To pi c

F ac t-

C he ck er

A ga dj an ia n

et al .c d ef g lp

20 19

1 18

82 -

- F al se

st at em

en t by

a S en at or

(M cC

on ne ll /R ei d)

P ol it iF ac t

A m az ee n

et al .a b cd eg k lo

20 18

1 67

7 .3 3

- F al se

st at em

en t by

a fi ct it io us

C on

gr es sm

an F ic ti ti ou

s

C ar na ha n et

al .b h m o

20 18

1 50

7 .3 7

- M is in fo rm

at io n ab ou

t C om

m on

C or e

po li cy

F ic ti ti ou

s

F ly nn

et al .a d ef h jm

p 20

17 1

64 8

.5 5

- F ra ud

al le ga ti on

s ag ai ns t a le gi sl at or

F ic ti ti ou

s F ri dk

in et

al .a d eh k m o 20

15 1

45 2

.4 8

.5 5

N eg at iv e ad s du

ri ng

a ca m pa ig n

F ic ti ti ou

s F ri dk

in et

al .a h k m o

20 16

1 45

2 .4 8

- N eg at iv e ad s du

ri ng

a ca m pa ig n

F ic ti ti ou

s G ar re tt &

W ee ks

b cf h jn o

20 13

1 57

4 .3 1

.6 0

E le ct ro ni c he al th

re co rd s

F ac tc he ck

G ar re tt et

al .h jo

20 13

1 30

0 .3 9

.3 4

Is la m ic

C ul tu ra l C en te r in

N ew

Y or k

C it y

F ac tc he ck

G ra ve s

et al .b d ef g k m p

20 18

1 18

76 .4 1

- G un

R ig ht s de ba te

P ol it iF ac t

Ja rm

an d eh m o

20 14

1 21

7 .4 1

- G un

R ig ht s de ba te

F ic ti ti ou

s Ja rm

an ad eh lp

20 15

1 18

7 .3 8

- O ba m ac ar e

F ic ti ti ou

s Ja rm

an ad eh lp

20 15

2 26

9 .3 7

- O ba m ac ar e

F ic ti ti ou

s Ja rm

an b d eh lp

20 16

1 19

8 .4 4

- Im

m ig ra ti on

P ol ic y

F ic ti ti ou

s N yh

an et

al .a b d ef h lo

20 17

1 16

80 .5 1

- C ri m e R at es

F ic ti ti ou

s N yh

an et

al .a b d ef h lo

20 17

2 52

7 .5 1

- C ri m e R at es

F ic ti ti ou

s P en ny

co ok

et al .a b cf ij n o

20 17

1 47

9 .5 5

- C la im

s ab ou

t a ca nd

id at e

F ic ti ti ou

s

P en ny

co ok

et al .a b cf ij n o

20 17

2 18

79 .5 5

- C la im

s ab ou

t a ca nd

id at e

F ic ti ti ou

s

P en ny

co ok

et al .a b cf ij n o

20 17

3 14

95 .5 5

- C la im

s ab ou

t a ca nd

id at e

F ic ti ti ou

s

P en ny

co ok

et al .a b cf ij n o

20 17

4 40

0 .5 5

- C la im

s ab ou

t a ca nd

id at e

F ic ti ti ou

s

P en ny

co ok

et al .a b cf ij n o

20 17

5 10

18 .5 5

- C la im

s ab ou

t a ca nd

id at e

F ic ti ti ou

s

S w ir e et

al .a d eh lo

20 17

1 17

76 .6 1

- S ta te m en ts

ab ou

t va cc in es

F ic ti ti ou

s S w ir e et

al .a d eh lo

20 17

2 32

0 .6 1

- S ta te m en ts

ab ou

t va cc in es

F ic ti ti ou

s T ho

rs on

ad h n p

20 16

1 12

2 .5 9

- A cc us at io ns

of m is co nd

uc t

F ic ti ti ou

s T ho

rs on

ad eh n p

20 16

2 34

5 .5 1

- A cc us at io ns

of m is co nd

uc t

F ic ti ti ou

s W ee ks

d eh jk lo

20 15

1 51

2 .4 0

.5 0

D ea th

pe na lt y/ im

m ig ra ti on

A P

W in te rs ie ck

ab fh k m o

20 15

2 45

2 .4 8

.5 5

N eg at iv e ad s du

ri ng

a ca m pa ig n

F ic ti ti ou

s W in te rs ie ck

ab d ef h k lp

20 15

3 13

4 .4 8

- P ol it ic al

de ba te

P ol it iF ac t

W in te rs ie ck

ab fh k lp

20 15

4 28

4 .4 8

- N eg at iv e ad s du

ri ng

a ca m pa ig n

P ol it iF ac t

Y or k et

al .fh

k n o

20 18

1 11 90

.5 9

- M an da to ry

im m un

iz at io n

F ic ti ti ou

s Y ou

ng et

al .d eh k lo

20 18

1 11 1

.3 2

.8 5

K ey st on

e P ip el in e

F ac tC he ck

N ot e.

a C am

pa ig n- re la te d;

b pa rt ia l re fu ta ti on ; c i nc lu si on

of gr ap hi ca l el em

en ts ; d co un te ra tt it ud in al

co rr ec ti on ; e p ro at ti tu di na l co rr ec ti on ; f u np ub li sh ed ; g ar ti cl e

+ ra ti ng ; h ar ti cl e on ly ; i la be l on ly ; j re fe re nc e to

ob je ct iv it y;

k lo go ; l p ol it ic ia n as

so ur ce ; m in te re st

gr ou p as

so ur ce ; n pr es s as

so ur ce ; °b el ie f ac cu ra cy ; p be li ef

fa vo ra bi li ty ; q co m po si te

sc or e ba se d on

fo ur

di ff er en t in di ca to rs

of le xi ca l co m pl ex it y, in cl ud in g so ph is ti ca ti on ,l ex ic al

w or d va ri at io n,

ve rb

va ri at io n,

an d no un

va ri at io n;

r s ta nd ar di ze d sc or e (0 –1 ) ba se d on

st ud ie s th at

m ea su re d po li ti ca l kn ow

le dg e.

played by ideology, we examined the interplay between fact-checking messages and participants’ political affiliation. Interestingly, when focusing only on Democratic/liberal participants, the recorded difference between fact-checking of counter-attitudinal and pro- attitudinal messages disappeared (Q(1) = 0.93, p = .34). Simply put, for Democrats/ liberals, there was no significant difference between counter-attitudinal fact-checking (d = 0.32, 95% CI [.17, .47], p = .005, k = 6) and pro-attitudinal fact-checking (d = 0.43, 95% CI [.26, .61], p = .001, k = 7). Yet, for Republicans/conservatives (Q (1) = 9.55, p = .001), there was a significant difference in favor of pro-attitudinal corrections (d = 0.52, 95% CI [.32, .73], p = .001, k = 5), compared with counter- attitudinal corrections (d = 0.17, 95% CI [.07, .26], p = .003, k = 8). In other words, and supporting H3b, Republicans/conservatives appear to be more susceptible to motivated processing compared to their Democratic/liberal counterparts. Figure 3 illustrates the interaction between fact-checking valence and political ideology. Additionally, as hypothesized in H4, fact-checking effects were significantly weaker for campaign- related statements (d = 0.24, 95% CI [.17, .30], p = .005, k = 20), as opposed to non- campaign fact-checking (d = 0.38, 95% CI [.26, .51], p = .005, k = 10; Q(1) = 4.07, p = .04).

Fact-Checking and Message Design

With respect to fact-checking design characteristics, a pertinent question deals with the inclusion of truth scales or graphical elements that can potentially facilitate fact-checking. The treatment of graphical elements as a moderator, resulted in a significant difference between fact-checkers that included visual accuracy cues and those that did not (Q (1) = 6.58, p = .01). Specifically, and contrary to H5, the inclusion of graphical elements appears to backfire and attenuate correction of misinformation, with visual fact-checkers producing significantly weaker effects (d = 0.19, 95% CI [.14, .24], p = .001, k = 8), as

Figure 3. The interaction effect of fact-checking on beliefs by political affiliation/ideology and pre-existing beliefs.

364 Nathan Walter et al.

opposed to fact-checking that does not include visual accuracy cues (d = 0.31, 95% CI [.23, .39], p = .001, k = 23).

Contrary to H6, there was no significant effect of message length on the efficacy of fact- checking; b = .01, SE = .02, (Q(1) = 0.30, p = .58). The role played by the lexical complexity of the fact-checking message was also tested with a meta-regression. Effect sizes associated with fact-checking were treated as the outcome and message length (i.e., number of words) and lexical complexity, a composite of lexical sophistication, lexical word variation, verb variation, and noun variation, were entered as predictors. The results of the meta-regression indicated that lexical complexity is a significant negative predictor of fact-checking efficacy; b = −1.43, SE = .43, p = .001 (Q(1) = 12.17, p = .001). Simply put, and in line with H7a, linguistic complexity had a taxing effect on fact-checking, so that more simplistic messages tended to achieve more desirable results. Further, there was no support for H7b as no significant interaction between lexical complexity and political sophistication was found; b = −6.64, SE = 27.87, p = .81; (Q(3) = 9.25, p = .02). Likewise, as a post-hoc test, we assessed whether the effects of lexical complexity on fact-checking follow a curvilinear pattern. The results did not support a potential curvilinear effect; b = −3.75, SE = 4.58, p = .41 (Q(3) = 12.68, p = .005). Finally, the analysis found support for the implied truth effect (H8; Q(1) = 4.30, p = .03). In particular, fact-checkers that attempted to refute an entire statement (d = 0.31, 95%CI [.23, .40], p = .001, k = 15) were substantiallymore effective than fact-checkers that sought to correct only parts of a statement (d = 0.19, 95% CI [.12, .27], p = .001, k = 15).9

Stimulus Characteristics

Finally, the analysis focused on several critical dimensions of fact-checking messages. In particular, there was a significant difference (Q(2) = 6.66, p = .04), between messages that included only fact-checking labels (d = 0.18, 95% CI [.11, .26], p = .001, k = 5), fact- checking articles (d = 0.32, 95% CI [.24, .40], p = .001, k = 22), and fact-checking articles with rating scales (d = 0.28, 95% CI [.10, .47], p = .003, k = 3). Further, there were no difference in effect sizes (Q(1) = 0.83, p = .36) between messages that targeted specific political actors (d = 0.27, 95% CI [.21, .34], p = .001, k = 23) as opposed to general false claims (d = 0.36, 95% CI [.19, .52], p = .001, k = 7). In addition, there was no significant difference between messages that explicitly referred to the objectivity/ impartiality of their fact-checking organization (d = 0.29, 95% CI [.13, .46], p = .001, k = 9) and those that did not (d = 0.29, 95% CI [.23, .36], p = .001, k = 21); (Q(1) = 0.01, p = .99). Likewise, effect sizes did not vary (Q(1) = 0.97, p = .32) by whether fact- checking messages included a logo (d = 0.37, 95% CI [.19, .54], p = .001, k = 7) or not (d = 0.27, 95% CI [.21, .34], p = .001, k = 23).

Publication Bias

A common method to detect a publication bias uses Kendall’s Tau to estimate the relation- ship between the standardized effects and their variance (Begg & Mazumdar, 1994). According to this estimate, there is a significant relationship between standardized effects and their variance (Tau = .30, p = .02), indicating some support for a potential file drawer problem. Further, a funnel plot (plots Cohen’s d effect sizes on the x-axis against a measure of study size on the y-axis; standard errors) is a useful approach to detect bias (Egger, Smith, Schneider, & Minder, 1997), with degree of funnel plot asymmetry indicating a possible publication bias. As Figure 4 illustrates, larger samples tend to produce

Fact-Checking 365

somewhat weaker effects compared with studies that utilized smaller samples. Duval and Tweedie (2000) method of trim and fill was used to numerically assess this asymmetry. Following this approach, smaller studies causing plot asymmetry are removed and “true” averaged effect is estimated, while the center of the plot is filled with the omitted studies. Based on the trim and fill assessment in comprehensive meta-analysis, nine studies had to be trimmed, resulting in a slightly weaker average effect of fact-checking (d = 0.21, 95%CI [.14, .27]) than the one observed in the data. Likewise, when directly comparing published studies (k = 16) with unpublished studies (k = 14) within our sample, the analysis recorded a significant difference [Q(1) = 11.03, p = .001], indicating that unpublished studies tended to retrieve weaker effects (d = 0.18, 95% CI [.12, .24]), compared to published studies (d = 0.39, 95% CI [.28, .49]). In total, this provides considerable evidence for a publication bias in the fact-checking literature.

Discussion

In its brief history, the fact-checking enterprise has drawn praise and criticism due to its role as an unbiased referee in a highly polarized political environment (Marietta et al., 2015). When disputed realities are involved, any effort to correct misinformation and inform the public by utilizing facts can be expected to both have effects and to encounter resistance. Though fact-checking organizations have successfully become a standard part of political reporting (Graves et al., 2016), their growing prominence raises important questions regarding effectiveness. Hence, the question at the center of the current study is the following: What is the ability of fact-checking messages to affect people’s beliefs?

There are two potential answers to this question offered in this meta-analysis. Striking a more optimistic tone, fact-checking messages positively affect beliefs, irre- spective of political ideology, preexisting positions, context (campaign vs. routine), and whether it refutes the entire false statement or just parts of a statement. In fact, as the forest plot clearly demonstrates, though not all fact-checking attempts are equally

Figure 4. Funnel plot for the detection of a potential publication bias (open circles represent observed effect sizes and full circles represent imputed effect sizes).

366 Nathan Walter et al.

effective, there is no evidence of factual backfire (Nyhan & Reifler, 2010). As argued by Wood and Porter (2019), “citizens heed factual information, even when such information challenges their ideological commitments” (p. 1). Simply put, the beliefs of the average individual become more accurate and factually consistent, even after a single exposure to a fact-checking message. To be sure, effects are heterogeneous and various contingencies can be applied, but when compared to equivalent control conditions, exposure to fact- checking carries positive influence.

However, the results also raise substantial concerns. In line with the motivated reasoning literature (Kunda, 1990; Nir, 2011), the effects of fact-checking on beliefs are quite weak and gradually become negligible the more the study design resembles a real-world scenario of exposure to fact-checking. For instance, though fact-checking can be used to strengthen preexisting convictions, its credentials as a method to correct misinformation (i.e., counter- attitudinal fact-checking) are significantly limited. Likewise, even if journalists and elites often hail the independent and objective nature of fact-checking organizations (Fridkin et al., 2015; Nyhan&Reifler, 2015), the results point to visible differences between howDemocrats/liberals and Republicans/conservatives interpret fact-checking information. Whereas Democrats/liber- als are equally receptive to information that supports or contradicts their ideology, Republicans/ conservatives are more eager to accept pro-attitudinal corrections and less likely to adopt ideologically inconsistent information. In agreement with previous research (Shin & Thorson, 2017), it seems that the quality of objectivity often ascribed to fact-checking organizations does not transcend political and ideological divides. Ostensibly, this finding may speak to inherent differences between Republicans/conservatives and Democrats/liberals, with the former being more likely to engage in biased processing. From a strategic communication point of view, however, these findings are perhaps indicative of an ongoing attempt to undermine fact- checking efforts that contradict party lines (Stencel, 2015). For example, President Trump – with an astonishingly poor record among professional fact-checkers (Graves, 2017) – has repeatedly dismissed discernible reality by labeling legacy media and the fact-checking of his statements as “fake news” (Jamieson & Taussig, 2017). Arguably, this aggressive approach to elite journalism and a growing disdain for fact-checking helps thwart criticism, making partisan bias among Republicans considerably worse.

In addition, fact-checking messages that included graphical elements tended to be less effective than fact-checking messages that did not integrate visuals. We initially expected graphical elements to facilitate corrections because they provide simple and straightforward summary of the fact-checking message, but the data did not support this line of reasoning. Rather, graphical elements appeared to interfere with the correction. It is difficult to speculate why the inclusion of visual elements tended to backfire, one potential cause has to do with motivated reasoning. As argued by Amazeen et al. (2018), truth scales can serve as a strong partisan cue, confirming a person’s pre-existing beliefs (either favorable or unfavorable) thus eliminating the motivation to centrally process the rest of the message.

Further, though misinformation is a constant problem, individuals are more likely to be exposed to misinformation or disinformation during election campaigns (Coddington et al., 2014). But it is during these times, when fact-checking is neededmost, that motivated reasoning is at its peak (Miller & Conover, 2015), and so fact-checking is less effective. The results also question the ability of fact-checking organizations to increase public knowledge. In fact, what sets fact-checking messages apart from most other approaches to correction of misinformation is their emphasis on providing more context that can help individuals accurately evaluate political entities. The focus on the political process and public policies often necessitates the use of nuanced and complex language, which can have both favorable and unfavorable

Fact-Checking 367

consequences. As a whole, lexical complexity appears to detract from fact-checking efforts, whereas simpler and less sophisticated messages prove to be more effective.

Finally, given the inherent complexities of our political reality, the ratings of statements made by political entities are more likely to be in a shade of gray rather than completely true or false (PunditFact, 2018). For example, an hour-long State of the Union address can strike a variety of false notes along with some exaggerations, as well as generate completely accurate statements. Keeping in mind, however, that fact-checkers focus solely on factual statements, even a ruling of “mostly false” can increase the credibility of various ungrounded statements. After all, if a long speech was thoroughly fact-checked and opinion statements were not directly refuted, perhaps, it means that they are accurate (Pennycook & Rand, 2017). Interestingly, there is some evidence to suggest that even different fact-checking organizations find it difficult to agree on an ambiguous scoring range (e.g., “mostly false”) (Lim, 2018); thus, the limited efficacy of partial corrections can be the result of political discourse that is designed to be ambiguous rather than audience’s cognitive biases (e.g., implied truth effect). Practically speaking, this may suggest that fact-checking organizations need to carefully consider their units of analysis. For example, instead of focusing on entire speeches that are likely to result in equivocal verdicts (e.g., mostly true/mostly false), an alternative approach can focus on specific statements leading to more conclusive judgment (e.g., true/false).

The limited utility of fact-checking is also evident when directly contrasting the findings with results from recent meta-analyses that addressed the psychological efficacy of messages countering misinformation (Chan et al., 2017; Walter & Murphy, 2018) and ways to protect eyewitness memory against the misinformation effect (Blank & Launay, 2014). In comparison to the weak effects recorded in the current study, other meta-analyses observed strong (Blank & Launay, 2014; Chan et al., 2017) and moderate effects (Walter & Murphy, 2018). Presumably, these discrepancies can be attributed to the challenging context in which fact-checking organizations operate. In particular, while previous analyses have aggregated the effects of political misinformation together with topics that are less likely to induce resistance and partisanship (e.g., health, marketing), the weaker effects obtained in the current study can be potentially attributed to the value-laden contexts that studies of fact- checking typically address. This explanation is supported by the fact that when focusing exclusively on political topics, there is virtually no difference between the main effects recorded in Walter and Murphy (r = .15, 2018) and the current meta-analysis (r = .1510).

At this point, it is important to situate the current results in the broader context of fact- checking as a journalistic practice. Though the current study focuses exclusively on the direct effects of political messages on citizens’ beliefs, fact-checking is not only about changing what citizens think – it is also, to a large extent, about holding political entities accountable by keeping false statements out of public discourse (Graves, 2016). According to this view, if political figures think that their statements will be subjected to robust scrutiny, they will be less likely to make, or repeat, false claims. Additionally, in line with the third-person effect, if politicians feel that their voters are likely to be affected by exposure to fact-checking messages, they, in turn, would attempt to change their rhetoric and actions (Cohen, Tsfati, & Sheafer, 2008). As these scenarios indicate, fact-checking can be a powerful tool of social influence even if it has only limited efficacy when it comes to changing public opinion.

Limitations and Future Directions

The most obvious limitation associated with meta-analysis is its dependence on primary studies, which is exacerbated when the analyses are based on a relatively small number of

368 Nathan Walter et al.

studies. In such cases, not all hypotheses can be adequately tested and the power to detect significant moderators is somewhat compromised. Likewise, the relatively small sample of studies has limited our ability to provide a more nuanced understanding of fact- checking. For instance, distinct constructs such as ideology (conservative vs. liberal) and partisanship (Republican vs. Democrat) were combined rather than analyzed sepa- rately. Further, the decision to focus on highly controlled experimental designs assisted in identifying the causal link between fact-checking and beliefs, but it also raises the question whether the current results can be generalized to more naturalistic settings. For example, does exposure to fact-checking messages produce the same effects when people are instructed to carefully read a message in a lab setting, as opposed to co- viewing a political debate in a living room with family and friends? What is the role of forced exposure to messages versus having the real-world choice of whether to consume a fact-check message (see Garrett, 2013)? Keeping in mind that extant literature lacks tests of such comparisons, these questions remain unanswered.

Further, the current analysis did not include empirical reports that tested the role played by fact-checking using correlational methods (e.g., Karlsson, Clerwall, & Nord, 2017). While including correlational studies, and treating methodology as a moderator, may increase the sample size, it also introduces bias into our results. Namely, experimental designs, where the effect of fact-checking is isolated, permit conclusions about whether the causal effect of fact- checking on beliefs. Correlational studies, however, permit conclusions about whether fact- checking is associated with beliefs, but cannot possibly speak to the effects of fact-checking. Considering the fact that correlational designs attempt to answer distinct questions, even if their inclusion is desirable in terms of comprehensiveness, conceptually and methodologically such syntheses are difficult to justify. Relatedly, the fact that only 27.6% of the studies in the sample employed authentic fact-checkers (the remaining studies used fictitious fact-checkers) hinders the external validity of the findings. Likewise, additional moderators such as issue-involvement and political efficacy are expected to provide a more fine-grained understanding of fact- checking; yet, such variables were largely ignored by primary studies.

As to be expected with a relatively young research program, there are numerous gaps to be filled in fact-checking research. First, future research should expand the palate of potential outcomes beyond beliefs and attitudes, to include effects on political knowledge, cynicism, efficacy, and information sharing. After all, if the primary goal of fact-checking organizations is to infuse public discourse with factual data, political knowledge or information seeking and sharing are perhaps better indicators of successful fact-checking. Second, while the number of fact-checkers around the world has more than tripled in the past few years (Stencel & Griffin, 2018) and currently fact-checking organizations can be found in 53 countries around the globe, the empirical literature focuses disproportionately on the United States. Hence, one promising stream for future research could advance a cross-cultural approach to fact- checking, contrasting well-established democracies with other types of political systems (see Van Aelst et al., 2017). Finally, given the struggle of social media and technology companies to keep up with the sophistication of viral hoaxes and disinformation campaigns, more attention should be given to understand the most effective ways in which fact-checking practices can be integrated into online platforms.

Conclusion

Only recently has the empirical study of political misinformation and fact-checking correc- tion produced a critical mass of work that is worthy of meta-analytical review. This area of

Fact-Checking 369

study is at a critical juncture in its maturation process while continuing to increase in social significance as fake news generation and dissemination become more pervasive in our digital media environments. Although much remains to be uncovered about the influence of fact-checking, we hope that the current study can serve as a roadmap to continue exploring the challenging terrain of political misinformation and its correction.

Notes

1. The typical procedure in the control condition involved exposure to a message that included misinformation and then exposure to a measurement of message-relevant beliefs. Importantly, message design did not account for significant differences in effect sizes [Q(1) = 0.24, p = .63]; between-subjects d = 0.28, 95% CI [.21, .36], k = 23 vs. within-subjects d = 0.32, 95% CI [.19, .46], k = 7.

2. Though the original plan was to include both favorable and unfavorable fact-checking, the paucity of favorable fact-checking research would have prevented us from analyzing the data in a statistically meaningful way.

3. Issue favorability d = 0.36, 95% CI [.23, .49] vs. issue accuracy d = 0.26, 95% CI [.19, .33]. 4. Economy d = 0.14, 95% CI [.08, .19] vs. education d = 0.15, 95% CI [−.13, .44] vs.

healthcare d = 0.28, 95% CI [.15, .40] vs. national security d = 0.26, 95% CI [.06, .47]. 5. Real-world issue d = 0.30, 95% CI [.23, .37] vs. fictional issue d = 0.26, 95% CI [.10, .42]. 6. Interest group d = 0.27, 95% CI [.15, .38] vs. politician d = 0.33, 95% CI [.01, .43] vs. the

press d = 0.22, 95% CI [.16, .28]. 7. Due to the fact that some variables were coded at the subsample level, several moderators add

up to more than 30 cases (see Table 1). 8. Due to the fact that lexical complexity cannot be meaningfully assessed for texts with less

than 50 words, one study (i.e., Agadjanian et al., 2019) was not coded for this moderator. 9. In terms of research design characteristics, there were no significant differences between real-

world fact-checkers and fictitious fact-checkers (Q(1) = 2.41, p = .12); fictional misinforma- tion and real-world misinformation (Q(1) = 0.29, p = .87).

10. Transformed from Cohen’s d = 0.29.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Jonathan Cohen http://orcid.org/0000-0003-1743-836X

References

An asterisk precedes references included in the meta-analysis. Achen, C. H., & Bartels, L. M. (2017). Democracy for realists: Why elections do not produce

responsive government. Princeton, NJ: Princeton University Press. *Agadjanian, A., Bakhru, N., Chi, V., Greenberg, D., Hollander, B., Hurt, A., ... & Pham, D.

(2019). Counting the Pinocchios: The effect of summary fact-checking data on perceived accuracy and favorability of politicians. Research & Politics, 6, 1-10. doi:10.1177/ 2053168019870351

Ai, H., & Lu, X. (2010, June 8–12). A web-based system for automatic measurement of lexical complexity. Paper presented at the 27th Annual Symposium of the Computer-Assisted Language Consortium (CALICO-10). Amherst, MA.

370 Nathan Walter et al.

Althaus, S. L. (2012). What’s good and bad in political communication research? Normative standards for evaluating media and citizen performance. In H. A. Semetko & M. Scammell (Eds.), The SAGE handbook of political communication (pp. 97–112). Thousand Oaks, CA: Sage.

Amazeen, M. A. (2013, October). Making a difference: A critical assessment of fact-checking in 2012 (New America Foundation Media Policy Initiative Research Paper). Retrieved from https://www.newamerica.org/new-america/making-a-difference/

Amazeen, M. A. (2016). Checking the fact-checkers in 2008: Predicting political ad scrutiny and assessing consistency. Journal of Political Marketing, 15, 433–464. doi:10.1080/ 15377857.2014.959691

*Amazeen, M. A., Thorson, E., Muddiman, A., & Graves, L. (2018). Correcting political and consumer misperceptions: The effectiveness and effects of rating scale versus contextual correction formats. Journalism & Mass Communication Quarterly, 95, 28–48. doi:10.1177/1077699016678186

Ashford, S., Edmunds, J., & French, D. P. (2010). What is the best way to change self-efficacy to promote lifestyle and recreational physical activity? A systematic review with meta-analysis. British Journal of Health Psychology, 15, 265–288. doi:10.1348/135910709X461752

Barberá, P., Jost, J. T., Nagler, J., Tucker, J. A., & Bonneau, R. (2015). Tweeting from left to right: Is online political communication more than an echo chamber? Psychological Science, 26, 1531–1542. doi:10.1177/0956797615594620

Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101. doi:10.2307/2533446

Bennett, W. L., & Pfestch, B. (2018). Rethinking political communication in a time of disrupted public spheres. Journal of Communication, 68, 243–253. doi:10.1093/joc/jqx017

Benoit, W. L., Hansen, G. J., & Verser, R. M. (2003). A meta-analysis of the effects of viewing U.S. presidential debates. Communication Monographs, 70, 335–350. doi:10.1080/ 0363775032000179133

Blank, H., & Launay, C. (2014). How to protect eyewitness memory against the misinformation effect: A meta-analysis of post-warning studies. Journal of Applied Research in Memory and Cognition, 3, 77–88. doi:10.1016/j.jarmac.2014.03.005

Bode, L., & Vraga, E. K. (2015). In related news, that was wrong: The correction of misinformation through related stories functionality in social media. Journal of Communication, 65, 619–638. doi:10.1111/jcom.12166

Bokhari, A. (2018, January). Google suspends ‘fact-check’ program after being … fact checked. Breitbart. Retrieved from https://www.breitbart.com/tech/2018/01/22/google-cancels-fact- check-program-fact-checked/

Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive meta-analysis version 3.0. Englewood, NJ: Biostat.

Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta- analysis. West Sussex, UK: Wiley.

Boulianne, S. (2009). Does internet use affect engagement? A meta-analysis of research. Political Communication, 26, 193–211. doi:10.1080/10584600902854363

Card, N. A. (2012). Applied meta-analysis for social science research. New York, NY: Guilford. *Carnahan, D., Hao, Q., Jiang, X., & Lee, H. (2018). Feeling fine about being wrong: The influence

of self-affirmation on the effectiveness of corrective information. Human Communication Research, 44, 274–298. doi:10.1093/hcr/hqy001

Chaiken, S., & Eagly, A. H. (1976). Communication modality as a determinant of message persuasiveness and message comprehensibility. Journal of Personality and Social Psychology, 34, 605–614. doi:10.1037//0022-3514.34.4.605

Chan, M. P. S., Jones, C. R., Hall Jamieson, K., & Albarracín, D. (2017). Debunking: A meta-analysis of the psychological efficacy of messages countering misinformation. Psychological Science, 28, 1531–1546. doi:10.1177/0956797617714579

Coddington, M., Molyneux, L., & Lawrence, R. G. (2014). Fact checking the campaign: How political reporters use Twitter to set the record straight (or not). International Journal of Press/ Politics, 19, 391–409. doi:10.1177/1940161214540942

Fact-Checking 371

Cohen, J., Tsfati, Y., & Sheafer, T. (2008). The influence of presumed media influence in politics: Do politicians’ perceptions of media power matter? Public Opinion Quarterly, 72, 331–344. doi:10.1093/poq/nfn014

Druckman, J. N. (2012). The politics of motivation. Critical Review, 24, 199–216. doi:10.1080/ 08913811.2012.711022

Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. doi:10.1111/ j.0006-341X.2000.00455.x

Dyer, R. F., & Kuehl, P. G. (1978). A longitudinal study of corrective advertising. Journal of Marketing Research, 15, 39–48. doi:10.2307/3150399

Ecker, U. K. H., Lewandowsky, S., Swire, B., & Chang, D. (2011). Correcting false information in memory: Manipulating the strength of misinformation encoding and its retraction. Psychonomic Bulletin & Review, 18, 570–578. doi:10.3758/s13423-011-0065-1

Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315, 629–634. doi:10.1136/bmj.315.7109.629

FactCheck.org. (2018). Our mission. Retrieved from https://www.factcheck.org/about/our-mission/ Flynn, D. J., Nyhan, B., & Reifler, J. (2017). The nature and origins of misperceptions:

Understanding false and unsupported beliefs about politics. Political Psychology, 38, 127–150. doi:10.1111/pops.12394

*Fridkin, K., Kenney, P. J., & Wintersieck, A. (2015). Liar, liar, pants on fire: How fact-checking influences citizens’ reactions to negative advertising. Political Communication, 32, 127–151. doi:10.1080/10584609.2014.914613

*Fridkin, K. L., Courey, J., Hernandez, S., & Spears, J. (2016). Gender differences in reactions to fact checking of negative commercials. Politics & Gender, 12, 369–390. doi:10.1017/ S1743923X16000076

Garrett, R. K. (2013). Selective exposure: New methods and new directions. Communication Methods & Measures, 7, 247–256. doi:10.1080/19312458.2013.835796

*Garrett, R. K., & Weeks, B. E. (2013, February). The promise and peril of real-time corrections to political misperceptions. In Proceedings of the 2013 Conference on computer supported cooperative work (pp. 1047–1058). San Antonio, TX: Association for Computing Machinery.

*Garrett, R. K., Nisbet, E. C., & Lynch, E. K. (2013). Undermining the corrective effects of media- based political fact checking? The role of contextual cues and naïve theory. Journal of Communication, 63, 617–637. doi:10.1111/jcom.12038

Grant, M. (1995). Greek and Roman historians: Information and misinformation. London, UK: Routledge.

Graves, L. (2016). Deciding what’s true: The rise of political fact-checking in American journalism. New York, NY: Columbia University Press.

Graves, L. (2017). Anatomy of a fact check: Objective practice and the contested epistemology of fact checking. Communication, Culture & Critique, 10, 518–537. doi:10.1111/cccr.12163

Graves, L., Nyhan, B., & Reifler, J. (2016). Understanding innovations in journalistic practice: A field experiment examining motivations for fact-checking. Journal of Communication, 66, 102–138. doi:10.1111/jcom.12198

*Graves, L., Gunther, A. C., Pelled, A., Su, M., Wang, Y., & Zhang, Y. (2018). Effects of fact checks on partisan beliefs and perceptions of bias. Paper presented at the annual meeting of the International Communication Association, Prague, Czech Republic.

Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504. doi:10.1037/1082-989X.3.4.486

Higgins, J., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558. doi:10.1002/sim.1186

Hovland, C., Janis, I., & Kelley, H. (1953). Communication and persuasion. New Haven, CT: Yale University Press.

Hovland, C., & Weiss, W. (1951). The influence of source credibility on communication effectiveness. Public Opinion Quarterly, 15, 635–650. doi:10.1086/266350

372 Nathan Walter et al.

Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.

Jackson, D., & Turner, R. (2017). Power analysis for random-effects meta-analysis. Research Synthesis Methods, 8, 290–302. doi:10.1002/jrsm.1240

Jamieson, K. H., & Taussig, D. (2017). Disruption, demonization, deliverance, and norm destruc- tion: The rhetorical signature of Donald J. Trump. Political Science Quarterly, 132, 619–650. doi:10.1002/polq.v132.4

*Jarman, J. W. (2014). A disturbing conclusion: The irrelevant of argument. In C. Palczewski (Ed.), Disturbing argument: Selected papers from the 18th biennial conference on argumentation (pp. 174-180). New York, NY: Routledge.

*Jarman, J. W. (2015). The failure of fact-checking. In B. Garssen, D. Gooden, G. Mitchell, & A. Snoeck Henkemans (Eds.), Proceedings of the 8th International Conference of the International Society for the Study of Argumentation (pp. 657–667). Amsterdam: Sic Sat.

*Jarman, J. W. (2016). Influence of political affiliation and criticism on the effectiveness of political fact-checking. Communication Research Reports, 33, 9–15. doi:10.1080/ 08824096.2015.1117436

Johnson, B. T., & Eagly, A. H. (1989). The effects of involvement on persuasion: A meta-analysis. Psychological Bulletin, 106, 290–314. doi:10.1037/0033-2909.106.2.290

Johnson, H. M., & Seifert, C. M. (1994). Sources of the continued influence effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 1420–1436. doi:10.1037/ 0278-7393.20.6.1420

Kahne, J., & Bowyer, B. (2017). Educating for democracy in a partisan age: Confronting the challenges of motivated reasoning and misinformation. American Educational Research Journal, 54, 3–34. doi:10.3102/0002831216679817

Karlsson, M., Clerwall, C., & Nord, L. (2017). Do not stand corrected: Transparency and users’ attitudes to inaccurate news and corrections in online journalism. Journalism & Mass Communication Quarterly, 94, 148–167. doi:10.1177/1077699016654680

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498. doi:10.1037/0033-2909.108.3.480

Levitan, L. C., & Visser, P. S. (2008). The impact of the social context on resistance to persuasion: Effortful versus effortless responses to counter-attitudinal information. Journal of Experimental Social Psychology, 44, 640–649. doi:10.1016/j.jesp.2007.03.004

Lewandowsky, S., Ecker, U. K., Seifert, C. M., Schwarz, N., & Cook, J. (2012). Misinformation and its correction continued influence and successful debiasing. Psychological Science in the Public Interest, 13, 106–131. doi:10.1177/1529100612451018

Lim, C. (2018). Checking how fact-checkers check. Research & Politics, 5, 1–7. doi:10.1177/ 2053168018786848

Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal, 96, 190–208. doi:10.1111/j.1540-4781.2011.01232_1.x

Marietta, M., Barker, D. C., & Bowser, T. (2015). Fact-checking polarized politics: Does the fact- check industry provide consistent guidance on disputed realities? The Forum, 13, 577–596. doi:10.1515/for-2015-0040

Media Bia/Fact Check. (2016 July). The 10 best fact checking sites. Retrieved from https:// mediabiasfactcheck.com/2016/07/20/the-10-best-fact-checking-sites/

Miller, P. R., & Conover, P. J. (2015). Red and blue states of mind: Partisan hostility and voting in the United States. Political Research Quarterly, 68, 225–239. doi:10.1177/1065912915577208

Neuman, W. R. (2016). The digital difference. Cambridge, MA: Harvard University Press. Nir, L. (2011). Motivated reasoning and public opinion perception. Public Opinion Quarterly, 75,

504–532. doi:10.1093/poq/nfq076 Nisbet, E. C., Cooper, K. E., & Ellithorpe, M. (2015). Ignorance or bias? Evaluating the ideological

and informational drivers of communication gaps about climate change. Public Understanding of Science, 24, 285–301. doi:10.1177/0963662514545909

Fact-Checking 373

*Nyhan, B., Porter, E., Reifler, J., & Wood, T. (2017). Taking corrections literally but not seriously? The effects of information on factual beliefs and candidate favorability. SSRN. https://papers. ssrn.com/sol3/papers.cfm?abstract_id=2995128.

Nyhan, B., & Reifler, J. (2010). When corrections fail: The persistence of political misperceptions. Political Behavior, 32, 303–330. doi:10.1007/s11109-010-9112-2

Nyhan, B., & Reifler, J. (2012). Misinformation and fact-checking: Research findings from social science. Washington, DC: New America Foundation.

Nyhan, B., & Reifler, J. (2015). The effect of fact-checking on elites: A field experiment on US state legislators. American Journal of Political Science, 59, 628–640. doi:10.1111/ajps.12162

Nyhan, B., Reifler, J., & Ubel, P. A. (2013). The hazards of correcting myths about health care reform. Medical Care, 51, 127–132. doi:10.1097/MLR.0b013e318279486b

O’Keefe, D. J. (2015). Message generalizations that support evidence-based persuasive message design: Specifying the evidentiary requirements. Health Communication, 30, 106–113. doi:10.1080/10410236.2014.974123

Paul, B., Salwen, M. B., & Dupagne, M. (2000). The third-person effect: A meta-analysis of the perceptual hypothesis. Mass Communication and Society, 3, 57–85. doi:10.1207/ S15327825MCS0301_04

*Pennycook, G., & Rand, D. G. (2017). The implied truth effect: Attaching warnings to a subset of fake news stories increases perceived accuracy of stories without warnings (Unpublished manuscript). Retrieved from: https://www.researchgate.net/profile/Gordon_Pennycook/publi- cation/ 321887941_The_Implied_Truth_Effect_Attaching_warnings_to_a_subset_of_fake_news_hea- dlines_increases_perceived_accuracy_of_headlines_without_warnings/links/ 5a38347b0f7e9b7c486f7f19/The-Implied-Truth-Effect-Attaching-warnings-to-a-subset-of- fake-news-headlines-increases-perceived-accuracy-of-headlines-without-warnings.pdf

Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York, NY: Springer- Verlag.

Pew Research Center. (2018, June). Distinguishing between factual and opinion statements in the news. Retrieved from: http://www.journalism.org/2018/06/18/distinguishing-between-factual- and-opinion-statements-in-the-news/

Pfau, M., & Louden, A. (1994). Effectiveness of adwatch formats in deflecting political attack ads. Communication Research, 21, 325–341. doi:10.1177/009365094021003005

PunditFact. (2018). CNN’s file. Retrieved from https://www.politifact.com/punditfact/tv/cnn/ Read, J. (2000). Assessing vocabulary. Oxford: Oxford University Press. Rokeach, M. (1968). Beliefs, attitudes, and values. San Francisco, CA: Jossey-Bass. Schumacher-Matos, E. (2012, October 28). Election 1: Fact checking the NPR fact checkers.

National Public Radio. Retrieved from http://www.washingtonpost.com/blogs/fact-checker /wp/2014/04/28/candidates-flinging-pinocchios-at-each-other

Shen, F., Sheer, V. C., & Li, R. (2015). Impact of narratives on persuasion in health communication: A meta-analysis. Journal of Advertising, 44, 105–113. doi:10.1080/00913367.2015.1018467

Shin, J., & Thorson, K. (2017). Partisan selective sharing: The biased diffusion of fact-checking messages on social media. Journal of Communication, 67, 233–255. doi:10.1111/jcom.12284

Shulman, H. C., & Sweitzer, M. D. (2018). Varying metacognition through public opinion ques- tions: How language can affect political engagement. Journal of Language and Social Psychology, 37, 224–237. doi:10.1177/0261927X17707557

Stencel, M. (2015). “Fact check this”: How U.S. politics adapts to media scrutiny. Reston, VA: American Press Institute. Retrieved from http://www.americanpressinstitute.org/factchecking- project/fact-checking-research/u-s-politics-adapts-media-scrutiny/

Stencel, M., & Griffin, R. (2018, February). Fact-checking triples over four years. Retrieved from https://reporterslab.org/fact-checking-triples-over-four-years/

Swift, A. (2016). Americans’ trust in mass media sinks to new low. Gallup. Retrieved from http:// www.gallup.com/poll/195542/americans-trust-mass-media-sinks-new-low.aspx.

374 Nathan Walter et al.

*Swire, B., Berinsky, A. J., Lewandowsky, S., & Ecker, U. K. (2017). Processing political misinformation: Comprehending the Trump phenomenon. Royal Society Open Science, 4, 160802. doi:10.1098/rsos.160802

Taber, C. S., Cann, D., & Kucsova, S. (2009). The motivated processing of political arguments. Political Behavior, 31, 137–155. doi:10.1007/s11109-008-9075-8

*Thorson, E. (2016). Belief echoes: The persistent effects of corrected misinformation. Political Communication, 33, 460–480. doi:10.1080/10584609.2015.1102187

Van Aelst, P., Strömbäck, J., Aalberg, T., Esser, F., De Vreese, C., Matthes, J., … Papathanassopoulos, S. (2017). Political communication in a high-choice media environment: A challenge for democracy? Annals of the International Communication Association, 41, 3–27. doi:10.1080/23808985.2017.1288551

Walter, N., & Murphy, S. T. (2018). How to unring the bell: A meta-analytic approach to correction of misinformation. Communication Monographs, 85, 423-441. doi:10.1080/ 03637751.2018.1467564

*Weeks, B. E. (2015). Emotions, partisanship, and misperceptions: How anger and anxiety mod- erate the effect of partisan bias on susceptibility to political misinformation. Journal of Communication, 65, 699–719. doi:10.1111/jcom.12164

Whitehead, J. L., Jr. (1968). Factors of source credibility. Quarterly Journal of Speech, 54, 59–63. doi:10.1080/00335636809382870

*Wintersieck, A. (2015). The rise and impact of fact-checking in U.S. campaigns (Doctoral dissertation). Arizona State University. Available from ProQuest database (UMI no. 3701377).

Wolfsfeld, G. (2011). Making sense of media and politics. New York, NY: Routledge. Wood, T., & Porter, E. (2019). The elusive backfire effect: Mass attitudes’ steadfast factual

adherence. Political Behavior, 41, 135-163. doi:10.1007/s11109-018-9443-y *York, C., Ponder, J., Humphries, Z., Goodall, C., Beam, M., & Winters, C. (2018). Beyond

(mis)perceptions: How fact-checking fake news affects perceptual accuracy, intent to share misinformation and EPE. Paper presented at the annual meeting of the National Communication Association, Salt Lake City, Utah.

*Young, D. G., Jamieson, K. H., Poulsen, S., & Goldring, A. (2018). Fact-checking effectiveness as a function of format and tone: Evaluating FactCheck.org and FlackCheck.org. Journalism & Mass Communication Quarterly, 95, 49–75. doi:10.1177/1077699017710453

Zaller, J. R. (1992). The nature and origins of mass opinion. New York, NY: Cambridge University Press.

Fact-Checking 375