English final assessment

Contents lists available at ScienceDirect

Library and Information Science Research

journal homepage: www.elsevier.com/locate/lisres

How college students evaluate and share “fake news” stories Chris Leeder⁎

School of Communication and Information, Rutgers, The State University of New Jersey, 4 Huntington Street, New Brunswick, NJ 08901-1071, USA

A B S T R A C T

The spread of “fake news” stories online has become a pressing concern in the United States and around the world in recent years. Social media platforms enable the rapid spread of such misinformation and also make evaluating the credibility of online information more difficult. Since college students are frequent users of social media, they are particularly likely to be exposed to fake news. A survey was conducted with 63 undergraduate students in which they identified and evaluated examples of both fake and real news stories and reported their associated information behaviors. Results showed correlations between accurate identification of fake news stories and specific critical evaluation behaviors and strategies. However, students were unable to accurately evaluate their own skills, and their willingness to share fake news stories on social media was not related to accurate identifications or evaluations of trustworthiness. This study contributes to the understanding of not just how accurately students evaluate fake news stories, but of the specific information-seeking behaviors and critical evaluation strategies that are associated with accurate identifications and evaluations and with willingness to share news stories on social media. Implications for educators and directions for future research are discussed.

1. Introduction

The growth and spread of misinformation and disinformation online have become an urgent concern in the United States and around the world in recent years. In the US there has been a nationwide discussion of “fake news,” especially in regard to the 2016 election and its po- tential implications for our democratic institutions and processes (Allcott & Gentzkow, 2017; Kristof, 2016; Ruddick, 2017; Tavernise, 2016). In a 2017 Gallup/Knight Foundation survey, 73% of Americans reported that the spread of inaccurate information on the Internet was a major problem, and a majority of U.S. adults consider fake news a threat to our democracy (Knight Foundation, 2018). Most Americans believe that fake news stories cause confusion about the basic facts of current issues and events (Barthel, Mitchell, & Holcomb, 2016). The rapid proliferation of online news sources and political opinion sites and the blurring of distinctions between them raises concerns about the vulnerability of democratic societies to fake news and other forms of misinformation (Lazer et al., 2017). Misinformation and fake news pose a threat to the existence of an informed electorate and citizenry which are crucial to an effective democracy (Caplan, Hanson, & Donovan, 2018; Legg & Kerwin, 2018). The ease at which disinformation about civic issues can spread and flourish online represents a threat to de- mocracy itself (Wineburg, McGrew, Breakstone, & Ortega, 2016).

A 2016 Pew Research Center survey found that nearly a quarter of American adults (23%) admit to sharing fake news in the past, either knowingly or not (Barthel et al., 2016). Many people who see fake news stories reported that they believe them (Silverman & Singer-Vine,

2016). One reason for this is that fake news articles are often published by sites that are deliberately designed to resemble or even impersonate legitimate, reputable news outlets (Knight Foundation, 2018; Marwick & Lewis, 2017; Ruddick, 2017). Not only is the existence and spread of fake news itself concerning, but evidence shows that coordinated campaigns to push fake news stories and other types of disinformation were conducted during the 2016 election (Timberg, 2017). Agents of the Russian Internet Research Agency participated in the intentional spread of fake news stories during the 2016 election, with the ostensible goal of interfering with the US election process (DiResta et al., 2018; Knight Foundation, 2018). In what has been called “an unprecedented foreign intervention in American democracy,” Russian agents used so- cial media platforms such as Facebook and Twitter to fuel distrust and increase political polarization (Shane, 2017).

In this confusing and potentially misleading media environment, college students are particularly likely to be exposed to fake news, since they are frequent users of social media. In a 2018 survey, 88% of 18- to 29-year-olds reported using social media (Smith & Anderson, 2018). In a study of 6000 college students at 11 universities in the US, 89% of respondents reported that they got their news from social media (Head, Wihbey, Metaxas, MacMillan, & Cohen, 2018). That study also found that 89% of respondents had followed news about national government and politics in the preceding week and 58% had shared or retweeted news in the preceding week (Head et al., 2018). Research also shows that 61% of millennials (ages 18–33) say they rely more heavily on Facebook for their political news than any other source (Gottfried & Barthel, 2015). Young adults are substantially more likely to use these

https://doi.org/10.1016/j.lisr.2019.100967 Received 2 March 2019; Received in revised form 16 June 2019; Accepted 1 July 2019

⁎ Corresponding author. E-mail address: [email protected].

Library and Information Science Research 41 (2019) 100967

Available online 11 July 2019 0740-8188/ © 2019 Elsevier Inc. All rights reserved.

T

social media platforms than adults in their mid- to late-20s (Smith & Anderson, 2018). For teens, the next generation of college students, social media use is nearly ubiquitous: 97% of 13- to 17-year-olds use at least one social media platform (Anderson & Jiang, 2018). Worryingly, a 2015 study found that over 60% of student respondents reported that they had shared misinformation online on social media (Chen, Sin, Theng, & Lee, 2015). This willingness to share misinformation means that students themselves may contribute to the spread of fake news.

2. Problem statement and research questions

Since college students are heavy users of social media, they are likely to often be exposed to fake news stories which circulate easily online. These students must make evaluative judgments about the be- lievability and trustworthiness of such stories and decide whether to share such stories with others. Previous research has studied college student's ability to identify fake news stories. However, their behaviors around identifying, evaluating, and sharing fake news stories on social media have not been studied. In addition, the ability of college students to accurately evaluate their own skills at identifying fake news stories has not been studied. Previous research has also documented college students' reliance on social media as a news source but has not speci- fically investigated any effect this behavior may have on their ability to accurately evaluate fake news. To address these gaps, this study in- vestigates the following research questions:

RQ1. What information-seeking behaviors relate to college students' ability to accurately identify real and fake news stories?

RQ2. What information-seeking behaviors relate to college students' ability to accurately evaluate the quality of real and fake news stories?

RQ3. How do college students' evaluations of news stories relate to their willingness to share them on social media?

These research questions were investigated through a survey of 63 college students in which they identified and evaluated examples of both fake and real news stories and reported their associated informa- tion-seeking behaviors.

3. Literature review

3.1. Defining “fake news”

Fake news is related to other types of incorrect information: mis- information (false or misleading information) and disinformation (false information that is purposely spread to deceive people) (Lazer et al., 2018). Misinformation may be unintentionally inaccurate, but disin- formation is deliberately false or misleading (Jack, 2017). Nearly all fake news can be considered disinformation, since it is created or spread with the intent to deceive (Knight Foundation, 2018). In recent times, the term “fake news” has become so broadly used as to lose its meaning (Borchers, 2017; Mikkelson, 2016). In the popular media, the term has become politicized with contested meanings that fall along partisan lines (Caplan et al., 2018). Recent studies have defined the term in various ways: as news articles that are intentionally and verifiably false and could mislead readers (Allcott & Gentzkow, 2017); problematic content using news signifiers (Caplan et al., 2018); misleading, decep- tive or incorrect information purporting to be real news (Howard, Kollanyi, Bradshaw, & Neudert, 2017); fabricated articles spreading falsehoods that nonetheless appeared to be credible news stories (Knight Foundation, 2018); fabricated information that mimics news media content in form but not in editorial norms and processes (Lazer et al., 2018); and sensational, partisan stories intended to increase readership and make money (Marwick & Lewis, 2017). Snopes.com, the well-known fact-checking website, defines fake news as fabricated stories intended to fool readers and generate advertising revenue (Mikkelson, 2016). For this study, fake news stories are considered to

be those that are fabricated with the intention to deceive viewers into thinking they are real news (Tandoc, Lim, & Ling, 2018).

3.2. Social media and fake news

Social media has been called “the lifeblood of fake news” since these platforms allow anyone to disseminate viral fake news to mass audi- ences easily and at low cost (Klein & Wueller, 2017). Concern about the spread of fake news focuses on the both the ubiquity of social media and the easy circulation of information that social media platforms afford due to their technical affordances (Allcott & Gentzkow, 2017). These digital platforms incentivize the spread of problematic information and enable it to circulate in novel and disorienting ways (Jack, 2017). The rapidity with which information travels through social media allows fake news to quickly spread unchecked and make it difficult to counter or correct (Lazer et al., 2017). These affordances of social media have created a new avenue for propaganda on the Internet, which is being used with increasing sophistication to spread disinformation (Ruddick, 2017).

A 2016 study showed that the most popular fake news stories were more widely shared on Facebook than the most popular mainstream news stories (Silverman, 2016a). A 2018 study reported that false in- formation on Twitter is typically retweeted by many more people, more rapidly and broadly, than true information, especially around political topics (Vosoughi, Roy, & Aral, 2018). Factors that influence the spread and virality of fake news include negative sentiment in the story (Hansen, Arvidsson, Nielsen, Colleoni, & Etter, 2011) and the novelty of the information (Vosoughi et al., 2018). Among the primary motiva- tions for people sharing misinformation are self-expression and socia- lizing, while accuracy and authoritativeness of the information are often not considered important (Chen et al., 2015) Youth in particular may prefer opinionated stories, which may be seen as more “authentic” than objective news (Marchi, 2012). In many cases, people share stories based on affective or emotional appeal more than factual accuracy, with the goal of supporting their pre-existing beliefs and signaling their identity to like-minded others (Marwick, 2018).

Social media users must make evaluative judgments about the credibility of information that they encounter online and make deci- sions about whether to share such information with others. Credibility has been defined as the believability of information and sources as evaluated by an information receiver (Fogg, 2003; Metzger, Flanagin, Markov, Grossman, & Bulger, 2015). Trustworthiness is also considered to be a key dimension of credibility perception, which relates to the perceived goodness of the source in terms of being truthful, fair, and unbiased (Fogg, 2003; Rieh, 2014). Individual users conceptualize and define credibility in their own terms and according to their own beliefs and understandings (Hilligoss & Rieh, 2008). However, assessing the credibility of information on social media is increasingly challenging due to structural changes in the information environment (Lazer et al., 2017). Online information sources often lack the filters and markers of institutional credibility and authority which promote reliability in tra- ditional print sources (Metzger, Flanagin, & Medders, 2010). Histori- cally, these markers of credibility were maintained by professional gatekeepers such as editors and reviewers (Rieh & Danielson, 2007). However, social media platforms enable the rapid, unchecked circula- tion of information among users with no significant third-party fil- tering, fact-checking, or editorial intervention (Allcott & Gentzkow, 2017). This leaves the responsibility of evaluating the credibility and trustworthiness of online information to the user rather than to tradi- tional expert intermediaries. The kinds of verification and fact-checking duties that used to be the responsibility of editors and journalists now fall on the shoulders of social media users themselves (Wineburg et al., 2016). In addition, distinctions between scholarly expert information and popular nonexpert information have become blurred on the In- ternet, due in part to the fact that all information online tends to look the same (Burkhardt, 2018). The Internet is also producing novel,

C. Leeder Library and Information Science Research 41 (2019) 100967

2

hybrid, and emergent genres that resist easy classification into the traditional categories of print formats and present even greater chal- lenges to users attempting to understand their meaning and relevance (Leeder, 2016). To add additional confusion and uncertainty to the problem, fake news sources often deliberately appropriate the look and feel of real news sources in order to add a veneer of legitimacy and credibility to their deception (Tandoc et al., 2018).

Critical information evaluation skills can help combat the effects of fake news by promoting more sophisticated information evaluation skills (Cooke, 2017). In universities and colleges, the most common way that students are introduced to these skills is through libraries and in- formation literacy classes taught by librarians. Critical evaluation skills have long been central to traditional information literacy instruction, but they are now receiving renewed interest with the increasing awareness of fake news online (Batchelor, 2017; Hoover, 2017; Huwe, 2017; Jacobson, 2017; Large, 2017). Several studies have found posi- tive associations between exposure to information literacy instruction and effective use of critical evaluation skills, including a positive cor- relation between exposure to credibility evaluation training and the use of analytical evaluation methods (Metzger et al., 2015), the use of a greater range of evaluation criteria in a more sophisticated way com- pared to a control group (Walton & Hepworth, 2011), more critical evaluation of sources (Hoffmann & LaBonte, 2012), and improved re- search behavior by students who have received critical evaluation training (Leeder & Shah, 2016). There is good news in that a recent Pew survey reported that 87% of Millennials view libraries as “a place that helps them find information that is trustworthy and reliable” (Geiger, 2017), and that 84% of youth surveyed nationally said they thought that they and their friends would benefit from instruction in how to tell if a given source of online news was trustworthy (Cohen, Kahne, Bowyer, Middaugh, & Rogowski, 2012).

3.3. Previous studies of fake news identification

Several studies have investigated people's ability to identify fake news. Metzger et al. (2015) conducted a study of 2747 11–18 year old Internet users, in which participants were shown a hoax website and asked how much they believed the information on the website. Results showed that 50.4% of students reported believing “at least some” to “a whole lot” of the information on the hoax website. Silverman and Singer-Vine (2016) surveyed 3015 US adults, in which participants were shown true and false news headlines related to the 2016 election and were asked to rate the accuracy of the claim in the headline. Results showed that 75% of respondents rated the fake news headline as ac- curate (Silverman & Singer-Vine, 2016). The survey also found that people who report relying on Facebook as a major source of news were more likely to rate fake news headlines as accurate than people who do not. Allcott and Gentzkow (2017) surveyed 1208 US adults, in which respondents were shown 15 news headlines about the 2016 election and asked if they believed the statements in the headlines were true. Analysis of demographic factors showed that people who spent more time consuming media, people with higher education, and older people have more accurate beliefs about news, while people who report relying on social media as their most important sources of election news were more likely to incorrectly believe false headlines. Pennycook and Rand (2017) conducted a series of three online studies with 1606 adults, in which participants were shown 15 fake and 15 real news headlines and asked to rate the headline's accuracy and report how willing they would be to share the news stories on social media. Results showed a positive correlation between the propensity to think analytically, as measured by the Cognitive Reflection Test, and the ability to differentiate fake news from real news. Wineburg et al. (2016) interviewed 25 college students at Stanford University as part of a large-scale study of 7804 students. Participants were shown two sites providing health informa- tion and were asked to determine whether the sites were trustworthy sources of information. One of the sites was from a legitimate non-

partisan medical organization and one was from a deceptive partisan advocacy organization. More than half of the participants concluded that the article from the partisan site was more reliable. The researchers concluded that students in the study were “easily duped” when at- tempting to evaluate information found on social media (p. 4). While these studies investigated people's ability to identify fake news stories, the participants' specific information-seeking behaviors associated with identifying, evaluating, and sharing fake news stories have not been studied. In addition, the ability of college students to accurately eval- uate their own skills at identifying fake news stories has not been stu- died. This study aims to address these gaps in the literature.

4. Method

4.1. Selection of news stories

Recent fake news stories were selected to present to participants for evaluation. Sources for listings of fake news items included Snopes.com, a well-known fact-checking website, and research published on Buzzfeed.com that listed the most frequently shared fake news stories on Facebook in the final three months of the 2016 election (Silverman, 2016b).1 Stories specifically related to the 2016 election were con- sidered outdated and not included. Also, stories with prurient or of- fensive headlines were excluded. Excluding stories with these topics resulted in a set of 6 fake news stories.

To balance out the fake stories, 6 real news stories were selected from reputable news sources that presented unusual or surprising headlines, with the goal that each story (fake or real) would require a comparable amount of effort to evaluate. Popular mainstream news sources such as CNN, MSNBC, or Fox were excluded to reduce the possibility of familiarity and potential bias toward the source. To pro- mote a balance of topics, both political and non-political stories were included, based on whether the story headline and content referred specifically to political topics or candidates. In total, 12 news stories were selected, 6 of which were real and 6 of which were fake. The story headlines, type, and IDs are shown below in Table 1. The URLs of the original stories are listed in Appendix 1.

Participants were shown 12 news stories (6 real and 6 fake) and were asked to rate each story as either “real” or “fake.” Correct iden- tifications were coded as 1 and incorrect identifications were coded as zero. Thus, the maximum score for correctly identifying all the stories was 12.

4.2. Participants

Study participants were recruited from two undergraduate in- troductory courses in Social Informatics and one undergraduate in- troductory course in Gender and Technology at a public university in the northeast U.S. Courses were selected through a convenience sample, since participants were to be compensated only by extra credit. A total of 72 participants began the study and 63 participants successfully completed it, for a completion rate of 88%. Participants took an online survey outside of class for extra credit. The survey took approximately 45–60 minutes to complete. Demographics of the participants are shown in Table 2. The age range of participants was 19 to 24 years with a mean of 20.78 years. By gender, participants were 79.37% male and 20.63% female. Most participants were in their junior year in college

1 Some of the most widely-shared election-related fake news stories have been subsequently debunked, such as the notorious “Pope Francis Endorses Donald Trump for President” and “I Was Paid $3500 To Protest Trump's Rally” hoaxes (Silverman, 2016b). Some of the fake news stories which have been debunked were deleted and are no longer available online. Many websites which supply fake news stories are short-lived, and some that delivered these notorious election-related stories no longer exist (Allcott & Gentzkow, 2017).

C. Leeder Library and Information Science Research 41 (2019) 100967

3

(52.4%) followed by those in their senior year (41.3%). By ethnicity, the highest percentages of participants were Asian or Asian-American (38.1%) and White (33.3%). The demographics were a reflection of the non-random selection of participants and the population of the classes from which participants were recruited.

4.3. Survey instrument

The study was conducted through the Qualtrics online survey tool. Participants responded to demographic questions regarding their age, gender, year in college, ethnicity, and their use of online news sources. Participants were also shown headlines from 6 fake news and 6 real news stories in random order. Each headline was hyperlinked to the original story in its original context as published online, which allowed participants to investigate the story and evaluate its quality. For each story, participants were asked to rate the story's believability and trustworthiness and to describe how they made their evaluations in an open-ended response field. They were also asked about their willingness to share the story on social media if they had received it from a friend. Next, participants were shown the complete list of news stories that they had evaluated and were asked to identify the stories were real or fake. Finally, they were asked to describe their online news reading habits. The time spent on each question was automatically recorded for all participants.

After evaluating the news stories, study participants responded to a questionnaire designed to measure the information-seeking behavior of undergraduate students developed by Timmers and Glas (2010). The questionnaire is based on activities related to information-seeking be- havior derived from the ACRL Information Literacy Competency Stan- dards for Higher Education: defining information problems, using sources, applying search strategies, evaluating information, and refer- ring to information. To develop the instrument, the authors conducted a literature review of prior information-seeking behavior research and

previous measurement instruments. Once an initial set of questions was collected, experts reviewed them to ensure content validity, factor analysis was used to ensure construct validity and reliability, and field testing was conducted. The instrument was found to be a reliable and valid measurement of information-seeking behavior (Timmers & Glas, 2010). For this study, only the sections of the questionnaire pertaining to “applying search strategies” and “evaluating information” were used, as these were relevant to the online information evaluation context of the study. Questions focus on specific behaviors during the online search process such as “I determine new search terms during the search process,” “I use the advanced search option,” and “I examine the results on subsequent result pages.” Response options were on a 4-point scale from “Rarely or never,” “Sometimes,” “Often,” and “Always.” The full text of the questionnaire used in this study is shown in Appendix 2.

The study instruments were pilot tested with 4 volunteer under- graduate student participants to verify the clarity and understandability of the instructions, questions, and format. Participants were recruited from introductory LIS courses and were paid $20. The pilot test em- ployed a think aloud method (Charters, 2003) in which participants completed the entire study procedure while being observed by the re- searcher and were asked to think aloud throughout, describing verbally their experience and any issues or challenges they encountered. Based on feedback from the participants, minor clarifications were made to the instructions and a few of the questions. The initial number of stories to be read during the survey (14) was reduced after the pilot test due to the amount of time required to complete the survey. The final survey consisted of 12 news stories (6 real, 6 fake) to be read and evaluated by participants.

5. Findings

5.1. News story identification

Overall, 63 subjects evaluated 12 stories each, for a total of 756 identifications (both real and fake stories). Results show that 472 or 62.43% of all identifications were correct and 284 or 37.57% were incorrect. Thus, nearly 40% of news stories were incorrectly identified as either real or fake. The most correctly identified story overall was “New $20 bill featuring Robert E. Lee to make early debut in southern U.S.” a fake news story correctly identified by 52 participants (82.54%). The least correctly identified story overall was “Harvard study proves Apple slows down old iPhones to sell millions of new models” a fake news story correctly identified by 29 participants (46.03%). The total correct identifications by story are shown in Table 3.

By story type (real or fake), 229 (60.58%) of real news stories identifications were correct and 243 (64.29%) of fake news story identifications were correct. Thus, participants performed slightly better at identifying fake news stories than they did at identifying real news stories. The most correctly identified real story was “National poll: A quarter of Millennials would prefer a meteor strike to 2016

Table 1 News story headline, story type, and story ID.

News story headline Story type Story ID

Hillary Clinton breaks news that 15,000+ women have contacted Emily's List about running for office Real Emily's List Meth-laced 7Up is killing people in Mexico Real 7 Up National poll: A quarter of Millennials would prefer a meteor strike to 2016 presidential candidates Real Millennials Trump's behavior similar to male chimpanzee, says Jane Goodall Real Goodall US Navy to replace joysticks on nuclear submarines with Xbox controllers Real Navy Woman gets life in prison for murder witnessed by parrot Real Parrot Fukushima to dump 770,000 tons of deadly nuclear waste into Pacific Ocean Fake Fukushima Harvard study proves Apple slows down old iPhones to sell millions of new models Fake iPhones Hillary Clinton in 2013: “I would like to see people like Donald Trump run for office; they're honest and can't be bought” Fake Clinton/Trump New $20 bill featuring Robert E. Lee to make early debut in southern U.S. Fake $20 bill Trump Mocks Trudeau For Celebrating Thanksgiving “6 Weeks Early” Fake Trump/Trudeau Trump plans to force Taliban to negotiating table by January 2025 Fake Taliban

Table 2 Participant demographics.

Category N %

Gender Male 50 79.37 Female 13 20.63

College year Sophomore 4 6.35 Junior 33 52.38 Senior 26 41.27

Ethnicity Asian/Asian American 24 38.10 White 21 33.33 Latino/Latina 8 12.70 Black/African American 2 3.17 Other/multiple races 8 12.70

C. Leeder Library and Information Science Research 41 (2019) 100967

4

presidential candidates” by 44 participants (69.84%) and the least correctly identified real story was “Meth-laced 7Up is killing people in Mexico” by 35 participants (55.56%). The most correctly identified fake story was “New $20 bill featuring Robert E. Lee to make early debut in southern U.S.” by 52 participants (82.54%) and the least correctly identified fake story was “Harvard study proves Apple slows down old iPhones to sell millions of new models” a fake news story correctly identified by 29 participants (46.03%). The total correct identifications by story type are shown in Table 4.

The total number of correct identifications by individuals was cal- culated. The mean was 7.49 (SD = 2.13) with results ranging from a high of 12 to a low of 4. Out of the 63 participants, only 3 correctly identified all 12 stories. The largest percentage of participants (25.40%) identified 6 out of the 12 total stories correctly. Thus, one quarter of the participants incorrectly identified half of the news stories. The total correct identifications by individual are shown in Table 5.

Overall, participants correctly identified 62.40% of all stories, with 60.58% of real news stories were correctly identified and 64.29% of fake news stories were correctly identified. Thus, nearly 40% of stories overall were incorrectly identified and participants performed slightly better at identifying fake news stories than real news stories.

5.2. News story evaluation

While evaluating the news stories, participants were asked to rate how believable and trustworthy they considered each story to be on a 4- point scale from “Very unbelievable/untrustworthy” to “Very believ- able/trustworthy.” Responses were coded inversely according to whe- ther they corresponded to a real or fake story. For example, ratings of “Very believable” for real stories were scored as 4 and ratings of “Very unbelievable” for real stories were rated as a 1. Ratings of “Very be- lievable/trustworthy” for fake stories were scored as 1 and ratings of “Very unbelievable/untrustworthy” for fake stories were scored as 4. Thus, the higher the score, the more accurately participants rated real stories as believable and trustworthy and fake stories as unbelievable and untrustworthy.

Participants were asked how likely they would be to share each story with friends on social media. Responses were on a 4-point scale from “Definitely Not” to “Definitely Yes” and were scored from 1 for “Definitely Not” to 4 for “Definitely Yes”. Results are shown in Table 5. Overall, fake news stories were rated as more believable and trust- worthy, but real stories were rated higher for willingness to share. Participants indicated that on average they would not be very likely to share the stories, with averages ranging between 1 (Definitely Not) and 2 (Probably Not). Thus, although they incorrectly evaluated fake stories on average, participants were more likely to share real news stories. The average scores for believability, trustworthiness, and willingness to share by story type are shown in Table 6.

Average scores for believability, trustworthiness, and willingness to share by individual story were calculated. The results are shown in Table 7. Of the 5 most accurately rated stories for both believability and trustworthiness (Clinton/Trump, $20 bill, Fukushima, Millennials, Ta- liban), 4 were fake and 1 was real (Millennials). Of the 5 highest-rated stories for willingness to share (Navy, Goodall, Parrot, iPhone, Millen- nials), 4 were real and 1 was fake (iPhone). The only story that received highly accurate ratings for believability and trustworthiness and high ratings for willingness to share was the real story Millennials (“National poll: A quarter of Millennials would prefer a meteor strike to 2016 presidential candidates”).

To determine if the participants' evaluation scores were consistent with their final determination of whether the story was real or fake, a Pearson two-tail correlation test was conducted on the accuracy of in- dividuals' believability and trustworthiness scores and their total cor- rect identifications. Results showed that the total number of correct identifications of stories correlated strongly to both the individual's accuracy in rating believability (r(61) = 0.669, p < .01) and trust- worthiness (r(61) = 0.707, p < .01). Thus, there was consistency in participants' evaluations of the stories between their first individual story evaluations and their later identifications of stories as real or fake. There was also a strong correlation between individual's average be- lievability and trustworthiness accuracy (r(61) = 0.824, p < .01) which shows that individuals were consistent in these two ratings.

A Pearson two-tail correlation test was conducted on individual's scores for willingness to share and their average accuracy of believ- ability and trustworthiness ratings. There was a strong negative corre- lation between average willingness to share and average accuracy of trustworthiness ratings (r(61) = −0.255, p = .043). There was no correlation between willingness to share and accuracy of believability ratings. This finding suggests that trustworthiness was not a factor in the participants' decision whether to share the news stories on social

Table 3 Correct identifications by story.

Story ID Story type N Correct % Correct

$20 bill Fake 52 82.54 Clinton/Trump Fake 46 73.02 Millennials Real 44 69.84 Goodall Real 43 68.25 Taliban Fake 43 68.25 Fukushima Fake 38 60.32 Navy Real 37 58.73 Parrot Real 36 57.14 Emily's List Real 35 55.56 Trump/Trudeau Fake 35 55.56 7 Up Real 34 53.97 iPhones Fake 29 46.03

Table 4 Correct identifications by story type.

Story ID Story type N Correct % Correct

Millennials Real 44 69.84 Goodall Real 43 68.25 Navy Real 37 58.73 Parrot Real 36 57.14 Emily's List Real 35 55.56 7 Up Real 34 53.97 $20 bill Fake 52 82.54 Clinton/Trump Fake 46 73.02 Taliban Fake 43 68.25 Fukushima Fake 38 60.32 Trump/Trudeau Fake 35 55.56 iPhones Fake 29 46.03

Table 5 Total correct identifications by individual.

N of correct identifications N of participants % of participants

12 3 4.76 11 4 6.35 10 6 9.52 9 6 9.52 8 8 12.70 7 11 17.46 6 16 25.40 5 5 7.94 4 4 6.35

Table 6 Average evaluation scores by story type.

Story type Believability Trustworthiness Willingness to Share

Real 2.65 2.57 1.94 Fake 2.86 3.08 1.59

C. Leeder Library and Information Science Research 41 (2019) 100967

5

media.

5.3. Information seeking behaviors

To investigate how individual behaviors may have affected in- dividual performance, the total amount of time that each of the 63 participants spent on evaluating the 12 news stories was calculated, and the total time spent was compared to their total number of correct story identifications. Since the time spent per individual was not normally distributed, a Spearman's correlation test was used. Results of the Spearman correlation indicated that there was a significant positive association between the amount of time spent on evaluation and the number of correctly identified news stories (rs(61) = 0.288, p = .022). Thus, those participants who spent more time evaluating tended to be more accurate in their judgments.

After evaluating the news stories, participants answered a survey about their information seeking behavior while searching for informa- tion for research assignments. Response options were on a 4-point scale from “Rarely or never,” “Sometimes,” “Often,” and “Always.” Responses were coded from 1 to 4 with “Always” being 4. To determine whether there was a correlation between reported behaviors and ability to correctly identify stories, a Pearson two-tail correlation test was conducted on each individual's responses to the behavioral ques- tionnaire and their total number of correct identifications. Results showed a significant positive correlation between the number of correct identifications of stories and the behavior “I examine the rest of the webpage to judge the reliability of the information” (r(61) =0.254, p = 0.046) and a significant negative correlation to the behavior “I use the top results from the list” (r(61) = −0.256, p = 0.043). Thus, par- ticipants who reported critically evaluating the source performed better at accurately identifying fake news stories.

A Pearson two-tail correlation test was also conducted on in- dividual's responses to the behavioral survey and their ratings of be- lievability, trustworthiness, and willingness to share the new stories. Results showed a significant positive correlation between the behavior “I use more than one source to answer my question” and accuracy of ratings for both believability (r(61) =0.258, p = .041) and trust- worthiness (r(61) =0.289, p = .022). The behavior “I scan through the results found” also showed a significant positive correlation to accuracy of ratings for both believability (r(61) =0.351, p = .005) and trust- worthiness (r(61) =0.309, p = .014). The behavior “I examine the rest of the webpage to judge the reliability of the information” showed a significant positive correlation to accuracy of ratings for trustworthi- ness (r(61) =0.335, p = .008). Thus, participants who reported criti- cally evaluation behaviors performed better at accurately evaluating the news stories.

The behavior “I select information that corresponds with my own opinion” showed a significant positive correlation to willingness to share (r(61) =0.313, p = .013) and a significant negative correlation to

accuracy of ratings of trustworthiness (r(61) = −0.263, p = .037). This result echoes the strong negative correlation between willingness to share and trustworthiness ratings above and suggests again that trust- worthiness is not an important factor in decisions to share news stories on social media, but rather personal beliefs and opinions are more important factors.

After completing the behavioral survey questions, participants gave open-ended text responses to the prompt “Thinking about your use of the Internet for finding information, how would you describe your online news reading habits?” Some of the responses showed dis- crepancies between participants' performance and their own self-rating of their abilities. For example, a participant who received a perfect score (12) on identifying stories commented: “After answering all these questions about my news reading habits, it makes me think that my habits aren't as good as I thought they were because I don't evaluate my sources that often and I never check the credibility of the author.” This participant expressed doubt about their abilities, although they per- formed perfectly, and mentions critical evaluation specifically. In con- trast, a participant who received the lowest score (4) commented: “I browse the headline and look at the format of the website. Links to other articles usually reveal the type of language and rhetoric the website uses. If it is politically biased or unreliable then the design of the website and the wording will show that.” This participant expresses self-confidence in their ability, although they performed poorly, and suggest an uncritical attitude toward online information because of the belief that quality is self-evident.

5.4. Use of online news sources

As part of the survey, participants were asked “Which of the fol- lowing online news sources do you follow on a regular basis?” For each option, the number of individuals who selected that particular option was totaled (out of 63 participants). Total responses and percentages are shown in Table 8.

Social media was by far the largest percentage of responses (88.89%), followed by national TV/cable news sites (44.44%), and major newspaper sites (41.27%). Participants were asked “How fre- quently do you use social media sites?” with response options from Never to Daily on a 5-point scale. The most frequent response was Daily (82.54%), followed by 4–6 times a week (9.52%), 2–3 times a week (6.35%), and Once a week (1.59%). No participants selected the re- sponse option Never. These findings support the findings of the research literature that students rely heavily on social media as a primary source for news.

In their open-ended responses to the prompt regarding online news reading habits, participants commented “I get all my news from social media,” “I just rely on social media to get my news,” and “I mostly get my news from social media.” This echoes the majority response above. In the comments, some specific social media sites not listed in the prompt were mentioned (Reddit, Snapchat, YouTube). Only seven participants (11.11%) mentioned broadcast news networks (CNN, MSNBC, ABC, Fox, BBC) in their open-ended comments. Only six par- ticipants (9.52%) mentioned mainstream journalistic sources (The New York Times, Washington Post, Wall Street Journal, Guardian, Reuters)

Table 7 Average evaluation scores by story.

Story ID Story type Believability Trustworthiness Willingness to Share

Clinton/Trump Fake 3.43 3.57 1.27 $20 bill Fake 3.16 3.29 1.44 Fukushima Fake 2.92 3.19 1.51 Taliban Fake 2.95 3.10 1.52 Millennials Real 2.89 2.84 1.87 Parrot Real 2.62 2.70 2.11 Trump/Trudeau Fake 2.35 2.68 1.87 Goodall Real 2.71 2.67 2.11 iPhone Fake 2.32 2.67 1.90 Navy Real 2.54 2.52 2.13 Emily's List Real 2.63 2.44 1.71 7Up Real 2.49 2.25 1.68

Table 8 Use of online news sources.

Type of news source N %

Social media sites (Facebook, Twitter) 56 88.89 National TV/cable news network sites (CNN, ABC, MSNBC, FOX) 28 44.44 Major newspaper sites (New York Times, Washington Post, LA

Times) 26 41.27

Blogs or other personal web pages 9 14.29 Local TV news station sites 9 14.29 Politically oriented websites (Politico, The Hill) 6 9.52

C. Leeder Library and Information Science Research 41 (2019) 100967

6

in their open-ended comments. A two-way factorial ANOVA was conducted on individuals' reported

use of news sources and their total correct identifications. No correla- tions were found, which may be due to the frequency of use of social media by most of the participants.

5.5. Performance by groups

To investigate individual differences in behavior that may relate to accuracy of story identification and evaluation, participants were di- vided into 3 equivalently-sized groups based on their total number of correct identifications. The high-performing group (N = 19) correctly identified 9–12 stories with an average of 10.2, the medium-performing group (N = 19) correctly identified 7–8 stories with an average of 7.4, and the low-performing group (N = 25) correctly identified 4–6 stories with an average of 5.5. A one-way between groups ANOVA test was conducted to compare the total correct identifications of the three groups. Results showed statistically significant difference between groups (F (2,60) = 181) at the p < .01 level. Post hoc comparisons using Tukey's HSD test indicated that the mean for correct identification for the high-performing group (M = 10.2, SD = 1.08) was significantly higher than the mean of the medium-performing group (M = 7.4, SD = 0.5) and the low-performing group (M = 5.5, SD = 0.8) at the p < .01 level. Thus, the high-performing group performed significantly better at accurately identifying the news stories.

The total amount of time spent on evaluation questions in minutes was compared between groups. Due to the skewed distribution of the data, a logarithmic transformation was used to normalize the data, resulting in normalized means of time spent for the high-performing group (M = 3.50, SD = 1.10), the medium-performing group (M = 3.21, SD = 1.01) and the low-performing group (M = 2.55, SD = 0.82). A one-way between groups ANOVA test showed that these normalized means were significantly different (F (2,58) = 5.50) at the p < .01 level. Post hoc comparisons using Tukey's HSD test indicated that the mean of time spent by the high-performing group was sig- nificantly higher than that of the low-performing group at the p < .05 level. Thus, the high-performing group spent significantly more time on conducting their evaluations.

The evaluation scores for believability, trustworthiness, and will- ingness to share were compared between groups. The average evalua- tion scores by group are shown in Table 9. As described above, higher scores are more accurate for both real and fake stories. A one-way be- tween groups ANOVA test showed that scores for believability were significantly different at the p < .01 level (F (2,58) = 15.61). Post hoc comparisons using Tukey's HSD test indicated that the mean score for believability of the high-performing group (M = 3.02, SD = 0.38) was significantly higher than the mean of the medium-performing group (M = 2.76, SD = 0.24) and the low-performing group (M = 2.54, SD = 0.21) at the p < .05 level. The ANOVA tests showed that the scores for trustworthiness were also significantly different at the p < .01 level (F (2,58) = 16.97). Tukey's HSD test indicated that the mean score for trustworthiness of the high-performing group (M = 3.10, SD = 0.30) was significantly higher than that of the medium-performing group (M = 2.80, SD = 0.28) and the low-per- forming group (M = 2.63, SD = 0.22) at the p < .05 level. Thus, the high-performing group performed significantly better at accurately

evaluating both the believability and trustworthiness of the news stories. There was no statistical difference between the groups on scores for willingness to share, which suggests that sharing is unrelated to accurate identifications of news stories as real or fake.

After responding to prompts on how believable and trustworthy they found each story, participants were asked “How did you decide?” with an open-ended response field. Some students responded that they had searched for other sources to double-check the stories (“I found other sources that supported their story,” “A quick search found this story on multiple news sites,” “a bit of background search on the sources”) or mentioned “Googling” to verify the story. Some also mentioned consulting the fact-checking site Snopes.com. In the high- performing group, 5 participants reported searching for other sources (26.32%), 4 reported using Google to search the topic (21.05%), and 4 reported using Snopes or fact-checking sites (21.05%). In the medium- performing group, 3 individuals reported searching for other sources (15.79%), 2 using Google (10.53%) and none reported using Snopes. No individuals in the low-performing group reported using any of these strategies. Additionally, 3 participants in the low-performing group reported that they based their decisions on “intuition” or “instinct” (12.00%) while no members of the other groups reported that strategy. Thus, while reported use of effective verification strategies overall was low, participants in the high-performing group were more likely to employ them. The number of individuals reporting these verification strategies per group are shown in Table 10.

In their open-ended responses to the prompt regarding online news reading habits, some participants in the high-performing group echoed the use of these verification strategies: “I tend to be fairly skeptical, which is why I often double check for the legitimacy of information I find online” and “I try not to believe everything that I find on the in- ternet unless I can verify from other news sources that the information I retained can be trusted.” In the high-performing group, 3 participants used the word “skeptical” (15.78%) and 2 participants used the word “verify” (10.52%). No members of the other groups used either of these terms. One member of the high-performing group stated “I like to ex- amine and critically think whether the info is real or not.” This was the only response that referenced those concepts.

The use of news sources reported by participants was analyzed by group to determine what relationships might exist between level of performance and news reading habits. The total number of participants out of each group who selected each option were calculated (partici- pants were allowed to select multiple responses so totals do not equal 100%). Results are shown in Table 11. A Chi-Square test was conducted on the reported use of news sources by the groups. No significant dif- ferences between the groups were found, which may be due to the frequency of use of social media by most of the participants.

While reliance on social media was high across all groups, the re- sponse data shows that a larger percentage of the low-performing group selected social media as a news source (92.00%) compared to the high- performing group (84.21%) and the medium-performing group (89.47%). In contrast, a larger percentage of the high-performing group selected national TV/cable news sites (57.89%) compared to the medium-performing (36.84%) and the low-performing group (40.00%). A larger percentage of the medium-performing group selected major newspaper sites as a news source (47.36%) than the higher-performing (36.84%) or the low-performing (40.00%) groups. Although the

Table 9 Average evaluation scores by group.

Evaluation High N = 19

Medium N = 19

Low N = 25

Believability 3.02 2.76 2.54 Trustworthiness 3.10 2.80 2.63 Willingness to Share 1.74 1.82 1.74

Table 10 Verification strategies by groups.

Strategy High N = 19

% Medium N = 19

% Low N = 25

%

Search/use other sources 5 26.32 3 15.79 0 0.00 Google the story 4 21.05 2 10.53 0 0.00 Snopes/fact-checking sites 4 21.05 0 0.00 0 0.00 Intuition/instinct 0 0.00 0 0.00 3 12.00

C. Leeder Library and Information Science Research 41 (2019) 100967

7

number of cases is small, the results suggest a difference in behavior between the groups regarding use of news sources. One possible ex- planation for this result is that exposure to news presented by outlets which follow traditional journalistic standards (national TV/cable news networks and major newspapers) may help participants learn to accu- rately identify fake news.

6. Discussion

6.1. Implications

The relation between college students' information-seeking beha- viors and their ability to accurately identify online news stories, their ability to accurately evaluate the believability and trustworthiness of online news stories, and their willingness to share these stories on social media was investigated. Research Question 1 addressed the relationship between information-seeking behaviors and college students' ability to accurately identify real and fake news stories. The findings showed correlations between the use of certain critical evaluation behaviors and accurate identifications of fake and real news stories: examining the rest of the webpage to judge the reliability of the information and spending more time on evaluation. In contrast, the behaviors of relying on the top items from a search results list and spending less time on evaluation correlated with incorrect identifications. These findings support the findings of previous research that more analytic thinkers are more likely to correctly identify fake stories (Pennycook & Rand, 2017). The results of this study provide support for educational efforts to improve and expand the use of these critical evaluation skills among college students. However, the participants' self-rating of their in- formation-seeking behaviors did not correlate to their actual perfor- mance in correctly identifying the news stories and thus they may not be able to accurately assess their own critical evaluation abilities. This finding is supported by research which shows that students with below- proficient IL skills hold inflated views of their own ability and sig- nificantly overestimate their performance (Georgas, 2014; Gross & Latham, 2007). It is an example of the Dunning-Kruger Effect (Kruger & Dunning, 1999), in which unskilled individuals overestimate their abilities while skilled individuals underestimate their performance. This phenomenon underscores the need for students to be taught ex- plicit strategies for credibility evaluation in order to accurately assess their own performance.

Research Question 2 addressed the relationship between college students' information-seeking behaviors and their ability to accurately evaluate the quality of real and fake news stories. The findings showed correlations between the use of certain critical evaluation behaviors and accurate evaluations of the believability and trustworthiness of the news stories. These behaviors include examining the rest of the web- page to judge the reliability of the information, using more than one source to answer a question, and scanning through the results found. Participants in the high-performing group were more likely to employ effective verification strategies such as searching for other sources on the topic, using Google to search the topic, and using Snopes or fact- checking sites. In contrast, the behavior of selecting information that corresponds with the participant's own opinion was negatively

correlated to accuracy of ratings of trustworthiness. These results pro- vide empirical evidence of the types of effective critical evaluation skills and strategies that educators should promote among college students when evaluating online news stories. This is supported by literature that has shown positive associations between exposure to information lit- eracy instruction and effective use of critical evaluation skills (Hoffmann & LaBonte, 2012; Leeder & Shah, 2016; Metzger et al., 2015; Walton & Hepworth, 2011).

Research Question 3 addressed the relationship between college students' evaluations of news stories and their willingness to share them on social media. The findings showed a negative correlation between their willingness to share and the accuracy of their trustworthiness ratings and a positive correlation between their willingness to share and the behavior of selecting information that corresponds with the parti- cipant's own opinion. There was no statistical difference between the groups on their willingness to share scores, which suggests that sharing is unrelated to accurate identifications of news stories as real or fake and to accurate evaluations of the stories' believability or trustworthi- ness. Thus, accuracy of evaluating the trustworthiness of a news story does not appear to influence the decision to share news stories online while selecting information that reinforces the students' own opinions does. This is supported by literature that suggests that factors unrelated to trustworthiness are more important in the decision to share stories through social media, such as emotional reaction and novelty (Hansen et al., 2011; Vosoughi et al., 2018). Given the ease with which mis- information can be spread through social media, these findings raise concerns around students' behavior around sharing news stories on social media. College students may not realize or understand the po- tential impacts of sharing news stories that they know are fake or un- trustworthy and may not consider the consequences for society of the rapid, unchecked spread of fake news stories through social media.

6.2. Limitations

This study has several limitations that affect its generalizability. Participants in this study may have been sensitized by the questions and the type of stories presented to pay particular attention to fake stories and may have thus been more critical than they would under real life conditions. This awareness may also have influenced their self-re- porting about their behavior. Another limitation on the findings is that it is impossible to determine how closely participants actually read the news stories, although lack of effort in this regard would echo findings that much of the public's engagement with news on social media in- volves only reading article headlines and not actually reading the story content (Gabielkov, Ramachandran, Chaintreau, & Legout, 2016). An additional limitation is that the demographic composition of the par- ticipants was unequal, with women, Hispanics, and African-Americans underrepresented. Since socio-demographic factors influence Internet usage (van Deursen & van Dijk, 2014) and choice in news sources (Tran, 2013), lack of demographic diversity limits the generalizability of the findings. A random sampling of students would provide a more diverse and representative group of participants and would increase the gen- eralizability of the findings.

Table 11 News sources used by groups.

New sources High N = 19

% Medium N = 19

% Low N = 25

%

National TV/cable news network sites (CNN, ABC, MSNBC, FOX) 11 57.89 7 36.84 10 40.00 Major newspaper sites (New York Times, Washington Post, LA Times) 7 36.84 9 47.36 10 40.00 Local TV news station sites 4 21.05 2 10.52 3 12.00 Politically oriented websites (Politico, The Hill) 1 5.26 2 10.52 3 12.00 Social media sites (Facebook, Twitter) 16 84.21 17 89.47 23 92.00 Blogs or other personal web pages 3 15.78 3 15.78 3 12.00

C. Leeder Library and Information Science Research 41 (2019) 100967

8

7. Conclusion

The findings of this study contribute to the field by identifying specific information-seeking behaviors that correlate with college stu- dents' ability to accurately identify and evaluate fake news stories and with their willingness to share these stories on social media. Previous research has examined how accurately people can identify fake news stories but has not identified specific behaviors associated with effective evaluations and what further actions participants would take after evaluating the story, i.e., whether to share it with others. The results of this study provide empirical evidence of specific information-seeking behaviors that support effective evaluations of online news stories: 1) examining the rest of the webpage to judge the reliability of the in- formation, 2) using more than one source to answer a question, 3) scanning through the search results found, and 4) spending adequate time identifying and evaluating online news stories. The results also identified effective verification strategies such as 1) searching for other sources to verify a story, 2) using Google to find information about a news story, and 3) using Snopes or fact-checking sites. These techniques of “lateral reading” (seeking sources of outside verification when evaluating information) are employed by professional fact checkers and are advocated for by the Stanford History Education Group as an ap- proach that students should employ when evaluating online informa- tion (Wineburg et al., 2016).

In addition, this study identified specific behaviors that related to poor performance in identifying and evaluating fake news stories: re- lying on the top items from a search results list and selecting in- formation that corresponds with the participant's own opinion. These ineffective behaviors may feel natural or instinctive to students, but educators should explain clearly and specifically why they are in- effective when dealing with online information, especially with news stories on social media. Educators should also draw upon college stu- dents' familiarity with “lateral reading” strategies and encourage stu- dents to integrate these strategies consistently as their standard practice when evaluating information online. While most of these skills and strategies are neither new nor surprising, taken together they provide an effective tool kit of critical evaluation behaviors for online news stories. Educators should emphasize the importance of consistently and systematically employing these behaviors when interacting with online information, especially news stories on social media.

Exposure to news presented by sources which follow traditional journalistic standards (national TV/cable news networks and major newspapers) may have helped the study participants to accurately identify fake news stories. Since studies have shown that college students rely on social media for their news more than any other sources, edu- cators should encourage students to reflect on the possible effects of this selective reliance. A study by Project Information Literacy found that students believe news is important to democracy and that traditional journalistic values still matter to them (Head et al., 2018). Educators should leverage these beliefs and values by helping students analyze and

understand what distinguishes different types of online news sources and why traditional journalistic standards are still important in the online information environment. Students should be encouraged to critically evaluate all types of online news sources, especially those found through social media. They should also be encouraged to reflect on their moti- vations are for sharing news stories that they know are fake or un- trustworthy and consider the consequences for society of the rapid spread of fake news stories through social media.

Several possible directions for future research arise from this study's findings. A larger study with a more diverse student population would allow for investigating college students' beliefs and attitudes toward evaluating the quality of online news stories in greater detail. A larger study would also allow for further investigation of any potential re- lationships between the types of news sources used by college students and their ability to accurately identify and evaluate fake news stories, taking into consideration the predominance of social media as a primary news source. Future research could investigate what characteristics of news stories may influence college students' decision to share them on social media. Future research could also investigate college students' understanding of the impacts of sharing misinformation online and the potential outcomes that such behavior may have on the attitudes, beliefs, and behaviors of the people that such stories are shared with.

The ability to locate, evaluate, and verify digital information about social and political issues has been defined as “civic online reasoning,” highlighting the importance of digital literacy to informed engagement in civic life (Breakstone, McGrew, Smith, Ortega, & Wineburg, 2018). The rapid spread of misinformation and fake news through social media poses a significant threat to the existence of an informed electorate which is fundamental to an effective democracy (Caplan et al., 2018; Legg & Kerwin, 2018). Unfortunately, the affordances of social media provide a fertile ground for the spread of misinformation about civic and political issues that is particularly dangerous for political debate in a democratic society (Lazer et al., 2017; Wineburg et al., 2016). The current structural environment of the Internet and social media make it difficult for students to accurately evaluate the quality of online in- formation and make it easy to spread misinformation. These challenges make it even more crucial that today's students learn to effectively evaluate the information they find and share online, especially news stories. As habitual users of social media platforms, today's college students must learn the critical evaluation skills and habits necessary to becoming informed citizens and empowered participants in our de- mocratic society.

Acknowledgements

Special thanks to Associate Professor Rebecca Reynolds for allowing her students to participate in this study.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Appendix 1. News story URLs

ID Source

Millennials https://www.uml.edu/News/press-releases/2016/odyssey-poll-10182016.aspx Goodall http://www.huffingtonpost.com/entry/trump-chimpanzee-behavior_us_57ddb84fe4b04a1497b4e512 Emily's List http://emilyslist.org/news/entry/hillary-clinton-breaks-news-that-15000-women-have-contacted-emilys-list-abo Navy https://www.stuff.co.nz/world/americas/97331833/us-navy-to-replace-joysticks-on-nuclear-submarines-with-xbox-controllers Parrot http://time.com/4920252/parrot-murder-woman-sentenced/ 7 Up http://www.grubstreet.com/2017/09/meth-laced-7up-is-killing-people-in-mexico.html Clinton/Trump http://therightists.com/hillary-clinton-in-2013-i-would-like-to-see-people-like-donald-trump-run-for-office-theyre-honest-and-cant-be-bought Taliban https://www.duffelblog.com/2017/08/trump-taliban-afghanistan-strategy/ Trump/Trudeau http://www.burrardstreetjournal.com/trump-mocks-trudeau-celebrating-thanksgiving-6-weeks-early/ $20 bill http://realnewsrightnow.com/2017/09/new-20-bill-featuring-robert-e-lee-make-early-debut-southern-u-s/ Fukushima http://yournewswire.com/fukushima-dump-nuclear-waste-ocean/ iPhones http://wikileaksnews.co/harvard-study-proves-apple-slows-old-iphones-sell-millions-new-models

C. Leeder Library and Information Science Research 41 (2019) 100967

9

Appendix 2: Information-seeking behavior scale (Timmers & Glas, 2010)

When I search for information for research assignments… [Applying Search Strategies] I search for general background information on the topic I use words from my questions as search terms I make a list of search terms before I start my search I determine the best places to search for this information I determine new search terms during the search process I use the advanced search option …and find little or no information on the topic, I adjust the question I examine the number of results found …and a search produces many results, I narrow my search I manage the information found so as to easily find it again later on [Evaluating Information] I scan through the information found I use the top results from the list I examine the results on subsequent result pages I select information that corresponds with my own opinion I select information which brings new thoughts to mind I select information which takes an effort (e.g. copying, visit library) I select information that is immediately accessible …on the Internet, I examine the date of the last update …on the Internet, I examine the rest of the webpage to judge the reliability of the information …on the Internet, I evaluate a website's purpose …on the Internet, I examine the qualifications of the author I carefully read the information found I use more than one source to answer my question I formulate the answer to the question in my own words

References

Allcott, H., & Gentzkow, M. (2017). Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31, 211–236.

Anderson, M., & Jiang, J. (2018). Teens' social media habits and experiences. Pew Research Center. November 28. Retrieved June 26, 2019, from: http://www.pewinternet.org/ 2018/11/28/teens-social-media-habits-and-experiences/#fn-21827-1.

Barthel, M., Mitchell, A., & Holcomb, J. (2016). Many Americans believe fake news is sowing confusion. Pew Research Center. December 15. Retrieved June 26, 2019, from: http://www.journalism.org/2016/12/15/many-americans-believe-fake-news-is- sowing-confusion.

Batchelor, O. (2017). Getting out the truth: The role of libraries in the fight against fake news. Reference Services Review, 45, 143–148.

Borchers, C. (2017). Fake news’ has now lost all meaning. The Washington Post.. February 9. Retrieved from https://www.washingtonpost.com/news/the-fix/wp/2017/02/09/ fake-news-has-now-lost-all-meaning.

Breakstone, J., McGrew, S., Smith, M., Ortega, T., & Wineburg, S. (2018). Teaching stu- dents to navigate the online landscape. Social Education, 82, 219–221.

Burkhardt, J. M. (2018). Truth, post-truth, and information literacy: Evaluating sources. In D. E. Agosto (Ed.). Information literacy and libraries in the age of fake news (94–105). Santa Barbara: Libraries Unlimited.

Caplan, R., Hanson, L., & Donovan, J. (2018). Dead reckoning: Navigating content mod- eration after fake news. Data & Society Research Institute. Retrieved from https:// datasociety.net/pubs/oh/DataAndSociety_Dead_Reckoning_2018.pdf.

Charters, E. (2003). The use of think-aloud methods in qualitative research: An in- troduction to think-aloud methods. Brock Education Journal, 12(2), 68–82.

Chen, X., Sin, S. C. J., Theng, Y. L., & Lee, C. S. (2015). Why students share mis- information on social media: Motivation, gender, and study-level differences. The Journal of Academic Librarianship, 41, 583–592.

Cohen, C. J., Kahne, J., Bowyer, B., Middaugh, E., & Rogowski, R. (2012). Participatory politics: New media and youth political action. Youth and Participatory Politics Research Network. Retrieved from http://ypp.dmlcentral.net/sites/all/files/publications/YPP_ Survey_Report_FULL.pdf.

Cooke, N. A. (2017). Post-truth, truthiness, and alternative facts: Information behavior and critical information consumption for a new age. The Library Quarterly, 87, 211–221.

DiResta, R., Shaffer, K., Ruppel, B., Sullivan, D., Matney, R., Fox, R., & Johnson B (2018). The tactics & tropes of the Internet Research Agency. New Knowledge. Retrieved June 26, 2019 from: https://www.newknowledge.com/disinforeport.

Fogg, B. J. (2003). Persuasive technology: Using computers to change what we think and do. San Francisco, CA: Morgan Kaufmann.

Gabielkov, M., Ramachandran, A., Chaintreau, A., & Legout, A. (2016). Social clicks: What and who gets read on Twitter? ACM SIGMETRICS Performance Evaluation Review, 44(1), 179–192.

Geiger, A. (2017). Most Americans - especially millennials - say libraries can help them find reliable, trustworthy information. Pew Research Center. August 30. Retrieved from http://www.pewresearch.org/fact-tank/2017/08/30/most-americans-especially- millennials-say-libraries-can-help-them-find-reliable-trustworthy-information/.

Georgas, H. (2014). Google vs. the library (part II): Student search patterns and behaviors when using Google and a federated search tool. Libraries and the Academy, 14, 503–532.

Gottfried, J., & Barthel, M. (2015). How millennials' political news habits differ from those of Gen X and Baby Boomers. Pew Research Center. Retrieved June 26, 2019, from: http://www.pewresearch.org/fact-tank/2015/06/01/political-news-habits-by- generation/.

Gross, M., & Latham, D. (2007). Attaining information literacy: An investigation of the relationship between skill level, self-estimates of skill, and library anxiety. Library and Information Science Research, 29, 332–353.

Hansen, L. K., Arvidsson, A., Nielsen, F.Å., Colleoni, E., & Etter, M. (2011). Good friends, bad news: Affect and virality in Twitter. In J. Park, L. Yang, T. Laurence, & C. Lee (Eds.). Future information technology: 6th International Conference on Future Information Technology, FutureTech 2011Berlin, Germany, Heidelberg: Springer (pp. 34-43).

Head, A. J., Wihbey, J., Metaxas, P. T., MacMillan, M., & Cohen, D. (2018). How students engage with news: Five takeaways for educators, journalists, and librarians. Project Information Literacy Research Institute. Retrieved June 26, 2019, from https://www. projectinfolit.org/news_study.html.

Hilligoss, B., & Rieh, S. Y. (2008). Developing a unifying framework of credibility as- sessment: Construct, heuristics, and interaction in context. Information Processing & Management, 44, 1467–1484.

Hoffmann, D. A., & LaBonte, K. (2012). Meeting information literacy outcomes: Partnering with faculty to create effective information literacy assessment. Journal of Information Literacy, 6, 70–85.

Hoover, A. (2017). The fight against fake news is putting librarians on the front line and they say they're ready. The Christian Science Monitor. February 15. Retrieved from https:// www.csmonitor.com/The-Culture/2017/0215/The-fight-against-fake-news-is- putting-librarians-on-the-front-line-and-they-say-they-re-ready.

Howard, P. N., Kollanyi, B., Bradshaw, S., & Neudert, L. M. (2017). Social media, news and political information during the US election: Was polarizing content concentrated in swing states? Oxford Internet Institute. Retrieved June 26, 2019, from: https://comprop.oii. ox.ac.uk/research/working-papers/social-media-news-and-political-information- during-the-us-election-was-polarizing-content-concentrated-in-swing-states/.

Huwe, T. (2017, March 13). Fake news and the librarian's duty of care. Information Today. Retrieved from http://www.infotoday.eu/Articles/Editorial/Featured-Articles/The- USA-Desk-Fake-news-and-the-librarians-duty-of-care-117248.aspx.

Jack, C. (2017). Lexicon of lies: Terms for problematic information. Data & Society Research Institute. Retrieved June 26, 2019, from: https://datasociety.net/output/lexicon-of- lies/.

Jacobson, L. (2017). The smell test: In the era of fake news, librarians are our best hope. School Library Journal, 63(1), 24–29.

C. Leeder Library and Information Science Research 41 (2019) 100967

10

Klein, D., & Wueller, J. R. (2017). Fake news: A legal perspective. Journal of Internet Law, 20(10), 6–13.

Knight Foundation (2018). Disinformation, “fake news” and influence campaigns on Twitter. Retrieved June 26, 2019, from https://knightfoundation.org/reports/ disinformation-fake-news-and-influence-campaigns-on-twitter.

Kristof, N. (2016). Lies in the guise of news in the Trump era. The New York Times. November 13. Retrieved from http://www.nytimes.com/2016/11/13/opinion/ sunday/lies-in-the-guise-of-news-in-the-trump-era.html.

Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in re- cognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121–1134.

Large, J. (2017). Librarians take up arms against fake news. The Seattle Times.. February 6. Retrieved from http://www.seattletimes.com/seattle-news/librarians-take-up-arms- against-fake-news.

Lazer, D., Baum, M., Benkler, Y., Berinsky, A., Greenhill, K., Menczer, F., ... Zittrain, J. (2018). The science of fake news. Science, 359(6380), 1094–1096 March 9.

Lazer, D., Baum, M., Grinberg, N., Friedland, L., Joseph, K., & Mattsson, C. (2017). Combating fake news: An agenda for research and action. Shorenstein Center on Media, Politics and Public Policy. Retrieved June 26, 2019, from: https://shorensteincenter. org/combating-fake-news-agenda-for-research.

Leeder, C. (2016). Student misidentification of online genres. Library & Information Science Research, 38, 125–132.

Leeder, C., & Shah, C. (2016). Practicing critical evaluation of online sources improves student search behavior. Journal of Academic Librarianship, 42, 459–468.

Legg, H., & Kerwin, J. (2018). The fight against disinformation in the U.S.: A landscape analysis. Shorenstein Center on Media, Politics, and Public Policy. Retrieved June 26, 2019, from: https://shorensteincenter.org/the-fight-against-disinformation-in-the-u- s-a-landscape-analysis/.

Marchi, R. (2012). With Facebook, blogs, and fake news, teens reject journalistic objec- tivity. Journal of Communication Inquiry, 36, 246–262.

Marwick, A. (2018). Why do people share fake news? A sociotechnical model of media effects. Georgetown Law Technology Review, 2, 474–512.

Marwick, A., & Lewis, R. (2017). Media manipulation and disinformation online. Data & Society Research Institute. Retrieved from https://datasociety.net/output/media- manipulation-and-disinfo-online/.

Metzger, M. J., Flanagin, A. J., Markov, A., Grossman, R., & Bulger, M. (2015). Believing the unbelievable: Understanding young people's information literacy beliefs and practices in the United States. Journal of Children and Media, 9, 325–348.

Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60, 413–439.

Mikkelson, D. (2016, November 17). We have a bad news problem, not a fake news problem. Retrieved June 26, 2019, from https://www.snopes.com/2016/11/17/we- have-a-bad-news-problem-not-a-fake-news-problem/

Pennycook, G., & Rand, D. G. (2017). Who falls for fake news? The roles of analytic thinking, motivated reasoning, political ideology, and bullshit receptivity. SSRN. Retrieved June 26, 2019, from: https://ssrn.com/abstract=3023545.

Rieh, S. Y. (2014). Credibility assessment of online information in context. Journal of Information Science Theory and Practice, 23, 6–17.

Rieh, S. Y., & Danielson, D. R. (2007). Credibility: A multidisciplinary framework. Annual Review of Information Science and Technology, 41, 307–364.

Ruddick, G. (2017). Experts sound alarm over news websites' fake news twins. The Guardian.

August 18. Retrieved from https://www.theguardian.com/technology/2017/aug/ 18/experts-sound-alarm-over-news-websites-fake-news-twins.

Shane, S. (2017). The fake Americans Russia created to influence the election. The New York Times. September 7. Retrieved from https://www.nytimes.com/2017/09/07/us/ politics/russia-facebook-twitter-election.html.

Silverman, C. (2016a). This analysis shows how fake election news stories outperformed real news on Facebook. BuzzFeed News. Retrieved June 26, 2019, from https://www. buzzfeed.com/craigsilverman/viral-fake-election-news-outperformed-real-news-on- facebook?utm_term=.uhMQvJOg0#.vtn9VLJ5m.

Silverman, C. (2016b). Here are 50 of the biggest fake news hits on Facebook from 2016. BuzzFeed News. Retrieved from https://www.buzzfeed.com/craigsilverman/top-fake- news-of-2016?utm_term=.te3Zo9871#.ha7EaQWV9.

Silverman, C., & Singer-Vine, J. (2016). Most Americans who see fake news believe it, new survey says. BuzzFeed News. December 6. Retrieved June 26, 2019, from https:// www.buzzfeed.com/craigsilverman/fake-news-survey?utm_term=.udX5WZ8wP#. orK39vJwR.

Smith, A., & Anderson, M. (2018). Social media use in 2018. Pew Research Center. March 1. Retrieved from http://www.pewinternet.org/2018/03/01/social-media-use-in- 2018/.

Tandoc, E. C., Lim, Z. W., & Ling, R. (2018). Defining fake news. Digital Journalism, 6(2), 137–153.

Tavernise, S. (2016). As fake news spreads lies, more readers shrug at the truth. The New York Times.. December 6. Retrieved from https://www.nytimes.com/2016/12/06/ us/fake-news-partisan-republican-democrat.html.

Timberg, C. (2017). Spreading fake news becomes standard practice for governments across the world. The Washington Post.. July 17. Retrieved from https://www. washingtonpost.com/news/the-switch/wp/2017/07/17/spreading-fake-news- becomes-standard-practice-for-governments-across-the-world.

Timmers, C. F., & Glas, C. A. (2010). Developing scales for information-seeking beha- viour. Journal of Documentation, 66, 46–69.

Tran, H. (2013). Does exposure to online media matter? The knowledge gap and the mediating role of news use. International Journal of Communication, 7, 831–852.

Van Deursen, A. J., & Van Dijk, J. A. (2014). The digital divide shifts to differences in usage. New Media & Society, 16, 507–526.

Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359, 1146–1151.

Walton, G., & Hepworth, M. (2011). A longitudinal study of changes in learners' cognitive states during and following an information literacy teaching intervention. Journal of Documentation, 67, 449–479.

Wineburg, S., McGrew, S., Breakstone, J., & Ortega, T. (2016). Evaluating information: The cornerstone of civic online reasoning. Stanford Digital Repository. Retrieved June 26, 2019, from: http://purl.stanford.edu/fv751yt5934.

Chris Leeder is a lecturer in the School of Communication and Information at Rutgers, The State University of New Jersey, USA. He holds a PhD from the University of Michigan. His research interests include information literacy, online credibility evaluation, educa- tional technology, and fake news. His research has been published in Library & Information Science Research, Journal of Information Science, Journal of Academic Librarianship, Reference and User Service Quarterly, and College & Research Libraries.

C. Leeder Library and Information Science Research 41 (2019) 100967

11