Please only use sources that provided from me. I have uploaded 6 sources, please choose 4 of them as 4 sources for this paper.

Contents lists available at ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier.com/locate/comphumbeh

“THE RUSSIANS ARE HACKING MY BRAIN!” investigating Russia's internet research agency twitter tactics during the 2016 United States presidential campaign

Darren L. Linvilla,∗, Brandon C. Boatwrightb, Will J. Grantc, Patrick L. Warrend

a Department of Communication, Clemson University, USA b School of Advertising and Public Relations, University of Tennessee Knoxville, USA c Center for Public Awareness of Science, Australia National University, Australia d John E. Walker Department of Economics, Clemson University, USA

A B S T R A C T

This study analyzed tweets from handles associated with the Russian Internet Research Agency in an effort to better understand the tactics employed by that organization on the social media platform Twitter in their attempt to influence U.S. political discourse and the outcome of the 2016 U.S. Presidential election. We sampled tweets from the month preceding the election and analyzed to understand the qualitative nature of these tweets as well as quantitative differences between how types of IRA Twitter accounts communicated. Seven categories of tweet behavior were identified: attack left, support right, attack right, support left, attack media, attack civil institutions, and camouflage. While camouflage was the most common type of tweet (52.6%), descriptive analyses showed it was followed by attack left (12%) and support right (7%). A variety of quantitative differences were shown between how account types behaved.

In February 2018, the U.S. Justice Department indicted 13 Russian nationals, listing them – and the organization they worked for – as central to a Russian state effort to interfere with the 2016 U.S. Presidential election (Barrett, Horwitz, & Helderman, 2018). According to the indictment, beginning in 2014 the Internet Research Agency (IRA) of St. Petersburg – owned by Russian oligarch Yevgeny Prigozhin and widely held to be a tool of the Russian state (Shane & Mazzetti, 2018, pp. 1–11) – began to sow discord in the U.S. political system.

With this indictment it is by now widely accepted that the IRA played a significant, and perhaps even game-changing, role in the 2016 United States' Presidential Election (Jamieson, 2018). At the heart of the IRA's efforts was a sophisticated campaign using social media platforms to sow division, discontent, and disconnection with reality among US political discussions. While these efforts draw on a long history of Russian (and indeed Soviet) attempts to infiltrate US political discourse, the 2016 campaign marked a radical shift in both tactics and impact. Never before had a foreign power been able to so successfully infiltrate American democracy; never before had social media been so weaponized.

Yet while general knowledge of this campaign and its goals is now widespread, details of the strategic and tactical choices made by the IRA remain far from fully documented. One narrative suggests the IRA were political opportunists, playing ideologies against one another in an effort to sow greater division and weaken both political parties

(Graff, 2018). A slightly different narrative of events, however, suggests that while the IRA did play both sides, they did not do so equally and they in fact worked to reinforce Donald Trump's rhetoric while weak- ening support for Hillary Clinton (Shane & Mazzetti, 2018, pp. 1–11). This article works to understand the truth of these differing narratives and presents an analysis of a sample of 3 million now deleted tweets and 3841 Twitter accounts identified by the U.S. House Intelligence Committee as associated with IRA activity (Permanent Select Committee on Intelligence, 2018), with the goal of exploring how the Kremilin's strategy manifested in specific actions on social media in 2016.

At heart, if security agencies and political actors throughout the democratic world are to detect and deter such action in the future, it is crucial that we understand the pattern of such strategic social media activity, and develop tools to resist it when it emerges.

1. The sword and the shield

In August 2018 the FBI launched a new website focused on com- bating foreign influence. Marking the occasion, the press release an- nouncing the launch noted that “Foreign influence operations—which include covert actions by foreign governments to influence U.S. poli- tical sentiment or public discourse—are not a new problem” (Federal Bureau of Investigations, 2018). While it is possible that the efforts of

https://doi.org/10.1016/j.chb.2019.05.027 Received 4 February 2019; Received in revised form 13 May 2019; Accepted 21 May 2019

∗ Correponding author. E-mail address: [email protected] (D.L. Linvill).

Computers in Human Behavior 99 (2019) 292–300

Available online 28 May 2019 0747-5632/ © 2019 Elsevier Ltd. All rights reserved.

T

Russia's IRA during the 2016 presidential election mark a high water mark of foreign influence in US politics, the FBI's assertion is worth noting. In particular, it is crucial to see the IRA's tactics as a direct continuation of the work – and thinking – of the Soviet intelligence services throughout the Cold War. Here “in addition to collecting in- telligence and producing politically correct assessments of it,” Andrew and Mitrokhin have noted, “the KGB also sought to influence the course of world events by a variety of ‘active measures’ (aktivinyye meropriatia) ranging from media manipulation to ‘special actions’ involving various degrees of violence” (1999, p. 292). Some of these ‘active measures’ can be seen in the work of the IRA.

One example of such “active measures” with direct parallels to the 2016 election supplied by Andrew and Mitrokhin (1999) took place in 1960. At the time Khurshchev “regarded the Republican candidate, Richard Nixon, as a McCarthyite friend of the Pentagon hawks, and was anxious that Kennedy should win” (p. 236). This anxiety led to the KGB's Washington resident, Aleksandr Semyonovich Feklisov being directed to propose diplomatic or propaganda initiatives to facilitate Kennedy's victory. To achieve this the residency “tried to make contact with Robert Kennedy but was politely rebuffed” (p. 236). Later in the decade this same desire to stop Richard Nixon winning the White House led to the Kremlin secretly offering to fund the campaign of the De- mocratic candidate Hubert Humphrey (Andrew & Mitrokhin, 1999).

Another historic case with possible parallels to IRA activity involves Soviet involvement in the U.S. civil rights movement (Ioffe, 2017). Andrew and Mitrokhin (1999) note that Martin Luther King Jr. was probably, “the only prominent American to be the target of active measures by both the FBI and the KGB” (1999, p. 308). For the KGB, the civil rights movement seemed ripe with opportunities for greater poli- tical unrest in the US – yet King himself was an obstacle, linking the movement not with a worldwide struggle against American im- perialism, but with the fulfilment of the American dream. In August 1967 the KGB commenced working to replace King with a more radical leader. The first step included “placing articles in the African press, which could then be reprinted in American newspapers, portraying King as an ‘Uncle Tom’ who was secretly receiving government sub- sidies to tame the civil rights movement and prevent it threatening the Johnson administration” (1999, p. 309). In line with this, a range of other active measures were conducted with the goal of further stoking racial tensions. Documents purporting to be from the John Birch So- ciety and the Jewish Defence League calling for violence against the black community were forged and sent to black community leaders (Andrew & Mitrokhin, 1999).

Soviet Intelligence agencies spent much of the Cold War engaged in a range of active measures against their adversaries in the West. Other stories pushed by these operations included conspiracy theories sur- rounding the Kennedy assassination in the sixties and later the AIDS epidemic in the eighties (Andrew & Mitrokhin, 1999; Boghardt, 2009; Romerstien, 2001). The point of this brief examination is not to provide exhaustive coverage, but instead to point thematic threads: lines of attack pursued to some effect by the Soviets, and returned to by the IRA. In particular, we can see rhetorical attacks on American media and American institutions of governance, a willingness to insert fraud and forgery into the news cycle, a fomenting of conspiratorial thinking for its own sake, and a rejection of scientific consensus. Andrew and Mi- trokhin believe this approach stemmed from a paranoid view of reality in the KGB – perhaps that is true. Yet it is also true that it was strate- gically useful in these attacks in the United States to undermine the very concept of a shared reality.

2. Disinformation and the 2016 U.S. Presidential campaign

Allcott and Gentzkow (2017) assessed the extent to which in- dividual users were exposed to fake news during the 2016 election cycle. They found that “the average US adult read and remembered on the order of one or perhaps several fake news articles during the

election period, with higher exposure to pro-Trump articles than pro- Clinton articles” (p. 232). Allcot and Gentzkow's findings are consistent with the broader narrative suggesting social media played a major role in how issues rose to societal prominence during the election. Salient for the purposes of this study is the allegation that it was strategically employed by the Russian government for the express purposes of un- dermining the democratic process in the U.S.

In his testimony before the Senate Armed Services Committee on April 27, 2017, Rand Waltzman of the RAND Corporation argued:

State sponsored propaganda and disinformation have been in ex- istence for as long as there have been states. The major difference in the 21st century is the ease, efficiency, and low cost of such efforts. Because audiences worldwide rely on the Internet and social media as a primary source of news and information, they have emerged as an ideal vector of information attack. (p. 4).

Whereas cybersecurity is primarily concerned with technical fea- tures (e.g., defenses against denial of service attacks, bots, intellectual property theft, and other attacks that typically take advantage of se- curity vulnerabilities), little attention has been paid to addressing psychosocial effects of misinformation and false influence. Waltzman suggested this concept is reflective of a separate but related form of cyber threat he called cognitive hacking, which is predicated on two primary factors: (1) the unprecedented speed and extent of disin- formation distribution, and (2) the audience's cognitive vulnerability, or the premise that the audience is already predisposed to accept [misinformation] because it appeals to existing fears or anxieties (p. 3).

Recent scholarship by Broniatowski et al. (2018) found that Russian trolls and Twitter bots promoted discord by spreading anti-vaccination messages while masquerading as legitimate users. The authors suggest exposure to misinformation online regarding vaccines has significant consequences, such as increased vaccination hesitancy or delay, de- creased trust among vaccine-hesitant parents of healthcare providers, and promotion of distrust in the scientific community shaking con- fidence in vaccination. Prior research has demonstrated the capacity for social media – and especially Twitter – users to participate in con- versations around salient issues (Starbird et al., 2015), propagate ru- mors (Maddock et al., 2015), and deliberately spread misinformation (Huang, Starbird, Orand, Stanek, & Pedersen, 2015).

Finally, Linvill and Warren (2019) take an overarching view of the IRA effort over the more than eight-year period of the IRA operation. They show that there are several distinct types of IRA troll accounts on Twitter, which are differentiated by content theme. They further show that these account types are specialized in that different types are especially active at different times, place themselves differentially in the social network, use different clients to post their content, and use different mixes of retweets versus original content. But other than to establish the account types, they perform no analysis on the content of the individual tweets.

3. Agenda building

History gives several examples of foreign governments attempting to influence populations of other nations. Japan's Tokyo Rose broad- casted to U.S. troops throughout World War II, and Voice of America has for decades been a global mouthpiece of the U.S. government aimed across borders. The work of the IRA, however, has extended propa- gandized broadcasts to social media in a new and covert way. Research exploring propaganda has, to a great extent, focused on the form and source of the messaging. As early as Lasswell, 1927 defined propaganda as “the management of collective attitudes by the manipulation of significant symbols” (p. 627). Lasswell focused his discussion of pro- paganda on the propagandist and the form their chosen symbols em- body. Herman and Chomsky (1988) discussed the systems that manu- facture propaganda. Their work focused on issues of ownership and funding of messages. In building an understanding of propaganda di- rected through social media, however, we feel it is important to

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

293

understand not simply the message form and source, but also the pro- cesses at work. With this in mind, we contend the purpose of much of the work of state sponsored distributers of disinformation, such as the IRA, is one of agenda-building.

Cobb and Elder (1971) defined agenda-building as the process by which actors endeavor to move issues from their own agenda onto the agendas of policy makers. Agenda-building theory emerged as a para- digm shift in political science away from classical democratic theory which focused predominantly on the assumption that public policy- makers advanced the interests of civically engaged constituents via an autonomous press. Instead, Cobb and Elder proposed:

We are raising the basic question of where public-policy issues come from … How is an agenda built, and who participates in the process of building it? Assuming that the balance of social forces influencing, if not controlling, the content of the political agenda at any point in time is necessarily biased to the advantages of some and the disadvantage of others, how may this balance be changed and with what consequences? (p. 905).

Agenda-building upended traditional assumptions of political issue salience – shifting the focus away from policymakers toward polities themselves. Cobb, Ross, and Ross (1976) extended this line of reasoning with the outside initiative model of agenda building, which applies to situations in which a group outside of the government structure (1) articulates a grievance, (2) tries to expand interest in the issue to en- ough other groups in the population to gain a place on the public agenda, in order to (3) create sufficient pressure on decision makers to force the issue onto the formal agenda for their serious consideration (p. 132). The focus on nongovernmental groups raising issues and jock- eying for salience among the general public transferred the agenda- building process to the populace rather than the policymakers.

Denham (2010) defined three distinct but overlapping categories of work being done using differing conceptualizations and foci of agenda building theory: policy agenda building, media agenda building, and public agenda building. Policy agenda building focuses on how issues are created, expanded upon, and subsequently enter the policy agenda. Media agenda building focuses on how media agendas are built, “re- flecting institutional imperatives and an ongoing negotiation between media personnel and their sources of information” (p. 311). Finally, public agenda building applies to “behavioral responses to mass and interpersonal communication. Examples of such responses might in- clude voting for a particular policy action, attending an event, or of- fering financial support to a social movement” (p. 316). It is this third category, public agenda building, that relates most directly to the work of the IRA.

While IRA social media activity was directed at both politicians (Gallagher, 2018) and journalists (Lukito & Wells, 2018), on balance it seems probable the IRA's primary goals were related to public agenda building. The IRA focused on divergent, often seemingly contrary agenda in their disinformation campaigns. Influencing votes, atten- dance at events, and support for particular social movements were central, however, in all of their social media efforts (Shane & Mazzetti, 2018, pp. 1–11). Parmelee (2014) found evidence showing that poli- tical posts on Twitter have the potential to influence mass publics. Si- milarly, Kahne and Bowyer (2018) found strong support for the premise that online activity fosters political participation and It is through this mechanism that it seems they hoped to ultimately influence the policy agenda of the United States government.

The purpose of this study is to expand upon previous work ana- lyzing IRA activity prior to the 2016 U.S. Presidential election and specifically focus on the content of the IRA output in the month prior to the election. Previous work by Linvill and Warren (2019) qualitatively analyzed IRA tweets at the account level and explained thematic forms IRA accounts took. In our current research we analyzed IRA tweets at the tweet level in an attempt to understand the activity in which these themed account types engaged in and if it differed between types. By analyzing IRA tweets both qualitatively and quantitatively we hope to

better understand the tactics employed by that organization in their attempt to influence U.S. political discourse and the outcome of the election. The goal of this research was to build a better understanding of the agenda setting goals of the IRA; which narrative of their activity is most correct? Did their public agenda building activity aim to sow division and weaken the agendas of both Republicans and Democrats or did the IRA work to foster greater support for one candidate and in- fluence U.S. policy through their election? With this in mind, we ex- plored the following overarching research question:

RQ: How, if at all, did Russia's IRA strategically employ Twitter to influence the United States public agenda prior to the 2016 Presidential election.

To address this question, we asked the following two sub-questions:

1. What tactics did Russia's IRA employ in the content of Tweets originating in the month prior to the 2016 U.S. Presidential elec- tion? 2. What differences, if any, exist between account types regarding the content of Russian IRA English language tweets in the month prior to the 2016 U.S. Presidential election?

4. Method

This study employed an exploratory sequential mixed methods de- sign (Creswell, 2014). Research began with an initial qualitative phase in which we performed a content analysis of IRA tweets. In phase one we placed tweets into qualitatively derived categories (see below) that facilitated the quantitative second phase of research. In this second phase, we chi-square tests explored how differing types of accounts employed the qualitatively derived categories to differing extents. This research design allowed us to understand both the nature of IRA mes- saging on Twitter and how they implemented this messaging in the month prior to the 2016 election.

In the first phase of research we conducted qualitative analysis of tweets originating from the IRA in the month prior to the 2016 U.S. Presidential election. On June 18, 2018, the U.S. House Intelligence Committee released a list of 3841 Twitter handles associated with IRA activity (Permanent Select Committee on Intelligence, 2018). These handles correspond to what are commonly called troll or sometimes sock puppet accounts, meaning they had human operators. These ac- counts differ from bot accounts which are to a greater extent computer automated. Linvill and Warren (2019) collected approximately three million cumulative tweets from these handles and made this data public at fivethirtyeight.com (Roeder, 2018).

Linvill and Warren's (2019) analysis of these tweets identified four major types of English-language accounts which were active in the month prior to the election: right trolls, left trolls, news feeds, and hashtag gamers. This typology was defined at the account level and captured the dominant persona that the account presented throughout its “life”. Right trolls tended to express nativist and right-leaning po- pulist messages, often employing hashtags employed by similar but genuine Twitter users, including #tcot, #MAGA, and #Red- NationRising. Left trolls expressed ideologically liberal views and had a focus on cultural identity, racial identity, and, often, the Black Lives Matter movement. News-feed accounts presented themselves as local news aggregators. They had names specific to a city, such as @Onli- neMemphis and @TodayPittsburgh, and tweeted out news specific to those cities. Finally, hashtag gamer accounts were dedicated almost exclusively to playing word games popular on Twitter. Hashtag games involve posting a hashtag and then answering the question implied by that hashtag, e.g. “#ThingsILearnedFromCartoons You can get your head blown off by a cannon and completely recover in five minutes.”

The Linvill and Warren (2019) account types are useful to us for two reasons. First, we use them to provide context for our coding procedures (as outlined, below). Second, we will document the degree to which the specific tactics employed by accounts varied by the account type.

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

294

5. Qualitative coding procedures

The data set included 100,396 English language tweets dated from October 7, 2016 to November 8, 2016. We conducted qualitative ana- lysis of random selections taken from this set as recommended by Corbin and Strauss (2015). First, we read several hundred tweets to get a sense of the data. From there, we conducted unrestricted open coding, working together to examine, compare, break down, and conceptualize data. From this process we identified meaningful, recurring patterns.

Second, we conducted axial coding by comparing and reducing these patterns. Seven distinct categories were identified through axial coding (see Results). As this process continued we worked to identify exemplar sample tweets and create definitions for each category to help clarify their meaning. It is important to clarify that we read tweets within the context of the account type that sent the tweet as identified by Linvill and Warren's (2019) previous work (see also Roeder, 2018). For instance, the tweet “What the Republicans did today was amazing!” could be interpreted differently if posted by a right troll as opposed to a left troll. Similarly, tweets were read in context with any content to which a tweet had an active link. In many cases linked content was necessary for proper interpretation. Tweets were coded using the in- formation available and categorized based on what the coder felt was the most likely intent of the tweet. It is possible that in some cases some higher order strategies were being employed by IRA operators, but for the purposes of coding individual tweets we attempted to err on the side of parsimony.

To help assure the reliability of our analysis we engaged in peer debriefing, the development and use of a code book, and intercoder reliability. Peer debriefing (Creswell & Miller, 2000) involves recruiting an individual familiar with the phenomenon being explored but ex- ternal to the research team to play the role of devil's advocate. Peer debriefing was conducted near the end of axial coding. A code book was developed using the definitions and example tweets developed and identified during axial coding. The use of a code book served as a stable representation of the coding analysis to serve as a reference throughout the coding process (Creswell & Poth, 2018). Employing this code book, three members of the research team coded randomly selected sets of 100 tweets. After each set was coded we compared results and refined our analysis. This process was continued until we met an acceptable Krippendorff's alpha reliability of 0.76 (Krippendorff, 2004). Following the completion of reliability analysis a random sample of 4200 tweets was selected and distributed among the research team for analysis. Of these tweets, 1017 (24%) were from right troll accounts, 1830 (47%) were from left troll accounts, 653 (16%) were from hashtag gamer accounts, and 700 (17%) were from news feed accounts.

For the subset of tweets that were identified through this process as partisan political (n = 1,338, see below), we further documented whether the tweets explicitly mentioned either Hillary Clinton or Donald Trump, and, if so, whether the mention was an attack or sup- portive of the mentioned candidate.

6. Results

6.1. Qualitative analysis

273 (6.5%) of the 4200 English language IRA tweets from the month prior to the 2016 U.S. Presidential election sampled for this study could not be placed into a category. These tweets were either part of a Twitter exchange that could no longer be viewed or had a link that was no longer active and therefore lacked sufficient context necessary to properly construe the nature of the tweet. Qualitative analysis placed the remaining tweets (n = 3927) into one of seven distinct categories: attack left, support right, attack right, support left, attack media, attack civil institutions, and camouflage. Tweets were placed into the category with which their message seemed most focused. Note, all example tweets are as originally written, including errors.

6.1.1. Attack left (n = 505, 12%) These tweets primarily attacked left leaning ideas, ideals, and/or

candidates. Most of these tweets (n = 397) explicitly attacked Hillary Clinton or her campaign. Such tweets included @KenzDonovan's re- tweet, October 8, 2016, “Hillary hates America & wants to see it crumble with open borders. She wants nothing more than to Merkel the United States. #PodestaEmails” and also @Pamela_Moore13, October 23, 2016, “The crowd at #CrookedHillary rallies are FAKE and paid for by the Clinton foundation! As I said absolutely no enthusiasm!“. These tweets frequently connected to FBI investigations into Clinton's use of email, e.g. @CooknCooks, October 28, 2016, “Comey better be on the watch now for Hillary's goon squad and mafia - to quote Colin Powell - as they will do anything to get her elected!!“. Many of these tweets also employed Hillary Clinton's husband, former President Bill Clinton, as a mechanism to attack her, e.g. @WorldofHashtags, October 19, 2016, “#RejectedDebateTopics how many of Bill's sexual assaults has hill covered up?”

Many tweets (n = 108) attacked liberal or progressive ideas and ideals without directly attacking the Clinton Campaign. These included racially charged tweets such as @AmelieBaldwin's retweet, November 7, 2016, “The #Black Community Is Leaving The #DEMOCRATIC PLANTATION. #MalcomX ADVISES IT! #BlackTwitter #Miami #Orlando #FL” and also JaydaAstonishin's retweet, November 1, 2016, “Black folks with criticism of President Obama's policies are not traitors to Blackness. Y'all need to cut that Messiah complex stuff out.” These tweets also included occasional tweets about specific issues and po- licies, including @KateRitterrrr's retweet, October 25, 2016, “Why the hell do liberals think the Govt can run a HC system for 300 million ppl, when they can't run one for veterans? I mean, wtf man!” and also @ DonnaBRivera's retweet, October 24, 2016, “.@KenBurns DNC engages in POLITICAL TERRORISM & VOTER FRAUD”.

6.1.2. Support right (n = 295, 7.0%) These tweets primarily supported right wing ideas, ideals, and/or

candidates. The majority of these tweets explicitly supported Donald Trump or the Trump campaign (n = 221). Such tweets included @ hyddrox's retweet, October 7, 2016, “The Trump Train never sleeps! We are more determined than ever to get Donald Trump elected! #MakeAmericaGreatAgain”. These tweets often suggested why certain demographic groups should support Trump's candidacy, e.g. @TEN_GOP's tweet, October 14, 2016, “Black lady perfectly explains why she supports Trump. Please help to spread her word! #WomenWhoVoteTrump https://t.co/0CdEaTeVGt” and also DorothieBell's retweet, October 21, 2016, “Dear Christians, we MUST all vote! We must have revival in the church and a spiritual awakening in our nation! It's time 4 @realDonaldTrump”.

Other tweets in this category (n = 75) more generally supported conservative ideas and ideals. Some of these were general support for conservative ideology, including @hipppo_’s retweet, November 7, 2016, “No matter what happens on Tuesday I am proud to be a de- plorable. It was a pleasure to meme with you gentleman #ElectionFinalThoughts”. Others focused on core conservative issues. Such tweets included @J0hnLarsen's retweet, October 10, 2016, “No, Australia Is Not An Example of The Effectiveness of Gun Control - https://t.co/Mfj8hHPrYB #2a #guns #gunrights #gunsense”. These tweets also included more fringe conservative issues, however, in- cluding @AmelieBaldwin's retweet, October 11, 2016, “BOOM! Allen West Publicly Declares O An Islamist http://t.co/m1iR79iKNn @ VRWCTexan #2A #Veterans #tcot @ChristiChat http://t.co/ HQouKMYQoH”.

6.1.3. Attack right (n = 227, 5.4%) These tweets primarily attacked right leaning ideas ideals, and/or

candidates. The majority of these tweets explicitly attacked Trump or the Trump campaign (n = 154), but mostly in the form of retweets such as @Jani_s_Jac, October 13, 2016, “You guys worry about Trans using

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

295

bathrooms but vote for a man who creeps In a room full of naked young girls. MIss Teen USA” and also @OGDeandre, October 14, 2016, “Trump's Team Breitbart now wavering between International Jewish Conspiracy & International Mexican Conspiracy. I need a brisket taco STET!“.

The remaining tweets in this category attacked conservative ideas and ideals more generally. These included tweets such as @ PamBLMDaniels' retweet, October 15, 2016, “'@Crystal1Johnson the NRA is a terrorist organization filled w very under/uneducated trash not fit to carry/own guns. Fuck #2A'“. Also indicative of these tweets is @BlackToLive's October 29, 2016, “All Lives Matter's rhetoric is just bullshit”.

6.1.4. Support left (n = 311, 7.4%) These tweets primarily supported left leaning ideas, ideals, and/or

candidates. While a minority of these tweets explicitly supported Hillary Clinton (n = 43), with tweets such as @staywoke88, October 14, 2016, “#WhyWomenDontReport #Hillary2016 #WomenStopTrumpParty #Delusionalclown #democrat #ImWithHer #NeverTrump #StrongerTogether #GrabThemByTheP”, these were nearly exclusively retweets and not originally composed by the IRA. The majority of tweets in this category (n = 267) more generally sup- ported liberal and progressive ideas and ideals. Many of these tweets supported issues central to the Black Lives Matter movement, e.g. @ camosaseko's retweet, October 8, 2016, “I see you rolling your eyes when you hear the word 'racism'. We die because of racism. I don't care how you feel #BlackToLive” and also @Crystal1Johnson, October 10, 2016, “Black Power Is Giving Power To People Who Have Not Had Power To Determine Their Destiny. #HueyNewton”. One outlier in the category originated with @AmelieBaldwin, a right troll, on October 7, 2016, who retweeted support for Green Party candidate Jill Stein: “Total foreign bases of all other nations combined… abt 30 #JillStein is #AntiWar #ImWithHer - > #JillStein2016”.

6.1.5. Attack media (n = 85, 2.0%) These tweets primarily attacked the validity of mainstream media

outlets. They often attacked a perceived liberal bias in the media such as @ TEN_GOP's tweet, October 8, 2016, “Media doesn't show any in- terest on murder of #SethRich! Spread to make sure everybody knows the story!” and also @CynthiaMHunter, October 14, 2016, “Can NBC be treated as giving in kind donations to Hillary? #askingforafriend https://t.co/A0bS4MkmL3”.

But there were also some attacks on the media from the left. These tweets often charged the media with racial bias, including @ BLMSoldier's retweet, November 2, 2016, “White man kills two cops n #Iowa and media or Trump give no attention. Only matters if a min- ority doing killing I guess. @NPR #Elections2016”. Some also con- tained accusations of a media bias in favor of Hillary Clinton, including @JavonHIDP's retweet of Jill Stein, October 27, 2016, “Read how @TheDailyBeast tried to smear me while it takes orders from @ChelseaClinton! https://t.co/h0qBDZToWC”.

6.1.6. Attack civil institutions (n = 293, 7.0%) These tweets primarily attacked civil and governmental institutions.

Such attacks were viewed as distinct from attacks on foundational ideals of an institution or specific political office holders advocating for such institutions which were placed in the attack left or attack right categories. Many seemed timed primarily undermined the validity of the electoral process using hashtags such as #BetterAlternativeToDebates, #RejectedDebateTopics, and #2016ElectionIn3Words. These included @IlikeBIGbuttand, October 10, 2016, “see who could best live like a voter on minimum wage for 6 months, close family aswell, better TV too #BetterAlternativeToDebates” and also @Gab1Aldana, October 19, 2016, “What is Aleppo? #RejectedDebateTopics”.

Other civil institutions which were the subject of tweets in this ca- tegory included U.S. policing and judicial systems, particularly as the pertain to people of color, e.g. @JassScott's retweet, October 17, 2016, “#BlackFistFACT: “White Man's Justice Is The Black Man & Black Woman's Hell. Always Has Been, Aways Will Be!" https://t.co/ 1CL15I8uzH” and also @Crystal1Johnson, October 23, 2016, “Niya was arrested for challenging police abuse in school. Her story is an important part of our struggle.”

Several tweets in this category suggested the U.S. electoral system may be rigged. These tweets included, @March_For_Trump's November 8, 2016 tweet “This election is being rigged! REPORT VOTER FRAUD: 888 486 8102 (Being Patriotic hotline) or 855 976 1200 (Trump Lawyer Team) #Elections2016”. @Pamela_Moore13's October 10, 2016 epon- ymous tweet, contained meaning almost surely missed by unaware readers, “THE RUSSIANS ARE HACKING MY BRAIN! #debate”.

6.1.7. Camouflage (n = 2211, 52.6%) These tweets had no clear or overt connection to IRA agenda

building activity. Such tweets may serve to help give credibility to the IRA account or to build Twitter followers and make connections with a potential audience. Tweets in this category ranged widely in topic. Camouflage included very mundane tweets such as @ JadonHutchinson's retweet, October 12, 2016, “Start each day with a grateful heart. #GoodMorning #happywednesday”. These tweets often addressed cultural topics such as television, music, and video games, including @RosieOnFleek's retweet, October 23, 2016, “Wu-Tang Clan ain't nuthing ta fuck wit https://t.co/4YHPVaAEwt”.

A second type of camouflage tweets were local news articles. They were presented in a journalistic fashion, mirroring headlines to legit- imate news stories shared in the tweet itself. These tweets included @ DailySanFran, October 24, 2016, “Big tree falls in Berkeley on College Avenue https://t.co/yaYlT5hKFY” and also @OnlineCleveland, October 27, 2016, “Chicago Cubs beat the Cleveland Indians, at least in terms of political donations https://t.co/mVZyzw2tr2 #politics”. Not all tweets in this category were without a degree of potential bias, however. While not addressing U.S. politics, tweets from @TodayInSyria tweeted con- tent linking directly to Syria state media. Such tweets included October 7, 2016, “Russian Defense Ministry: Lists provided by Washington don't tell apart “moderate opposition” from terror org … https://t.co/ rNZarPdNax”.

Table 1 Tweet type share by account type.

Attack Left Attack Right Support Left Support Right Attack Media Attack Institutions Camouflage N

Right Troll 0.413 0.020 0.010 0.260 0.050 0.042 0.133 1017 Left Troll 0.027 0.090 0.153 0.007 0.014 0.038 0.570 1830 Newsfeed 0.013 0.017 0.016 0.011 0.000 0.011 0.929 700 Hashtag Gamer 0.041 0.046 0.015 0.015 0.012 0.265 0.587 653 N 505 227 311 295 85 293 2211

Notes: Each row reports the distribution of tweet types originating in from the given account type. Rows do not sum to one, because 273 tweets were of none of the identified types. A chi-squared test rejects the null of identical distributions with p < 0.001.

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

296

6.2. Quantitative analysis

6.2.1. Distribution by account type The seven tweet types described above were not equally utilized by

the four account types. Table 1 lays out the distribution of tweet type by account type. Each row reports the distribution of tweet types origi- nating in from the given account type, with the total number of tweets from each account type given in the final column, and the total number of tweets of each type in the final row. The residual category, not presented, are those (273) tweets that we were unable to code into any of these categories.

Overall, we can soundly reject the null of an identical distribution of tweet types across these account types ≈ <χ p( (15) 1900, 0.001).2 This statistically significant difference in distribution is also true for every pairwise comparison (p < 0.001). This overall difference is not sur- prising, given that account types were assigned based on the persona that the accounts were representing, and a large part of that persona is formed by the character of their performance. But the degree and di- rection that the distributions differ are, perhaps, more informative about how the strategies of these account types diverge.

First, compare the behavior of the left and right trolls. The first two marked differences are in left/right orientation and attack/defense or- ientation. As you would expect, the right trolls use a large fraction (41 percent) of their tweets attacking the left and (26 percent) supporting the right. The left trolls spend more time (15 percent) supporting the left than they do attacking the right (9 percent). Neither spends much time being “balanced” (attacking their own side or supporting the other). A third difference between these two account types is the amount of time they spend on non-political camouflage, where the left trolls spend much more time (57 percent) than the right trolls do (13 percent). Turning to attacks on institutions and the media, there is marked similarity between the two types. At first appearances, the right trolls seem to do more of it (9 percent) than the left trolls do (5 per- cent), but if you adjust for the amount of camouflage the two groups do they fall much more in line. Both groups spend about 11 percent of their non-camouflage tweets attacking the media and institutions, although the right trolls focus relatively more on the media and the left trolls focus more on other institutions (mostly institutions related to criminal justice).

Turning to the other two account types, news feeds and hashtag gamers, their tweet distributions look entirely different from each other and from the trolls. The news feeds essentially tweeted out nothing but real local news content, scraped or retweeted from other legitimate news accounts, all coded as “camouflage” in our analysis. The hashtag gamers, on the other hand, told jokes. Most (59 percent) were in- nocuous, while a large fraction (about 26 percent) were derogatory toward institutions, and a small fraction (about 4–5 percent each) at- tacked the left or the right.

Table 2 provides an initial analysis of the ideological political content coded as partisan attack/support in the first analysis into whether that they specifically attacked or supported one presidential

candidate. As in Table 1, each row reports the distribution of tweet types (here, partisan tweets, only) originating from the given account type. The residual category (522 tweets, not presented) are tweets that do not refer to either of the specific candidates. As the news feed ac- counts produced so few partisan tweets, they have been excluded from this analysis.

Overall, there is, again, a statistically significant difference in the distribution types of partisan tweets among these account types, both overall ≈ <χ p( (8) 640, 0.001)2 , and pairwise (p < 0.001).

Beginning, again, with the right and left trolls, there are dramatic differences in their behavior. First, obviously, right trolls support Trump and attack Clinton more than left trolls do, and left trolls support Clinton and attack Trump more than the right trolls do. Second, right trolls spend a much greater fraction of their tweets, overall, on candi- date-specific messaging (78 percent) than left trolls do (35 percent). Third, while the right trolls overwhelmingly support their candidate, with 15 times as many tweets supporting Trump as attacking him, the left trolls are much more lukewarm in their support, with nearly as many tweets attacking Clinton as supporting her (1.3 times more in support). The same pattern is true when looking at how the trolls dis- cuss the opposing candidate. Right trolls never tweet in support of Clinton, while attacking her more than they do anything else. Left trolls often attack Trump (20 percent, although not as much as they post non- candidate political commentary), but they also support him with 2 percent of their tweets.

The hashtag gamers spend more of the partisan tweets discussing specific candidates (81 percent) than either of the Troll types, and they overwhelmingly use these tweets to attack (66 percent) rather than support (15 percent). The mix of attack/support varies among candi- dates. When discussing Clinton, the hashtag gamers attacked over 7 times as much as they supported. When discussing Trump, they at- tacked about 3.6 times as much as they supported. They also discussed the candidates unequally, mentioning Trump about 1.5 times as much as they mentioned Clinton.

It is important to note regarding the above quantitative analyses, the goal was not to uncover the general causal effect of account theme on the distribution of tweets. After all, account type and tweet dis- tribution are codetermined by more-or-less the same actor (the IRA). Rather, we are trying to document the correlation of theme strategies with tweet-type strategies. When the IRA chooses to use a left-troll account, do they attack with it or defend with it or neither?

The key question is what we can infer from these correlations. The major threat to inference is omitted variables that are correlated with account type. If these unidentified characteristics are the real drivers of the correlations we uncover, we might wrongly infer that the IRA is choosing different messages by account type, but, in reality, that re- lationship is almost accidental. For example, if it were true that left troll account profiles were overwhelmingly represented as women, and right troll account profiles were overwhelmingly represented as men, but all “man" accounts attack about the same amount and at a rate more than the “woman" accounts do, we might falsely infer that right trolls attack more because they are right trolls, while in fact they attack more for reasons related to the gender they are portraying.

It might seem natural to try to condition on other factors that might be correlated with account type and tweet strategy, either by subsetting on those factors or by running a regression. But the male/female ex- ample illustrates why this approach is misguided. When creating the left- and right-troll accounts in this hypothetical example, the IRA decided to make the right-troll accounts more “male", which are used to attack. Conditioning on “male" would actually distort the analysis, be- cause it would obscure this strategic choices and would lead us to wrongly infer that there is no difference in desire to attack across troll types. The simple bivariate analysis is actually the more robustly valid approach.

Table 2 The relationship between account type and candidate targeting.

Attack Clinton

Support Clinton

Attack Trump

Support Trump

N

Right Troll 0.480 0.000 0.018 0.279 714 Left Troll 0.053 0.071 0.201 0.020 507 Newsfeed 0.150 0.100 0.250 0.100 40 Hashtag Gamer 0.286 0.039 0.377 0.104 77 N 398 43 154 221

Notes: Each row reports the distribution of tweet types originating in from the given account type. Rows do not sum to one, because 522 tweets mentioned neither candidate. A chi-squared test rejects the null of identical distributions with p < 0.001.

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

297

7. Discussion

This study offers further empirical description of the tactics Russian Twitter trolls played in the 2016 presidential election. Moreover, it sheds light on the growing threat of state-sponsored public agenda building efforts that seek to undermine democratic processes. These findings provide insight into how a foreign misinformation campaign functioned at a practical level. Our findings suggest that in the debate between the two narratives of IRA activity, sowing discord and chaos versus supporting Trump's candidacy, both may be true and neither are mutually exclusive. In the run up to the 2018 U.S. Presidential election the IRA worked to spread distrust in institutions and extremist view- points but did so in a manner that, on balance, favored Donald Trump.

Broadly, we have shown that analysis of state sponsored disin- formation campaigns cannot take a narrow view but must analyze campaigns in their full breadth to understand an actor's goals. Previous research exploring propaganda has focused largely on the form and structure of messaging. Propaganda on social media is more grassroots, more bottom-up than the top-down propaganda employed through other media. This research has shown that in the world of social media it is important to take a broader view and work to understand the agenda-building processes at play. This researcher benefitted by looking at a range of accounts and account types. Had we analyzed a narrower data set we would have received a blinkered view of IRA activity. Looking at only the source of these messages or the form of particular sets of messages could lead one to make incorrect assump- tions about IRA goals. Looking only at the account @southlonestar, for instance, may lead one to believe the IRA was actively engaged in the Texas secessionist movement. Similarly, looking solely at the account @ blmsoldier could lead one to infer the IRA was passionate about police violence. It is only by looking at the messaging as a whole that we can understand IRA tactics and how they work together to influence dif- fering audiences. Social media is far more interactive and multi-chan- neled than media previously employed for the purposes of propaganda and the lenses we use to understand it must necessarily adapt.

Early democratic theory posited that policymakers advanced poli- tical interests to engaged publics via an autonomous press. Cobb and Elder (1971) upended the conventional wisdom of democratic theory by advancing agenda-building theory, wherein actors move issues from their own agendas to those of the policymakers. Agenda building is predicated on the notion that engaged publics determine the salience of issues. They then bring those issues up to policymakers with the hope of securing legislative attention and action. Of course, democratic elec- tions are the cornerstone of this process, making them ideal targets for coordinated disinformation campaigns. Here we have demonstrated how tools employed by a foreign government actively worked to sub- vert and undermine authentic public agenda-building efforts by en- gaged publics. Accounts disguised as U.S. citizens infiltrated normal political conversations and inserted false, misleading, or sensationa- lized information. These practices create an existential threat to the very democratic ideals that grant the electorate confidence in the po- litical process. Our findings suggest that this state-sponsored public agenda building attempted to achieve those effects prior to the 2016 U.S. Presidential election in two ways. First, the IRA destabilized au- thentic political discourse and focused support on one candidate in favor of another and, as their predecessors had done historically, worked to support a politically preferred candidate (Shane & Mazzetti, 2018, pp. 1–11). Second, the IRA worked to delegitimize knowledge. Just as the KGB historically spread conspiracies regarding the Kennedy assassination and the AIDS epidemic, our findings support previous research (Broniatowski et al., 2018) that IRA messaging attempted to undermine scientific consensus, civil institutions, and the trustworthi- ness of the media. These attacks could have the potential for societal damage well beyond any single political campaign.

It is important to note the manner in which the IRA engaged within the growing political divide in America. Pew Research (2017) found

that “the shares of Republicans and Democrats who express very un- favorable opinions of the opposing party have increased dramatically since the 1990s” (p. 5, emphasis in original). Political polarization is especially prevalent on social media. Previous research has shown that users with similar political ideologies band together in echo chambers (Garrett, 2009; Barberá, Jost, Nagler, & Tucker, Bonneau, 2015; Boutyline & Willer, 2017).

Political echo chambers are particularly evident on Twitter. A Pew Research study, for example, found that in a polarized conversation around #my2k, only 0.65% of tweets were shared between con- servative and liberal groups (Smith, Rainie, Himelboim, & Shneiderman, 2014). Steward, Arif, and Starbird (2018) performed a network analysis of identified Russian Twitter accounts and found a high degree of polarization was characteristic of the coordinated effort to propel various issues into the public spotlight. Within this context, our results showed Russian troll accounts which created content that attacked the left/supported the right (19%) produced more tweets than those that attacked the right/supported the left (12.8%). Moreover, those tweets that attacked the left/supported the right were more likely to be about candidate Clinton then those tweets which attacked the right/supported the left were to be about candidate Trump. This be- comes especially significant when considering that research has shown that conservative individuals have demonstrated greater political homophily on Twitter (Boutyline & Willer, 2017), a fact that can lead to “advantages for diffusing political behaviors that require normative pressure or social confirmation—including behaviors like turning out to vote, attending political protests, and engaging in potentially con- tentious political speech” (p. 552).

Communicating within only a single ideological group, and among only a single online echo chamber, would have limited the reach of IRA messaging. Our results demonstrate how IRA operatives used accounts mimicking communication within differing ideological networks, and likely differing echo chambers. Account types were employed by the IRA in differing ways and these differences seemed to be constructed so as to achieve the same overarching strategic goals among differing U.S. ideological groups. The IRA made coordinated efforts to drive a wedge between conservatives and liberals, further separating individuals by their political ideologies. Extant research has shown that polarization among political bases decreases productivity in the legislature; “as we have seen repeatedly in recent years, ideologically divided parties are an impediment to policymaking” (Iyengar, 2016, p. 219). This focus harks directly back to the focus paid by the KGB to the divide in the US during the Civil Rights era. IRA efforts in 2016 went beyond ideological division, however, to include supporting the candidacy of Donald Trump. The IRA accomplished both of these goals by communicating with different ideological groups and their accompanying social media echo chambers using different tactics. In this way, they turned echo chambers, which may have otherwise limited the reach of their mes- saging, into a tool that may have benefited them.

Even users that were not part of a political echo chamber, however, were targeted by Russian trolls. Several of the Russian accounts were identified by Linvill and Warren (2019) as ‘hashtag gamers’ (see also Roeder, 2018) which reflects a sort of call/response form of tweeting “wherein a user will post some kind of gag within a tightly defined genre, usually some kind of pun-based mashup of two different things, like #UpdatedTVShows or #BreakfastFilms” (Alexander, 2011; para. 2). Trolls participated in various hashtag games primarily to attack civil institutions, including the electoral system. Some accounts used hash- tags such as #BetterAlternativeToDebates, and #2016ElectionIn3- Words to undermine the validity of the process. Parker (2016) sug- gested that “hashtag games can generate thousands of tweets as people come up with creative ways to expand on a topic” (para. 1). As such, Russian trolls' use of hashtag games was strategic in that they were used to (1) reach those that may not necessarily follow politics to the same extent as politically engaged users and (2) further undermine their perceptions of the election through political quips and jabs.

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

298

7.1. Limitations and future research

This study was reliant on data collected from Twitter handles as- sociated with IRA activity released on June 18, 2018 by the U.S. House Intelligence Committee (Permanent Select Committee on Intelligence, 2018). It is possible, if not probable, that this set of Twitter handles is not the complete population of IRA associated handles operating to influence U.S. political discourse in the month before the Presidential election. The list was dependent on information supplied to the House Intelligence Committee by Twitter, and the organization has not been forthcoming in how the handles were identified. It is possible that the dataset employed for this study is not representative. Given the number of tweets available in the dataset from between October 7 and No- vember 8 (n = 100,396), however, we argue that while our findings may not be representative of everything the IRA was doing prior to the election, they clearly point to important tactics employed by the or- ganization.

None-the-less, should more robust Twitter data become available, it should be employed to expand our research. It would be particularly interesting if future data may help us better understand how users en- gage with differing disinformation tactics. An attack tweet, for instance, may better serve the IRA's goals as a means to coalesce support among like-minded users, enflame anger among those who disagree, or some combination of both. Additional data will be necessary to explore how differing users engage through retweets, likes, mentions, and replies and what types of messages reach differing audiences.

Findings could also be expanded through computer-assisted quali- tative data analysis to magnify this analysis of a single month to the full IRA data set. This approach may also help address inherent validity concerns regarding the fundamental subjective nature of qualitative research such as that conducted here. Further, data from other social media platforms should be analyzed to understand how, if at all, plat- forms were employed differently by the IRA and how the nature of platforms influenced their use. Looking only at Twitter has given us a blinkered view of the full IRA operation prior to the 2016 election. We know Russian disinformation to have been active on Facebook, Instagram, and Reddit, to name just a few social media platforms. More work is necessary to capture the full scope of the specific tactics they employed in this period. Varying platforms should be examined also in the time since 2016 to better understand how the IRA tactics may have evolved in the face of greater public awareness of their activity on so- cial media. Finally, it should be noted this research only examined disinformation originating from Russia's IRA. This may or may not tell us anything regarding disinformation originating from other sources, and a growing number of nations are engaging in disinformation on social media across platforms (Lapowsky, 2018; Thaker, 2018). All of these elements should be looked at together, however, to better un- derstand how social media platforms and their users might better guard against disinformation.

Understanding how state sponsored disinformation serves to influ- ence public discourse is becoming increasingly important. Ongoing ef- forts by the Russian state to distract, divide, and demoralize popula- tions not only in the U.S. but across the globe have been called political war (Galeotti, 2018). If true, it is a form of warfare waged with in- creasing intensity and by a growing number of global powers (Lapowsky, 2018; Thaker, 2018). The healthy future of free and open democracies may depend on how well we engage in this new form of warfare.

References

Allcott, H., & Gentzkow, M. (2017). Social media and fake news in the 2016 election. The Journal of Economic Perspectives, 31, 211–236. https://doi.org/10.1257/jep.31.2.211.

November 11 Alexander, J. (2011). The hashtag game: Waste of time or art form? Time. Retrieved from http://ideas.time.com/2011/11/11/the-hashtag-game-waste-of- time-or-art-form/.

Andrew, C., & Mitrokhin, V. (1999). The mitokhin archive: The KGB in Europe and the west.

London: Penguin Books (Published as The Sword and the Shield: The Mitrokhin Archive and the Secret History of the KGB in the U.S.).

Barberá, P., Jost, J. T., Nagler, J., Tucker, J. A., & Bonneau, R. (2015). Tweeting from left to right: Is online political communication more than an echo chamber. Psychological Science, 26, 1531–1542. https://doi.org/10.1177/0956797615594620.

Barrett, D., Horwitz, S., & Helderman, R. S. (2018). February 17). Russians Indicted in 2016 election interference. The Washington Post.1A.

Boghardt, T. (2009). Soviet bloc intelligence and its AIDS disinformation campaign. Studies in Intelligence, 53, 1–24.

Boutyline, A., & Willer, R. (2017). The social structure of political echo chambers: Variation in ideological homophily in online networks. Political Psychology, 38, 551–569. https://doi.org/10.1111/pops.12337.

Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., et al. (2018). Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate. American Journal of Public Health, 108(10), 1378–1384.

Cobb, R. W., & Elder, C. D. (1971). The politics of agenda-building: An alternative per- spective for modern democratic theory. Journal of Politics, 33, 892–915. https://doi. org/10.2307/2128415.

Corbin, J., & Strauss, A. (2015). Basics of qualitative research. Thousand Oaks, CA. Sage. Cobb, R., Ross, J., & Ross, M. (1976). Agenda building as a comparative political process.

American Political Science Review, 70, 126–138. https://doi.org/10.2307/1960328. Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods ap-

proaches. Thousand Oaks, CA: Sage. Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory

Into Practice, 39, 124–130. https://doi.org/10.1207/s15430421tip3903_2. Creswell, J. W., & Poth, C. N. (2018). Qualitative inquiry & research design: Choosing among

five approaches. Thousand Oaks, CA: Sage. Denham, B. E. (2010). Toward a conceptual consistency in studies of agenda-building

processes: A scholarly review. The Review of Communication, 10, 306–323. https:// doi.org/10.1080/15358593.2010.502593.

Federal Bureau of Investigations (2018 August 30). The FBI launches a combating foreign influence webpage. Retrieved from https://www.fbi.gov/news/pressrel/press- releases/the-fbi-launches-a-combating-foreign-influence-webpage.

Garrett, R. K. (2009). Politically motivated reinforcement seeking: Reframing the selec- tive exposure debate. Journal of Communication, 59, 676–699. https://doi.org/10. 1111/j.1460-2466.2009.01452.x.

Galeotti, M. (2018). I'm sorry for creating the ‘Gerasimov Doctrine’. Foreign Policy. Retrieved from http://foreignpolicy.com/2018/03/05/im-sorry-for-creating-the- gerasimov-doctrine/.

Gallagher, N. K. (2018 August 2). Russian-backed troll Twitter accounts targeted Maine politicians. Portland Press Herald. Retrieved from https://www.pressherald.com/ 2018/08/02/russian-backed-troll-twitter-bots-targeted-maine-politicians/.

Graff, G. M. (2018 October 19). Russian trolls are still playing both sides—even with the Mueller probe. WIRED. Retrieved from https://www.wired.com/story/russia- indictment-twitter-facebook-play-both-sides/.

Herman, E. S., & Chomsky, N. (1988). Manufacturing consent: The political economy of the mass media. New York, NY: Pantheon Books.

Huang, Y. L., Starbird, K., Orand, M., Stanek, S. A., & Pedersen, H. T. (2015). February). Connected through crisis: Emotional proximity and the spread of misinformation online. Proceedings of the 18th ACM conference on computer supported cooperative work & social computing (pp. 969–980). ACM.

Iyengar, S. (2016). E pluribus pluribus, or divided we stand. Public Opinion Quarterly, 80, 219–224. https://doi.org/10.1093/poq/nfv084.

Ioffe, J. (2017 October 21). The history of Russian involvement in America's race wars. The Atlantic. Retrieved from https://www.theatlantic.com/international/archive/2017/ 10/russia-facebook-race/542796/.

Jamieson, K. H. (2018). Cyber-War: How Russian hackers and trolls helped Elect a president. New York: Oxford University Press.

Kahne, J., & Bowyer, B. (2018). The political significance of social media activity and social networks. Political Communication, 1–24. https://doi.org/10.1080/10584609. 2018.1426662.

Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30, 411–433. https://doi. org/10.1111/j.1468-2958.2004. Tb00738.x.

Lapowsky, I. (2018 August 24). What we know about Iran's global propaganda campaign. WIRED. Retrieved from https://www.wired.com/story/iran-global-propaganda- fireeye/.

Lasswell, H. D. (1927). The theory of political propaganda. American Political Science Review, 21, 627–631.

Linvill, D. L., & Warren, P. L. (2019). Troll factories: Manufacturing specialized disin- formation on Twitter. Retrieved from http://pwarren.people.clemson.edu/Troll_ Factories_v2_Linvill_Warren.pdf.

Lukito, J., & Wells, C. (2018). Most major outlets have used Russian tweets as sources for partisan opinion: Study. Columbia Journalism Review. Retrieved from https://www. cjr.org/analysis/tweets-russia-news.php.

Maddock, J., Starbird, K., Al-Hassani, H. J., Sandoval, D. E., Orand, M., & Mason, R. M. (2015). Characterizing online rumoring behavior using multi-dimensional signatures. Proceedings of the 18th ACM conference on computer supported cooperative work & social computing (pp. 228–241). ACM.

Parmelee, J. H. (2014). The agenda-building function of political tweets. 16, New Media & Society434–450. https://doi.org/10.1177/1461444813487955.

Permanent Select Committee on Intelligence (2018). Schiff statement on release of Twitter ads, accounts and data. Retrieved from https://democrats-intelligence.house.gov/ news/documentsingle.aspx?DocumentID=396.

Roeder, O. (2018 July 31). Why we’re sharing 3 million Russian troll tweets. FiveThirtyEight. Retrieved from https://fivethirtyeight.com/features/why-were-sharing-3-million-

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

299

russian-troll-tweets/. Romerstein, H. (2001). Disinformation as a KGB weapon in the Cold war. Journal of

Intelligence History, 1, 54–67. https://doi.org/10.1080/16161262.2001.10555046. February 20 Smith, M. A., Rainie, L., Shneiderman, B., & Himelboim, I. (2014). Mapping

Twitter topic networks: From polarized crowds to community clusters. Retrieved from https://www.pewinternet.org/2014/02/20/mapping-twitter-topic-networks-from- polarized-crowds-to-community-clusters/.

Shane, S., & Mazzetti, M. (2018). The plot to subvert an election. The New York Times, Center insert.

Starbird, K., Dailey, D., Walker, A. H., Leschine, T. M., Pavia, R., & Bostrom, A. (2015). Social media, public participation, and the 2010 BP Deepwater Horizon oil spill. Human and Ecological Risk Assessment: An International Journal, 21(3), 605–630.

Stewart, L. G., Arif, A., & Starbird, K. (2018). Examining trolls and polarization with a retweet network. In: Proc. ACM WSDM, workshop on misinformation and misbehavior mining on the web.

Thaker, A. (2018). An Iranian influence campaign has been targeting Indians on Twitter. Quartz India. Retrieved from https://qz.com/india/1446534/iran-may-be-targeting- indians-in-a-twitter-influence-campaign/.

D.L. Linvill, et al. Computers in Human Behavior 99 (2019) 292–300

300