helpfn

15

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning

NADIA FELIX F. DA SILVA, LUIZ F. S. COLETTA, and EDUARDO R. HRUSCHKA, University of Sao Paulo (USP) at Sao Carlos, Brazil

Twitter is a microblogging platform in which users can post status messages, called “tweets,” to their friends. It has provided an enormous dataset of the so-called sentiments, whose classification can take place through supervised learning. To build supervised learning models, classification algorithms require a set of representative labeled data. However, labeled data are usually difficult and expensive to obtain, which motivates the interest in semi-supervised learning. This type of learning uses unlabeled data to complement the information provided by the labeled data in the training process; therefore, it is particularly useful in applications including tweet sentiment analysis, where a huge quantity of unlabeled data is accessible. Semi-supervised learning for tweet sentiment analysis, although appealing, is relatively new. We provide a comprehensive survey of semi-supervised approaches applied to tweet classification. Such approaches consist of graph-based, wrapper-based, and topic-based methods. A comparative study of algorithms based on self-training, co-training, topic modeling, and distant supervision highlights their biases and sheds light on aspects that the practitioner should consider in real-world applications.

CCS Concepts: � Information systems → Expert systems; Sentiment analysis; Clustering and classification; Information extraction

Additional Key Words and Phrases: Co-training, self-training, semi-supervised learning, topic modeling, tweet sentiment analysis

ACM Reference Format: Nadia Felix F. da Silva, Luiz F. S. Coletta, and Eduardo R. Hruschka. 2016. A survey and comparative study of tweet sentiment analysis via semi-supervised learning. ACM Comput. Surv. 49, 1, Article 15 (June 2016), 26 pages. DOI: http://dx.doi.org/10.1145/2932708

1. INTRODUCTION

An increasing amount of content derived from social networking platforms, such as blogs, forums, and microblogs has been observed [Wang et al. 2013]. Twitter is a fa- mous microblogging service that enable users to post status messages called “tweets” with no more than 140 characters. Tweets represent one of the biggest and most chang- ging datasets of user-generated content, with approximately 288 million active users posting 500 million tweets per day.1 These short texts can express opinions on different topics, which can help to direct marketing campaigns because consumers share their opinions concerning brands and products [Jansen et al. 2009]. Outside the realm of

1https://about.twitter.com/company.

This work was supported by Brazilian Research Agencies Capes (Proc. DS-7253238/D), CNPq (Proc. 303348/2013-5), and FAPESP (Proc. 2013/07375-0 and 2010/20830-0). Authors’ addresses: N. F. F. da Silva, L. F. S. Coletta, and E. R. Hruschka, Institute of Mathematics and Computer Science; emails: {nadia, luizfsc, erh}@icmc.usp.br. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or [email protected]. c© 2016 ACM 0360-0300/2016/06-ART15 $15.00 DOI: http://dx.doi.org/10.1145/2932708

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:2 N. F. F. da Silva et al.

business applications, tweets can make it possible to identify bullying outbreaks [Xu et al. 2012], events that generate insecurity [Cheong and Lee 2011], and acceptance or rejection of politicians [Diakopoulos and Shamma 2010], all using an electronic word- of-mouth method. Given the huge amount of data that is typically available in the outlined scenarios, actionable insight can be derived from human-machine systems, in which both human expertise and data-driven approaches are intelligently combined. To intelligently combine the data, particularly considering our application scenario, four relevant issues must be addressed. Specifically, these issues are as follows:

—Although a tweet can have up to 140 characters, people tend to use far fewer than this limit. Indeed, the average length of a tweet is 28 characters.2 This characteristic makes the analyses of tweets based on the so-called bag-of-words harder to perform because the data matrix is very sparse.

—The frequency of misspellings and slang in tweet messages is much higher than that in other domains because users typically post messages from many different electronic devices, such as cell phones and tablets [Saif et al. 2016]. Furthermore, in this type of environment, users develop their own culture with a specific vocabulary. From the perspective of length, although the content (e.g., in characters) is limited, a message may convey rich meanings.

—Unlike blogs, news, and other sites that are tailored to specific topics, Twitter users post messages on a variety of topics.

—Most tweet sentiment analysis techniques fall into two approach categories: lexicon based and corpus based. As with all supervised tasks, these categories require labeled sentiment data to build a machine-learning model [Sebastiani 2002] and/or need labeled sentiment data for evaluation. The more labeled sentiment data that are available, the more robust the machine-learning model and the more accurate the evaluation scores.

Our work focuses on the development of tools for tweet sentiment analysis, where labeled data are typically scarce. In this scenario, particular attention must be given to the role of the (human) experts who help build, monitor, and maintain the system. However, although manual annotation is necessary, it is tedious, expensive, and error prone [Zhu 2005; Chapelle et al. 2006]. In Go et al. [2009] the authors suggested obtaining labels from emoticons and hashtags but noted that these are not part of every tweet. Therefore, this and other related approaches [Mohammad 2012; Wang et al. 2012; Qadir and Riloff 2013; Suttles and Ide 2013; Davidov et al. 2010] have limited use in practice.

Semi-Supervised Learning (SSL) techniques take advantage of using unlabeled data in their training processes and are able to improve classification in applications where labeled data are scarce [Balcan and Blum 2010; Goldberg 2010; Zhu and Goldberg 2009]. In this context, SSL-based approaches show promise in dealing with tweet sen- timent analysis because an overwhelming number of unannotated tweets is accessible, in contrast to the limited number of annotated ones [Xiang and Zhou 2014; Becker et al. 2013; Baugh 2013; Liu et al. 2013c]. The acquisition of labeled tweets often requires a costly process that involves skilled experts, whereas the acquisition of unlabeled ones is relatively inexpensive. From this perspective, systems based on SSL are of great practical value.

This article provides a survey of SSL approaches for tweet sentiment analysis. Furthermore, the comparative study conducted offers instructive guidelines for users (experts) interested in practical applications. This type of study is not available in

2http://thenextweb.com/twitter/2012/01/07/interesting-fact-most-tweets-posted-are-approximately-30- characters-long/.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:3

Fig. 1. Overview of tweet sentiment analysis approaches.

the literature. In contrast to the work of Thakkar and Patel [2013], which presumes plenty of labeled data, our work focuses on scenarios where labeled data are scarce. Our work also differs from others that address general approaches for sentiment analysis [Medhat et al. 2014; Feldman 2013; Tsytsarau and Palpanas 2012; Liu 2012; Montoyo et al. 2012; Vinodhini and Chandrasekaran 2012]. In particular, our comparative study considers self-training, co-training, topic modeling, and distant supervision. As a complementary contribution, and to better position our work with respect to the existing literature, we also provide a compact overview of unsupervised and supervised approaches for tweet sentiment analysis.

Our article is organized as follows. In Section 2, we give a brief overview of the liter- ature on supervised and unsupervised approaches for tweet sentiment analysis. This overview describes the detailed and systematic survey of SSL approaches in Section 3. In Section 4, we report an experimental comparative analysis performed on representa- tive approaches that are surveyed in Section 3. Finally, in Section 5, we summarize our study and conclusions as well as we address some important issues for future research.

2. A BRIEF OVERVIEW OF SUPERVISED AND UNSUPERVISED SENTIMENT ANALYSIS

Most of the studies about tweet sentiment analysis utilize supervised learning al- gorithms to produce sentiment classification models (see Figure 1). Such algorithms require a training set formed by labeled data, where the labels are the classes (e.g., positive, neutral, and negative) of each tweet. Some studies propose the use of emoti- cons and hashtags for building the training set, including Go et al. [2009] and Davidov et al. [2010], who identified tweet polarity by using emoticons as class labels—this type of strategy is known as distant supervision–based classification. Deep learning approaches have also used emoticons as class labels to refine the embeddings on a large distant supervised corpus [Severyn and Moschitti 2015; Tang et al. 2015, 2014a, 2014b, 2014c]. Other algorithms use the characteristics of the social network as networked data, as in Hu et al. [2013].

The lexicon-based approaches depend on the availability of a sentiment lexicon, which is a collection of known and previously created sentiment words. These ap- proaches can be categorized into two different groups: (i) dictionary based, which use

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:4 N. F. F. da Silva et al.

Fig. 2. Typical methods of learning according to required human efforts. Semi-Supervised Learning estab- lishes a synergy between supervised and unsupervised learning by compensating for the lack of labeled instances with unlabeled ones and is thus particularly useful for building sentiment classification models.

dictionaries as lexical resources [Taboada et al. 2011; Qiu et al. 2010; Hu and Liu 2004; Kim and Hovy 2004], and (ii) corpus-based, which use statistical or semantic methods to find sentiment polarity [Turney 2001, 2002].

Approaches that integrate opinion mining lexicon-based techniques and machine- learning-based techniques have also been investigated (known as hybrid approaches). For example, Agarwal et al. [2011], Read [2005], Zhang et al. [2011], and Mohammad et al. [2013] used lexicons, part-of-speech, and writing style as linguistic resources. In a similar context, Saif et al. [2012] introduced an approach to add semantics to the training set as an additional feature. More recently, classifier ensembles have been successfully used [da Silva et al. 2014; Silva et al. 2014; Lin and Kolcz 2012; Clark and Wicentwoski 2013; Rodrıguez-Penagos et al. 2013; Hassan et al. 2013].

The seminal work on sentiment classification that does not depend on labeled data was proposed by Turney [2002], in which a document is predicted as either positive or negative by taking into account the semantic orientation of its phrases that contain adjectives or adverbs. His approach was assessed on automobile reviews and movie reviews, which are data sources that differconsiderably from short texts such as those found in tweets. Along the same line, Read and Carroll [2009] put forward different forms to quantify the similarity between words and polarity words (based on lexical association, semantic spaces, and distributional similarity). Because labeled data are not used by unsupervised learning approaches, they are expected to be less accurate than those based on supervised learning. From this aspect, prominent human-machine systems for sentiment analysis should address the scarcity of labeled tweets (taking advantage of unlabeled ones, such as is done by unsupervised models) and provide better classification (as those usually reached by supervised models).

3. SEMI-SUPERVISED LEARNING FOR SENTIMENT ANALYSIS

Semi-Supervised Learning takes advantage of both unlabeled and labeled data during the training phase [Balcan and Blum 2010; Goldberg 2010; Zhu and Goldberg 2009]. Therefore, as shown in Figure 2, SSL fits in between supervised and unsupervised learning. For supervised approaches, all training instances must be labeled and more

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:5

interactivity with the users (experts) is required. This dependency decreases in SSL approaches, in which a balance between supervised and unsupervised learning is found [Chapelle et al. 2006].

Given a labeled data set, Dl = {(xi , yi )|(xi, yi ) ∈ X × Y, i = 1, . . . , l}, and an unlabeled data set, Du = {x j |x j ∈ X, j = l + 1, . . . , l + u}, in which X denotes the input space of data instances and Y is the label space, according to Zhu et al. [2009] “a semi- supervised algorithm aims to train a classifier f from Dl ∪ Du, that is, from both the labeled and unlabeled data, such that it is better than a supervised learner induced on the labeled data alone” (p. 9). Thus, SSL is particularly appropriate in cases where obtaining an unlabeled sample is cheap and easy, while labeling the sample is expensive or difficult [Chapelle et al. 2006]—as a consequence, typically unlabeled data is much more accessible and available than labeled data, that is, u � l. This is the case of several sentiment analysis applications, especially when the data source comes from social networks (e.g., Twitter).

We identify three categories of semi-supervised approaches for tweet sentiment anal- ysis: (i) graph-based methods, (ii) wrapper-based methods (e.g., self-training and co- training), and (iii) topic-based methods. To address these categories and understand the context and the development of research on the subject, we also provide an overview of approaches that manipulate other types of data sources, including web pages, online news, Internet discussion groups, online reviews, and web blogs.

3.1. Graph-Based Methods

Graph-based methods propagate labels to unlabeled data. The label propagation pro- cess requires the computation of similarities among the data instances. Similarities are captured through a graph G = 〈V, E〉, where each vertex vi from the vertex set V represents an instance xi ∈ X and each edge (vi, v j ) from the edge set E is associated with a non-negative weight wi j . Such a weight indicates the similarity between vi and v j . In addition, V = Vl

⋃ Vu, where each vertex in Vl has an initial label y ∈ Y and all

vertices in Vu are unlabeled. In the area of sentiment analysis, the literature on graph-based algorithms focuses

on sentiments related to either sentences or full documents. The investigated appli- cations range from document polarity classification [Sandler et al. 2008; Sindhwani and Melville 2008; Ren et al. 2014], document rating prediction [Goldberg and Zhu 2006; Zhu and Goldberg 2007], and identification of political affiliation [Lu et al. 2010] to algorithms that learn sentiment polarity lexicons from a few seed words [Rao and Yarowsky 2009].

The use of a suitable similarity measure, usually dependent on the specific task of interest, is the key to the successful application of graph-based algorithms because it determines the distance between two data instances and, as a consequence, how similar the probability distributions of their labels should be. In other words, such algorithms work only if a proper similarity measure exists such that the assumption holds. For document-level sentiment analysis, finding the similarity measure is non-trivial. Typ- ically, cosine similarity based on bag-of-words representation is employed. However, this favors topic similarity rather than sentiment similarity. Indeed, a high similarity value often suggests that two documents share numerous content words rather than similar sentiments. As shown in Goldberg and Zhu [2006], using the cosine similarity with bag-of-words representation, algorithms performed worse than the support vector regression [Joachims 1999] for rating prediction of movie reviews. By changing that measure to the positive-sentence percentage-based similarity, which is computed as the percentage of positive sentences in a document, graph-based algorithms outper- form their supervised counterparts when the quantity of labeled documents is limited [Pang and Lee 2005].

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:6 N. F. F. da Silva et al.

Instead of defining a proper similarity measure to construct a similarity graph, Sandler et al. [2008] and Sindhwani and Melville [2008] proposed different methods for constructing similarity graphs. Sandler et al. [2008] encoded prior knowledge into a graph of word features, in which the vertices represent words and the edges represent similarities between them.

On the other hand, the use of graph-based methods has been motivated by the avail- able social information, which can help to capture sentiments of particular users [Tan et al. 2011; Pozzi et al. 2013]. Users that can somehow be categorized as “follower” or “followee” are more likely to hold similar sentiments. Accordingly, relationship infor- mation can help what can be extracted about users perspectives that are originated from textual features only. It is worth noting that similarity measures still have a key role in these approaches. However, they are now based on users’ characteristics. In particular, the graph should capture the fact that some users share similar opinions. These approaches are supported by many social studies [Lazarsfeld and Merton 1954; McPherson et al. 2001; Thelwall 2010].

Tan et al. [2011] proposed a formulation of a “Twitter Graph,” where it is considered a “query” topic that includes users who have tweeted about this, while omitting users who have never expressed themselves about the query topic. The goal is to distinguish between users who show positive feelings about the topic and those who have negative feelings about the topic. A connection edge between two users is set up if one follows or mentions the other.

Johnson et al. [2012] investigated a graph-based method called “label propagation.” The general idea behind their method is to build a weighted graph where the users, tweets, and other features are the set of vertices. The edges connecting the vertices are derived from retweets and their weights are related to the relative frequency ratio of the unigram or bigram in the training data. Given such a graph structure, a label distribution is initially seeded to a subset of vertices and then spread across the graph.

Calais Guerra et al. [2011] proposed a transfer-learning approach for tweet senti- ment analysis. It is based on textual resources and the prediction of social media user bias. Transfer learning is applicable when classification is hampered (e.g., because of outdated data and lack of labeled instances) and improvements can be reached through supplementary knowledge that is derived from similar concepts [Li et al. 2010; Pan and Yang 2010]. According to Calais Guerra et al. [2011], it is possible to use “social media endorsements from retweets to quantify user bias towards a topic. Endorsements may be represented as a directed graph, where an edge represents that a user endorsed or retweeted a tweet from a user.”

3.2. Wrapper-Based Methods

A wrapper-based method uses a supervised learning algorithm in an iterative fashion. In each iteration, a certain amount of unlabeled instances is labeled by the decision function that is learned and incorporated into the training data. From its own predic- tions and the labeled data already available, the classification model is retrained for the next iteration. The well-known representatives of this category are self-training [Scudder 1965] and co-training [Blum and Mitchell 1998]:

3.2.1. Self-Training. Essentially, Zhu and Goldberg [2009] state that “the learning pro- cess uses its own predictions to teach itself ” (p. 15). Self-training starts with a super- vised learner that is trained on available labeled data and then iterates several times. In each iteration, it selects a subset of predicted instances to augment the training data. Typically, this subset contains instances for which the predictions have shown higher confidence levels. Then, the new training data are used to update or retrain the supervised learner for the next iteration. This iterative learning process implies that

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:7

the method only works if the highest confidence predictions are effectively correct [Zhu et al. 2009].

Self-training has been applied in several contexts. For example, the algorithm AROW [Crammer et al. 2009] makes use of self-training for large-scale reviews of polarity prediction. Haimovitch et al. [2012] show that AROW can reduce test errors by more than half compared to the supervised classifier trained on the initial labeled data.

Another sentiment classification approach based on self-training was proposed by Zagibalov and Carroll [2008]. In this work, self-training is used to add sentiment lexical items into the vocabulary for Chinese text. Liu et al. [2013a] also used a self-training approach for sentiment analysis in a Chinese microblog.

Similarly to Zagibalov and Carroll [2008], Qiu et al. [2009] utilized an iterative process based on lexicon to increase a sentiment dictionary. The approach considers a massive Chinese sentiment dictionary instead of employing a one-word seed dictionary as in Zagibalov and Carroll [2008]. Documents predicted in the initial phase are used as the training data to build Support Vector Machines (SVMs), which are subsequently employed in the refinement of the primary results.

Finally, approaches that employ self-training for increasing the size of the feature space can be found in Becker et al. [2013], Baugh [2013], and Zhao et al. [2014], in which the training process leads to the inclusion of additional polarity lexicons. The main motivation of these approaches is to adapt a static polarity lexicon with the help of an unlabeled tweet set.

3.2.2. Co-Training. It adopts an iterative learning process similar to self-training, but instead of using a single supervised learner, it uses two learners that teach each other [Blum and Mitchell 1998]. The two learners operate on different feature sets, which are referred to as (independent) views of the data. The most confident classifications from each learner are then used to iteratively build more labeled data which, in turn, are used for training. The process finishes when all unlabeled data have been used or a specific number of iterations has been reached.

As in self-training, co-training has successful applications in sentiment analysis. For the cross-lingual document polarity classification [Wan 2009], each view used by co- training is a set of language-based features. For problems where the class distribution is imbalanced, Li et al. [2011] proposed an under-sampling method to generate balanced datasets of different views. Yu [2014] revisited co-training in depth, discussing several strategies for sentiment analysis in three domains: news articles, online reviews, and blog posts. Liu et al. [2013c] also designed a two-view (textual and non-textual views) approach for tweets classification based on the co-training framework. In their ap- proach, two classifiers are trained on a common set of labeled tweets. According to Liu et al. [2013b], they proposed a semi-supervised adaptive SVM model that augments the labeled set and expands topic-adaptive features based on the unlabeled data available.

3.3. Topic-Based Methods

The methods reviewed above only consider features that capture local information in the data (e.g., lexicons, unigrams,3 bigrams,4 and part-of-speech). Specifically, they do not consider global, higher-level information, such as topic information that may some- how influence sentiments. In particular, the same word may have different sentiment polarities in different domains. For instance, though the adjective “complex” in the sentence “The book is complex and exciting!” may have a positive orientation in a book review, it could also have a negative orientation in the sentence “It is hard to use such

3Words or tokens. 4Sequence of two adjacent words in a text of tokens.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:8 N. F. F. da Silva et al.

Fig. 3. Topic-based approach for tweet sentiment analysis proposed in Xiang and Zhou [2014].

a complex cellphone” in an electronic review. Therefore, it is more suitable to analyze topics and sentiments simultaneously.

Topic information has been applied in different domains of sentiment analysis. In the seminal studies [Mei et al. 2007; Jo and Oh 2011; He et al. 2012], all of the training data are required to infer the classes of unlabeled instances. More recently, Si et al. [2013] used a continuous Dirichlet Process Mixture model to learn daily topic sets. Then, for each topic, the sentiment is derived according to an opinion word distribution aiming to build a sentiment time series. The sentiment was estimated based on a lexicon (a list of positive and negative opinion words, e.g., “good” and “bad”).

In Xiang and Zhou [2014], a topic-based SSL is used to analyze sentiments from tweets. The authors proposed building a topic model on labeled tweets5 so specific sentiment models can be induced on each cluster that was found. Figure 3 summarizes such an approach. First, a classifier is inferred from labeled tweets, and then it is used to estimate the class probabilities for each unlabeled tweet (this primary step occurs only once). After this (in Step 4), a subset of tweets with class probability higher than a confidence threshold is selected. They are then included in the labeled tweets set. In step 5, the labeled tweets are used to build a topic model, from which topic distributions are stored for each tweet. Then, clusters based on the topic distributions are inferred and a particular sentiment model is trained for each cluster. The resulting sentiment mixture model is used to classify the unlabeled tweets (Steps 7 and 3). An iterative

5The topic information is generated through topic modeling based on the implementation of Latent Dirichlet Allocation (LDA) [Blei et al. 2003].

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:9

Table I. Overview of Literature on SSL for Tweet Sentiment Analysis. Evaluations Were Carried Out on Proprietary Unlabeled Data though the Training Data Were Typically Public. F-Scores Are Shown

for Illustrative Purposes

Approach Work Dataset #labeled #unlabeled F-score

Self-training Becker et al.

[2013] SemEval 2013

[Nakov et al. 2013] 9,829 – Public

485,112 – Proprietary

0.641

Baugh [2013]

SemEval 2013 [Nakov et al. 2013]

8,750 – Public

910,000 – Proprietary

0.543

Zhao et al. [2014]

SemEval 2013 [Nakov et al. 2013]

8,471 – Public

N/A – Proprietary

0.637

Co-training Liu et al. [2013c]

TREC 2011 Microblogging and proprietary dataset

16,000,000 – Proprietary

N/A – Proprietary

N/A

Liu et al. [2013b]

“Taco Bell”, Sanders-Twitter

Sentiment, and 2008 Presidential debate corpus (all together)

10,537 – Public

N/A – Proprietary

The performance was assessed on different

sample ratios Topic

Modeling Xiang and

Zhou [2014] SemEval 2013

[Nakov et al. 2013] 9,684 – Public

2,000,000 – Proprietary

0.703

process takes place until a certain number of iterations has been reached or no more tweets have been promoted to the set of labeled tweets.

4. COMPARATIVE STUDY

A comprehensive comparison on SSL methods for tweet sentiment analysis is not an easy task. The main difficulties come from the fact that there is no consensus about which features are the best or which proportion of unlabeled data should be used. Most of the datasets have limited use because they are not publicly available or because they involve proprietary data.

Our goal was to conduct controlled experiments with fair and instructive comparisons among the different methods.6 To do so, we used a set of standardized features from public data, using no additional data to formulate the semi-supervised phase. Table I shows an overview of the studies on SSL for tweet sentiment analysis that was surveyed in Section 3. These studies employ evaluations performed on proprietary (unlabeled) data; thus, reproducibility is obviously an issue. In addition, we did not include graph- based approaches because they require information about the user network.

4.1. Datasets

Table II summarizes the datasets used in our comparative study. We utilize datasets employed by the organizers of the International Workshop on Semantic Evaluation (SemEval),7 which is a leading scientific event in this field. As suggested by the orga- nizers of SemEval 2013 (task 2) and SemEval 2014 (task 9) competitions, the dataset known as SemEval 2013 was used to induce classification models. Actually, this dataset is currently the most used for tweet sentiment analysis, in addition to being represen- tative and publicly available with a considerable size [Nakov et al. 2013; Rosenthal et al. 2014]. The induced models were then assessed on five test sets, namely Live- Journal, SMS2013, Twitter2013, Twitter2014, and Twitter Sarcasm 2014. Essentially, the datasets LiveJournal and SMS2013 were included to determine how systems that are trained on Twitter perform on other sources (particularly from web blogs and cell

6Software is available on request from the authors. 7http://en.wikipedia.org/wiki/SemEval.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:10 N. F. F. da Silva et al.

Table II. Class Distributions of Training and Test Sets That Were Used. Sentiment Classification Models Were Induced on a Set of Labeled Tweets (SemEval 2013 [Nakov et al. 2013]). The ResultsWere Obtained on Five

Test Sets, Namely LiveJournal, SMS2013, Twitter2013, Twitter2014, and Twitter Sarcasm 2014

Training set Name Positive Negative Neutral Total

SemEval 2013 [Nakov et al. 2013] 4,215 (37%) 1,807 (15%) 5,325 (48%) 11,338 Test sets

LiveJournal [Rosenthal et al. 2014] 427 (37%) 304 (27%) 411 (36%) 1,142 SMS2013 [Nakov et al. 2013] 492 (23%) 394 (19%) 1,207 (58%) 2,093

Twitter2013 [Nakov et al. 2013] 1,572 (41%) 601 (16%) 1,640 (43%) 3,813 Twitter2014 [Rosenthal et al. 2014] 982 (53%) 202 (11%) 669 (36%) 1,853

Twitter Sarcasm 2014 [Rosenthal et al. 2014] 33 (38%) 40 (47%) 13 (15%) 86

phone messages). They were labeled by the Amazon Mechanical Turk8 annotators. Twitter2013 was obtained in a process formed by three phases: First, named enti- ties were extracted from millions of tweets that were collected over a one-year period spanning from January 2012 to January 2013 using the public streaming Twitter API. Then, popular topics, such as those named entities that were frequently mentioned in association with a specific date, were identified. Finally, given this set of automatically identified topics, tweets were gathered from the same time period related to the named entities. Twitter2013 has different topics from training and spanned later periods. Twitter2014 and Twitter Sarcasm 2014 were obtained more recently. The latter was collected by the #sarcasm hashtag with the goal of determining how sarcasm affects the tweet polarity.

4.2. Feature Engineering

Different studies have used different features to represent tweet messages. In fact, it is expected that a set of the chosen features properly fits the classification model adopted. For example, approaches in Table I have employed Ngrams and emoticons [Liu et al. 2013c, 2013b] or only Ngrams [Baugh 2013]. Others adopted a more complex feature space also containing part-of-speech tags, lexicons, and hashtags [Becker et al. 2013; Xiang and Zhou 2014; Zhao et al. 2014]. The feature set used in our experiments was inspired from Mohammad et al. [2013], whose authors ranked first in SemEval 2013 [Nakov et al. 2013]. Such a feature set also achieved the highest scores on LiveJournal, Twitter Sarcasm 2014, and SMS2013 in SemEval 2014 [Rosenthal et al. 2014]. It is composed of the following:

(i). Ngrams: unigrams, bigrams, and trigrams. (ii). Negation: the number of negated contexts. A negated context according to Pang

et al. [2002] is a segment of a tweet that starts with a negative word (e.g., “no” and “shouldn’t”) and ends with a punctuation as a comma, period, colon, semicolon, exclamation mark, or question mark. A negated context affects the ngram and lexicon features, so it added the suffix “NEG” to each word following the negation word (e.g., “good” became “good NEG”). A list of negation words was adopted from Christopher Potts’ sentiment tutorial.9

(iii). Part of Speech: a part-of-speech tagging was carried out by using Ark-twitter NLP [Owoputi et al. 2013] and the number of occurrences of each part-of-speech tag was computed.

8https://www.mturk.com. 9http://sentiment.christopherpotts.net/lingstruc.html.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:11

(iv). Writing Style: we considered the presence of three or more repeated characters in the words, the sequence of three or more punctuation marks, and the number of words with all letters in uppercase.

(v). Lexicons: the number of positive and negative words computed by the lexicon- based method [Mohammad et al. 2013]

(vi). Microblogging features: the total number of sentiment hashtags in the text provided by sentiment lexicons and emoticons [Hu and Liu 2004; Thelwall et al. 2010; Mohammad et al. 2013].

4.3. Experimental Setup

To perform a fair comparison among the existing semi-supervised tweet sentiment analysis methods, we used only public datasets, as shown in Table II. The literature indicates that Naive Bayes, SVM with linear kernel, and Logistic Regression are the most used algorithms in tweet sentiment analysis [Nakov et al. 2013]. We have chosen SVM to perform our experiments because it provides a good out-of-sample general- ization, usually providing better classification accuracy compared to Naive Bayes and Logistic Regression in tweet classification applications. As in Mohammad et al. [2013], we used a linear kernel with parameter C = 0.005.

Two widely known SSL approaches were used in our comparative study, namely self-training [Scudder 1965] and co-training [Blum and Mitchell 1998]. The advantages and disadvantages of the topic-based approach introduced in Xiang and Zhou [2014] are also analyzed. In addition, we performed a tweet sentiment classification using distant supervision [Go et al. 2009], where the sentiment classes in the training set are replaced by positive and negative emoticons and hastags, and in cases where there are no emoticons the tweets are considered as neutral. Note that, in this method, human effort is not required to annotate the training set. Thus, this is indeed a useful baseline for comparison purposes. For self-training and distant supervision–based classification, we considered all features mentioned in Section 4.2. Because the model being constructed learns iteratively by aggregating new reliable data, we adopted specific confidence thresholds according to Scudder [1965]. In particular, because the classifier gives confidence scores when it labels instances from unlabeled data, those instances with confidence scores higher than the predefined threshold are promoted to the labeled set. In our experiments, we evaluated the following values of this parameter: 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9. For co-training, we considered the textual features (i.e., unigrams, bigrams, and trigrams) as one view, and the features from (ii) to (vi) in Section 4.2 as the second view (also referred as lexicon view). Figure 4 illustrates the use of these views by the co-training approach. Based on Blum and Mitchell [1998], we set the number of samples per class, which are classified with the best confidence levels, as p = 3, n = 2, and ne = 4 for positive, negative, and neutral classes, respectively. These parameters were defined from the distribution of classes in the training set. The size of the smaller pool U ′ was set to 10% of the training set.

To evaluate how the algorithms perform with different amounts of initial labeled data, we randomly sampled a proportion, s, of labeled tweets from the training set (maintaining the balance of the three classes). The remaining (1 − s) tweets were used in the learning phase—in which a certain number of instances were incorporated and used for adapting the classification model (in our experiments such a phase consisted of 40 iterations of the algorithms). The proportion of initial labeled tweets, s, was varied as a percentage of the number of instances from the training set as 1%, 5%, 10%, 20%, and 40%.

The F-score (F1) was adopted for evaluating the accuracy of the algorithms. We computed the F-score for each class (positive, negative, and neutral) and the overall

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:12 N. F. F. da Silva et al.

Fig. 4. Conceptual schema of textual and lexicon views being used in the co-training method. The classifier models cooperate between themselves.

F-score (F1), which was obtained by (F1 positive + F1negative)/2 [Becker et al. 2013; Baugh 2013; Zhao et al. 2014; Xiang and Zhou 2014]. Algorithms were run 20 times and the averages and standard deviations were reported.

The computational implementation uses the Natural Language Toolkit (nltk)10 for preprocessing tweet messages, the Scikit-Learn (sklearn)11 for classification (SVM), and gensim12 for topic modeling with hierarchical LDA.

4.4. Results and Discussion

4.4.1. Topic-Based Approach. The performance of the topic-based approach (Section 3.3) is highly dependent on the choice of the confidence threshold and the inference of the number of topics (clusters). If the number of labeled tweets is small and the chosen confidence threshold is high, then the learning process will be hampered, whereas if the threshold is low, the model will learn wrong classes. In Xiang and Zhou [2014], the authors chose an arbitrary value for the number of topics. We adopt a more principled approach, based on the Hierarchical Dirichlet Process (HDP) [Gordon et al. 2011], which is widely used in applications where different groups of data may share the same settings of partitions. The adopted approach does not require the number of topics to be provided in advance, that is, the number is estimated directly from data.

Because the training set was collected over a 1-year period, spanning from January 2012 to January 2013, we may have different samplings with a wide variety of tweets. In this scenario, the HDP tends to obtain clusters with tweets from only one class or two classes, especially if the threshold is high. After clustering the training set based on topic distributions, the next phase is to train a separate sentiment model for each

10http://www.nltk.org. 11http://scikit-learn.org/. 12http://radimrehurek.com/gensim/.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:13

cluster. However, if the clusters only have tweets from one or two classes, then the algorithm cannot proceed because the groups of tweets must reflect the probability distribution over the three classes under study.

The algorithm was unable to learn the classes with 1%, 5%, 10%, 20%, and 40% of the training set and confidence thresholds ranging from 0.4 to 0.9. Taking into account samplings with 60% of the training set and the threshold set to 0.9, some learning progress was observed, but it is still not compatible with the self-training and co-training algorithms. It is likely that the algorithm worked well in Xiang and Zhou [2014] because the authors used 2M tweets as additional unlabeled data, with a threshold of 0.96, and all the training set consisted of labeled tweets. By doing this, it is possible to have representative clusters that, in turn, are good models of classification.

4.4.2. Self-Training and Co-Training. We focus on the overall F-score curves as the number of promoted instances increases over the iterations, as well as on the different amounts of the initial labeled instances. In particular, we explored the F-score relation between positive and negative classes.

Figure 5 illustrates the overall F-score curves when 1% of the training set—SemEval 2013 [Nakov et al. 2013]—was used as initial labeled data. Algorithms run for 40 iterations. Co-training resulted in significant learning, demonstrated by the increasing F-score curves, particularly for the datasets LiveJournal, SMS2013, Twitter2013, and Twitter2014. For self-training, low confidence thresholds (such as 0.6) are likely to have deteriorated the learning process because the promoted instances are more likely to have been misclassified. However, higher thresholds (such as 0.9) allow us to select more useful and noise-free data but are typically more difficult to obtain. Self-training with a threshold set at 0.9 achieved good F-scores for Twitter 2014. By setting the threshold at 0.7 (i.e., a more balanced threshold), self-training resulted in a competitive performance on LiveJournal and better results on Twitter Sarcasm 2014. This occurs because LiveJournal has texts from a web blog, where there is less slang and typos, and no character limit for the user.

Figure 6 shows the overall F-scores achieved by different proportions of initial labeled tweets (1%, 5%, 10%, 20%, and 40%) after 40 iterations of the algorithms. As can be seen, with limited labeled instances (e.g., 1% of the training set) the best choice is to use co-training. However, if more labeled instances are available, self-training can obtain better results, being potentially more useful for these scenarios. Typically, self- training is an algorithm with a sensitive parameter; however, in our experiments we observed that self-training with a confidence threshold equal to 0.9 offered the best results in general (i.e., considering different proportions of initial labeled tweets). It is worth mentioning that in the presence of irony and sarcasm (i.e., considering the Twitter Sarcasm 2014 dataset), self-training was the best choice when the size of initial labeled data was 5%, 10%, 20%, and 40%.

Figures 7 and 8 show the F-scores for positive and negative classes (individually) after 40 iterations of the algorithms and different amounts of initial labeled instances. These results show that co-training performed better with limited data and without the presence of irony and sarcasm. With at least 10% of the training set as initial data, self-training is the best choice to solve the problem.

Table III shows specific results for the self-training (with confidence threshold of 0.9) and co-training approaches. The F-score measure for each class (positive, negative, and neutral) and the overall F-score, which was obtained by averaging the F-scores from positive and negative classes, are presented. Better results are highlighted in bold and the best results found in the literature are also reported. From this table, we can extract some interesting findings:

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:14 N. F. F. da Silva et al.

Fig. 5. Overall F-score curves for 40 iterations of the self-training and co-training approaches. For self- training, four different confidence thresholds were assessed. The size of initial labeled data corresponds to 1% of the original training sets.

(1) Because the LiveJournal set is composed of formal texts with no slang, with 40% of the training data the self-training approach obtained an overall F-score of 66.68%, which is close to the overall F-score achieved by SVM that was implemented with the whole training set (67.34%).

(2) Because SMS2013 has a well-defined vocabulary (with known slang), with only 1% of labeled instances, the co-training approach obtained an overall F-score of 52.39%, which is close to the same result that was obtained for the whole training set (55.50%) with SVM. Better results were achieved by using 40% of the training set and self-training, with an overall F-score equal to 59.36%.

(3) By using 40% of the training set, an overall F-score of 59.13% was obtained by self-training on Twitter2013. For Twitter2014, in the same scenario, the overall F-score is 58.95%. Such values are close to the results observed when the whole training set is used (66.05% and 63.69%, respectively).

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:15

Fig. 6. Overall F-score for different sizes of initial labeled data, which correspond to 1%, 5%, 10%, 20%, and 40% of the original training sets. Such results were obtained after 40 iterations of the algorithms.

(4) Zhu et al. [2014] achieved the best results on LiveJournal, SMS2013, and Twitter Sarcasm 2014 by training a SVM classifier with all labeled instances available and using cluster features as extra features that were inferred from 56 million English language tweets. The 1,000 clusters found are an alternative represen- tation of tweet content. The authors emphasized that this strategy reduces the sparsity of the token space, because the n-grams-based features are replaced by the representative elements of the data partition.

(5) Miura et al. [2014] yielded the best results on Twitter2013 and Twitter2014 by using Logistic Regression on all labeled data. Several lexicons and pre-processors were employed to enhance the lexical information. In addition, because the distri- bution of sentiment on training set is previously known, the authors proposed a weighting scheme that biases the learning process.

(6) Although self-training with 40% of the training set provided competitive results (39.65%) in comparison with SVM on all labeled tweets (41.08%), the modest results

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:16 N. F. F. da Silva et al.

Fig. 7. F-scores for the positive class and different percentages of initial labeled data. Such results were obtained after 40 iterations of the algorithms.

on Twitter Sarcasm 2014 suggest that more research efforts are necessary. In particular, the study of features that can properly represent irony and sarcasm is required. Such features may leverage the performance of human-machine systems in these scenarios. It is also possible that by using a different evaluation measure (other than F-score), one might get more competitive results on Twitter Sarcasm 2014.

(7) Table IV summarizes the results for self-training and co-training by providing the percentages of higher F-scores and lower standard deviations found in Table III for each test set. Note that self-training in general provides better F-scores than co-training and is more stable in most cases as well.

4.4.3. Distant Supervision-Based Classification. We run experiments with an unsupervised approach known as tweet sentiment classification using distant supervision [Go et al. 2009]. In this approach, the sentiment classes from the training set are replaced by

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:17

Fig. 8. F-scores for the negative class and different percentages of initial labeled data. Such results were obtained after 40 iterations of the algorithms.

positive and negative emoticons13 and hashtags [Mohammad et al. 2013]. A tweet is considered as neutral when it does not contain emoticons or hashtags. Therefore, from distant supervision there is no human effort in the annotation of the data. In our experiments, new training sets were created with the same tweets, but their classes were based on the presence or absence of emoticons and sentiment hashtags.

Table V summarizes the results with tweet sentiment classification using distant supervision. All results are worse compared to self-training and co-training (even con- sidering only 1% of labeled tweets). This occurs due to the small percentage of tweets in this data set with emotions and sentiment hashtags, since only 842 tweets had emoticons or sentiment hashtags (what represents 7.4% of the training set).

As mentioned in Section 2, the distant supervision–based classification has been widely used. However, it requires large datasets to achieve satisfactory prediction

13http://en.wikipedia.org/wiki/List_of_emoticons.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:18 N. F. F. da Silva et al.

Ta b le

II I.

F -S

co re

s fo

r D

iff e re

n t

P e rc

e n ta

g e s

o f th

e In

iti a lL

a b e le

d D

a ta

fo r

S e lf-

Tr a in

in g ,

C o -T

ra in

in g ,

a n d

S u p e rv

is e d

A p p ro

a ch

e s

fr o m

th e

L ite

ra tu

re

L iv

eJ o u

rn a

l A

p p

ro a

ch 1

% 5

% 1 0

% 2 0

% 4 0

% 1 0 0

% F

1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

S el

f- tr

a in

in g

4 2 .2

4 ±1

1 .7

8 2 9 .0

2 ±1

1 .0

3 5 0 .6

8 ±1

4 .8

1 3 5 .6

3 ±7

.5 8

6 1 .8

2 ±3

.7 4

4 7 .3

5 ±5

.2 5

6 2 .3

9 ±4

.6 8

5 4 .5

9 ±3

.3 8

6 6 .3

4 ±1

.5 4

5 4 .5

7 ±4

.0 6

6 6 .8

0 ±0

.7 5

6 0 .4

5 ±2

.1 9

6 9 .3

9 ±0

.8 7

6 0 .1

5 ±3

.9 0

6 8 .6

2 ±0

.9 2

6 4 .7

7 ±2

.0 3

6 9 .5

4 ±1

.3 9

6 3 .8

1 ±5

.1 8

7 0 .2

6 ±0

.7 3

6 6 .6

8 ±3

.0 6

– –

– –

C o-

tr a in

in g

6 5 .9

7 ±1

.5 1

4 2 .4

9 ±1

6 .4

0 6 3 .2

8 ±5

.2 8

5 4 .2

3 ±8

.2 1

6 4 .5

9 ±4

.4 6

4 3 .2

3 ±8

.7 9

6 4 .9

9 ±1

.3 5

5 3 .9

1 ±5

.8 4

6 2 .0

2 ±3

.4 9

4 6 .7

3 ±9

.2 5

6 4 .1

9 ±1

.4 1

5 4 .3

8 ±4

.9 3

6 2 .6

2 ±3

.1 3

5 2 .4

5 ±4

.1 9

6 5 .5

5 ±0

.8 5

5 7 .5

4 ±3

.2 3

6 0 .5

4 ±1

.2 8

5 5 .0

2 ±3

.0 1

6 5 .7

1 ±0

.7 8

5 7 .7

8 ±1

.4 9

– –

– –

S V

M –

– –

– –

– –

– –

– –

– –

– –

– –

– –

– 6 8 .8

4 6 1 .7

5 5 5 .3

7 6 7 .3

4 L

it er

a tu

re S

u p

er vi

se d

a p

p ro

a ch

p ro

p os

ed b y

Z h

u et

a l.

[2 0 1 4 ]

7 4 .8

4

S M

S 2 0 1 3

A p

p ro

a ch

1 %

5 %

1 0

% 2 0

% 4 0

% 1 0 0

% F

1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

S el

f- tr

a in

in g

3 9 .4

8 ±3

.7 1

2 8 .1

5 ±4

.3 8

5 4 .7

2 ±2

0 .2

8 3 3 .8

1 ±2

.8 8

5 0 .8

7 ±3

.8 3

4 2 .1

3 ±4

.1 8

6 4 .8

1 ±7

.5 5

4 6 .5

0 ±3

.5 1

5 6 .7

8 ±1

.6 7

4 8 .2

1 ±2

.5 9

7 2 .1

8 ±1

.7 5

5 2 .5

0 ±1

.5 0

5 9 .8

0 ±1

.1 5

5 3 .6

4 ±1

.7 3

7 5 .6

0 ±1

.7 5

5 6 .7

2 ±1

.1 2

6 2 .9

3 ±1

.9 5

5 5 .7

9 ±2

.7 3

7 6 .2

8 ±4

.2 0

5 9 .3

6 ±0

.7 0

– –

– –

C o-

tr a in

in g

5 9 .2

3 ±3

.1 7

4 3 .1

3 ±1

7 .2

8 7 5 .9

3 ±2

.9 7

5 2 .3

9 ±1

2 .1

4 5 9 .3

2 ±2

.5 7

4 0 .6

6 ±7

.0 7

7 9 .0

3 ±0

.7 5

4 9 .9

9 ±4

.1 6

5 7 .9

0 ±1

.4 7

4 2 .9

3 ±6

.4 8

7 9 .5

3 ±0

.6 3

5 0 .4

2 ±3

.2 8

5 9 .0

0 ±1

.2 1

4 7 .3

8 ±3

.1 3

8 0 .1

3 ±0

.3 7

5 3 .1

9 ±1

.7 6

5 7 .8

8 ±2

.1 2

4 9 .4

7 ±1

.4 0

8 0 .5

6 ±0

.2 3

5 3 .6

7 ±1

.3 5

– –

– –

S V

M –

– –

– –

– –

– –

– –

– –

– –

– –

– –

– 5 9 .3

1 5 1 .6

9 5 7 .6

4 5 5 .5

0 L

it er

a tu

re S

u p

er vi

se d

a p

p ro

a ch

p ro

p os

ed b y

Z h

u et

a l.

[2 0 1 4 ]

7 0 .2

8

T w

it te

r2 0

1 3

A p

p ro

a ch

1 %

5 %

1 0

% 2 0

% 4 0

% 1 0 0

% F

1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

S el

f- tr

a in

in g

4 0 .9

7 ±6

.1 8

2 6 .1

0 ±6

.8 8

6 1 .2

7 ±2

.5 9

3 8 .0

3 ±4

.8 1

6 0 .6

3 ±2

.0 6

4 1 .5

5 ±1

.7 9

6 7 .3

7 ±0

.8 7

5 1 .0

9 ±1

.7 7

6 3 .8

5 ±1

.0 1

4 4 .9

6 ±1

.7 6

6 9 .2

5 ±0

.3 2

5 4 .4

1 ±1

.2 2

6 6 .4

6 ±0

.6 3

4 9 .0

9 ±1

.0 5

7 0 .2

5 ±0

.3 1

5 7 .7

7 ±0

.6 7

6 7 .6

9 ±0

.4 4

5 0 .5

7 ±0

.7 0

7 1 .2

7 ±0

.4 9

5 9 .1

3 ±0

.3 5

– –

– –

C o-

tr a in

in g

6 6 .8

0 ±1

.4 7

3 5 .7

6 ±1

3 .1

3 6 7 .5

1 ±2

.0 8

5 1 .2

8 ±7

.1 4

6 5 .2

7 ±3

.1 7

3 6 .5

5 ±5

.9 2

6 8 .8

7 ±0

.4 7

5 0 .9

1 ±3

.6 9

6 4 .5

9 ±1

.8 8

3 8 .0

9 ±4

.2 0

6 8 .6

8 ±0

.5 0

5 1 .3

4 ±2

.1 0

6 5 .2

1 ±1

.1 9

4 2 .3

2 ±2

.1 9

6 9 .3

8 ±0

.4 0

5 3 .7

6 ±1

.5 4

6 5 .5

4 ±0

.7 9

4 3 .7

6 ±1

.3 7

7 0 .2

5 ±0

.3 9

5 4 .6

5 ±0

.9 3

– –

– –

S V

M –

– –

– –

– –

– –

– –

– –

– –

– –

– –

– 7 0 .7

0 6 1 .4

1 6 5 .8

1 6 6 .0

5 L

it er

a tu

re S

u p

er vi

se d

a p

p ro

a ch

p ro

p os

ed b y

M iu

ra et

a l.

[2 0 1 4 ]

7 2 .1

2

T w

it te

r2 0

1 4

A p

p ro

a ch

1 %

5 %

1 0

% 2 0

% 4 0

% 1 0 0

% F

1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

S el

f- tr

a in

in g

5 0 .2

2 ±8

.1 6

2 7 .9

2 ±5

.8 3

5 4 .0

4 ±2

.2 4

3 9 .0

7 ±5

.9 9

5 9 .5

1 ±2

.1 3

4 1 .9

2 ±1

.7 0

5 9 .4

1 ±0

.7 5

5 0 .7

1 ±1

.5 6

6 2 .7

2 ±1

.3 0

4 5 .9

8 ±0

.8 9

6 1 .3

5 ±0

.5 4

5 4 .3

5 ±0

.7 7

6 6 .6

1 ±0

.6 7

4 8 .1

2 ±1

.6 4

6 3 .2

9 ±0

.4 4

5 7 .3

6 ±0

.8 8

6 8 .0

7 ±0

.3 6

4 9 .8

2 ±1

.3 0

6 4 .3

9 ±0

.3 8

5 8 .9

5 ±0

.6 2

– –

– –

C o-

tr a in

in g

6 7 .9

3 ±2

.2 6

3 8 .6

9 ±1

4 .0

0 6 1 .4

0 ±1

.4 4

5 3 .3

1 ±7

.6 5

6 6 .4

1 ±2

.9 5

4 1 .1

7 ±6

.6 4

6 2 .9

9 ±0

.6 7

5 3 .7

9 ±4

.1 3

6 6 .1

4 ±2

.3 1

4 3 .4

4 ±4

.4 4

6 3 .0

4 ±0

.6 1

5 4 .7

9 ±2

.2 5

6 7 .0

5 ±1

.5 4

4 5 .7

7 ±1

.2 6

6 3 .7

8 ±0

.6 2

5 6 .4

1 ±0

.9 8

6 6 .9

2 ±0

.8 9

4 5 .1

7 ±2

.1 8

6 4 .2

1 ±0

.5 6

5 6 .0

4 ±1

.3 3

– –

– –

S V

M –

– –

– –

– –

– –

– –

– –

– –

– –

– –

– 7 1 .8

7 5 5 .5

1 5 7 .7

3 6 3 .6

9 L

it er

a tu

re S

u p

er vi

se d

a p

p ro

a ch

p ro

p os

ed b y

M iu

ra et

a l.

[2 0 1 4 ]

7 0 .9

6

T w

it te

r S

a rc

a sm

2 0 1 4

A p

p ro

a ch

1 %

5 %

1 0

% 2 0

% 4 0

% 1 0 0

% F

1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

F 1 -

p os

F 1 -

n eg

F 1 -

n eu

F 1

S el

f- tr

a in

in g

4 1 .0

8 ±1

1 .1

3 1 6 .9

7 ±1

0 .6

8 3 2 .2

6 ±4

.7 0

2 9 .0

2 ±7

.3 2

4 9 .7

6 ±4

.8 7

1 9 .5

4 ±6

.4 6

3 7 .0

0 ±5

.4 6

3 4 .6

5 ±4

.2 3

4 9 .9

0 ±1

.5 3

1 9 .9

0 ±4

.9 3

3 7 .6

2 ±1

.1 1

3 4 .9

0 ±2

.2 6

5 4 .1

8 ±2

.0 9

2 2 .4

4 ±2

.4 5

4 2 .6

6 ±1

.5 0

3 8 .3

1 ±1

.9 1

5 4 .5

4 ±2

.1 7

2 4 .7

6 ±2

.3 6

4 4 .8

3 ±3

.3 9

3 9 .6

5 ±1

.3 0

– –

– –

C o-

tr a in

in g

5 1 .1

3 ±4

.7 1

1 0 .5

0 ±4

.2 5

4 0 .9

0 ±6

.1 9

2 9 .8

2 ±3

.9 2

4 8 .8

1 ±5

.5 8

1 0 .2

9 ±4

.0 6

3 8 .1

5 ±3

.3 9

2 9 .5

5 ±3

.4 8

4 7 .8

2 ±6

.4 5

9 .3

4 ±4

.8 2

3 6 .4

9 ±1

.8 6

2 8 .5

8 ±3

.4 9

4 5 .6

0 ±4

.0 8

1 0 .7

9 ±3

.0 7

3 7 .0

4 ±1

.6 9

2 8 .2

0 ±3

.0 6

4 3 .4

2 ±4

.1 2

1 3 .4

7 ±3

.0 2

3 6 .9

0 ±1

.3 8

2 8 .4

4 ±2

.4 8

– –

– –

S V

M –

– –

– –

– –

– –

– –

– –

– –

– –

– –

– 5 7 .1

5 2 5 .0

0 5 0 .0

0 4 1 .0

8 L

it er

a tu

re S

u p

er vi

se d

a p

p ro

a ch

p ro

p os

ed b y

Z h

u et

a l.

[2 0 1 4 ]

5 8 .1

6

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:19

Table IV. Number of Higher F-Scores and Lower Standard Deviations (%) for Self-Training and Co-Training According to the Results from Table III

Self-training Co-training Dataset Higher F1s Lower Stds Higher F1s Lower Stds LiveJournal 70% 65% 30% 35% SMS2013 65% 50% 55% 50% Twitter2013 65% 80% 35% 20% Twitter2014 50% 80% 50% 20% Twitter Sarcasm 2014 80% 60% 20% 40%

Table V. Distant Supervision Results on the Five Test Sets

LiveJournal2014 positive negative neutral

precision recall F1 precision recall F1 precision recall F1 6.56 62.22 11.86 0.33 100.00 0.66 96.84 36.31 52.82

F1: 6.26 SMS2013

positive negative neutral precision recall F1 precision recall F1 precision recall F1 5.49 60.00 10.06 0.51 40.00 1.00 98.51 58.20 73.17

F1:5.53 Twitter2013

positive negative neutral precision recall F1 precision recall F1 precision recall F1 11.45 76.60 19.92 0.33 40.00 0.66 97.38 44.70 61.27

F1: 10.29 Twitter2014

positive negative neutral precision recall F1 precision recall F1 precision recall F1 11.41 86.82 20.16 1.49 75.00 2.91 98.06 38.14 54.92

F1:11.54 Twitter Sarcasm 2014

positive negative neutral precision recall F1 precision recall F1 precision recall F1 3.03 33.33 5.56 0.00 0.00 0.00 92.31 14.46 25.00

F1:2.78

power. For example, Severyn and Moschitti [2015] collected 60M tweets over a 2-month period and Tang et al. [2014c] collected 10 million tweets in April, 2013.

5. CONCLUSIONS

We surveyed SSL approaches applied to tweet sentiment analysis. Tweet applications with semi-supervised settings are motivated by the fact that labeled tweets are typi- cally expensive and difficult to obtain, whereas unlabeled tweets are generally widely available at low cost. Aiming to provide additional instructive guidelines for those inter- ested in SSL-based tweet classification approaches, we also reported an experimental comparative analysis on real-world data from state-of-the-art algorithms. From this perspective, our study is helpful for new developments of human-machine systems for tweet sentiment analysis.

We empirically compared three SSL approaches namely: Self-training, Co-training, and Topic Modeling. In general, Co-training performed better without the presence of irony and sarcasm and with limited data (i.e., using at most 5% of the labeled

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:20 N. F. F. da Silva et al.

data available). However, Self-training is the best choice when a significant amount of labeled tweets are available. In addition, the Self-training approach was observed to be more useful when irony and sarcasm are present. Considering samplings with 60% of the training set and a confidence threshold of 0.9, some learning progress has been observed for the Topic-based approach, but such a performance is still not compatible with those shown by the Self-training and Co-training algorithms.

As an emerging research topic, the use of SSL to address tweet sentiment analysis faces many challenges that motivate relevant future work such as:

(1) When selecting a portion of available data for the initial training phase, some features may not make sense for the purpose of building classifiers. Therefore, it is important to evaluate the impact of the chosen features in a semi-supervised setting. In our experiments, the selected features were inspired in Mohammad et al. [2013]; these authors ranked first in SemEval 2013 [Nakov et al. 2013]. However, this feature set was defined based on the entire training set. It is more realistic to select features on the fly as an intrinsic part of the SSL. In addition, techniques for dimensionality reduction as principal component analysis (PCA), information gain, correlation-based feature selection (CFS) [Chong et al. 2014; O’Keefe and Koprinska 2009], and feature hashing [da Silva et al. 2014] are worth studying.

(2) To increase the performance of SSL methods in applications where sarcasm and irony are present, it is interesting to study specific features, such as in Carvalho et al. [2009], González-Ibáñez et al. [2011], Vanin et al. [2013], and de Freitas et al. [2014]. The feature set used in our experiments did not consider this particular scenario.

(3) In Xiang and Zhou [2014], a topic model must be inferred from training data. The process of learning and updating the model in semi-supervised phase has a high computational cost, so this approach has shown to be ineffective in our experiments. We believe that combining classification and clustering for tweet sentiment analysis in a semi-supervised approach is promising and should be explored with a topic model from the testing set instead of from the training set as (unrealistically) done in Xiang and Zhou [2014]. One idea is to capture the similarities among the tweets that are being classified, such that the classifier can be refined from additional information provided by clusterers, as proposed in da Silva et al. [2016] and Coletta et al. [2015].

(4) Most SSL algorithms have been designed for binary classification. In tweet senti- ment analysis, these approaches have been extended to multi-class classification without any adaptation. Additional problems with extending semi-supervised bi- nary classifiers to multi-class problems include imbalanced classification and differ- ent output scales of different binary classifiers. To adopt a binary SSL algorithm to problems with more than two classes, such as speech recognition and object recog- nition, multi-class problems are usually decomposed into a number of independent binary classification problems using techniques such as one-versus-the-rest, one- versus-one, and error-correcting output coding [Dietterich and Bakiri 1995]. This type of solution has not yet been applied in tweet sentiment analysis and is a promising future work.

(5) Given that the learning process of semi-supervised classifiers still depends on a small initial sample of labeled data, another interesting research topic involves studying sampling methods. Zhu et al. [2003] combines active and SSL in a Gaussian random field model and indicates that the active learning scheme requires a much smaller number of queries to active high accuracy compared to random query selection. Recently, active learning has been applied in sentiment

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:21

classification of movies and product reviews [Dasgupta and Ng 2009] with success, thereby suggesting that they can be applied to tweet sentiment analysis.

(6) A dynamic and online tweet sentiment analysis is also an interesting research area that was not addressed in this article. It has been studied for different applications [Dahal et al. 2015; Domingos and Hulten 2000], with few studies for sentiment analysis [Lourenco Jr. et al. 2014; Kim et al. 2013; Mejova and Srinivasan 2012; Bifet et al. 2011; Bifet and Frank 2010], especially when a semi-supervised setting is considered [Kmieciak and Stefanowski 2011; Masud et al. 2008].

REFERENCES

Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of twitter data. In Proceedings of the Workshop on Languages in Social Media (LSM’11). Association for Computational Linguistics, Stroudsburg, PA, 30–38.

Maria-Florina Balcan and Avrim Blum. 2010. A discriminative model for semi-supervised learning. J. ACM 57, 3, Article 19 (2010), 46 pages.

Wesley Baugh. 2013. bwbaugh: Hierarchical sentiment analysis with partial self-training. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh Inter- national Workshop on Semantic Evaluation (SemEval 2013). Association for Computational Linguistics, Atlanta, GA, 539–542.

Lee Becker, George Erhart, David Skiba, and Valentine Matula. 2013. AVAYA: Sentiment analysis on twitter with self-training and polarity lexicon expansion. In Second Joint Conference on Lexical and Compu- tational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). Association for Computational Linguistics, Atlanta, GA, 333–340.

Albert Bifet and Eibe Frank. 2010. Sentiment knowledge discovery in twitter streaming data. In Proceedings of the 13th International Conference on Discovery Science (DS’10). Springer-Verlag, Berlin, 1–15.

Albert Bifet, Geoffrey Holmes, and Bernhard Pfahringer. 2011. MOA-TweetReader: Real-time analysis in twitter streaming data. In Proceedings of the 14th International Conference on Discovery Science (DS’11). Springer-Verlag, Berlin, 46–60.

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. J. Mach. Learn. Res. 3 (March 2003), 993–1022.

Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory (COLT’98). ACM, New York, NY, 92–100.

Pedro Henrique Calais Guerra, Adriano Veloso, Wagner Meira, Jr., and Virgı́lio Almeida. 2011. From bias to opinion: A transfer-learning approach to real-time sentiment analysis. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’11). ACM, New York, NY, 150–158.

Paula Carvalho, Luı́s Sarmento, Mário J. Silva, and Eugénio de Oliveira. 2009. Clues for detecting irony in user-generated contents: Oh...!! It’s “So Easy” ;-). In Proceedings of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion (TSA’09). ACM, New York, NY, 53–56.

O. Chapelle, B. Schölkopf, and A. Zien. 2006. Semi-Supervised Learning. MIT Press, Cambridge, MA. Marc Cheong and Vincent C. Lee. 2011. A microblogging-based approach to terrorism informatics: Explo-

ration and chronicling civilian sentiment and response to terrorism events via twitter. Informat. Syst. Front. 13, 1 (March 2011), 45–59.

Terence Tai-Leung Chong, Bingqing Cao, and Wing-Keung Wong. 2014. A new principal-component approach to measure the investor sentiment. In IGEF Working Paper.

Sam Clark and Rich Wicentwoski. 2013. SwatCS: Combining simple classifiers with estimated accuracy. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). Association for Computational Linguistics, Atlanta, GA, 425–429.

Luiz F. S. Coletta, Eduardo R. Hruschka, Ayan Acharya, and Joydeep Ghosh. 2015. Using metaheuristics to optimize the combination of classifier and cluster ensembles. Integr. Comput.-Aid. Eng. 22, 3 (2015), 229–242.

Koby Crammer, Alex Kulesza, and Mark Dredze. 2009. Adaptive regularization of weight vectors. In NIPS. 414–422.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:22 N. F. F. da Silva et al.

Nadia F. F. da Silva, Luiz F. S. Coletta, Eduardo R. Hruschka, and Estevam R. Hruschka Jr. 2016. Using unsupervised information to improve semi-supervised tweet sentiment classification. Informat. Sci. (2016), 1–18. DOI:http://dx.doi.org/10.1016/j.ins.2016.02.002

Nadia F. F. da Silva, Eduardo R. Hruschka, and Estevam R. Hruschka Jr. 2014. Tweet sentiment analysis with classifier ensembles. Decision Support Syst. 66 (2014), 170–179.

N. Dahal, O. Abuomar, R. King, and V. Madani. 2015. Event stream processing for improved situational awareness in the smart grid. Expert Syst. Appl. 42, 20 (2015), 6853–6863.

Sajib Dasgupta and Vincent Ng. 2009. Mine the easy, classify the hard: A semi-supervised approach to automatic sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2 (ACL’09). Association for Computational Linguistics, Stroudsburg, PA, 701–709.

Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced sentiment learning using twitter hashtags and smileys. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters (COLING’10). Association for Computational Linguistics, Stroudsburg, PA, 241–249.

Larissa A. de Freitas, Aline A. Vanin, Denise N. Hogetop, Marco N. Bochernitsan, and Renata Vieira. 2014. Pathways for irony detection in tweets. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (SAC’14). ACM, New York, NY, 628–633.

Nicholas A. Diakopoulos and David A. Shamma. 2010. Characterizing debate performance via aggregated twitter sentiment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’10). ACM, New York, NY, 1195–1198.

Thomas G. Dietterich and Ghulum Bakiri. 1995. Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 2 (1995), 263–286.

Pedro Domingos and Geoff Hulten. 2000. Mining high-speed data streams. In Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’00). ACM, New York, NY, 71–80. DOI:http://dx.doi.org/10.1145/347090.347107

Ronen Feldman. 2013. Techniques and applications for sentiment analysis. Commun. ACM 56, 4 (April 2013), 82–89.

Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. Unpublished Manuscript, Stanford University (2009), 1–6.

A. B. Goldberg. 2010. New Directions in Semi-Supervised Learning. Ph.D. Dissertation. University of Wisconsin—Madison. http://books.google.com.br/books?id=fO3RXwAACAAJ.

Andrew B. Goldberg and Xiaojin Zhu. 2006. Seeing stars when there aren’t many stars: Graph-based semi- supervised learning for sentiment categorization. In Proceedings of the First Workshop on Graph Based Methods for Natural Language Processing (TextGraphs-1). Association for Computational Linguistics, Stroudsburg, PA, 45–52.

Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguis- tics: Human Language Technologies: Short Papers - Volume 2 (HLT’11). Association for Computational Linguistics, Stroudsburg, PA, 581–586.

Geoffrey J. Gordon, David Blei Dunson, and Miroslav Dudı́k (Eds.). 2011. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011. JMLR Proceedings, Vol. 15. JMLR.org.

Yoav Haimovitch, Koby Crammer, and Shie Mannor. 2012. More is better: Large scale partially-supervised sentiment classification - appendix. CoRR abs/1209.6329 (2012).

Ammar Hassan, Ahmed Abbasi, and Daniel Zeng. 2013. Twitter sentiment analysis: A bootstrap ensemble framework. In SocialCom. IEEE, Los Alamitos, CA, 357–364.

Yulan He, Chenghua Lin, Wei Gao, and Kam-Fai Wong. 2012. Tracking sentiment and topic dynamics from social media. In ICWSM.

Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’04). ACM, New York, NY, 168–177.

Xia Hu, Lei Tang, Jiliang Tang, and Huan Liu. 2013. Exploiting social relations for sentiment analysis in microblogging. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining.

Bernard J. Jansen, Mimi Zhang, Kate Sobel, and Abdur Chowdury. 2009. Twitter power: Tweets as electronic word of mouth. J. Am. Soc. Inf. Sci. Technol. 60, 11 (Nov. 2009), 2169–2188.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:23

Yohan Jo and Alice H. Oh. 2011. Aspect and sentiment unification model for online review analysis. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (WSDM’11). ACM, New York, NY, 815–824.

Thorsten Joachims. 1999. Advances in kernel methods. MIT Press, Cambridge, MA, 169–184. Christopher Johnson, Parul Shukla, and Shilpa Shukla. 2012. On Classifying the Political Sentiment of

Tweets. www.cs.utexas.edu. Hwi-Gang Kim, Seongjoo Lee, and Sunghyon Kyeong. 2013. Discovering hot topics using twitter streaming

data: Social topic detection and geographic clustering. In Proceedings of the 2013 IEEE/ACM Interna- tional Conference on Advances in Social Networks Analysis and Mining (ASONAM’13). ACM, New York, NY, 1215–1220.

Soo-Min Kim and Eduard Hovy. 2004. Determining the sentiment of opinions. In Proceedings of the 20th International Conference on Computational Linguistics (COLING’04). Association for Computational Linguistics, Stroudsburg, PA.

Milosz R. Kmieciak and Jerzy Stefanowski. 2011. Handling sudden concept drift in enron messages data stream. Control Cybernet. 40(3) (2011), 667–695.

P. F. Lazarsfeld and R. K. Merton. 1954. Friendship as a social process: A substantive and methodological analysis. In Freedom and Control in Modern Society, M. Berger, T. Abel, and C. Page (Eds.). Van Nostrand, New York, NY, 18–66.

Shoushan Li, Zhongqing Wang, Guodong Zhou, and Sophia Yat Mei Lee. 2011. Semi-supervised learning for imbalanced sentiment classification. In IJCAI. 1826–1831.

Tao Li, Vikas Sindhwani, Chris H. Q. Ding, and Yi Zhang. 2010. Bridging domains with words: Opinion analysis with matrix tri-factorizations. In SDM. 293–302.

Jimmy Lin and Alek Kolcz. 2012. Large-scale machine learning at twitter. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD’12). ACM, New York, NY, 793– 804.

Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Morgan & Claypool Publishers, London. Shenghua Liu, Fuxin Li, Fangtao Li, Xueqi Cheng, and Huawei Shen. 2013b. Adaptive co-training SVM

for sentiment classification on tweets. In Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management (CIKM’13). ACM, New York, NY, 2079–2088.

Shenghua Liu, Wenjun Zhu, Ning Xu, Fangtao Li, Xue-qi Cheng, Yue Liu, and Yuanzhuo Wang. 2013c. Co-training and visualizing sentiment evolvement for tweet events. In Proceedings of the 22nd Interna- tional Conference on World Wide Web Companion (WWW’13 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 105–106.

Zhiguang Liu, Xishuang Dong, Yi Guan, and Jinfeng Yang. 2013a. Reserved self-training: A semi-supervised sentiment classification method for chinese microblogs. In Proceedings of the Sixth International Joint Conference on Natural Language Processing. Asian Federation of Natural Language Processing, Nagoya, Japan, 455–462.

Roberto Lourenco Jr., Adriano Veloso, Adriano Pereira, Wagner Meira Jr., Renato Ferreira, and Srinivasan Parthasarathy. 2014. Economically-efficient sentiment stream analysis. In Proceedings of the 37th Inter- national ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR’14). ACM, New York, NY, 637–646.

Yue Lu, Panayiotis Tsaparas, Alexandros Ntoulas, and Livia Polanyi. 2010. Exploiting social context for review quality prediction. In Proceedings of the 19th International Conference on World Wide Web (WWW’10). ACM, New York, NY, 691–700.

Mohammad Masud, Jing Gao, Latifur Khan, Jiawei Han, and Xiaohu Li. 2008. A practical approach to classify evolving data streams: Training with limited amount of labeled data. Proc. 2008 Int. Conf. on Data Mining (ICDM’08), Pisa, Italy, Dec. 2008 (December 2008).

Miller McPherson, Lynn Smith-Lovin, and James M. Cook. 2001. Birds of a feather: Homophily in social networks. Annu. Rev. Sociol. 27, 1 (2001), 415–444.

Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applications: A survey. Ain Shams Eng. J. 5, 4 (2014), 1093–1113.

Qiaozhu Mei, Xu Ling, Matthew Wondra, Hang Su, and ChengXiang Zhai. 2007. Topic sentiment mixture: Modeling facets and opinions in weblogs. In Proceedings of the 16th International Conference on World Wide Web (WWW’07). ACM, New York, NY, 171–180.

Yelena Mejova and Padmini Srinivasan. 2012. Political speech in social media streams: YouTube comments and twitter posts. In Proceedings of the 4th Annual ACM Web Science Conference (WebSci’12). ACM, New York, NY, 205–208.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:24 N. F. F. da Silva et al.

Yasuhide Miura, Shigeyuki Sakaki, Keigo Hattori, and Tomoko Ohkuma. 2014. TeamX: A sentiment ana- lyzer with enhanced lexicon mapping and weighting scheme for unbalanced data. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics and Dublin City University, Dublin, Ireland, 628–632.

Saif Mohammad. 2012. #Emotional tweets. In *SEM 2012: The First Joint Conference on Lexical and Com- putational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). Association for Computational Linguistics, Montréal, Canada, 246–255.

Saif Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the Seventh International Workshop on Semantic Evaluation Exercises (SemEval-2013). Atlanta, GA.

AndréS Montoyo, Patricio Martı́Nez-Barco, and Alexandra Balahur. 2012. Subjectivity and sentiment anal- ysis: An overview of the current state of the area and envisaged developments. Decis. Support Syst. 53, 4 (Nov. 2012), 675–679. DOI:http://dx.doi.org/10.1016/j.dss.2012.05.022

Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. SemEval-2013 task 2: Sentiment analysis in twitter. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). Association for Computational Linguistics, Atlanta, GA, 312– 320.

Tim O’Keefe and Irena Koprinska. 2009. Feature selection and weighting methods in sentiment analysis. In Proceedings of 14th Australasian Document Computing Symposium.

Olutobi Owoputi, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved part- of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL.

Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 10 (Oct. 2010), 1345–1359.

Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL. 115–124.

Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of EMNLP. 79–86.

FedericoAlberto Pozzi, Daniele Maccagnola, Elisabetta Fersini, and Enza Messina. 2013. Enhance user- level sentiment analysis on microblogs with approval relations. In AI*IA 2013: Advances in Artificial Intelligence, Matteo Baldoni, Cristina Baroglio, Guido Boella, and Roberto Micalizio (Eds.). Lecture Notes in Computer Science, Vol. 8249. Springer International Publishing, Berlin, 133–144.

Ashequl Qadir and Ellen Riloff. 2013. Bootstrapped learning of emotion hashtags #hashtags4you. In Pro- ceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. Association for Computational Linguistics, Atlanta, GA, 2–11.

Guang Qiu, Xiaofei He, Feng Zhang, Yuan Shi, Jiajun Bu, and Chun Chen. 2010. DASA: Dissatisfaction- oriented advertising based on sentiment analysis. Expert Syst. Appl. 37, 9 (2010), 6182–6191.

Likun Qiu, Weishi Zhang, Changjian Hu, and Kai Zhao. 2009. SELC: A self-supervised model for sentiment classification. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM’09). ACM, New York, NY, 929–936.

Delip Rao and David Yarowsky. 2009. Ranking and semi-supervised classification on large scale graphs using map-reduce. In Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4). Association for Computational Linguistics, Stroudsburg, PA, 58–65.

Jonathon Read. 2005. Using emoticons to reduce dependency in machine learning techniques for sentiment classification. In Proceedings of the ACL Student Research Workshop (ACLstudent’05). Association for Computational Linguistics, Stroudsburg, PA, 43–48.

Jonathon Read and John Carroll. 2009. Weakly supervised techniques for domain-independent sentiment classification. In Proceedings of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion (TSA’09). ACM, New York, NY, 45–52.

Yong Ren, Nobuhiro Kaji, Naoki Yoshinaga, and Masaru Kitsuregawa. 2014. Sentiment classification in under-resourced languages using graph-based semi-supervised learning methods. IEICE Trans. 97-D, 4 (2014), 790–797.

Carlos Rodrıguez-Penagos, Jordi Atserias, Joan Codina-Filba, David Garcıa-Narbona, Jens Grivolla, Patrik Lambert, and Roser Saurı. 2013. FBM: Combining lexicon-based ML and heuristics for social media polarities. In Proceedings of SemEval-2013 – International Workshop on Semantic Evaluation Co-located with *Sem and NAACL. Atlanta, GA.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

A Survey and Comparative Study of Tweet Sentiment Analysis via Semi-Supervised Learning 15:25

Sara Rosenthal, Preslav Nakov, Alan Ritter, and Veselin Stoyanov. 2014. SemEval-2014 task 9: Sentiment analysis in twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 14), Preslav Nakov and Torsten Zesch (Eds.). Dublin, Ireland.

Hassan Saif, Yulan He, and Harith Alani. 2012. Semantic sentiment analysis of twitter. In Proceedings of the 11th International Conference on The Semantic Web - Volume Part I (ISWC’12). Springer-Verlag, Berlin, 508–524.

Hassan Saif, Yulan He, Miriam Fernández, and Harith Alani. 2016. Contextual semantics for sentiment analysis of Twitter. Inf. Process. Manage. 52, 1 (2016), 5–19. http://dx.doi.org/10.1016/j.ipm.2015.01.005.

Ted Sandler, John Blitzer, Partha Pratim Talukdar, and Lyle H. Ungar. 2008. Regularized learning with networks of features. In NIPS, Daphne Koller, Dale Schuurmans, Yoshua Bengio, and Lon Bottou (Eds.). Curran Associates, Inc., Red Hook, NY, 1401–1408.

H. Scudder, III 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Inform. Theor. 11, 3 (Jul 1965), 363–371. DOI:http://dx.doi.org/10.1109/TIT.1965.1053799

Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM Comput. Surv. 34, 1 (March 2002), 1–47.

Aliaksei Severyn and Alessandro Moschitti. 2015. Twitter sentiment analysis with deep convolutional neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). ACM, New York, NY, 959–962.

Jianfeng Si, Arjun Mukherjee, Bing Liu, Qing Li, Huayi Li, and Xiaotie Deng. 2013. Exploiting topic based twitter sentiment for stock prediction. In ACL (2). 24–29.

Nádia Silva, Estevam Hruschka, and Eduardo Hruschka. 2014. Biocom Usp: Tweet sentiment analysis with adaptive boosting ensemble. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics and Dublin City University, Dublin, Ireland, 123–128.

Vikas Sindhwani and Prem Melville. 2008. Document-word co-regularization for semi-supervised sentiment analysis. In ICDM. 1025–1030.

Jared Suttles and Nancy Ide. 2013. Distant supervision for emotion classification with discrete binary values. In Computational Linguistics and Intelligent Text Processing, Alexander Gelbukh (Ed.). Lecture Notes in Computer Science, Vol. 7817. Springer Berlin, 121–136.

Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Comput. Linguist. 37, 2 (June 2011), 267–307.

Chenhao Tan, Lillian Lee, Jie Tang, Long Jiang, Ming Zhou, and Ping Li. 2011. User-level sentiment analysis incorporating social networks. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’11). ACM, New York, NY, 1397–1405.

Duyu Tang, Bing Qin, and Ting Liu. 2015. Deep learning for sentiment analysis: Successful approaches and future challenges. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 5, 6 (2015), 292–303. DOI:http://dx.doi.org/10.1002/widm.1171

Duyu Tang, Furu Wei, Bing Qin, Ting Liu, and Ming Zhou. 2014a. Coooolll: A deep learning system for twitter sentiment classification. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics and Dublin City University, Dublin, Ireland, 208–212.

Duyu Tang, Furu Wei, Bing Qin, Ming Zhou, and Ting Liu. 2014b. Building large-scale twitter-specific sentiment lexicon: A representation learning approach. In COLING 2014, 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23–29, 2014, Dublin, Ireland. 172–182.

Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014c. Learning sentiment-specific word embedding for twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22–27, 2014, Baltimore, MD, USA, Volume 1: Long Papers. 1555–1565.

Harsh Thakkar and Dhiren Patel. 2013. Approaches for sentiment analysis on twitter: A state-of-art study. In International Network for Social Network Analysis Conference (INSNA). Xi’an, China.

Mike Thelwall. 2010. Emotion homophily in social network site messages. First Monday 15, 4 (2010). Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas. 2010. Sentiment in short

strength detection informal text. J. Am. Soc. Inf. Sci. Technol. 61, 12 (Dec. 2010), 2544–2558. Mikalai Tsytsarau and Themis Palpanas. 2012. Survey on mining subjective data on the web. Data Min.

Knowl. Discov. 24, 3 (May 2012), 478–514.

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.

15:26 N. F. F. da Silva et al.

Peter D. Turney. 2001. Mining the web for synonyms: PMI-IR versus LSA on TOEFL. In Proceedings of the 12th European Conference on Machine Learning (EMCL’01). Springer-Verlag, London, 491–502. http://dl.acm.org/citation.cfm?id=645328.650004

Peter D. Turney. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classifica- tion of reviews. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL 02). Association for Computational Linguistics, Stroudsburg, PA, 417–424.

Aline A. Vanin, Larissa A. Freitas, Renata Vieira, and Marco Bochernitsan. 2013. Some clues on irony detection in tweets. In Proceedings of the 22nd International Conference on World Wide Web Companion (WWW’13 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 635–636.

G. Vinodhini and R. M. Chandrasekaran. 2012. Sentiment analysis and opinion mining: A survey. Int. J. 2, 6 (2012).

Xiaojun Wan. 2009. Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Con- ference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1 (ACL’09). Association for Computational Lin- guistics, Stroudsburg, PA, 235–243.

Changbo Wang, Zhao Xiao, Yuhua Liu, Yanru Xu, Aoying Zhou, and Kang Zhang. 2013. SentiView: Sentiment analysis and visualization for internet popular topics. IEEE T. Hum.-Mach. Syst. 43, 6 (2013), 620–630.

Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P. Sheth. 2012. Harnessing twitter ”big data” for automatic emotion identification. In SocialCom/PASSAT. IEEE, Los Alamitos, CA, 587–592.

Bing Xiang and Liang Zhou. 2014. Improving twitter sentiment analysis with topic-based mixture modeling and semi-supervised training. In Proceedings of the 52nd Annual Meeting of the Association for Com- putational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Baltimore, MD, 434–439.

Jun-Ming Xu, Kwang-Sung Jun, Xiaojin Zhu, and Amy Bellmore. 2012. Learning from bullying traces in social media. In HLT-NAACL. The Association for Computational Linguistics, Baltimore, MD, 656–666.

Ning Yu. 2014. Exploring co-training strategies for opinion detection. J. Assoc. Inform. Sci. Technol. 65, 10 (2014), 2098–2110. http://dx.doi.org/10.1002/asi.23111.

Taras Zagibalov and John Carroll. 2008. Unsupervised classification of sentiment and objectivity in Chi- nese text. In Proceedings of the Third International Joint Conference on Natural Language Processing (IJCNLP).

Lei Zhang, Riddhiman Ghosh, Mohamed Dekhil, Meichun Hsu, and Bing Liu. 2011. Combining lexicon-based and learning-based methods for twitter sentiment analysis. HP Laboratories. Technical Report.

Jiang Zhao, Man Lan, and Tiantian Zhu. 2014. ECNU: Expression- and message-level sentiment orientation classification in twitter using multiple effective features. In Proceedings of the 8th International Work- shop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics and Dublin City University, Dublin, Ireland, 259–264.

Xiaojin Zhu. 2005. Semi-supervised learning literature survey. (2005). Technical Report 1530, Computer Sciences, University of Wisconsin—Madison.

Xiaojin Zhu and Andrew B. Goldberg. 2007. Kernel regression with order preferences. Proc. Natl. Conf. Artif. Intell. 22, 1 (2007), 681.

Xiaojin Zhu and Andrew B. Goldberg. 2009. Introduction to Semi-Supervised Learning. Morgan & Claypool Publishers, London.

Xiaojin Zhu, Andrew B. Goldberg, Ronald Brachman, and Thomas Dietterich. 2009. Introduction to Semi- Supervised Learning. Morgan and Claypool Publishers, London.

Xiaodan Zhu, Svetlana Kiritchenko, and Saif Mohammad. 2014. NRC-Canada-2014: Recent improvements in the sentiment analysis of tweets. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics and Dublin City University, Dublin, Ireland, 443–447.

Xiaojin Zhu, John Lafferty, and Zoubin Ghahramani. 2003. Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. In ICML 2003 Workshop on The Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining. 58–65.

Received June 2015; revised December 2015; accepted March 2016

ACM Computing Surveys, Vol. 49, No. 1, Article 15, Publication date: June 2016.