helpfn

IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021 815

Fast Supervised Topic Models for Short Text Emotion Detection

Jianhui Pang , Yanghui Rao , Member, IEEE, Haoran Xie, Member, IEEE, Xizhao Wang , Fellow, IEEE,

Fu Lee Wang , Senior Member, IEEE, Tak-Lam Wong, Member, IEEE, and Qing Li , Senior Member, IEEE

Abstract—With the development of social network platforms, discussion forums, and question answering websites, a huge num- ber of short messages that typically contain a few words for an individual document are posted by online users. In these short messages, emotions are frequently embedded for communicating opinions, expressing friendship, and promoting influence. It is quite valuable to detect emotions from short messages, but the corresponding task suffers from the sparsity of feature space. In this article, we first generate term groups co-occurring in the same context to enrich the number of features. Then, two basic supervised topic models are proposed to associate emotions with topics accurately. To reduce the time cost of parameter estimation, we further propose an accelerated algorithm for our basic models. Extensive evaluations using three short corpora val- idate the efficiency and effectiveness of the accelerated models for predicting the emotions of unlabeled documents, in addition to generate the topic-level emotion lexicons.

Index Terms—Accelerated algorithm, emotion detection, short text analysis, topic model.

Manuscript received January 17, 2019; revised May 16, 2019 and August 11, 2019; accepted August 26, 2019. Date of publication September 30, 2019; date of current version January 15, 2021. This work was supported in part by the National Natural Science Foundation of China under Grant 61972426, in part by the Interdisciplinary Research Scheme of the Dean’s Research Fund 2018-19 under Grant FLASS/DRF/IDS-3, in part by the Departmental Collaborative Research Fund 2019 under Grant MIT/DCRF-R2/18-19, in part by the Top-Up Fund (TFG-04) and Seed Fund (SFG-10) for General Research Fund/Early Career Scheme of the Dean’s Research Fund (DRF) 2018-19, in part by the General Research Fund Proposal under Grant RG 39/2019- 2020R, in part by the Internal Research Grant of the Education University of Hong Kong under Grant RG 90/2018-2019R, and in part by the Collaborative Research Fund by the Research Grants Council of the Hong Kong under Project C1031-18G. The work of X. Wang was supported by the National Natural Science Foundation of China under Grant 61732011. This article is an extended journal version of a conference paper published at DASFAA 2017 [50]. This article was recommended by Associate Editor S. Ozawa. (Corresponding author: Yanghui Rao.)

J. Pang and Y. Rao are with the School of Data and Computer Science, Sun Yat-sen University, Guangzhou 510006, China (e-mail: [email protected]).

H. Xie is with the Department of Computing and Decision Sciences, Lingnan University, Hong Kong.

X. Wang is with the College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China.

F. L. Wang is with the School of Science and Technology, Open University of Hong Kong, Hong Kong.

T.-L. Wong is with the Department of Computing Studies and Information Systems, Douglas College, New Westminster, BC V3M 5Z5, Canada.

Q. Li is with the Department of Computing, Hong Kong Polytechnic University, Hong Kong.

Color versions of one or more of the figures in this article are available online at https://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TCYB.2019.2940520

I. INTRODUCTION

W ITH the development and popularization of socialmedia services, users are increasingly inclined to com- municate and share emotions on social network platforms, such as Twitter, Facebook, Sina Weibo, and WeChat. By using mobile devices, it is convenient for users to express com- ments on news or personal events, which generates large-scale short messages that are limited in length, usually spanning sev- eral sentences or less. Emotion detection on short messages is therefore quite valuable to capture the emotional tendency of social media users, for example, happy, sad, or surprise, toward entities, brands, or events. However, the feature sparsity of short texts brings huge challenges to traditional word-level algorithms [1], [2]. This is because two short documents may semantically related to each other without sharing any com- mon words. Furthermore, a word can have multiple meanings depending on its context [3]. Thus, another solution to emo- tion detection attempts to extract topics first [4], [5], in which, a topic can represent a real-world event and the topic-level feature space is coherent by grouping semantically related words. Then, the emotions are associated with the topics for the emotion detection of unlabeled documents. Although the aforementioned issue of word-level algorithms can be allevi- ated by mapping the sparse word space to a coherent topic space, a traditional topic model, such as the latent Dirichlet allocation (LDA) [6], fails to generate accurate topics over short messages. This is because a short document lacks enough word occurrence patterns to draw statistical conclusions for such kind of topic models [7]. Recently, Cheng et al. [8] proposed the biterm topic model (BTM) to extract high-quality topics from short messages. BTM assumed that two words that co-occurred in a context (e.g., in the same document) are likely to belong to the same topic. However, the generated topic fea- tures of BTM may be unsuitable to predict emotions without any guidance from labels in the training corpus. Furthermore, BTM is too time consuming to model such large-scale word pairs.

To address the aforementioned issues, we here develop a weighted labeled topic model (WLTM) and an X-term emotion-topic model (XETM) to detect emotions toward cer- tain topics. In the generative process of WLTM, we first define a one-to-many mapping among each emotion and multiple top- ics, by assuming that a single emotion may be evoked by sev- eral topics. Second, we use the emotion distributions of labeled documents to constrain the topic probability for each feature

2168-2267 c© 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

816 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021

during the training process. Finally, we employ the support vector regression (SVR) [9] to predict emotion distributions of unlabeled documents given the estimated topic probabil- ity for each feature. In the generative process of XETM, we draw the emotion-topic probability which exploits abundant user scores over multiple emotions. Then, the topic-feature probability is derived for estimating the emotion probabilities of unlabeled documents. The main characteristics of WLTM and XETM are summarized as follows. First, both WLTM and XETM are supervised topic models which align the generated topics to emotions using the emotion distributions of train- ing documents for guidance. Second, the abundant features are generated by jointly modeling emotion labels and term groups. Particularly, a term group with X words co-occurring in the same context is called X-term. With abundant features, the proposed models allow us to draw statistical conclusions for short documents. Although the sparse feature issue of short messages can be alleviated by WLTM and XETM, the time cost of estimating parameters is high due to the large-scale term groups and the sampling algorithm [10]. To improve the efficiency, we further propose the accelerated models dubbed fWLTM and fXETM for WLTM and XETM by combining the Alias method [11] and the Metropolis–Hastings (MH) sam- pling [12]. Experiments using a sensibly small and unbalanced news headlines with six emotions, a larger and balanced sen- tences annotated with seven emotions, and a Chinese corpus with eight emotions validate the effectiveness of the proposed methods.

The remainder of this article is organized as follows. In Section II, we summarize the related works on emotion detec- tion and short text analysis. In Section III, we detail the basic WLTM and XETM methods, and corresponding accel- erated models called fWLTM and fXETM for short text emotion detection. The experimental evaluations are shown in Section IV, and we draw the conclusions in Section V.

II. RELATED WORK

As one of the basic tasks of affective computing and sen- timent analysis [13], emotion detection aims to identify and extract the attitudes of a subject (i.e., an opinion holder, a com- mentator, and so forth) toward either a topic, an aspect, or the overall tone of a document [14]. Methods of emotion detec- tion are mainly based on the lexicons, supervised learning, and unsupervised learning algorithms. The lexicon-based meth- ods [5], [15]–[19] construct the word-level, concept-level, or topic-level emotional/sentimental dictionaries to detect emo- tions. For example, the emotion-term method [4] associated words with emotions and used the word-emotion dictionary for prediction. The contextual sentiment topic model (CSTM) [20] mined connections between topics and emotions by distilling context-independent information, which were further applied to social emotion classification. The models based on super- vised learning used traditional classification algorithms (e.g., naïve Bayes [21], maximum entropy [22], and support vec- tor machines [23]) or deep learning models (e.g., sentiment embedding-based method [24], deep memory network [25],

hybrid neural network [26], and Sentic LSTM and H-Sentic- LSTM [27]) to detect emotions or sentiments from documents. The unsupervised learning methods detected the sentimental or emotional orientation by counting the co-occurrence frequency between words and positive/negative terms [28]. However, the aforementioned methods were mainly suitable to long articles which typically contain abundant features.

With the prevalence of tweets, questions, instant-messages, and news headlines, several strategies have been proposed to tackle the feature sparse issue of short messages. One solution expanded the content of short documents by trans- ferring topical knowledge from large-scale data collections or auxiliary long texts [29], [30], but it only achieved a good topical distribution when the auxiliary data are closely related to the original corpus. Furthermore, it is difficult to determine the suitable size of external data collections. Another solution to short text analysis exploited the aggre- gated word co-occurrence patterns in the entire corpus for topic learning [8], [31]. For a short document with N words, C2N unordered word pairs, namely, biterms, can be extracted by assuming that two words from the same document share a single topic. Unlike most existing document-level topic mod- els, the above method learns topic components for a corpus using the generated rich biterms. However, it was unsuitable to model labeled documents due to the lack of supervision during the training process. Furthermore, Gibbs sampling was employed by the above model and many other topic mod- els to estimate parameters [8], [32], which is quite time consuming with the increase of the number of documents, features/biterms, or topics. Therefore, we detect emotions of short text by two supervised topic models and further develop an MH sampling in conjunction with the Alias method for accelerating parameter estimation.

III. FAST SUPERVISED TOPIC MODELS

Here, we first present the basic supervised topic models, namely, WLTM and XETM for detecting emotions over short messages. To make the topic sampling more efficient without reducing much topic quality, we further develop accelerated algorithms for both WLTM and XETM.

A. Problem Definition

Before illustrating our supervised topic models for short text emotion detection, we summarize notations, variables, and terms in Table I. Taking a collection of ND short documents {d1, d2, . . . , dND } as an example, the issue of emotion detection is defined as predicting the emotion distribution of unlabeled documents conditioned to labeled data. For each labeled doc- ument d, there are Nd words and scores/ratings over NE emotions, which are denoted as ωd = {ω1, ω2, ω3, . . . , ωNd } and Ed = {Ed,1, Ed,2, . . . , Ed,NE }, respectively. Using each text as a context, we can generate NG unordered term groups that are represented by G = {gi}NGi=1. For instance, a short document with four words will get six term groups when X is 2: (ω1, ω2, ω3, ω4) ⇒ {(ω1, ω2), (ω1, ω3), (ω1, ω4), (ω2, ω3), (ω2, ω4), (ω3, ω4)}. We represent the emotion anno- tation information by a real-valued matrix γ with the size of

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

PANG et al.: FAST SUPERVISED TOPIC MODELS FOR SHORT TEXT EMOTION DETECTION 817

TABLE I NOTATIONS

Fig. 1. Label-topic projection with τ = 5.

ND × NE . Each row of γ is a document’s real-valued vec- tor over NE emotion labels, for example, {1, 0} means that the document is associated with the first emotion, and {3, 1} indicates that the document is tagged to both emotions with strengths of 3 and 1, respectively.

In the first model called WLTM, we assume that each emo- tion can be associated with multiple topics. Take the following two short messages as an example: “I feel surprised about my Christmas gift” and “The examination results surprised me.” Although both messages trigger the emotion of “sur- prise,” we can observe that the distinct topics of “Christmas gift” and “examination” are embedded. To this end, we define a multiplier τ to represent how many topics per emotion involves. Specifically, Fig. 1 presents the projection of emo- tion labels and topics when τ equals 5, in which constant mapping of an emotion to τ topics is adopted. This is con- sistent to LDA’s assumption that a document can be mapped to a given number of topics [6]. We leave the infinite map- ping method, for example, in hierarchical Dirichlet processes to further research, because the parameter estimation is quite time consuming [33]. Through the above mapping of emotions to topics, we can conveniently develop the supervised mech- anism in WLTM. In the second model called XETM, we use an NE × Nz matrix ϕ to denote the multinomial distributions of emotions to topics.

B. Weighted Labeled Topic Model

The graphical representation of WLTM is shown in Fig. 2, where observed and unobservable data are represented by shaded and blank nodes, respectively.

After mapping each emotion to multiple topics via multiplier τ , we could incorporate the supervision of

Fig. 2. Graphical representation of WLTM. � is the emotion label prior for X-terms. τ indicates the number of topics associated with each emotion. � represents the topic binary (presence/absence) indictor. i means the ith X-term.

emotion labels of training documents into topic generation. Furthermore, the number of topics Nz can be tuned by setting different values of τ , that is, Nz = NE ×τ . Although there were supervised topic models, such as labeled LDA (LLDA) [32] being proposed, existing methods mainly exploited the one-to- one correspondence between labels and topics, which renders the number of topics must equal the size of the label set. On the other hand, the label-topic projection in our WLTM is one-to-many. Thus, different aspects can be discovered for each emotion label as mentioned earlier.

To explore document labels in generating topics effectively, we propose to extract an NG × Nz indicator matrix � for all X-terms. For the above matrix, each row �gi is a list of binary topic indictors (i.e., presence/absence) related to the emotion labels of the document that contains X-term gi. In particular, the generation of �gi is as follows. Given τ and emotion label prior information �, for each X-term gi, �gi is the prior emo- tion label with size of 1 × NE . Then, we construct an NE × τ matrix Lgi which means each emotion label is linked to τ topics, as follows:

{ Ljgi

}NE j=1

= {

{1}τ if �jgi ! = 0 {0}τ if �jgi = 0

(1)

where L j gi is the jth row of Lgi and �

j gi is the jth element

of �gi . {1}τ and {0}τ are τ -dimensional vectors with 1 and 0, respectively. Then, we transform Lgi to a 1-D vector �gi with NE × τ (i.e., Nz) elements by appending the vector of following rows to the first row in turn. Take τ = 2, NE = 2, Nz = 4, and a labeled document that contains gi with emotion ratings �gi = {2, 0} as an example, we obtain Lgi =

[ 1 1 0 0

]

according to (1) and �gi = {1, 1, 0, 0}. The generative process of WLTM is presented as follows:

1) For each topic z ∈ [1, Nz], draw φz ∼ Dirichlet (β); 2) For each X-term gi ∈ G: 3) For each topic z ∈ [1, Nz]: 4) Generate �zgi ∈ {0, 1} ∼ Bernoulli(·|�gi , τ ); 5) Generate λgi = {z|�zgi = 1};

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

818 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021

6) Draw θ ∼ Dirichlet (·|α, λ); 7) For each X-term gi ∈ G: 8) Generate zi ∈ λgi ∼ Multinomial(·|θ, α); 9) Generate ωi,1, . . . , ωi,X ∈ gi ∼ Multinomial(φzi ). After generating �gi in step 4, we obtain the related topic

distribution for gi in step 5. Then, the topic assignment zi is drawn from the above distribution according to step 8, which indicates that this restriction limits all topic assignments to the labels of X-term. In the above key steps, note that we explore the topics of each X-term gi in the range of label relative topics and discard the topics not assigned to λgi .

To estimate model parameters, an approximate inference method based on Gibbs sampling [34]–[36] can be used. First, the conditional probability of X-term gi is estimated as follows:

P (

gi|θ̂, φ̂, λgi )

= Nz∑

z=1 P ( ωi,1, . . . , ωi,X , zi = z|θ, φ, λgi

)

= Nz∑

z=1 θz

X∏ x=1

φz,ωi,x . (2)

Second, the likelihood function of all X-terms that should be maximized is given as follows:

P (

G|θ̂ , φ̂, � )

= NG∏ i=1

Nz∑ z=1

θz

X∏ x=1

φz,ωi,x . (3)

Finally, the topic of each X-term gi is sampled by the following conditional probability:

P ( zi = z, z ∈ λgi | ˆz−i, G

)

∝ (n−i,z + α ) ×

γdi,| zτ |∑ z′ γdi,

∣∣∣ z′τ ∣∣∣

X∏ x=1

( n−i,ωi,x|z + β

) ( n−i,·|z + Nωβ

) (4)

where ˆz−i denotes the assigned topics for the group of X-terms, n−i,z represents the number of X-terms that are assigned to topic z, n−i,ω|z is the number of times that word ω is assigned to topic z, n−i,·|z is the number of times for all words that are assigned to topic z, and the notation −i indicates that the number does not include the current assignment of X-term gi. We use di to represent the document from which gi is sampled, and the absolute value of z divides by τ (i.e., |(z/τ )|) to achieve the emotion index. Since the sampling of topics for the ith X-term is restricted according to the emotion labels of documents containing gi (i.e., z ∈ λgi ), the label information is injected into the probability distribution to supervise the topic generation through a weighted mechanism.

After a given number of iterations, we record the number of X-terms that are assigned to topic z, that is, nz, and the number of times word ω being assigned to topic z, that is, nω|z. Then, the probabilities of words conditioned to topics φ and the probabilities of topics over the corpus θ are, respectively, calculated as follows:

φz,ω = nω|z + β

n·|z + Nωβ , θz =

nz + α NG + Nzα

. (5)

Based on the generated topic of each X-term, WLTM cal- culates the topic proportion via computing each document’s posterior topic probability. For each document d, the topic of

Algorithm 1 Gibbs Sampling Algorithm for WLTM Input:

1: τ : Multiplier between topic and emotion numbers; 2: NE : Number of emotion labels; 3: α: Hyperparameter of θ ; 4: β: Hyperparameter of φz; 5: G: The X-term groups in the training set;

Output: 6: φ: Multinomial distributions of words for topics; 7: θ : Multinomial distributions of topics for the corpus; 8: procedure BUILD WLTM 9: Calculate topic numbers Nz by τ × NE ;

10: Randomly initialize topic assignments for all X-terms; 11: repeat 12: for all gi = (ωi,1, ..., ωi,X ) ∈ G do 13: Draw topic z according to Equation (4); 14: Update nz, nωi,1|z, ..., and nωi,X |z; 15: end for 16: until Niter times 17: Compute φ and θ by Equation (5). 18: end procedure

X-term g(d)i = (ω(d)i,1 , . . . , ω(d)i,X ) is assumed to be conditionally independent with each other. After the generation of X-terms, we have P(z|d) = ∑i P(z|g(d)i )P(g(d)i |d), where P(g(d)i |d) is the frequency of X-term gi in document d, and P(z|g(d)i ) can be calculated by the following Bayes rule:

P (

zi = z|g(d)i )

= θz∑ z′ θz′

X∏ x=1

φ z,ω

(d) i,x∑

z′ φz′,ω(d)i,x . (6)

We present the Gibbs sampling algorithm that is used for WLTM in Algorithm 1. After computing the topic probabil- ity of each document P(z|d) as mentioned earlier, we employ the SVR [9] to predict the emotion distributions of unlabeled documents using P(z|d) as the input.

C. X-Term Emotion-Topic Model

Fig. 3 presents the graphical model of XETM, in which, an emotion label ε is first generated under the constraint of prior emotion frequencies. Second, a topic related to emotion label ε is sampled. Finally, we generate an X-term (i.e., a group of X words) for each document.

The generative process of XETM is shown as follows: 1) For emotion ε ∈ [1, NE ], draw ϕε ∼ Dirichlet (α); 2) For each topic z ∈ [1, Nz], draw φz ∼ Dirichlet (β); 3) For each document d ∈ D: 3) For each X-term gi ∈ d: 4) Generate εi ∼ Multinomial(γd ); 5) Generate zi ∼ Multinomial(δϕi ); 6) Generate ωi,1, . . . , ωi,X ∈ gi ∼ Multinomial(φzi ). In the above, εi ∈ E and zi ∈ Z are the assigned emotion

and topic for X-term gi, respectively. Specifically, ε, which is normalized and summed up to 1, is sampled from a multino- mial distribution with emotion ratings that are parameterized by γ . Accordingly, we can estimate the joint probability of all

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

PANG et al.: FAST SUPERVISED TOPIC MODELS FOR SHORT TEXT EMOTION DETECTION 819

Fig. 3. Graphical representation of XETM.

variables for each document as follows:

P(γ , ε, z, G, φ, ϕ; α, β) = P(ϕ; α)P(φ; β)P(γ ) × P(ε|γ )P(z|ε, φ)P(G|z, ϕ). (7)

Particularly, the posterior probability of emotion ε for term gi conditioned to topics is given as follows:

P ( εi = ε|ε̂−i, ẑ, γ , G; α, β

) ∝ α + n−i,zi|ε Nzα +

∑ z n−i,z|ε

× γdi,ε∑ ε′ γdi,ε′

. (8)

Then, we sample a new topic conditioned to the set of X- terms G as follows:

P ( zi = z|ẑ−i, ε̂, γ , G; α, β

) ∝ α + n−i,z|εi Nzα +

∑ z′ n−i,z′|εi

× X∏

x=1

β + n−i,ωi,x|z Nωβ +

∑ ω′ n−i,ω′|z

(9)

where the candidate topic and emotion for sampling are, respectively, denoted as z and ε, the number of times that topic z assigned to emotion ε is represented by nz|ε, the number of times that word ω assigned to topic z is denoted as nω|z, and each X-term gi in G contains X words (i.e., ωi,1, . . . , ωi,X ). The subscript −i is used for nz|ε and nω|z to indicate that the count does not include the current ith assignment of emotions or topics.

After the sampling of topics and emotions, the posterior probabilities of ϕ and φ can be calculated as follows:

ϕε,z = α + nz|ε

Nzα + ∑

z′ nz′|ε (10)

and

φz,ω = β + nω|z

Nωβ + ∑

ω′ nω′|z . (11)

Finally, the predicted emotion distribution for a testing document d can be estimated by

P(ε|d) = P(ε) ∏

ω,ω∈d P(ω|ε)∑ ε P(ε)

∏ ω,ω∈d P(ω|ε)

(12)

Algorithm 2 Gibbs Sampling Algorithm for XETM Input:

1: Nz: Number of topics; 2: NE : Number of emotion labels; 3: α: Hyperparameter of ϕ; 4: β: Hyperparameter of φ; 5: G: The X-term groups in the training set;

Output: 6: P(ε|d): The emotion proportion of document d; 7: procedure BUILD XETM 8: Randomly initialize topic assignments for all X-terms; 9: Randomly initialize emotion assignments for X-terms;

10: repeat 11: for all gi = (ωi,1, ..., ωi,X ) ∈ G do 12: Draw emotion ε according to Equation (8); 13: Draw topic z according to Equation (9); 14: Update nz|ε , nωi,1|z, ..., and nωi,X |z; 15: end for 16: until Niter times 17: Estimate P(ε|d) by Equation (12). 18: end procedure

where P(ε) is the emotion probability distribution for the entire training set, and the probability of word ω conditioned to emotion ε can be estimated by integrating the latent topic z: P(ω|ε) = ∑z ϕε,zφz,ω. To detail the estimation of parameters, we present the Gibbs sampling algorithm in Algorithm 2.

D. Accelerated Algorithm

Due to the high complexity of Gibbs sampling, we pro- pose an accelerated algorithm for WLTM and XETM via a supervised MH sampling [12] in conjunction with the Alias method [11].

1) Alias Method: The number of topics Nz is one of the factors that determine the time complexity in topic modeling. The sampling procedure is very time consuming when Nz is large. Particularly, a general discrete probability distribution P = {p1, p2, . . . , pNz } will take O(Nz) operations to generate a sample. On the other hand, it will take just O(1) operations if the discrete probability distribution is a uniform distribution. Inspired by the above property, the Alias method simulates the characteristics of a uniform distribution by building up an Alias table and a probability table [11]. The generation processes of these two tables are shown in Fig. 4.

Take the sample generation from a discrete probability dis- tribution P = {0.1, 0.2, 0.3, 0.4} as an example, the objective is to make each entry in P to be equal to 1. We first multiply each entry in P by 4, thus the third and the fourth entries of P are larger than 1, while the first and the second entries of P are less than 1. Then, we use the third and the fourth entries to supplement the first and the second entries. During the process, the values of the probability table (ProbTable) are from the value of each relative entry. Furthermore, the Alias table (AliasTable) is the index number of the supplement entry. After the above process, we can sample an entry from these two tables with O(1) operations as shown in Algorithm 3.

2) Metropolis–Hastings Sampling: As mentioned earlier, we implement the Gibbs sampling algorithm for our basic models WLTM and XETM, but with a high time cost (the

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

820 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021

Fig. 4. Illustration of generating the alias table and the probability table.

Algorithm 3 Sampling Process of the Alias Method Input:

1: AliasTable and ProbTable with Nz iterms 2: procedure SAMPLING 3: x = randint(Nz) 4: y = random(0, 1) 5: if y < ProbTable[x] then 6: return x 7: else 8: return AliasTable[x] 9: end if

10: end procedure

running time will be shown in Table IX). In the Gibbs sam- pling algorithm for WLTM and XETM, we need to draw a topic for each X-term in each iteration. This process will be very time consuming if the number of X-terms NG is too large. Moreover, if we only build up an Alias table for Gibbs sam- pling, we have to save two matrices in the Alias and probability table for all X-terms with a total size of NG × Nz. Since NG is quite large, the above operation not only costs time but also wastes storage space. In light of this consideration, we use the MH sampling [12], [37] in conjunction with the Alias method to estimate model parameters, in which, we only need to build up an Alias table and a probability table for each word. The accelerated models are, respectively, called fWLTM and fXETM, and the parameter derivation is shown in the next part.

3) Parameter Derivation: As for the parameter deriva- tion of fWLTM, we decompose (4) into separated parts: [(n−i,ωi,1|z + β)/(n−i,·|z + Nωβ)],. . . , [(n−i,ωi,X |z + β)/(ni,·|z + Nωβ)] and n−i,z + α for each X-term gi in the conditional distribution. According to the MH sampling method, these parts are called proposal distributions. Specifically, we denote [(n−i,z + α)/(NG + Nzα)] as the corpus proposal pz,c and [(n−i,ω|z + β)/(n−i,·|z + Nωβ)] as the word proposal pω|z. The MH sampling algorithm draws a topic from pz,c, and pωi,1|z,. . . , pωi,X |z in turns, thus pgi (z) ∝ pz,c

∏X x=1 pωi,x|z, where X is the

number of words in a term group gi, and it is called “cycle proposal” [38].

For the corpus proposal distribution, we have

pz,c ∝ (nz + α) (13)

where the acceptance probability is min(1, π s→tc ) for topic translation s → t, and π s→tc is given as follows:

π s→t c =

(n−i,t + α)( n−i,s + α

) ( n−i,·|s + Nωβ

) ( n−i,·|t + Nωβ

) (ns + α) (nt + α)

× X∏

x=1

( n−i,ωi,x|t + β

) ( n−i,ωi,x|s + β

) (14)

where ns is the number of X-terms assigned to topic s. During the corpus proposal sampling, we do not need to

build the Alias table and the probability table. Particularly, we store the topic that is assigned to the ith X-term gi as ZGi, which can be considered as an NG length vector. After randomly sampling a topic ZGj of an X-term gj from ZG, the current assigned topic ZGj of gj can be considered as the translation state. Because the probability of sampling entry from vector ZG is equal with each other, ZG is a uniform distribution and the time complexity is O(1). Considering the hyperparameter α in corpus proposal, we randomly set a float number f in the range of (0, NG + Nzα). If f is less than NG, we set an integer fint = f , else fint = f − NG . Then, the translation state/topic is ZGfint .

For the word proposal distribution (e.g., ωi,x), we have

pωi,x|z ∝ ( nωi,x|z + β

) ( n·|z + Nωβ

) (15)

where the acceptance probability is min(1, π s→tωi,1 ) when topic s translates to topic t, and π s→tωi,1 is estimated as follows:

π s→t ωi,x

= ( n−i,t + α

) ( n−i,s + α

) ( n−i,·|s + Nωβ

)2 ( n−i,·|t + Nωβ

)2 ( nωi,x|s + β

) ( nωi,x|t + β

)

× ( n·|t + Nωβ

) ( n·|s + Nωβ

) X∏

x=1

( n−i,ωi,x|t + β

) ( n−i,ωi,x|s + β

) . (16)

During the word proposal topic sampling, we restrict the states/topics of the gi to its relative topics via only sampling from its label-related topic set λgi as mentioned earlier.

As for the parameter deviation of fXETM, we decom- pose the conditional distribution (9) into separated parts: [(α+n−i,z|εi )/(Nzα+

∑ z′ n−i,z′|εi )] and [(β+n−i,ωi,1|z)/(Nωβ+∑

ω′ n−i,ω′|z)],. . . , [(β + n−i,ωi,X |z)/(Nωβ + ∑

ω′ n−i,ω′|z)]. Similarly, the first part is the topic-emotion proportion which

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

PANG et al.: FAST SUPERVISED TOPIC MODELS FOR SHORT TEXT EMOTION DETECTION 821

is called emotion proposal pz|εi , the remaining parts are word proposal pωi,1|z,. . . , pωi,X |z, respectively. Specifically, the MH sampling for XETM draws a topic from these three proposal in turns, thus pgi (z) ∝ pz|εi

∏X x=1 pωi,x|z.

For the emotion proposal distribution, we have

pz|εi ∝ α + n−i,z|εi

Nzα + ∑

z′ n−i,z′|εi (17)

where the acceptance probability is min(1, π s→tεi ) when topic s translates to topic t, and π s→tεi is estimated as follows:

π s→t εi

= ( α + ns|εi

) ( α + nt|εi

) ( α + n−i,t|εi

) ( α + n−i,s|εi

) ( Nωβ +

∑ ω′ n−i,ω′|s

)2 ( Nωβ +

∑ ω′ n−i,ω′|t

)2

× X∏

x=1

( β + n−i,ωi,x|t

) ( β + n−i,ωi,x|s

) . (18)

For the word proposal distribution (e.g., ωi,x), we have

pωi,x|z ∝ β + n−i,ωi,x|z

Nωβ + ∑

ω′ n−i,ω′|z (19)

where the acceptance probability is min(1, π s→tωi,1 ) when topic s translates to topic t, and π s→tωi,1 for the proposed fXETM is estimated as follows:

π s→t ωi,x

= ( β + nωi,x|s

) ( β + nωi,x|t

) ( Nωβ +

∑ ω′ nω′|t

) ( Nωβ +

∑ ω′ nω′|s

) ( α + n−i,t|εi

) ( α + n−i,s|εi

)

× ( Nωβ +

∑ ω′ n−i,ω′|s

)2 ( Nωβ +

∑ ω′ n−i,ω′|t

)2 X∏

x=1

( β + n−i,ωi,x|t

) ( β + n−i,ωi,x|s

) . (20)

During each iteration of topic generation, we first sample an emotion εi according to (8), which is less time consuming because there are generally a few emotion labels in the datasets (e.g., 6–8 for our employed datasets in the experiment). We apply the above MH sampling method for the topic genera- tion based on emotion εi, which alleviates the time-consuming problem under a large number of topics.

E. Complexity Analysis

In the sampling process of each term group gi, nω|z in WLTM or XETM changes slow, that is, there are only two counters reduced and two counters added for old and new topics, respectively. Therefore, it is unnecessary to update the Alias table and the probability table for each sample, which will reduce much running time. Especially, the Alias method keeps the MH proposal (i.e., corpus proposal, emo- tion proposal, and word proposal) over one iteration, rather than modify it after every sampling. For the MH sampling, the acceptance probability can be computed in O(1) time. To achieve a better mixing rate, we combine the proposals into a cycle proposal, such as pgi (z) ∝ pz,c

∏X x=1 pωi,x|z for the

fWLTM and pgi (z) ∝ pz|εi ∏X

x=1 pωi,x|z for the fXETM, where a sequence is constructed for each token by alternating between corpus proposal and word proposal. Such cycle proposals are theoretically guaranteed to converge as shown in [38].

According to the above formulas, we summarize different models’ time complexity in Table II. For the accelerated mod- els (i.e., fWLTM and fXETM), we update the Alias tables

TABLE II TIME COMPLEXITY OF DIFFERENT MODELS

TABLE III STATISTICS OF DATASETS

over each iteration rather than each sampling. In this table, Niter is the number of iteration, NG is the number of gener- ated term groups, Nω is the number of distinct words in the corpus, and NE and Nz are the numbers of emotion labels and topics, respectively. During each iteration of the proposed WLTM, we have to compute the topic probability distribu- tion for each term group using (4) and sample one topic, so its time complexity is O(Niter × NG × Nz). As for that of fWLTM, after the initialization of topic assignment for each term group, we build up the Alias table and the prob- ability table for each word, which takes O(Nω × Nz) time, then we update these two tables over each iteration. Thus, the time complexity of fWLTM is O(Niter × (Nω × Nz + NG)). In each iteration of XETM, (8) computes the emotion probabil- ity distribution for each topic to sample one emotion, and (9) computes the topic probability distribution for each term group to sample one topic. So the time complexity of XETM is O(Niter × NG × (NE + Nz)). As for fXETM, the MH sampling is applied in the topic sampling process according to (9). In each iteration of the topic sampling step, we update the Alias and probability tables for emotion and word proposal distri- butions, respectively. Thus, the time complexity of fXETM is O(Niter × (NG × NE + Nω × Nz + NE × Nz + NG)).

Specifically, we can observe that NG is always larger than Nω when the value of X is larger than 1, and the running time of WLTM and XETM will increase when the number of

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

822 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021

TABLE IV PERFORMANCE OF WLTM WITH VARIOUS X. (a) AP ON SemEval. (b) AP ON ISEAR. (c) AP ON RenCECps. (d) HD ON SemEval. (e) HD ON ISEAR.

(f) HD ON RenCECps. (g) Accuracy ON SemEval. (h) Accuracy ON ISEAR. (i) Accuracy ON RenCECps

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

TABLE V PERFORMANCE OF XETM WITH VARIOUS X. (a) AP ON SemEval. (b) AP ON ISEAR. (c) AP ON RenCECps. (d) HD ON SemEval. (e) HD ON ISEAR.

(f) HD ON RenCECps. (g) Accuracy ON SemEval. (h) Accuracy ON ISEAR. (i) Accuracy ON RenCECps

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

topics Nz getting larger. On the other hand, we first update the Alias table over each iteration for fWLTM and fXETM, then we can use the MH sampling method to sample a topic in O(1) time for each term group. Furthermore, the accelerated models, fWLTM and fXETM, only sample the topics from the related topics of each term group. Thus, the actual value of Nz in fWLTM is smaller than other topic models which compute each topic probability for training.

IV. EXPERIMENTS

This section presents the experimental results on the proposed models and baselines in terms of effectiveness and efficiency.

A. Datasets

SemEval: This dataset contains 1246 news headlines with the total score larger than 0, which is used in the 14th task

of the 4th International Workshop on Semantic Evaluations (SemEval-2007) [1] and officially divided into a training set with 246 documents and a testing set with 1000 documents. The emotion labels include anger, disgust, fear, joy, sad, and surprise, which are posited to be basic [39].

ISEAR: This dataset contains 7666 sentences annotated by 1096 participants manually according to seven emotions [40]. The emotion categories are anger, disgust, fear, joy, sadness, shame, and guilt. For this dataset, 60%, 20%, and 20% of sen- tences are selected randomly as the training set, the validation set, and the testing set, respectively.

RenCECps: This corpus contains 1487 Chinese blogs with a total of 35 096 sentences [41]. The emotion categories are joy, hate, love, sorrow, anxiety, surprise, anger, and expect. We limit the number of words in a sentence from 5 to 20, so as to generate a labeled short text dataset with 19 338 sentences. For this dataset, 60% and 40% of sentences are selected randomly as the training set and the testing set, respectively.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

PANG et al.: FAST SUPERVISED TOPIC MODELS FOR SHORT TEXT EMOTION DETECTION 823

TABLE VI PERFORMANCE STATISTICS OF DIFFERENT MODELS. (a) AP OVER SemEval. (b) AP OVER ISEAR. (c) AP OVER RenCECps. (d) HD OVER SemEval.

(e) HD OVER ISEAR. (f) HD OVER RenCECps. (g) Accuracy OVER SemEval. (h) Accuracy OVER ISEAR. (i) Accuracy OVER RenCECps

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

TABLE VII PERFORMANCE OF THE WORD-LEVEL BASELINE MODELS. (a) Semeval. (b) ISEAR. (c) RenCECps

(a)

(b)

(c)

Table III summarizes the statistics of these three datasets, where the number of documents and mean words of each emotion label are calculated based on the sum of documents having the largest score over that emotion. Note that an emo- tion can be assessed for both categories and the strength in SemEval and RecCECps. For example, the users annotated four categories (i.e., “Joy,” “Fear,” “Surprise,” and “Sad”) for a single news headline—“Test to predict breast cancer relapse is approved” in SemEval, and scores of these categories are 38, 15, 11, and 9, respectively. Therefore, it is suggested to take all emotion scores into account for evaluation [42], rather than only concern about the emotion with the largest score.

B. Experimental Design

We denote WLTM and XETM that incorporate our acceler- ated algorithm as fWLTM and fXETM, respectively. The term

groups are generated for the above three datasets. For instance, when X is equal to 2, there are 5123 and 1 571 829 2-terms in SemEval and ISEAR, respectively. Since the scale of SemEval is too limited, we employ ISEAR to evaluate the effi- ciency of fWLTM, fXETM, and other models. Some classical approaches that do not exploit topics [1], [4], [9], and topic- level baselines, including LLDA [32], BTM [8], emotion-topic model (ETM) [4], CSTM [20], sentiment latent-topic model (SLTM) [15], and siamese network-based supervised topic model (SNSTM) [43] are implemented as baselines.

For BTM, WLTM, fWLTM, XETM, and fXETM, all term groups are generated by taking each short text as an individual context unit. We employ SVR [9] with radial basis function (RBF) as the kernel function to predict emotion distributions of unlabeled documents for WLTM, fWLTM, LLDA, and BTM. To tune the parameters of SVR, five-fold cross-validation is

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

824 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021

performed on the training set for SemEval and RenCECps and on the validation set for ISEAR. For XETM and fXETM, the emotion distribution of each testing document is estimated by (12). Similar to the previous studies [4], [32], [34], the hyperparameters α and β are, respectively, set to symmetric Dirichlet priors with values of 0.1 and 0.01, and the number of Gibbs sampling iteration is set to 500. The running time is recorded on a 24 core high-performance computational node with 64G memory. To ensure the effectiveness of MH sam- pling, we set MH sampling times to 2, which means that the topic of an X-term is sampled twice at each iteration.

To take emotion scores into account, two fine-grained met- rics, the averaged Pearson’s correlation coefficients (AP) and the averaged Hellinger distance (HD), are used for evalua- tion [1], [20], [44]. Given two vectors p and q with element x, AP and HD are estimated as follows:

AP(p, q) = ∑

x(p(x) − p)(q(x) − q)√∑ x(p(x) − p)2

√∑ x(q(x) − q)2

HD(p, q) = √

1

2

∑ x

(√ p(x) −

√ q(x)

)2 .

In the above, p and q are the mean values of p and q, respectively. For completeness, both AP and HD are mea- sured between the predicted probabilities and the actual votes over the document level (APdocument and HDdocument), and over the emotion level (APemotion and HDemotion), respectively. The value of AP ranges from −1 to 1, where 1 indicates a perfect prediction with the maximum correlation coefficient, and the value of HD ranges from 0 to 1, where 0 indicates a perfect prediction with the minimum Hellinger distance.

We also compare the performance of different models by a coarse-grained metric, that is, Accuracy@N (N = 1, 2, 3) [3]. Specifically, given a document d, an actual emo- tion set EtopN@d which includes N top-ranked emotions, and the top-ranked predicted emotion εp, Accuracyd @N is first calculated as

Accuracyd @N = {

1 if εp ∈ EtopN@d 0 else.

Then, Accuracy@N for the testing set D is

Accuracy@N = ∑ d∈D

Accuracyd @N

|D| .

As mentioned earlier, the topic number of WLTM that indicates documents’ latent aspects depends on the multiplier (i.e., τ ) between topic and emotion numbers. To evaluate the performance of our models with different numbers of topics, we vary τ from 1 to 15 for three datasets in our experiments, thus the topic numbers of SemEval, ISEAR, and RenCECps range from |NESemEval ∗ 1| = 6 to |NESemEval ∗ 15| = 90, from |NEISEAR ∗ 1| = 7 to |NEISEAR ∗ 15| = 105, and from |NERenCECps ∗ 1| = 8 to |NERenCECps ∗ 15| = 120, respectively.

C. Influence of X

In the first part of experiments, we evaluate the influence of X (i.e., the number of words for each term group) on the

TABLE VIII PERFORMANCE OF THE ACCELERATED MODELS. (a) AP OVER ISEAR.

(b) HD OVER ISEAR. (c) Accuracy OVER ISEAR

(a)

(b)

(c)

model performance by setting X to 1, 2, 3, and 4. The exper- imental results in Tables IV and V indicate that the proposed models perform the worst when X is set to 1 in most cases. It is reasonable because of the number of words in a short mes- sage is limited. We can also observe that the proposed models perform the best when X is set to 2 mostly, which indicates that two words are more likely to form a phrase (i.e., a seman- tically related term) than others for these three datasets. Unless otherwise specified, we set X to 2 in the following experiments.

D. Comparison With Baselines

Table VI presents the mean and variance of model performance in terms of AP, HD, and Accuracy, where the top values of each metric are highlighted in boldface.

According to the AP results, the proposed WLTM achieves better performance than baselines on these three datasets in most cases, except for a sightly worse performance than some other models in terms of APdocument over SemEval. A possible reason is that there are 28 words appearing in the 1000 test- ing documents but not in the 246 training documents. Since the lack of samples in tuning parameters, WLTM, LLDA, and BTM which employ SVR for prediction may underfit emotional distributions at the document level. By generating emotion-topic and topic-word probabilities without parameter tuning, the proposed XETM yields competitive performance on APdocument. In terms of APemotion, WLTM achieves the best mean value of 0.2411 and XETM ranks top 3 with a value of 0.1995. Particularly, the variances of WLTM and XETM indicate the performance stability of our two mod- els. According to the results over ISEAR, WLTM yields competitive performance on both evaluation metrics and the corresponding variances rank top 3. On the other hand, XETM cannot achieve the best results on AP, but its variances with different multiplier values also rank top 3.

Note that the Hellinger distance measures the similarity between two probability distributions. Table VI shows that WLTM achieves the best performance except for HDdocument on SemEval, in which CSTM is slightly better. These results indicate that the predicted emotion distribution for WLTM is quite close to the prior emotion label distribution. This is because the generation of topics is constrained by one-to-many

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

PANG et al.: FAST SUPERVISED TOPIC MODELS FOR SHORT TEXT EMOTION DETECTION 825

TABLE IX RUNNING TIME WITH DIFFERENT VALUES OF τ OVER ISEAR (Niter = 500), UNIT: SECOND

projection between emotions and topics for WLTM, which renders the extracted topics corresponding to relative emo- tions. On the other hand, XETM achieves modest performance among these three datasets. The reason may be that XETM first samples one emotion label ε and then generates a topic conditioned to ε. However, there are more than one emotion label for most sentences in both SemEval and RenCECps.

As for the metrics of Accuracy@1, Accuracy@2, and Accuracy@3, the proposed two models also perform compet- itively. WLTM outperforms other models on both SemEval and RenCECps. However, ETM performs better on ISEAR, in which there are only one label for each document. As mentioned earlier, the sampling of topics is constrained by one emotion for ETM, thus ETM mostly samples an emotion which is the actual label of the document.

To compare the performance of our supervised topic mod- els on short text emotion detection statistically, we conduct t-tests to test the assumption that the difference in performance between paired models has a mean value of zero. T -test is con- ducted on the proposed models (i.e., WLTM and XETM) and the baseline models. The results indicate that the proposed WLTM outperforms the baselines of LLDA, BTM, ETM, CSTM, SLTM, and SNSTM significantly with p-values much less than 0.05. The p-values between XETM and most of baselines, except BTM, are less than 0.05. The difference in performance between XETM and BTM is not statistically sig- nificant with a p-value equal to 0.2856 in terms of HDdocument. Similar results are observed for other metrics.

We also implement some word-level baseline models for comparison, that is, ET [4], SVR [9], and SWAT [1]. The experimental results are shown in Table VII. ET computes P(ε) and P(w|ε) by extending naïve Bayes, and SWAT pre- dicts emotions via aggregating the emotion of each word. Since the input is quite sparse for both datasets, SVR based on words achieves the worst performance in terms of AP. On the other hand, our WLTM that employs SVR using topic distributions as the input can outperform the conventional SVR by a large margin. These results indicate that our mod- els effectively extract valuable features for short text emotion detection.

E. Evaluation on the Accelerated Algorithm

In this part, we evaluate the performance of fWLTM and fXETM on ISEAR in terms of AP, HD, and Accuracy, and compare the corresponding running time with topic models based on 2-terms, i.e., WLTM, XETM, and BTM. For all metrics, we vary τ from 1 to 15 and present the mean and variance values in Table VIII. Specifically, the APdocument of fWLTM reaches the best value of 0.3943 with τ = 5 and

TABLE X EMOTION LEXICON SAMPLES FROM WLTM AND FWLTM OVER ISEAR

has an averaged value of 0.3519. Although the above mean value is less than the averaged APdocument value of WLTM (i.e., 0.4299), it is better than BTM with an averaged value of 0.3327. Furthermore, the averaged APemotion value of fWLTM is 0.3519, which is very close to that of BTM (i.e., 0.3590). Particularly, the best value of APemotion is 0.4175 for fWLTM when τ equals to 4 or 5, which is higher than the best value of BTM. Although WLTM achieves the highest values of APdocument and APemotion, fWLTM is much less time consum- ing and more efficient than WLTM as shown in the following evaluation. On the other hand, the results of fXETM indicate that although the averaged APdocument value of fXETM (i.e., 0.2744) is less than that of XETM with a value of 0.2977, the averaged APemotion reaches a value of 0.3806, which out- performs XETM and BTM. The performance variances of fWLTM and fXETM show that both of them have good sta- bility. In terms of HD, fWLTM achieves a smaller value than those of baselines, and outperforms that of XETM, which means fWLTM generates better topic distribution conditioned to emotion labels during the supervised training process. As for Accuracy, the results of fWLTM are slightly lower than WLTM but still competitive for those of baselines above.

To evaluate the above results statistically, we conduct t-tests between the performance of fWLTM and those of WLTM and BTM. The p-values of BTM are almost larger than 0.05, and the mean values and variances of AP are closed to fWLTM. Therefore, the performance of fWLTM is as competitive as BTM. Although WLTM achieves the best performance of cor- relation coefficients, it is expensive when training on a lager scale of documents or features.

The t-tests between the performance of fXETM and those of XETM and BTM is also evaluated. Specifically, the p-values between the APemotion of fXETM and that of XETM are less than 0.05, which means that the performance on APemotion of fXETM is better than that of XETM statistically. Compared to BTM, the proposed fXETM also achieves better performance on the metric of APemotion.

Table IX presents the running time of those models with different values of τ . In the experiment, we set the iteration time Niter to 500, vary τ from 1 to 15, and record how many seconds are used for these models. Although BTM performs

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

826 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021

TABLE XI EMOTION LEXICON SAMPLES FROM XETM AND FXETM OVER ISEAR

competitively, it is quite time consuming. In general, the run- ning time of fWLTM is always less than that of BTM and WLTM when τ is larger than 1. Specifically, in terms of a larger τ value, when τ increases by 1, the running time of fWLTM increases by 10 s while that of WLTM increases by more than 100 s. On the other hand, the running time of fXETM is almost 3300 s with different values of τ , but that of XETM is more than 4000 s finally. Moreover, with τ get- ting larger, the values of running time of fWLTM and fXETM increase slowly while those of the WLTM, XETM, and BTM increase fast and become expensive. This is because the accel- erated models spend stable time to construct the Alias table with different τ values. As mentioned above, we construct the Alias table for each word in each iteration, so the running time relies on the number of different words instead of the number of topics Nz. After constructing the Alias table, we can spend O(1) time to sample a topic for each term group. As for Gibbs sampling, we need compute the topic probability distribution for each topic, which has a time complexity of O(Nz). Thus, Gibbs sampling is computationally prohibitive under a large Nz. In this article, by employing the Alias method and devel- oping the supervised MH sampling, both fWLTM and fXETM are efficient with competitive performance.

F. Emotion Lexicon Samples

As stated earlier, both WLTM and XETM are supervised topic models using prior emotion scores to restrict the topic probability during each sampling process. Specifically, the proposed WLTM directly maps topics to emotion labels, while XETM employs a topic-emotion layer to connect words and topics, from which we can conduct a topic-emotion probability distribution using (11). Therefore, for the example as shown in Fig. 1, each topic of WLTM is mapped to the corresponding emotion label. Each topic of XETM performs a probability distribution for each emotion after sampling. In the follow- ing text, we show the emotion lexicon samples over ISEAR for WLTM, fWLTM, XETM, and fXETM, in which the value of τ is 5, so the number of topics is τ × NEISEAR = 42. For XETM and fXETM, the distribution of an emotion specific to each topic can be estimated by (10). For these four proposed models, the probabilities of words conditioned to each topic are estimated according to (5) or (11).

Table X shows the emotion lexicon samples that are gen- erated by WLTM and fWLTM. In the second and the third columns, we present sample topics and their representative words. The last column is the relative emotion label from our one-to-many mapping method. As shown in the sample results, it is convinced that both WLTM and fWLTM can effectively

generate the emotion-related words for each topic. For exam- ple, in topic 23 from WLTM, the sample words are “win, team, competition, prize, game,” which mostly means “a team win in a competition and win the price,” and the emotion of that is exactly “joy.” In topic 2 of fWTLM, the sample word “corrupt” means the phenomenon of corruption and the related emotion is “anger.”

Table XI shows the emotion lexicon samples from XETM and fXETM, where the sample topics and their representative words with the largest conditional probabilities are presented in the second and the third columns. The distributions of seven emotions for each topic are listed in the other columns, and the largest values are boldfaced. First, the samples indicate that the topics are strongly relative to one emotion label, for example, the topic 6 from XETM has a probability of 99% relating to the emotion of “joy,” the topic 1 from fXETM is almost 90% relating to the emotion of “anger.” Second, the sample words have the exact emotional expression like the topic-related emotion. For example, the word “carnival” in topic 6 from XETM is mostly implied in a festival event, the word “skid” in topic 20 from fXETM means “stop the car,” which is probably used for a traffic accident news with the emotion of “fear.”

V. CONCLUSION

Emotion detection aims to predict emotional responses embedded in documents. This article proposed two models, WLTM and XETM, to address the issue of feature sparsity in detecting emotions over short messages. In this article, we evaluated the influence of the number of words in a term group and compare the performance with state-of-the-art base- lines. To reduce the time cost of estimating parameters, we proposed the accelerated methods, fWLTM and fXETM to generate topics and detect emotions efficiently. The experi- mental results indicated that the accelerated models were quite less time consuming without reducing much quality, especially for the proposed fWLTM. Considering that users often use sarcasm for emphasizing their sentiment [45], our future work will focus on incorporating sarcasm detection into our method. Furthermore, we intend to evaluate the model performance on multimodal sentiment analysis [46]. We also plan to extend the fast parametric topic models to nonparametric ones [47]–[49], so as to handle text streams where the number of topics is hard to be specified manually.

ACKNOWLEDGMENT

Some contents from the conference version are reused in this journal article as this article is a follow-up work of the conference paper. The new contents of this article are more

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

PANG et al.: FAST SUPERVISED TOPIC MODELS FOR SHORT TEXT EMOTION DETECTION 827

than 70% according to the regulation of the published journal. The new contents can be summarized in the following aspects.

1) We extend the basic proposed models by setting the length of term groups as a flexible variable.

2) To reduce the time complexity of the generation process, we newly propose an accelerated algorithm for our basic models.

3) In the experiments, we evaluate and compare the performance of our models with different lengths of a term group.

4) To conduct in-depth analysis, we present the emotion lexicon samples that are generated by our models.

5) We add a Chinese corpus and two metrics (i.e., the Hellinger distance and accuracy) to evaluate the effec- tiveness of different models comprehensively.

REFERENCES

[1] P. Katz, M. Singleton, and R. H. Wicentowski, “SWAT-MP: The SemeVal-2007 systems for task 5 and task 14,” in Proc. 4th Int. Workshop Semantic Eval., 2007, pp. 308–313.

[2] C. Strapparava and R. Mihalcea, “SemeVal-2007 task 14: Affective text,” in Proc. 4th Int. Workshop Semantic Eval., 2007, pp. 70–74.

[3] E. Cambria, S. Poria, A. Gelbukh, and M. Thelwall, “Sentiment anal- ysis is a big suitcase,” IEEE Intell. Syst., vol. 32, no. 6, pp. 74–80, Nov./Dec. 2017.

[4] S. Bao et al., “Mining social emotions from affective text,” IEEE Trans. Knowl. Data Eng., vol. 24, no. 9, pp. 1658–1670, Sep. 2012.

[5] Y. Rao, J. Lei, W. Liu, Q. Li, and M. Chen, “Building emotional dictio- nary for sentiment analysis of online news,” World Wide Web, vol. 17, no. 4, pp. 723–742, 2014.

[6] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,” J. Mach. Learn. Res., vol. 3, pp. 993–1022, Jan. 2003.

[7] Y. Song, H. Wang, Z. Wang, H. Li, and W. Chen, “Short text conceptu- alization using a probabilistic knowledgebase,” in Proc. 22nd Int. Joint Conf. Artif. Intell., 2011, pp. 2330–2336.

[8] X. Cheng, Y. Lan, J. Guo, and X. Yan, “BTM: Topic modeling over short texts,” IEEE Trans. Knowl. Data Eng., vol. 26, no. 12, pp. 2928–2941, Dec. 2014.

[9] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, pp. 389–396, 2011.

[10] X. He, H. Xu, J. Li, L. He, and L. L. Yu, “FastBTM: Reducing the sampling time for biterm topic model,” Knowl. Based Syst., vol. 132, pp. 11–20, Sep. 2017.

[11] A. J. Walker, “New fast method for generating discrete random numbers with arbitrary frequency distributions,” Electron. Lett., vol. 10, no. 8, pp. 127–128, Apr. 1974.

[12] W. K. Hastings, “Monte Carlo sampling methods using Markov chains and their applications,” Biometrika, vol. 57, no. 1, pp. 97–109, 1970.

[13] E. Cambria, “Affective computing and sentiment analysis,” IEEE Intell. Syst., vol. 31, no. 2, pp. 102–107, Mar./Apr. 2016.

[14] A. Gangemi, V. Presutti, and D. R. Recupero, “Frame-based detection of opinion holders and topics: A model and a tool,” IEEE Comput. Intell. Mag., vol. 9, no. 1, pp. 20–30, Feb. 2014.

[15] Y. Rao, Q. Li, X. Mao, and W. Liu, “Sentiment topic models for social emotion mining,” Inf. Sci., vol. 266, pp. 90–100, May 2014.

[16] Y. Rao, Q. Li, W. Liu, Q. Wu, and X. Quan, “Affective topic model for social emotion detection,” Neural Netw., vol. 58, pp. 29–37, Oct. 2014.

[17] Q. Yang, Y. Rao, H. Xie, J. Wang, F. L. Wang, and W. H. Chan, “Segment-level joint topic-sentiment model for online review analysis,” IEEE Intell. Syst., vol. 34, no. 1, pp. 43–50, Jan./Feb. 2019.

[18] M. Dragoni, S. Poria, and E. Cambria, “OntoSenticNet: A common- sense ontology for sentiment analysis,” IEEE Intell. Syst., vol. 33, no. 3, pp. 77–85, May/Jun. 2018.

[19] E. Cambria, S. Poria, D. Hazarika, and K. Kwok, “SenticNet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings,” in Proc. 32nd AAAI Conf. Artif. Intell., 2018, pp. 1795–1802.

[20] Y. Rao, “Contextual sentiment topic model for adaptive social emo- tion classification,” IEEE Intell. Syst., vol. 31, no. 1, pp. 41–47, Jan./Feb. 2016.

[21] S.-B. Kim, K.-S. Han, H.-C. Rim, and S. H. Myaeng, “Some effective techniques for naive Bayes text classification,” IEEE Trans. Knowl. Data Eng., vol. 18, no. 11, pp. 1457–1466, Nov. 2006.

[22] J. Li, Y. Rao, F. Jin, H. Chen, and X. Xiang, “Multi-label maxi- mum entropy model for social emotion classification over short text,” Neurocomputing, vol. 210, pp. 247–256, Oct. 2016.

[23] B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up? Sentiment classifi- cation using machine learning techniques,” in Proc. ACL Conf. Empirical Methods Nat. Lang. Process., 2002, pp. 79–86.

[24] D. Tang, F. Wei, B. Qin, N. Yang, T. Liu, and M. Zhou, “Sentiment embeddings with applications to sentiment analysis,” IEEE Trans. Knowl. Data Eng., vol. 28, no. 2, pp. 496–509, Feb. 2016.

[25] D. Tang, B. Qin, and T. Liu, “Aspect level sentiment classification with deep memory network,” in Proc. Conf. Empirical Methods Nat. Lang. Process., 2016, pp. 214–224.

[26] X. Li, Y. Rao, H. Xie, R. Y. K. Lau, J. Yin, and F. L. Wang, “Bootstrapping social emotion classification with semantically rich hybrid neural networks,” IEEE Trans. Affective Comput., vol. 8, no. 4, pp. 428–442, Oct./Dec. 2017.

[27] Y. Ma, H. Peng, T. Khan, E. Cambria, and A. Hussain, “Sentic LSTM: A hybrid network for targeted aspect-based sentiment analysis,” Cogn. Comput., vol. 10, no. 4, pp. 639–650, 2018.

[28] P. D. Turney, “Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews,” in Proc. 40th Annu. Meeting Assoc. Comput. Linguist., 2002, pp. 417–424.

[29] X.-H. Phan, L.-M. Nguyen, and S. Horiguchi, “Learning to classify short and sparse text & Web with hidden topics from large-scale data collections,” in Proc. 17th Int. Conf. World Wide Web, 2008, pp. 91–100.

[30] O. Jin, N. N. Liu, K. Zhao, Y. Yu, and Q. Yang, “Transferring topi- cal knowledge from auxiliary long texts for short text clustering,” in Proc. 20th ACM Int. Conf. Inf. Knowl. Manag., 2011, pp. 775–784.

[31] Y. Rao, H. Xie, J. Li, F. Jin, F. L. Wang, and Q. Li, “Social emotion classification of short text via topic-level maximum entropy model,” Inf. Manag., vol. 53, no. 8, pp. 978–986, 2016.

[32] D. Ramage, D. Hall, R. Nallapati, and C. D. Manning, “Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora,” in Proc. Conf. Empirical Methods Nat. Lang. Process., 2009, pp. 248–256.

[33] M. A. Taddy, “On estimation and selection for topic models,” in Proc. 15th Int. Conf. Artif. Intell. Stat., 2012, pp. 1184–1193.

[34] S. Bao et al., “Joint emotion-topic modeling for social affective text mining,” in Proc. 9th IEEE Int. Conf. Data Min., 2009, pp. 699–704.

[35] T. L. Griffiths and M. Steyvers, “Finding scientific topics,” Proc. Nat. Acad. Sci. USA, vol. 101, no. s1, pp. 5228–5235, 2004.

[36] R. Y. K. Lau, Y. Xia, and Y. Ye, “A probabilistic generative model for mining cybercriminal networks from online social media,” IEEE Comput. Intell. Mag., vol. 9, no. 1, pp. 31–43, Feb. 2014.

[37] J. Geweke and H. Tanizaki, “Bayesian estimation of state-space mod- els using the Metropolis–Hastings algorithm within Gibbs sampling,” Comput. Stat. Data Anal., vol. 37, no. 2, pp. 151–170, 2001.

[38] L. Tierney, “Markov chains for exploring posterior distributions,” Ann. Stat., vol. 22, no. 4, pp. 1701–1728, 1994.

[39] P. Ekman, “Facial expression and emotion,” Amer. Psychol., vol. 48, no. 4, pp. 384–392, 1993.

[40] K. R. Scherer and H. G. Wallbott, “Evidence for universality and cul- tural variation of differential emotion response patterning,” J. Pers. Soc. Psychol., vol. 66, no. 2, pp. 310–328, 1994.

[41] C. Quan and F. Ren, “Sentence emotion analysis and recognition based on emotion words using REN-CECPS,” Int. J. Adv. Intell. Paradigms, vol. 2, no. 1, pp. 105–117, 2010.

[42] D. Zhou, X. Zhang, Y. Zhou, Q. Zhao, and X. Geng, “Emotion distribu- tion learning from texts,” in Proc. Conf. Empirical Methods Nat. Lang. Process., 2016, pp. 638–647.

[43] M. Huang, Y. Rao, Y. Liu, H. Xie, and F. L. Wang, “Siamese network- based supervised topic modeling,” in Proc. Conf. Empirical Methods Nat. Lang. Process., 2018, pp. 4652–4662.

[44] L. Le Cam and G. L. Yang, Asymptotics in Statistics: Some Basic Concepts. New York, NY, USA: Springer, 2012.

[45] N. Majumder, S. Poria, H. Peng, N. Chhaya, E. Cambria, and A. Gelbukh, “Sentiment and sarcasm classification with multitask learning,” IEEE Intell. Syst., vol. 34, no. 3, pp. 38–43, Jan. 2019.

[46] S. Poria, N. Majumder, D. Hazarika, E. Cambria, A. Gelbukh, and A. Hussain, “Multimodal sentiment analysis: Addressing key issues and setting up the baselines,” IEEE Intell. Syst., vol. 33, no. 6, pp. 17–25, Nov./Dec. 2018.

[47] Y. W. Teh, M. I. Jordan, M. J. Beal, and D. M. Blei, “Sharing clusters among related groups: Hierarchical Dirichlet processes,” in Proc. 17th Adv. Neural Inf. Process. Syst., 2004, pp. 1385–1392.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.

828 IEEE TRANSACTIONS ON CYBERNETICS, VOL. 51, NO. 2, FEBRUARY 2021

[48] S. J. Gershman and D. M. Blei, “A tutorial on Bayesian nonparametric models,” J. Math. Psychol., vol. 56, no. 1, pp. 1–12, 2012.

[49] J. Xuan, J. Lu, and G. Zhang, “A survey on Bayesian nonparametric learning,” ACM Comput. Surveys, vol. 52, no. 1, pp. 1–36, 2019.

[50] Y. Rao et al., “Supervised intensive topic models for emotion detection over short text,” in Proc. 22nd Int. Conf. Database Syst. Adv. Appl., 2017, pp. 408–422.

Jianhui Pang received the Bachelor of Engineering degree in computer science from Sun Yat-sen University, Guangzhou, China.

His current research interests include topic modeling and emotion detection.

Yanghui Rao (M’18) received the master’s degree from the Graduate University of the Chinese Academy of Science, Beijing, China, in 2010, and the Ph.D. degree from the City University of Hong Kong, Hong Kong, in 2014.

He is an Associate Professor with the School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China. He has published over 20 ref- ereed journals and conference papers, including the ACM Transactions on Information Systems, the IEEE TRANSACTIONS ON AUTOMATIC CONTROL,

IEEE INTELLIGENT SYSTEMS, ACL, EMNLP, CIKM, and DASFAA. His current research interests include topic modeling, emotion detection, and natural language processing.

Haoran Xie (M’15) received the Ph.D. degree in computer science from the City University of Hong Kong, Hong Kong.

He is an Associate Professor with Lingnan University, Hong Kong. He has totally published 170 research publications, including 71 journal articles. His current research interests include artificial intel- ligence, big data, and educational technology.

Dr. Xie was a recipient of ten research awards, including the Golden Medal and British Innovation Award from International Invention Innovation

Competition in Canada and the Second Prize Winner from Multimedia Grand Challenges of ACM Multimedia in 2019. His proposed LSGAN pub- lished in the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE and ICCV, with over 700 citations in two years, has been included in a course in Stanford University and implemented by Google TensorFlow.

Xizhao Wang (M’03–SM’04–F’12) received the Ph.D. degree from the Harbin Institute of Technology, Harbin, China, in 1998.

He was a Research Fellow with Hong Kong Polytechnic University, Hong Kong, from 1998 to 2001, and served with Hebei University, Baoding, China, as a Professor and the Dean of the School of the Mathematics and Computer Sciences before from 2001 to 2014. After 2014, he was a Professor with the Big Data Institute of ShenZhen University, Shenzhen, China. He has edited over 10 special

issues and published 3 monographs, 2 textbooks, and over 200 peer-reviewed research papers. As a Principle Investigator (PI) or Co-PI, he has completed over 30 research projects. He has supervised over 150 M.Phil. and Ph.D. students. His current research interests include uncertainty modeling and machine learning for big data.

Prof. Wang was a recipient of the IEEE SMCS Outstanding Contribution Award in 2004 and the IEEE SMCS Best Associate Editor Award in 2006. He is the General Co-Chair of the 2002–2018 International Conferences on Machine Learning and Cybernetics, cosponsored by IEEE SMCS. He was a Distinguished Lecturer of the IEEE SMCS. He is the previous BoG Member of IEEE SMC Society, the Chair of the IEEE SMC Technical Committee on Computational Intelligence, the Chief Editor of the Machine Learning and Cybernetics, and an associate editor for a couple of journals in the related areas.

Fu Lee Wang (SM’15) received the B.Eng. degree in computer engineering and the M.Phil. degree in computer science and information systems from the University of Hong Kong, Hong Kong, and the Ph.D. degree in systems engineering and engineering man- agement from the Chinese University of Hong Kong, Hong Kong.

He is with the School of Science and Technology, Open University of Hong Kong, Hong Kong. He has published over 150 academic articles in refer- eed journals and conference proceedings. His current

research interests include educational technology, information retrieval, com- puter graphics, and bioinformatics.

Prof. Wang is a fellow of BCS and a Senior Member of ACM. He was the Chair of the IEEE Hong Kong Section Computer Chapter and ACM Hong Kong Chapter.

Tak-Lam Wong (M’09) received the Bachelor of Engineering and M.Phil. degrees in systems engineering and engineering management, the Postgraduate Diploma of Education degree in math- ematics, and the Ph.D. degree in systems engineer- ing and engineering management from the Chinese University of Hong Kong, Hong Kong.

He is a Professor with the Department of Computing Studies and Information Systems, Douglas College, New Westminster, BC, Canada. He has published papers in different journals, such

as the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE and the ACM Transactions on Information Systems, and con- ferences, including SIGIR, SIGKDD, AAAI, and WWW. His current research interests include Web mining, data mining, information extraction, machine learning, e-learning, programming education, and knowledge management.

Prof. Wong also served as the Chair of the IEEE Hong Kong Section Computer Chapter in 2016 and 2017.

Qing Li (SM’07) received the B.Eng. degree in computer science from Hunan University, Changsha, China, and the M.Sc. and Ph.D. degrees in computer science from the University of Southern California, Los Angeles, CA, USA.

He is a Chair Professor with the Department of Computing, Hong Kong Polytechnic University, Hong Kong. His current research interests include multimodal data management, conceptual data modeling, social media, Web services, and e-learning systems. He has authored/coauthored

over 400 publications in the above areas. Dr. Li is actively involved in the research community and has served as

an Associate Editor of a number of major technical journals, including the IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, the ACM Transactions on Internet Technology, Data Science and Engineering, World Wide Web, and the Journal of Web Engineering, in addition to being a Conference and Program Chair/Co-Chair of numerous major international conferences. He also sits in the Steering Committees of DASFAA, ER, ACM RecSys, IEEE U-MEDIA, and ICWL. He is a fellow of IEE/IET, U.K., and a Distinguished Member of CCF, China.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:18:42 UTC from IEEE Xplore. Restrictions apply.