helpfn

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 20, NO. 4, JULY 2016 987

Systematical Approach for Detecting the Intention and Intensity of Feelings on Social Network

Chih-Hua Tai, Member, IEEE, Zheng-Han Tan, and Yue-Shan Chang, Member, IEEE

Abstract—Online posts not only represent the records of people’s lives but also reveal their satisfaction with life and relationships as well as potential mental illnesses. The detection of (strong or gen- eral) negative as well as (strong or general) positive feelings of people from online posts can keep us from carelessly missing their important moments, difficult or great, due to the overloaded infor- mation in the daily life and lead to a better society. Therefore, in this paper, we build a Feeling Distinguisher system based on supervised Latent Dirichlet Allocation (sLDA), Latent Dirichlet Allocation, and SentiWordNet methodologies for detecting a person’s intention and intensity of feelings through the analysis of his/her online posts. Experimental results on posts collected from five social network websites demonstrate the effectiveness of FeD. The performance of FeD is about 1.08–1.18 folds that of SVM and sLDA.

Index Terms—Feeling intensity, latent Dirichlet allocation (LDA), supervised LDA (sLDA).

I. INTRODUCTION

T HE rise of social network websites such as Facebook andTwitter has significantly changed how people live. Cur- rently, many people are accustomed to recording their daily life online, such as frequently share their thoughts, opinions, and ideas, and communicate and interact with friends online. Abun- dant data resulted provide great opportunities for understanding human behavior and inspire many new applications for improv- ing people’s lives.

Many studies in recent years have sought to understand how people feel by investigating online posts, messages, and diaries [1]–[3]. They have considered the possibility of a relationship among linguistic expressions, emotion, and feelings. Several studies have even explored the characteristics of the emotional and linguistic expressions used by particular groups of peo- ple [4]–[7] (such as those suffering from major depression [5] and post traumatic stress disorder (PTSD) [6]). Thus, new ap- plications such as the service satisfaction survey and mental healthcare can be developed to benefit the society.

Inspired by such thoughts, we observed that online posts/diaries not only represent the records of life and com- munication, but also reveal the satisfaction with life and re- lationships as well as potential mental illnesses. Consider the

Manuscript received November 02, 2015; revised February 01, 2016 and February 18, 2016; accepted February 19, 2016. Date of publication February 29, 2016; date of current version July 06, 2016. This work was supported by the Ministry of Science and Technology of Taiwan, Republic of China, under Grant MOST 103-2221-E-305-007, Grant MOST 103-2218-E-305-002, Grant MOST 104-2221-E-305-010, and Grant MOST 104-2221-E-305-013.

The authors are with the Department of Computer Science and Infor- mation Engineering, National Taipei University, New Taipei 23741, Taiwan (e-mail: [email protected]; [email protected]; ysc@mail. ntpu.edu.tw).

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/JBHI.2016.2535721

2014 Taipei Metro Attack1 and the 2014 Isla Vista killings2 as examples. The attackers might have posted suggestive expres- sions of strong dissatisfaction with their lives and relationships for extended periods. Therefore, if we could detect and trace their strongly negative thoughts on life and provide timely sup- port such as comfort, screenings for mental illness, or proper treatments [8], [22]–[26] at early stages, such tragedies could be avoided. The detection of the (strongly) negative feelings or thoughts can call people’s attention to their friends succumbing to sadness. On the other hand, to share the love for friends, many people may want to proactively partake of the enjoyable expe- riences of their friends. The detection of the (strongly) positive feelings or thoughts can keep people from carelessly missing their friends’ enjoyable moments because of the overloaded in- formation in daily life. Consequently, a better society can be expected.

Therefore, this paper addresses the need for detecting how people feel about their daily life through their online posts. The feelings are presented in four categories: strongly nega- tive (SN), generally negative (GN), generally positive (GP), and strongly positive (SP). Here, we address SN from GN and SP from GP because SN feelings may indicate the significance of events that induce sadness or potential mental illness, whereas SP feelings may be associated with particularly enjoyable ex- periences. Therefore, we develop a new method, referred to as FeD (Feeling Distinguisher), for detecting a person’s intention and intensity of feelings through the analysis of his/her online posts. FeD is designed based on the supervised latent Dirichlet allocation (sLDA) [9], [10], latent Dirichlet allocation (LDA) [11], and SentiWordNet [12], [21] methodologies, where sLDA and LDA in natural language processing are generative mod- els explaining observed documents according to latent topics, whereas SentiWordNet is a lexical resource for opinion mining and sentiment analysis. In FeD, we use sLDA to first learn the discriminative linguistic terms between negative and positive feelings. LDA cooperating with SentiWordNet is then utilized to infer the representative terms of different intensities of neg- ative\positive feelings (i.e., SN versus GN and SP versus GP). Thus, in addition to the detection of negative\positive feelings, FeD can further distinguish the feelings into different inten- sities, strongly or generally, according to the intensity scores. Experiments on real online posts collected from five websites demonstrate the effectiveness of FeD. The performance of FeD is about 1.08–1.18 folds that of SVM and sLDA in terms of precision, recall, accuracy, and F1-scores.

1https://en.wikipedia.org/wiki/2014_Taipei_Metro_attack 2https://en.wikipedia.org/wiki/2014_Isla_Vista_killings

2168-2194 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

988 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 20, NO. 4, JULY 2016

In summary, the contribution of this work includes: 1) ad- dressing the problem of detecting the feeling intention and in- tensity for new applications on social network; 2) proposing the FeD method; and 3) demonstrating the effectiveness of FeD on real online posts.

II. RELATED WORK

As understanding the emotion in online messages and posts is frequently important for better expressing friendship, online arguments, opinion toward products, to name a few, emotion ex- traction from online text has attracted attention in recent years [1]–[3]. In [3], Yassine and Hajj introduced a framework for studying the relation between friendship and emotion expres- sion in online social networks. The framework includes several important aspects such as data collection, database schemas, data preprocessing and data mining steps to identify whether or not the online posts express any emotion and feelings. Unlike di- aries or blogs, online posts are usually short and informal, which increases the difficulty of sentiment analysis. In [2], Thelwall et al. proposed the SentiStrength algorithm to extract positive and negative sentiment strength from informal English text. By exploiting a range of human-recognized textual expressions, SentiStrength is able to overcome the problem of nonstandard spelling, which appears frequently in online posts. Bao et al. [1] studied the connection between emotion and affective terms, and proposed an emotion-topic model by adding an additional emo- tion layer on LDA. The emotion-topic model can distinguish the topics with strong emotion from background topics.

Very recently, there was also a group of works being partic- ularly interested in detecting emotion from Twitter messages [13]–[16]. Hasan et al. [14] proposed a supervised learning classifier Emotex for classifying Twitter messages into emo- tion classes annotated by hashtag. The Emotex uses Circum- plex model of affect [15] and defines the expressed emotion of users into four classes. Volkova et al. [16] introduced an ap- proach that can automatically infer latent personal attributes, emotion, and sentiment from Twitter text. With the inferred features, it was shown that detecting six kinds of emotion anno- tated by Twitter hashtag is achievable. In [13], Chikersal et al. developed a sentiment analysis system, SeNTU, by combin- ing SVM and a rule-based classifier. The rule-based classifier can classify emoticons and opinion words into positive, nega- tive, and unknown categories, while SVM can classify tweets into positive, negative, and neutral. The work showed that the rule-based classifier could refine the sentiment prediction of SVM.

Still another group of research focused on understanding the emotional expressions of particular groups who suffer from de- pression [4]–[7]. Choudhury et al. [5] compared the behaviors of identified depression cohort with that of standard users in terms of the volume of posts per day, insomnia index, linguistic style, and depression language on Twitter. Based on the ob- servations, a major depressive disorder classifier was built to predict whether or not an individual is vulnerable to depres- sion. In [7], Nguyen et al. provided a blogger view to predict depression by articles on communities. The relevant prediction

Fig. 1. LDA model.

rules were extracted through the analysis of the sentiment in- formation, topic interests and language styles of the articles collected from Clinical and Control communities on LiveJour- nal. In [4], Choudhury et al. first collected post-partum depres- sion (PPD) self-reported data from Facebook and differentiated PPD mothers from non-PPD mothers by their activities, inter- actions, and emotional and linguistic expression. After that, a series of statistical models based on the observations were de- veloped for predicting whether or not a post-partum woman is getting PPD. Coppersmith et al. [6] found out that PTSD users have different linguistic style from random users on Twitter, and presented a PTSD classifier by training Unigram Language Model, Character Language Model (CLM), and Linguistic In- quiry Word Count lexicon to classify whether or not a user is getting PTSD.

III. PRELIMINARY BACKGROUND

This section introduces the sLDA, LDA, and SentiWordNet methodologies proposed by previous studies.

A. LDA [11]

LDA is a generative probabilistic model of a text corpus. By regarding each document as a collection of words, LDA assumes that each document is formed by the mixture of hidden topics and that each hidden topic is represented by words with different probabilities. Fig. 1 shows the concept of the LDA model. The inner rectangle represents a document with N words, and the outer rectangle is the corpus D consisting of M documents. The generative process of a document d is simulated given back- ground parameters ζ, α, and β as follows.

1) Choose N from a Poisson distribution Poisson(ζ) to deter- mine the length of the document d.

2) Choose θ from a Dirichlet distribution Dir(α), indicating the mixture of topics forming the d.

3) To generate each word wn in the document d,

a) choose a topic zn from a Multinomial distribution Multinomial(θ);

b) choose a word wn with a probability p(wn |zn , β) conditioned on the chosen topic zn .

Accordingly, the joint distribution of a topic mixture θ, a set of topics z, and the document d is

p (θ, z, d|α, β) = p(θ|α) N∏

n = 1

p (zn |θ) p (wn |zn , β).

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

TAI et al.: SYSTEMATICAL APPROACH FOR DETECTING THE INTENTION AND INTENSITY OF FEELINGS 989

Fig. 2. sLDA model.

The margin distribution of the document d can thus be ex- pressed as follows:

p (d|α, β) = ∫p (θ|α) (

N∏

n = 1

∑

zn

p (zn |θ) p (wn |zn , β) )

dθ.

Subsequently, the probability of a corpus D consisting of M documents is inferred as follows:

p (D|α, β) = M∏

d= 1

∫p (θd|α) ( N d∏

n = 1

∑

zd , n

p(zd,n |θd )

× p(wd,n |zd,n , β) )

dθd .

The training of LDA is unsupervised learning. Expectation– maximization (EM) with variational inference [11], EM with expectation propagation [17], and Gibbs sampling [18] are the most commonly used strategies for learning LDA. Using these learning methods, given a corpus D and a specified number k of possible hidden topics in D for training, LDA can determine the probabilities of topics in each document and the words describ- ing each topic.

B. sLDA [9]

sLDA is a statistical model of labeled documents. It simulates a generative process of a corpus D similarly to LDA, but asso- ciates each document d with a response variable y. Therefore, sLDA can train a fitted model to infer and predict the label of a new coming unlabeled document. Fig. 2 shows the concept of the sLDA model. The inner rectangle represents a document d with N words, and the outer rectangle is the corpus D. The top rectangle is the same set of K topics shared by documents in D, where each topic is tied to a vector of word probabilities βZn . Under the sLDA model with the Dirichlet parameter α and the response parameters η and δ, the generative process of each document and response is as follows.

1) Choose θ from a Dirichlet distribution Dir(α), indicating the mixture of topics forming a document d.

2) To generated each word wn in d, a) choose a topic zn from a Multinomial distribution

Multinomial(θ); b) choose a word wn from a Multinomial distribution

Multinomial(βZn ). 3) Choose a response variable y from a generalized lin-

ear model (GLM) distribution GLM (z̄, η, δ), where z̄ := 1 N

∑N n = 1 zn .

The distribution of response is a GLM that can be expressed as follows:

p (y|z1:N , η, δ) = h (y, δ) exp {

ηT (z̄y) − A ( ηT z̄

)

δ

} .

When α, β1:K , η and δ are treated as unknown constants to be estimated, an approximate maximum-likelihood estimation can be achieved using a variational EM [9]. For a new unseen document, the response (label) can then be estimated using

E [Y |w1:N , α, β1:K , η, δ] ≈ Eq [ μ

( ηT Z̄

)] .

C. SentiWordNet [12], [21]

SentiWordNet is a lexical resource for opinion mining and sentiment analysis based on WordNet, an English lexical database. According to the degrees of how “positive,” “nega- tive,” or “objective” synonyms are from automatic annotation of speech, SentiWordNet associates every WordNet synonym collection to three numerical scores Pos, Neg, and Obj. Each of the three scores is normalized between 0 and +1, and their sum is equal to 1. This indicates that SentiWordNet allows a collection of synonyms to have nonzero Pos, Neg, and Obj scores simulta- neously, because many synonyms yield different interpretations depending on the context of use.

IV. FEELINGS DISTINGUISHER

A. Overview

The purpose of FeD is to detect how people feel (i.e., SN, GN, GP, or SP), about their lives through their online diaries. For de- tecting feeling intention and intensity, the rationale of FeD is to learn the discriminative expression terms for differentiating negative and positive feelings, and to learn the representative expression terms and the represented emotion intensity accord- ing to these terms for each intensity level of feelings. Thus, FeD is designed as a multilayer model based on sLDA, LDA, and SentiWordNet, where sLDA is trained to learn the discrimina- tive terms, LDA is utilized to learn the representative terms, and SentiWordNet is used to identify the sentiment terms and in- fer the intensity of feelings. sLDA is utilized for discriminating negative and positive feelings because supervised topic models are demonstrated to perform effectively, especially for limited training data (e.g., short texts, infrequent topics, and skewed distributions of labels, which are the general characteristics of online posts) [19], [20]. LDA, the unsupervised learning model (instead of sLDA), is used to identify representative terms of different intensity of feelings because LDA can associate key- words to latent topics, and the differences between sensitive terms expressing strong and general feelings are generally vague (compared with those between negative and positive feelings).

Fig. 3 shows the overview of FeD. The online posts collected from social network websites could contain various noises, which typically significantly reduce the effectiveness of anal- ysis. To focus on the analysis of feeling intensity interpreted by the wordings, SentiWordNet is first used as a dictionary for identifying sentiment terms from posts in both training and

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

990 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 20, NO. 4, JULY 2016

Fig. 3. System overview of FeD.

evaluation phases. FeD then discriminates between sentiment terms expressing negative and positive feelings by sLDA. In the training phase, an sLDA classifier is first trained by online posts with negative and positive labels. The well-trained sLDA classi- fier then assumes responsibility for discriminating the intention of feelings that a new post reveals in the evaluation phase. Fol- lowing the (predicted) feeling intention, FeD further extracts representative terms through LDA and calculates the intensity scores of feelings represented by the representatives based on SentiWordNet. Specifically, in the training phase, FeD learns the representative terms (denoted as TSN , TGN , TGP , and TSP ) and the corresponding intensity scores (denoted as SSN , SGN , SGP , and SSP ) of SN, GN, GP, and SP feelings. Subsequently, these terms and scores are used as guidelines in the evaluation phase for determining the intensity level of the feelings that a new post reveals.

In the following two sections, we explain the details of the training and evaluation phases.

B. Training Phase

In the training phase, FeD learns the discrimination of sen- timent terms between negative and positive feelings, the repre- sentative terms TSN , TGN , TGP , and TSP and the represented intensity scores SSN , SGN , SGP , and SSP for each intensity level of feelings. To focus on the analysis of feeling intensity in- terpreted by the wordings, all the learnings are performed under the consideration of sentiment terms alone. Therefore, the first step of the training phase in FeD is to identify sentiment terms from the posts based on SentiWordNet. FeD then performs the learnings as follows.

To learn the discriminative sentiment terms between negative and positive feelings, FeD trains an sLDA classifier according to a training corpus consisting of labeled online posts. Specif- ically, a training corpus C = {d1 , d2 , . . .} contains a group of online posts, where each post di consists of sentiment terms and

associates with one and only one label li

li =

{ N, if di reveals negative feelings

P, otherwise.

Given the corpus C for training, FeD then derives a fitted sLDA prediction model for discriminating between posts re- vealing negative feelings and those revealing positive feelings by the wordings.

To learn the representative terms for each intensity level of feelings, FeD applies LDA to further distinguish strong feelings from general feelings in corpuses revealing spe- cific feeling intentions. Specifically, to learn the represen- tative terms TSN = {tSN−1 , tSN−2 , . . . , tSN−k } and TGN = {tGN−1 , tGN−2 , . . . , tGN−k } of SN and GN feelings, FeD as- sumes two latent topics in a corpus revealing negative feelings (i.e., C = {d1 , d2 , . . .} where all li = N ) and utilizes LDA to extract the top k frequent concept keywords (associating with the latent topics) as the representatives. Similarly, the k representatives TGP = {tGP−1 , tGP−2 , . . . , tGP−k } and TSP = {tSP−1 , tSP−2 , . . . , tSP−k } of GP and SP feelings, respectively, are learned from a positive corpus (i.e., C = {d1 , d2 , . . .} where all li = P ) through LDA with the number of latent topics set at two.

Finally, FeD learns the represented intensity scores for each intensity level of feelings based on the corresponding represen- tative terms learned. Specifically, FeD regards SentiWordNet as a dictionary, which associates sentiment terms with intensity scores as introduced in the preliminary background, and derives the score of a specific level of feelings by averaging the inten- sity scores over all the learned representative terms of that level. In other words, for an intensity level l as SN, GN, GP or SP, the intensity score Sl is calculated based on the corresponding representatives Tl as

Sl =

∑ tl ∈T l s(tl ) |Tl|

= ∑k

i= 1 s(tl−i ) k

where s(tl ) is the intensity score SentiWordNet gives to a sen- timent term tl .

Example 1. Suppose that, given a corpus revealing negative feelings, LDA extracts the top 5 frequent concept keywords as the representatives, TSN and TGN , of SN and GN feelings as follows:

TSN (d) = {hate, fighting, normal, divorced, rejected}, TGN (d) = {mum, understand, hospital, today, hurt}.

Then, based on the score given to each term by SentiWordNet, the represented intensity scores are calculated accordingly as follows:

SSN = ((−1) + 0 + (−0.186) + 0.25 + (−0.5))/5 = − 0.287

SGN = (0.128 + 0.163 + 0.125 + 0.235 + (−1.081))/5 = − 0.086

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

TAI et al.: SYSTEMATICAL APPROACH FOR DETECTING THE INTENTION AND INTENSITY OF FEELINGS 991

Fig. 4. Pseudocodes of the evaluation phase of FeD.

C. Evaluation Phase

In the evaluation phase, given a new online post d, FeD outputs the evaluated intensity level of feelings ld with an evaluated intensity score Sd .

Fig. 4 shows the pseudocodes of the evaluation phase in- cluding four major steps: Sentiment Terms Identification, Feel- ing Concept Discrimination, Representatives Identification, and Feeling Intensity Evaluation. First, to focus on the feeling in- tensity, sentiment terms are identified from the new online post d based on SentiWordNet (Line 1). These identified sentiment terms are then passed to the well-trained sLDA classifier for detecting whether the feeling intention revealed by the post is negative or positive (Line 2). “ld = N ” indicates negative feel- ings, and “ld = P ” represents positive feelings. Next, according to the predicted feeling intention ld , FeD identifies the repre- sentative terms of strong feelings and general feelings of that intention in the new post d, denoted as TSld (d) and TGld (d), respectively, for evaluating the feeling intensity that d reveals (Lines 3–10). Specifically, to infer TSld (d) and TGld (d), FeD first extracts the concept keywords denoted as T (d) of d by assuming one latent topic represented in d and applying LDA. FeD then examines which of the keywords is regarded as the representative of specific feeling intensity in the training phase to determine TSld (d) and TGld (d); that is,

TSld (d) = T (d) ∩ TSld TGld (d) = T (d) ∩ TGld

where the predicted feeling intention ld ∈ {N, P }. Finally, FeD evaluates the feeling intensity d revealed by calculating the intensity scores Sd based on d’ representatives TSld (d) and TGld (d), and determines the intensity level of the feelings in

d by comparing Sd with the intensity scores of intensity levels learned in the training phase (Lines 11–24). Specifically, based on TSld (d) and TGld (d), the intensity score Sd of d is computed as

Sd = |TSld (d)|

∑ t∈T S l d (d)

s(t) + |TGld (d)| ∑

t∈T G l d (d) s(t)

|TSld (d)| + |TGld (d)| (1)

where ld ∈ {N, P } is the feeling intensity predicted previously and s(t) is the intensity score of a sentiment term t according to SentiWordNet. If |Sd − SSld | < |Sd − SGld |, then FeD predicts the intensity revealed by d to be general. Otherwise, FeD predicts the intensity to be strong.

Example 2. Follow example 1 (i.e., TSN , TGN , SSN , and SGN ). Suppose that a new post d is predicted as “ld = N ” by a well-trained sLDA and the concept keywords is

T (d) = {fighting, normal, hate, divorced understand} .

Then, the representative terms of d are

TSN (d) = T (d) ∩ TSN = {fighting, normal, hate, divorced,} , TGN (d) = T (d) ∩ TGN = {understand} .

Accordingly, the intensity score of d is

Sd = 4 × (0 + (−0.186) + (−1) + 0.25) + 1 × (0.163)

4 + 1 = − 0.7162.

Since “ld = N ” and |Sd − SSN | = 0.4292 < |Sd − SGN | = 0.6302, FeD evaluates the feelings that d presents as SN. �

V. EVALUATION

In this section, we evaluate the proposed FeD system through extensive experiments on 800 real online diaries and posts collected from five websites: Depression Forum,3 NetDoctor,4

Diary.com,5 MyDiary,6 and Facebook.7 The feeling each post revealed is labeled manually. In this work, we used the sLDA program built by Mcauliffe and Blei [9] for training an sLDA classifier to infer the latent topics and the probabilities of words appearing in the documents, and used Gibbs sampling [18] to learn the parameters of LDA classifiers.

In the following sections, we evaluate the performace of FeD by 1) showing the top 15 concept keywords identified as the rep- resentative terms of SN, GN, GP, and SP feelings; 2) comparing FeD with SVM and sLDA; and 3) exploring the effectiveness of FeD through case studies.

For evaluation, we consider the common aspects of precision, recall, F1-score, and accuracy. The specific definitions are given as follows.

3http://www.mentalhealthforum.net/forum/forum28.html 4http://www.netdoctor.co.uk/ 5http://diary.com/ 6http://www.my-diary.org/surf/ 7https://www.facebook.com/

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

992 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 20, NO. 4, JULY 2016

TABLE I TOP 15 CONCEPT KEYWORDS OF SN, GN, GP AND SP WITH THE APPEARANCE PROBABILITIES AND INTENSITY SCORES

Negative feelings Positive feelings

Strongly negative feelings General negative feelings General positive feelings Strongly positive feelings

Term Probability Score Term Probability Score Term Probability Score Term Probability Score

know 0.03 0.66 feel 0.014 0.04 love 0.033 0.72 love 0.023 0.72 feel 0.02 0.04 sleep 0.012 −0.08 know 0.018 0.66 day 0.019 0.017 people 0.02 0 feeling 0.012 −0.014 feel 0.016 0.04 time 0.019 0.079 time 0.01 0.08 depression 0.012 −0.16 thing 0.013 −0.006 back 0.013 0.126 make 0.01 0.29 back 0.012 0.137 heart 0.011 −0.004 heart 0.013 −0.004 things 0.01 0 time 0.011 0.08 people 0.011 0 night 0.011 −0.01 day 0.01 0.02 hope 0.011 0.128 life 0.011 0.0035 good 0.009 1.093 Lot 0.009 −0.018 today 0.008 0.235 things 0.011 0 better 0.008 1.875 house 0.009 0 work 0.008 0.022 happy 0.010 0.48 person 0.008 0

life 0.009 0.004 tired 0.008 0.068 smile 0.009 0.125 inside 0.008 0 talk 0.008 −0.056 head 0.007 0.02 friends 0.009 0.073 thought 0.007 0.013 anymore 0.008 0 good 0.006 1.09 way 0.008 0.0029 hard 0.007 −0.332 bad 0.008 −1.35 life 0.006 0.004 world 0.008 −0.375 long 0.006 −0.01 way 0.007 0.003 hard 0.006 −0.33 never 0.008 −0.46 well 0.006 0.714 hate 0.007 −1 night 0.006 −0.01 loved 0.007 0.5 made 0.006 0.045 Average Emotion Score −0.088 Average Emotion Score 0.082 Average Emotion Score 0.118 Average Emotion Score 0.289

1) Precision is the ratio of the number of relevant posts suc- cessfully identified to the total number of posts identified as relevant ones.

2) Recall is the ratio of the number of relevant posts suc- cessfully identified to the total number of truly relevant posts.

3) Accuracy is the proportion of the right judgment. 4) F1-score takes the review point of the harmonic mean of

precision and recall. In addition, the feeling intensity level a post presents is often

vague. One may even think that the separation of close lev- els of feeling intensity is not specific. Then, evaluting a post that reveals SN feelings as GN may be somewhat acceptable, whereas a post presenting SN feelings should not be evalu- ated as SP. Hence, in addition to the aforementioned common asepcts, we propose the significant error e∗ for evaluating the unacceptable judgments. Specifically, the significant error e∗ at an error-tolerant level i is defined as

e∗[i] = #(

{ d|i ≤

∣∣F (ld ) − F ( ld

)∣∣}) #({d})

where

F (ld ) =

⎧ ⎪⎪⎪⎨

⎪⎪⎪⎩

1, if ld = SN 2, if ld = GN 3, if ld = GP 4, if ld = SP,

ld represents the actual intensity of a post d, 1̄d is the judged intensity of d, and #(A) denotes the cardinality of a set A. Thus, the larger the value of i, the more tolerant of misjudgments is.

The following evaluations of precision, recall, accuracy, F1- score, and the significant error are reported using tenfold cross validation. As scientific supports for comparing FeD with SVM and sLDA, the p-values of two-tailed paired t-tests testing the

hypothesis that the difference in the evaluated performance of the compared methods is zero are also calculated.

A. Implicit Semantics of Representative Terms

The top concept keywords extracted from training posts play crucial roles in FeD for evaluating feeling intensity scores. The question remains whether the extracted keywords can properly specific feeling intensities. Therefore, we list the top concept keywords interpreting the SN, GN, GP, and SP feelings, and ex- amine whether the implicit semantics of these keywords comply with human intuition.

Table I lists the top 15 concept keywords with the appearance probabilities and intensity scores defined by the SentiWordNet dictionary. Table I shows that the negative feelings are expressed through many negative sentiment terms in the posts. In par- ticular, GN feelings are expressed in words such as “feeling,” “depression,” “tired,” and “hard,” whereas SN feelings are repre- sented by terms that are more negative, such as “bad” and “hate.” However, posts showing positive feelings mention “love” and use words that are more positive to describe what is appreciated in life. In particular, the posts showing GP feelings mention “smile” and “friends,” whereas the posts presenting SP feelings use encouraging words such as “good,” “better,” and “well.”

B. Performance of FeD

To evaluate the performances of FeD, we compare FeD with SVM and sLDA in terms of precision, recall, F1-score, accuracy, and error e∗.

Table II shows the confusion matrices of detecting feeling intensity using FeD, SVM, and sLDA, where each cell reports (for the posts belonging to each intensity level) the percentage of posts being classified to each intensity level. Fig. 5 presents the comparisons of the average precision, recall, F1-score, and accuracy of FeD, SVM, and sLDA, where the error bars

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

TAI et al.: SYSTEMATICAL APPROACH FOR DETECTING THE INTENTION AND INTENSITY OF FEELINGS 993

TABLE II CONFUSION MATRICES OF DETECTING FEELING INTENSITY

USING FED, SVM, AND SLDA

Evaluated feeling intensity

FeD SN GN GP SP total

Actual feeling intensity SN 22.5% 42% 23.5% 12% 100% GN 6% 58.5% 23.5% 12% 100% GP 11% 25% 37% 27% 100% SP 13% 27.5% 27% 32.5% 100%

Evaluated Feeling Intensity

SVM SN GN GP SP total

Actual feeling intensity SN 26.5% 35% 22% 16.5% 100% GN 20% 35% 29% 16% 100% GP 15.5% 21% 40.5% 23% 100% SP 15% 15% 36% 34% 100%

Evaluated Feeling Intensity

sLDA SN GN GP SP total

Actual feeling intensity SN 24% 35% 22% 19% 100% GN 16% 37% 30% 17% 100% GP 11.5% 23% 39.5% 26% 100% SP 14% 26% 31% 29% 100%

Fig. 5. Comparisons of performance of FeD, SVM, and sLDA.

represent the standard deviation of each mean value. Accord- ingly, the precision of FeD is 3.89% higher than that of SVM (p = 0.0050) and 5.76% higher than that of sLDA (p = 0.0019), where the standard deviations of FeD, SVM, and sLDA are 3.75%, 4.64%, and 5.49%, respectively. The recall of FeD is 3.63% higher than that of SVM (p = 0.0105) and 5.25% higher than that of sLDA (p = 0.0002), where the standard deviations of FeD, SVM, and sLDA are 3.09%, 5.52%, and 4.19%, re- spectively. For the F1-score, FeD outperforms SVM and sLDA 2.64% (p = 0.0528) and 4.36% (p = 0.0048), respectively. The standard deviations of FeD, SVM, and sLDA, respectively, are 3.35%, 5.16%, and 4.72%. For accuracy, FeD outperforms SVM and sLDA 3.63% (p = 0.0105) and 5.25% (p = 0.0002), respectively. The standard deviations of FeD, SVM, and sLDA, respectively, are 3.09%, 5.52%, and 4.19%. In summary, FeD has the smallest standard deviation, and the performance of FeD is on average 1.08–1.18 folds that of SVM and sLDA.

Table III shows the significant errors e∗[1], e∗[2], and e∗[3] of FeD, SVM, and sLDA together with the corresponding p-values for comparing FeD with SVM and sLDA. Accordingly, FeD

TABLE III COMPARISONS OF SIGNIFICANT ERRORS OF FED, SVM, AND SLDA

e∗[1] e∗[2] e∗[3]

FeD 62.38% ± 3.09% 24.75% ± 4.67% 6.25% ± 0.83% SVM 66% ± 5.52% 25% ± 3.63% 7.88% ± 2.5%

(p = 0.0251) (p = 0.8939) (p = 0.1599) sLDA 67.63% ± 4.19% 27.38% ± 6.19% 8.25% ± 4.94%

(p = 0.0004) (p = 0.1273) (p = 0.3702)

TABLE IV CONFUSION MATRICES OF DETECTING FEELING INTENSITY

USING FED(SLDA) AND SVM

Evaluated feeling intensity

FeD(sLDA) Negative Positive total

Actual feeling intensity Negative 64.5% 35.5% 100%

Positive 38.25% 61.75% 100%

Evaluated Feeling Intensity

SVM Negative Positive total

Actual feeling intensity Negative 42.25% 57.75% 100%

Positive 32.75% 67.25% 100%

commits the fewest errors at all error-tolerant levels. As a low number of errors correspond with higher method performance, FeD demonstrates the highest performance in the evaluation of feeling intensity.

These results above show that FeD outperforms SVM and sLDA in evaluating the feeling intensity (SN, GN, GP, or SP) that the online posts reveal, from the aspects of precision, recall, F1-score, accuracy, and the significant errors. This is because FeD learns not only the discriminative terms but also the feel- ing intensity the representative terms present as well as these representatives.

Furthermore, note that FeD is a multilayer model consisting of sLDA, LDA and SentiWordNet. In order to provide further insights of FeD and to answer the question “whether replacing sLDA with SVM in FeD can improve the performance,” we also compare FeD (sLDA) with SVM at the 2-class problem that con- siders only negative and positive feelings. Table IV shows the confusion matrices of detecting negative and positive feelings using FeD (sLDA) and SVM, where each cell reports (for the posts belonging to each category of feelings) the percentage of posts being classified to each category. Fig. 6 shows the preci- sion, recall, F1-score, and accuracy of FeD (sLDA) and SVM for distinguishing the negative and positive feelings. Accordingly, the precision, recall, F1-score and accuracy of FeD (sLDA) are 7.05%, 22.25%, 0.18%, and 8.38% higher than those of SVM (where the values of p are 0.0678, 0.0001, 0.9660, and 0.0149, respectively). Therefore, adopting sLDA in FeD is better.

C. Case Studies

To demonstrate the effectiveness of FeD, we also apply FeD to analyze other 4000 online articles collected from 20 different

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

994 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS, VOL. 20, NO. 4, JULY 2016

Fig. 6. Comparisons of performance of FeD(sLDA) and SVM.

TABLE V TWENTY COMMUNITIES IN JOURNALLIVE: COMMUNITY NAME, COMMUNITY

DESCRIPTION, AND EVALUATED FEELING INTENSITY

Community name

Community description Predicted intensity

5 Gratitude’s Share 5 things which made you happy today

General Positive

Add Me A community to meet new friends. Strong Positive Bad Services Complaints about bad service, and

related discussion Strong Negative

Bipolar Survival Bipolar, depression, mania and mood swings, surviving them all

General Negative

Co-worker Sucks

A place to vent about your sucky co-workers & bad bosses!

Stronger Negative

Customer Sucks Customers Who Suck General Positive Dear You A place for unsent letters Strong Positive Depression Support for people dealing with

depression General Negative

ED Recovery Eating Disorder Recovery & Help Strong Negative Friends I Want Friends General Positive I am Thankful We are (thankful) everyday General Positive I Wish A community for people to share their

hopes, dreams and wishes. Strong Negative

Momless Daughter

Support for parentless people Strong Negative

New Friends A Place to Meet New Friends Strong Positive Panic Anxiety A resource for those who experience

persistent anxiety or received an anxiety disorder diagnosis.

General Negative

The Nicest Things

Post the nicest things people say to you General Positive

Think Positive 30

Post thirty positive thoughts over 30 consecutive days and help to change your mindset.

Strong Positive

Unsent Letters Unsent Letters Strong Positive Wedding Plans Wedding Plans General

Negative What Do You Hate

Fucking hell just feel like ragin yano General Negative

communities of LiveJournal.8 (The first two columns of Table V list the community names and the given descriptions on Live- Journal for the 20 communities. From each community, 200 articles are collected.) The topics of the articles posted in each community are associated with the name of the community in certain aspects. In the analysis, we then classify each of the articles into one of the four categories (i.e., SN, GN, GP, and SP feelings) according to the feeling intensity interpreted in the articles, determine the label of each community according to

8http://www.livejournal.com/

its articles, and examine whether the predicted label intuitively comply with the implicit feelings interpreted by the name of the communities.

At the third column of Table V, it shows the predicted la- bels using FeD for the 20 communities. Accordingly, 17 of 20 communities are given expected or understandable labels. For example, the communities “Bad Services,” “Bipolar Survival,” “Friends I Want,” and “Think Positive 30” are labeled as SN, GN, GP and SP respectively. However, the “I Wish,” “Wedding Plans,” and “Unsent Letters” communities, to which we may expect a label of SP, SP, and GN, respectively, are assigned as SN, GN, and SP. To figure out the reasons, we read the arti- cles and observed that 52.5% articles collected in the “I Wish” community made wishes that something had not happened (in- stead of making blessings). For “Wedding Plans,” 54.5% articles complain about the details of preparing a wedding. For “Unsent Letters,” 70% articles collected share love or thanks that was not told.

VI. CONCLUSION

This paper addressed the problem of detecting how people feel about their daily life through their online posts, and catego- rized the feelings into four categories: SN, GN, GP, and SP. For the problem, we proposed a FeD methodology based on sLDA, LDA and SentiWordNet. Experimental results showed that the performance of FeD is about 1.08–1.18 folds that of SVM and sLDA.

REFERENCES

[1] S. Bao, S. Xu, L. Zhang, R. Yan, Z. Su, D. Han, and Y. Yu, “Mining social emotions from affective text,” IEEE Trans. Knowl. Data Eng., vol. 24, no. 9, pp. 1658–1670, Sep. 2012.

[2] M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, and A. Kappas, “Sentiment strength detection in short informal text,” J. Amer. Soc. Inf. Sci. Technol., vol. 61, no. 12, pp. 2544–2558, 2010.

[3] M. Yassine and H. Hajj, “A framework for emotion mining from text in online social networks,” in Proc. IEEE Int. Conf. Data Mining Workshops, 2010, pp. 1136–1142.

[4] M. D. Choudhury, S. Counts, E. J. Horvitz, and A. Hoff, “Characteriz- ing and predicting postpartum depression from shared Facebook data,” in Proc. 17th ACM Conf. Comput. Supported Cooperative Work Soc. Comput., 2014, pp. 626–638.

[5] M. D. Choudhury, M. Gamon, S. Counts, and E. Horvitz, “Predicting depression via social media,” presented at the 7th International AAAI Conference on Weblogs and Social Media, Boston, MA, USA, 2013.

[6] G. Coppersmith, C. Harman, and M. Dredze, “Measuring post traumatic stress disorder in Twitter,” presented at the International AAAI Conference on Weblogs and Social Media, Ann Arbor, MI, USA, 2014.

[7] T. Nguyen, D. Phung, B. Dao, S. Venkatesh, and M. Berk, “Affective and content analysis of online depression communities,” IEEE Trans. Affective Comput., vol. 5, no. 3, pp. 217–226, Jul.–Sep. 2014.

[8] American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders V. Washington, DC, USA: Amer. Psychiatric Press, 2013.

[9] J. D. Mcauliffe and D. M. Blei, “Supervised topic models,” presented at the Advances in Neural Information Processing Systems, Vancouver, B.C., Canada, 2007.

[10] D. Ramage, D. Hall, R. Nallapati, and C. D. Manning, “Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora,” in Proc. Conf. Empirical Methods Natural Language Process., 2009, pp. 248–256.

[11] D. Blei, A. Ng, and M. Jordan, “Latent Dirichlet allocation,” J. Mach. Learning Res., vol. 3, pp. 993–1022, 2003.

[12] A. Esuli and F. Sebastiani, “Sentiwordnet: A publicly available lexical resource for opinion mining,” presented at the Conference on Language Resources and Evaluation, Genoa, Italy, 2006.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.

TAI et al.: SYSTEMATICAL APPROACH FOR DETECTING THE INTENTION AND INTENSITY OF FEELINGS 995

[13] P. Chikersal, S. Poria, and E. Cambria, “SeNTU: Sentiment analysis of tweets by combining a rule-based classifier with supervised learning,” presented at the International Workshop on Semantic Evaluation, Denver, Colorado, USA, 2015.

[14] M. Hasan, E. Agu, and E. Rundensteiner, “Using hashtags as labels for supervised learning of emotions in twitter messages,” presented at the ACM SIGKDD Workshop on Health Informatics, New York, USA, 2014.

[15] J. A. Russell, “A circumplex model of affect,” J. Personality Soc. Psychol., vol. 39, no. 6, pp. 1161–1178, 1980.

[16] S. Volkova, Y. Bachrach, M. Armstrong and V. Sharma, “Inferring latent user properties from texts published in social media,” presented at the 29th AAAI Conference on Artificial Intelligence, Austin, Texas, USA, 2015.

[17] T. Minka and J. Lafferty, “Expectation-propagation for the generative aspect model,” presented at the Conference on Uncertainty in Artificial Intelligence, Alberta, Canada, 2002.

[18] T. L. Griffiths and M. Steyvers, “Finding scientific topics,” Proc. Nat. Acad. Sci., vol. 101, no. 1, pp. 5228–5235, 2004.

[19] D. Quercia, H. Askham, and J. Crowcroft, “TweetLDA: Supervised topic classification and link prediction in twitter,” in Proc. Annu. ACM Web Sci. Conf., 2012, pp. 247–250.

[20] T. N. Rubin, A. Chambers, P. Smyth, and M. Steyvers, “Statistical topic models for multi-label document classification,” Mach. Learning, vol. 88, nos. 1/2, pp. 157–208, 2012.

[21] S. Baccianella, A. Esuli, and F. Sebastiani, “SentiWordNet 3.0: An en- hanced lexical resource for sentiment analysis and opinion mining,” presented at the Conference on International Language Resources and Evaluation, Malta, 2010.

[22] Y. S. Chang, C. T. Fan, W. T. Lo, W. C. Hung and S. M. Yuan, “Mobile cloud-based depression diagnosis using an ontology and a Bayesian network,” Future Gener. Comput. Syst., vol. 43–44, pp. 87–98, 2015.

[23] S. Crystal, U. Sambamoorthi, J. T. Walkup, and A. Akincigil, “Diagnosis and treatment of depression in the elderly medicare population: Predictors, disparities, and trends,” J. Amer. Geriatrics Soc., vol. 51, no. 12, 1718– 1728, 2003.

[24] Z. Rihmer, “Can better recognition and treatment of depression reduce suicide rates? A brief review,” Eur. Psychiatry, vol. 16, no. 7, pp. 406– 409, 2001.

[25] B. W. Rovner, R. J. Casten, M. T. Hegel, B. E. Leiby, and W. S. Tasman, “Preventing depression in age-related macular degeneration,” Arch. Gen. Psychiatry, vol. 64, no. 8, pp. 886–892, 2007.

[26] B. G. Tiemens, J. Ormel, J. A. Jenner, K. Van Der Meer, T. W. D. P. Van Os, R. H. S. Van Den Brink, A. Smit, and W. Van Den Brink, “Training primary-care physicians to recognize, diagnose and manage depression: Does it improve patient outcomes?” Psychol. Med., vol. 29, no. 4, pp. 833–845, 1999.

Authors’ photographs and biographies not available at the time of publication.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on September 25,2021 at 02:30:25 UTC from IEEE Xplore. Restrictions apply.