helpfn

Received June 10, 2019, accepted July 15, 2019, date of publication August 12, 2019, date of current version August 23, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2934529

Semantic-Emotion Neural Network for Emotion Recognition From Text ERDENEBILEG BATBAATAR 1, MEIJING LI2, AND KEUN HO RYU 3,4, (Life Member, IEEE) 1School of Electrical and Computer Engineering, Chungbuk National University, Cheongju 28644, South Korea 2College of Information Engineering, Shanghai Maritime University, Shanghai 200136, China 3Faculty of Information Technology, Ton Duc Thang University, Ho Chi Minh City 700000, Vietnam 4Database and Bioinformatics Laboratory, School of Electrical and Computer Engineering, Chungbuk National University, Cheongju 28644, South Korea

Corresponding author: Keun Ho Ryu ([email protected])

This work was supported in part by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, Information and Communication Technology (ICT) and Future Planning under Grant 2017R1A2B4010826 and Grant NRF-2019K2A9A2A06020672, and in part by the National Natural Science Foundation of China in People’s Republic of China under Grant 61702324.

ABSTRACT Emotion detection and recognition from text is a recent essential research area in Natural Language Processing (NLP) which may reveal some valuable input to a variety of purposes. Nowadays, writings take many forms of social media posts, micro-blogs, news articles, customer review, etc., and the content of these short-texts can be a useful resource for text mining to discover an unhide various aspects, including emotions. The previously presented models mainly adopted word embedding vectors that represent rich semantic/syntactic information and those models cannot capture the emotional relationship between words. Recently, some emotional word embeddings are proposed but it requires semantic and syntactic information vice versa. To address this issue, we proposed a novel neural network architecture, called SENN (Semantic-Emotion Neural Network) which can utilize both semantic/syntactic and emotional information by adopting pre-trained word representations. SENN model has mainly two sub-networks, the first sub- network uses bidirectional Long-Short Term Memory (BiLSTM) to capture contextual information and focuses on semantic relationship, the second sub-network uses the convolutional neural network (CNN) to extract emotional features and focuses on the emotional relationship between words from the text. We conducted a comprehensive performance evaluation for the proposed model using standard real-world datasets. We adopted the notion of Ekman’s six basic emotions. The experimental results show that the proposed model achieves a significantly superior quality of emotion recognition with various state-of-the- art approaches and further can be improved by other emotional word embeddings.

INDEX TERMS Emotion recognition, natural language processing, deep learning.

I. INTRODUCTION Emotion recognition will play a promising role in the field of artificial intelligence and human-computer interaction [1]. Various types of techniques are used to detect emotions from a human being like facial expressions [2], body movements [3], blood pressure [4], heartbeat [5] and textual information [6]. In computational linguistics, the detection of human emotions in a text is becoming increasingly impor- tant from an applicative point of view. Nowadays within the internet, there’s an enormous amount of textual data. It’s fascinating to extract emotion from various goals like those of

The associate editor coordinating the review of this article and approving it for publication was Irene Amerini.

business [7]. As an example, in luxury merchandise, the emo- tional aspect as brand, individuality and prestige for purchas- ing confirmations, are a lot necessary than other aspects such as technical, functional or price [8]. There are basic emotion theories that have been developed on how some emotions are considered more than others [9], [10]. This study adopted basic six emotions [9] including joy, fear, anger, sadness, surprise and disgust.

There are many works which have achieved reasonable results in the field of emotion recognition from text. Improv- ing the previous result and emotion recognition using real- world data still remains a huge challenge for several reasons. Most machine learning methods overly rely on handcrafted features which require lots of manual design and adjustment,

111866 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ VOLUME 7, 2019

E. Batbaatar et al.: SENN for Emotion Recognition From Text

and it is time-consuming and cost intensive. Though the problem is helped greatly by the proposal of deep learning in recent years.

Word embeddings are widely used for many NLP tasks, such as sentiment analysis [11], question answering [12], and machine translation [13]. Existing word embedding learning approaches mostly represent each word by predicting the target word through its context [14], [15] and map words of similar semantic roles into nearby points in the embedding space. For example, the words ‘‘good’’ and ‘‘bad’’ are seman- tically similar and mapped into embedding space closely. It is confused, however, in emotional condition. Recently, some emotion embeddings are proposed for solving this issue and achieved better performance in emotion-related tasks [16]–[19].

Previous studies mostly used semantic based word embed- dings and achieved good results [20]–[22] by training a single model which can adopt either semantic or emotion word embeddings. As mentioned, these neural network approaches cannot encode and learn both semantic and emotional relationship in short text efficiently.

In order to address the above limitations, this paper pro- posed a novel neural network architecture, called semantic- emotion neural network (SENN) which can utilize both semantic and emotion information by adopting existing pre-trained word embeddings. We divided SENN into two sub-networks. The first network uses BiLSTM to capture semantic information map it into semantic-sentence space, the second network uses CNN to capture emotion information and map it into emotion-sentence space. CNN is supposed to be good at extracting position invariant features such as emotion terms and BiLSTM at modeling units in sequence of long semantics in whole sentence. Then we combine the final representation together for further emotion recognition. Experimental results show that the SENN model outperforms most of the baseline methods and state-of-the-art approaches. The main contributions of this work are as follows:

a) Semantic and emotion word embeddings are adopted separately in two sub-networks for same text input.

b) Under the framework of deep neural network, we use BiLSTM and CNN for designing semantic and emotion sentence encoder respectively. BiLSTM is designed to capture contextual information and CNN is designed to extract emotional information effectively.

c) A novel dual neural network model is proposed. We respectively use BiLSTM and CNN for encoding semantic and emotion text. Then we combine the final representation by concatenating semantic and emotion sentence encoding for further emotion recognition.

d) To get a better knowledge of semantic and emo- tion information on a specific dataset, we used the fine-tuning technique on pre-trained word embeddings which improves the performance of emotion recogni- tion models from text efficiently. Then we concate- nated the sentence-level encoded vectors to recognize emotion from the text.

The rest of the paper is organized as follows. Section II presents related works. Section III describes the detail of the proposed SENN model architecture. Section IV outlines the experimental setup, and Section V discusses the empirical results and analysis. Finally, Section VI presents the conclu- sion and introduces the next research direction.

II. RELATED WORK This section reviews recent advances in emotion recognition analysis. We roughly categorize the existing studies into two types: 1) Emotion recognition based on the traditional method; 2) Emotion recognition based on deep learning approaches.

A. EMOTION RECOGNITION BASED ON TRADITIONAL METHOD The traditional methods on emotion recognition analysis can be roughly subdivided into two categories: 1) lexicon- based method; 2) machine learning method. Lexicon based methods utilize pre-defined lists of terms that are cate- gorized according to different emotions [23]. On the one hand, these lexicons are often compiled manually, a fact which can later be exploited for keyword matching. For instance, the NRC Word-Emotion Association lexicon was derived analogously but with the help of crowdsourcing rather than involving experts from the field of psychol- ogy research, they annotated around 14000 words in the English language [24]. Another popular emotion lexicon is WordNet-Affect dictionary [25]. They tried to create a lexical representation of effective knowledge by starting from the WordNet lexical database [26]. WordNet-Affect dictionary starts with a set of seed words labelled as effect and then assigns scores to all other words based on their proximity to the seed words. Another attempt to generate an emotional lexicon is DepecheMood [27]. They used crowdsourcing to annotate 35,000 words. They showed that lexicons could be used in several approaches in sentiment analysis, as features for classification in machine learning methods [28], or to generate an affect score for each sentence, based on the scores of the words which are higher in the parse tree [29]. Other emotional lexicons frequently used in the literature are LIWC lexicon [30] consisting of 6,400 words annotated for emotions, and also ANEW (Affective Norm for English Words) [31]. This dataset has near 2,000 words which have been annotated based on the dimensional model of emotions, with three dimensions of valance, arousal and dominance. Lexicon based approaches are generally known for their straightforward use and out-of-the-box functionality. How- ever, manual labelling is error-prone, costly, and inflexible as it impedes domain customization. Conversely, the vocabulary from the heuristics is limited to a narrow set of dimensions that were selected a priori and, as a result, this procedure has difficulties when generalizing to other emotions [32].

Machine learning can infer decision rules for recognizing emotions based on a corpus of training samples with explicit labels [33], [34]. Previous researches have experimented

VOLUME 7, 2019 111867

E. Batbaatar et al.: SENN for Emotion Recognition From Text

with different models for inferring emotion from narrative materials. Examples include methods that explicitly exploit the flexibility of machine learning such as Naïve Bayes (NB) [35], Random Forest (RF) [36], [37], Support Vector Machine (SVM) [33], [35], [38]–[40], and Logistic Regres- sion (LR) [37]. All of which have commonly been deployed in literature. These classifiers are occasionally, but infre- quently, restricted to the subset of affect cues from emo- tion lexicons [41]. The more common approaches rely upon general linguistic features, bag-of-words (BOW) with subse- quent term frequency-inverse document frequency (TF-IDF) weighting [42], [43]. However, these features are often not suitable for document distances due to their frequent near- orthogonality [44], [45].

B. EMOTION RECOGNITION BASED ON DEEP LEARNING In the following, we discuss the few attempts at applying deep learning to emotion recognition, but find that actual perfor- mance evaluations are scarce. CNN with a sliding window and subsequent max-pooling are used to predict aggression expressed through NLP [36]. However, this approach is sub- ject to several limitations as the network is designed to handle only a single dimension and it is thus unclear how it gener- alizes across multi-class predictions or even regression tasks that appear in dimensional emotion models. Even though the approach utilizes a deep network, its network architecture can only handle texts of a predefined size, analogous to traditional machine learning. In this respect, it differs from recurrent networks, which iterate over sequences and thus can handle texts of arbitrary size.

There are many recurrent neural network (RNN) methods that are introduced for emotion recognition tasks. Due to the lack of emotion-labelled datasets, many supervised clas- sification algorithms for emotions have been done on data gathered from microblog such as Twitter, using hashtags or emoticons as the emotion label for the data. The current state- of-the-art methods are introduced by using Gated Recurrent Unit (GRU) network [20] for fine-grained emotion recogni- tion. They firstly built a large dataset for emotion recognition automatically from Twitter, then extended the classification to eight primary emotion dimensions situated in the psycho- logical theory of emotion.

A Long-Short Term Memory (LSTM) [19] is utilized that is pre-trained with tweets based on the appearance of emoti- cons. However, this work does not report a comparison of their LSTM against a baseline from traditional machine learning. A different approach [46] utilizes a custom LSTM architecture in order to assign emotion labels to complete conversations in social media. However, this approach is tai- lored to the specific characteristics and emotions of this type of conversational-style data. In addition, the conclusion from their numerical experiments cannot be generalized to emotion recognition, since the authors labelled their dataset through a heuristic procedure and then reconstructed this heuristic with their classifier.

A BiLSTM is also utilized to recognize emotions in cross- lingual texts [47] which employs the cross-lingual feature and the lexical level feature to analyze texts with multi- lingual forms. To incorporate a context-dependent word, an attention-based BiLSTM model [48] is introduced, which helps to decide the importance of each word for the emotion recognition task. They used the three modalities such as text, emoji and images to encode different information to express emotions.

Deep learning based methods mostly uses distributed word vectors, commonly used methods are Word2Vec [14], GloVe [15], and FastText [49]. Word2Vec is one of the very first models to learn word representations from trillions of words with relatively low computational costs. It has sig- nificantly outperformed various n-gram models [50]. Later GloVe was released, which stands for global vectors for word representations. It was an improvement over Word2Vec as it trains on global co-occurrence counts instead of separate local context windows in Word2Vec. Lately, the FastText was created for classification and learning of word representation. Word2Vec and Glove treat words as the smallest atomic units. FastText uses a different approach where it treats each individual word as being made of n-gram characters and it is more powerful than other two models as it can effectively handle rare words which are not present in the dictionary. Moreover, contextualized word embeddings are proposed, called ELMo [51] and BERT [52], to incorporate context information and solve the polysemy issues in conventional word embeddings. However, these word embeddings are gen- eralized on various tasks and limited to provide emotion information, therefore learning task-specific emotion embed- ding with the neural network has been proven to be effective. Emotion-enriched word embedding (EWE) is learned [17] on product reviews, with the much smaller corpus. This embed- ding could be easily applied to emotion-related tasks, which could largely overcome the limitations of emotion dictionary. It is, however, limited to provide a semantic and syntactic relationship.

Recently, other combined architectures are proposed for text-based emotion recognition tasks. For instance, the com- bined recurrent convolutional neural network (RCNN) approach [53] are introduced and achieved competitive results with fine-tuned contextual and emotional word embeddings [19], [51], [52]. The experimental results show that the fine-tuned GloVe embeddings perform noticeably better than contextual word embeddings, due to the emotion recognition task highly depends on emotion extraction and size of word dictionary.

There is some comparative study, which compared the traditional machine learning algorithms and deep learning based algorithms on large Twitter data [21]. They compared SVM, NB, LR, and RF algorithms with basic deep learn- ing algorithms CNN and RNN with GloVe word embed- dings. Generally, deep learning approaches outperformed the machine learning approaches in the emotion recognition task.

111868 VOLUME 7, 2019

E. Batbaatar et al.: SENN for Emotion Recognition From Text

FIGURE 1. SENN model architecture.

However, they collected a large number of training data, twit- ter data is more biased and noisy compared than hand- aggre- gated corpus [54], which is released for emotion analysis research, the combined datasets are collected from different domains and different label set.

In this paper, we propose a novel SENN model with BiLSTM and CNN sub-networks to conduct the emotion recognition task. We used the word embeddings Word2Vec, GloVe, and FastText to capture the semantic relationship between words and emotional word embedding to extract emotional features and evaluated the SENN model and the other compared models.

III. SENN MODEL In the paper, we propose a novel SENN model for emotion recognition from text and the structure is shown in Figure 1. It consists of two sub-networks: 1) BiLSTM network for semantic encoder between words and 2) CNN network for emotion encoder. The outputs of the sub-networks are used to recognize emotions from the text. Both of the two sub- networks are fed by the same sequence of N words and each word is transformed to a d dimensional word vector. Ultimately, the word embedding layer encodes the sequence representation as two matrices Zemo (emotion word embed- ding) and Zsem (semantic word embedding),

Zemo = [w emo 1 , . . . , w

emo t , . . . , w

emo N ] ∈ R

N×d (1)

Zsem = [w sem 1 , . . . , w

sem t , . . . , w

sem N ] ∈ R

N×d (2)

where wet and w s t are the emotion and semantic word vectors

of the word wt in the sequence, respectively,

wemot = [et1, . . . , etk, . . . , etd] (3)

wsemt = [st1, . . . , stk, . . . , std] (4)

FIGURE 2. BiLSTM structure.

Since we have enough data and our task slightly differs from unsupervised task used for training pre-trained word embedding, we speculated that further fine-tuning during the training process might improve the embedding. Whether to do this or not was an additional parameter we optimized.

A. BILSTM NETWORK FOR SEMANTIC ENCODER To better model the semantic information of text, we used bidirectional LSTM [55] to derive the hidden state of each word by summarizing the information from both forward and backward directions. An architecture of BiLSTM used in this paper is shown in Figure 2. An input for an LSTM is represented as Zsem matrix and the LSTM is fed by semantic word vectors wsemt . Forward LSTM and backward LSTM are denoted as

−−−→ LSTM and

←−−− LSTM, whereas

−−−→ LSTM reads words

VOLUME 7, 2019 111869

E. Batbaatar et al.: SENN for Emotion Recognition From Text

from left to right and ←−−− LSTM in reverse direction,

−→ ht =

−−−→ LSTM

( wsemt ,

−−→ ht−1

) , t ∈ [1, N] (5)

←− ht =

←−−− LSTM

( wsemt ,

←−− ht+1

) , t ∈ [1, N] (6)

We get a representation of each wsemt by concatenating the forward hidden state

−→ ht and the backward hidden state

←−− ht+1.

ht = [ −→ ht , ←− ht ] (7)

Finally, semantic sequence vector hsem encoded from the last hidden state is fed into the hidden layer.

hsem = hN (8)

gsem = f (wsemhsem +bsem) (9)

where gsem is the output of the semantic encoder, wsem and bsem are parameters of the f activation function.

FIGURE 3. CNN structure.

B. CNN NETWORK FOR EMOTION ENCODER To better extract emotion features from emotion-based word embeddings, we used CNN [56] to utilize layers with con- volving filters that are applied to local features. An archi- tecture of the CNN used in this paper is shown in Figure 3. An input for a CNN is represented as Zemo matrix and the CNN is fed by emotion word vectors wemot . Then the word embedding vectors are concatenated as the feature vector v of the sequence.

v = wemo1 ⊕ . . . w emo t . . .⊕wN

emo (10)

where ⊕ is the concatenation operator of vectors. In the first convolution layer, convolution calculation is performed using employ multiple filters with variable window size s and generate a local emotion feature vector xi for each possible word window size. And the bias term b ∈ R and transition matrix W ∈ Rsu×sN are generated for each filter, where su

is the number of hidden units in the convolution layer. Each convolution operation generates a new context local feature vector xsi in a word window s.

xsi = f (W·vi:i+s−1 +b) (11)

where f is a non-linear activation function and vi:i+s−1 is the local vector from position i to position i + s − 1 in the vector v. The convolution filter generates a local fea- ture mapping vector for each possible word window in the input sequence, which is followed by the completion of the convolution operation to generate a new vector that can be expressed as:

xs = [xs1, . . . , x s i , . . . , x

s N−s+1] (12)

Afterwards the convolution operation, max pooling operation is employed on the new feature vector xsi generated by the convolution layer. Max pooling mapped the vector xsi to a fixed length vector. The length of the vector is a hyperpa- rameter to be determined by the user and corresponds to the number of hidden units in the convolution layer. The local sentence features are integrated into all the features. For emotion recognition, the most decisive word or phrase is often only a few, but not uniformly scattered throughout the text. The max pooling is just some of the most discriminative language fragments. The max pooling selects the top number of features corresponding to multiple hidden layers so that the most important emotion feature information can be retained. At the same time, the sequence of words and the context information of each word are also taken into consideration in the pooling operation.

xsmax = max{x s 1, . . . , x

s i , . . . , x

s N−s+1} (13)

Since there are multiple feature maps, we have a vector after the pooling operation. All vectors which are output from the max-pooling layer are concatenated into a single feature vector hemo.

hemo = [ xsmax

] , s ∈ [smin, smax] (14)

Finally, the emotion sequence vector hemo is fed into the hidden layer.

gemo = f (wemohemo +bemo) (15)

where gemo is the output of the emotion encoder, wemo and bemo are parameters of the non-linear f activation function.

C. EMOTION RECOGNITION Finally, after producing emotion encoding gemo and semantic encoding gsem, we concatenated the vectors as c and fed it into the feedforward layer,

c = [gemo, gsem] (16)

o = f (woc+bo) (17)

where f is the feed-forward layer, wo and bo are weight and bias parameters respectively. o is the output of the feed-forward layer.

111870 VOLUME 7, 2019

E. Batbaatar et al.: SENN for Emotion Recognition From Text

Softmax classifier takes the output at the last step and o serves as its input. As noted above, given sequence with N words, we predict the emotion y for each sequence. Emo- tion annotations of sequences are represented by Y(Y = Y1, Y2, . . . , YM). The predicted values y′ can be calculated by:

p(y|X) = softmax(wpo+bp) (18)

and

y′ = arg max y

p(y|X) (19)

where p is the predicted probability of emotion label, wp and bp are parameters of the classification layer. We then use the cross-entropy to train the loss function. We first derive the loss of each labelled sequence and the final loss is averaged over all the labelled sequences by the following equation:

Loss =− 1 M

∑M m=1

Ym · logp(yn|Xn) (20)

where the subscript n indicates the nth input sequence.

IV. EXPERIMENTAL SETUP A. EXPERIMENTAL ENVIRONMENT In this paper, the experimental hardware platform is Intel Xeon E3, 32G memory, GTX 1080 Ti. The experimental soft- ware platform is Ubuntu 17.10 operating system and devel- opment environment is Python 3.5 programming language. The Pytorch library and the Scikit-learn library of python are used to build the proposed emotion recognition model and comparative experiments, respectively.

B. EVALUATION MEASURES For evaluating the SENN model, an exact matching crite- rion was used to examine three different result types. False negative (FN) and False positives (FP) are incorrect negative and positive predictions. True positives (TP) results corre- sponded to correct positive predictions, which are actual cor- rect predictions. The evaluation is based on the performance measures precision (P), recall (R) and F-score (F1). Recall denotes the percentage of correctly labelled positive results overall positive cases and is calculated as:

R = TP

TP+FN (21)

P = TP

TP+FP (22)

F1 = 2×P×R P+R

(23)

C. BASELINE We then compare the proposed method with other baseline models in terms of the F1-score. For this purpose, we imple- ment the following baseline models: • Naïve Bayes (NB) [35]: A basic multinomial Naïve

Bayes classifier based on probability theory.

• Random Forest (RF) [36]: A basic random forest classifier based on ensemble learning method. We built forests with 10, 100 and 500 trees.

• Support Vector Machine (SVM) [33]: A basic sup- port vector classifier based on hyperplane separator. We tested the parameters: Penalty (1, 10 and 100), ker- nel (linear and RBF) and gamma (0.001 and 0.0001).

• Logistic Regression (LR) [37]: A basic logistic regres- sion classifier based on statistic method. We tested the parameters: Regularizer (0.001, 0.01, 0.1, 1, 10, 100, 100) and penalty (l1 and l2).

• Convolutional Neural Network (CNN) [36]: Only convolutional neural networks are used. CNN models show the advantages of extracting complicated emotion features.

• Long-Short Term Memory (LSTM) [19]: A model used GRU or BiGRU. It utilizes the last hidden state for emotion recognition. The models show the advantages of learning contextual semantic knowledge.

• Gated Recurrent Unit (GRU) [20]: A model used GRU or BiGRU. It utilizes the last hidden state for emotion recognition. The models show the advantages of learning contextual semantic knowledge.

• RCNN [17]: A combination of CNN and LSTM. The LSTM layer is used before CNN layer.

• CNN+LSTM [57]: Another combination of CNN and LSTM. The CNN layer used before LSTM layer.

For machine learning methods, regarding representing texts, we used Bag-of-Words and Term Frequency-Inverse Document Frequency as the feature of each text. We used a grid search algorithm to find optimal parameters. And for deep learning based methods, regarding to representing texts, we used Word2Vec (3 billion 300 dimension word vectors), GloVe (840 billion 300 dimension word vectors), and Fast- Text (2 million 300 dimension word vectors) semantic word embeddings and EWE (10 thousand 300 dimension word vectors) emotion word embedding. For a fair comparison, we used the same hyperparameters settings [20] in default shown in Table 1.

TABLE 1. Hyperparameter setting.

D. EMOTION RECOGNITION DATASET In the experiment, we used the emotion-annotated datasets [54] which are from multiple domains (dialogues, tweets, fairy tales, blogs, and news headlines). We choose ten emotion recognition datasets to create our experimental data.

VOLUME 7, 2019 111871

E. Batbaatar et al.: SENN for Emotion Recognition From Text

TABLE 2. Datasets descriptions (D: DailyDialogs; C: CrowdFlower; T: TEC; TE: Tales-Emotions; I: ISEAR; E: EmoInt; ET: Electoral-Tweets; G: Grounded-Emotions; EC: Emotion-Cause; S: SSEC).

• Dailydialogs (D): The dataset which is built on con- versations. The annotation schema follows Ekman and non-emotional sentences [58].

• CrowdFlower (C): The twitter dataset published by CrowdFlower. The set of labels is non-standard (see details in [54]). The tweets are annotated via crowd- sourcing [59].

• TEC (T): The twitter dataset which is built on social media. The annotation schema corresponds to Ekman’s model of basic emotions [60].

• Tales-Emotions (TE): The tales dataset which is built on literature. The annotation schema consists of Ekman’s six basic emotions [42].

• ISEAR (I): The dataset which is built on collecting questionnaires answered by people with different cul- tural backgrounds. The labels are joy, fear, anger, sad- ness, disgust, shame, and guilt [61].

• EmoInt (E): The dataset which is built on social media content. The tweets are annotated via crowdsourcing with intensities of anger, joy, sadness, and fear [62].

• Electoral-Tweets (ET): The twitter dataset which tar- gets the domain of elections. The set of labels is non- standard (see details in [54]). The tweets are annotated via crowdsourcing [63].

• Grounded-Emotions (G): The dataset which is built on social media. The set of labels is happy and sad. The tweets are annotated by the authors [54].

• Emotion-Cause (EC): The dataset which is annotated both with emotions and their causes. The set of labels used for annotation consists of Ekman’s basic emotions to which shame is added [64].

• SSEC (S): The stance sentiment emotion dataset which is an annotation of the ‘‘SemEval 2016 Twitter’’ stance and sentiment dataset. It is annotated via expert anno- tation with multiple emotion labels per tweet following Plutchik’s fundamental emotions [65].

We only selected single label annotated sentences and removed the sentences with emotional intensity. We only selected Ekman’s six emotions including joy, fear, sadness, surprise, anger, and disgust. Table 2 shows the dataset infor- mation in detail.

E. DATA PREPROCESSING All user-generated data need to be preprocessed before clas- sification. Since text are written by the general public, there

is a possibility, indeed a reasonable probability, that a large number of casual words, abbreviations and short forms, spe- cial characters and spelling mistakes, are present in user- generated data. These add to the noise in the input data, which will be used for the classification in learning-based approaches. For reasons previously mentioned, it is important to clean user-generated data before classification. In this work, we clean the data without any subtasks like spelling mistakes, handling casual words, abbreviations and short forms. During preprocessing the following simple steps are followed for better performance of emotion recognition.

• All numbers and special character are removed. • All twitter IDs (starts with @ ) are removed. • All uppercase characters are changed into lowercase

characters.

F. HYPERPARAMETER AND TRAINING The hyperparameters in the proposed emotion recognition method include hidden layer size, number of layers, batch size, learning rate, dropout in BiLSTM and CNN. The hyper- parameters with the best classification effect of the model are studied. We used Adam optimizer to update parameters while training. And we used dropout and an early stopping strategy with patience 20 to avoid overfitting and early stopping moni- tored weighted F1-score on validation sets. The experimental dataset is randomly divided into 90:10 training set and testing set. We used 10% of the training set as a validation set.

V. EXPERIMENTAL RESULT In this section, we evaluate our approach and report empirical results. We compare the proposed SENN model with several baselines including traditional machine learning and deep learning methods on several emotion recognition datasets. The experimental results for emotion recognition are listed in Table 3. When comparing the performance of the three variants of SENN model, it is observed that FastText performs better than Word2Vec and GloVe word embeddings. One pos- sible reason is that the FastText word embedding can capture the meaning of shorter words and allows the embeddings to understand suffixes and prefixes. It works well with rare words and out-of-vocabulary words. Compared with traditional machine learning models,

we proved that deep learning based models outperformed the machine learning models as reported in previous studies.

111872 VOLUME 7, 2019

E. Batbaatar et al.: SENN for Emotion Recognition From Text

TABLE 3. Comparison with the baseline models. The best results are indicated in bold with (∗) and the second best result are indicated in only bold.

Logistic regression and support vector machine shows the comparative result using bag-of-word and tf-idf vectors.

Compared with the state-of-the-art models in emotion clas- sification, SENN gives the best performance on nine out of ten datasets except Tales-Emotion dataset. It performs F1-scores of 84.8%, 51.1%, 61.3%, 74.6%, 91.0%, 56.3%, 59.3%, 98.8% and 70.8% on real-world datasets. And CNN gives the best result on Tales-Emotion dataset using FastText word embedding.

The convolution based emotion encoder of SENN is simi- lar to the traditional CNN, but SENN emotion encoder only used the EWE emotion word embedding to extract emotion features. By concatenating semantic encoder, it improves the generalization of the model.

To the contrary, BiLSTM based semantic encoder of SENN is similar to the BiLSTM model. The SENN semantic encoder

work with semantic word vectors and then the output is concatenated with emotion encoded vectors. It also general- izes both view of semantic and emotional information well. Depends on the characteristic of the datasets, size of the datasets, and vocabulary, the other deep learning based base- line models shows the comparative results. From the results, we can see that bidirectional GRU and bidirectional LSTM models outperformed the GRU and LSTM models respec- tively. And the combined RCNN and CNN-LSTM models work better than the other single architectures.

As shown in Table 4, we compared the execution time of the proposed model and the baseline models to process the tasks. However, deep learning algorithms are slower than machine learning algorithms, the accuracy is higher as shown in Table 3. The models CNN, GRU, BiGRU, LSTM, and BiLSTM are single subnetwork models which consist of

VOLUME 7, 2019 111873

E. Batbaatar et al.: SENN for Emotion Recognition From Text

TABLE 4. Execution time comparison (seconds).

CNN or RNN subnetworks. Generally, the two subnetwork models are slower than single subnetwork models. The single LSTM and BiLSTM architectures execute the tasks slower than other compared models.

The proposed SENN model achieves the results in com- parative execution time with RCNN and CNN LSTM two subnetwork architectures. The CNN architecture efficiently works in case of execution time.

From the above results, we selected FastText word embed- ding (also EWE word embedding for SENN model) and well-balanced dataset ISEAR for evaluating the impact of parameters in the next sub-section.

A. IMPACT OF PARAMETERS We next evaluate the impacts of parameters of the proposed SENN model and compared with other deep learning based baseline models. In particular, we consider the following parameters: 1) the batch size; 2) the learning rate; 3) the dropout 4) the hidden size in RNN; 5) the number of layers in RNN; 6) the number of filters in CNN; 7) the filter sizes in CNN.

1) IMPACT OF BATCH SIZE We first investigate the impact of the batch size of the SENN model and other deep learning based models. The batch size is an important parameter that influences the dynamics of the learning algorithm. We compared the several different batch size between 50 and 250. The performance of the proposed SENN model is constant on different size of the batch as shown in Figure 4. LSTM and GRU models are highly sensitive to batch size. It shows the low performance when changing the batch size. On the contrary, bidirectional LSTM and GRU shows the constant result on all experiments. LSTM and GRU perform good result when batch size is low such as 4, 8 and 16. It is, however, very slow during training. Generally, the proposed SENN model achieved the highest results on different size of the batch. The default batch size in this paper is 128.

FIGURE 4. Relationship between batch size and F1 score.

FIGURE 5. Relationship between learning rate and F1 score.

2) IMPACT OF LEARNING RATE The appropriate choice of learning rate is important for the optimization of weights and offsets. If the learning rate is too large, it is easy to exceed the extreme point, making the sys- tem unstable. If the learning rate is too small, the training time is too long. Figure 5 shows the performance comparison of the proposed SENN model and other baseline models. SENN model achieved the highest result on a different configuration

111874 VOLUME 7, 2019

E. Batbaatar et al.: SENN for Emotion Recognition From Text

FIGURE 6. Relationship between dropout and F1 score.

of learning rate. When the learning rate is 0.01, the CNN model is performed better result than others. We considered the learning rates between values 0.002 and 0.01 for testing the impact of learning rate. LSTM and GRU models are also very sensitive to different learning rates. We set the default learning rate in this paper as 0.001.

3) IMPACT OF DROPOUT In order to prevent the over-fitting phenomenon in the train- ing process, the dropout mechanism was introduced, and our SENN model performs the best result when dropout is changed. Figure 6 is the relationship between dropout and F1 score. We tested the models with different size of dropout between 0.0 and 1.0. SENN model has two different dropouts for each sub-network, the one is at the end of CNN emotion encoder, and the other is at the end of BiLSTM semantic encoder. On the comparison, SENN model performs better result than other baseline models. We used the same value for both sub-networks to evaluate the impact of dropout. The default dropout in this paper is 0.5.

4) IMPACT OF HIDDEN SIZE IN RNN The number of hidden layer nodes influences the complexity and effect of the method. If the number of nodes is too small, the network learning ability will be limited. If the number of nodes is too large, the complexity of the network structure is large. At the same time, it is easier to fall into local minimum points during the training process, and the network learning speed will decrease. Figure 7 shows the comparison of models in case of a different number of hidden layer in RNN. We tested the different hidden sizes between 100 and 500. The proposed SENN model achieved a better result in all experiments. We set the default number of the hidden layer as 256.

5) IMPACT OF NUMBER OF LAYERS IN RNN It can be seen from Figure 8 that the proposed SENN model performs better results constantly compared with other base- line models. We tested the impact of a number of layers on

FIGURE 7. Relationship between the hidden size in RNN and F1 score.

FIGURE 8. Relationship between a number of layers in RNN and F1 score.

FIGURE 9. Relationship between a number of filters in CNN and F1 score.

different values between 1 and 5. We set the default hidden layer size in this paper as 2. LSTM and GRU models also perform low F1 score on all experiments.

6) IMPACT OF NUMBER OF FILTERS IN CNN We next investigate the impact of hidden layer size. In partic- ular, we first fix the hidden layer size between 100 and 500. Figure 9 shows the relationship between the number of fil- ters and F1 score. The experimental result shows that when increasing the number of filters, the proposed SENN model

VOLUME 7, 2019 111875

E. Batbaatar et al.: SENN for Emotion Recognition From Text

FIGURE 10. Relationship between filter sizes in CNN and F1 score.

performs the highest results. When a number of filters are 100 and 200, the combined CNN-LSTM performs higher result than the other models. We set the default number of filters in this paper as 100.

7) IMPACT OF FILTER SIZES IN CNN It can be seen from Figure 10 that when filter size is set of 4, 5, and 6, the SENN model achieved the highest F1 score of 75.3%. We tested the different set of filter sizes such as [1,2,3], [2,3,4], [3,4,5], [4,5,6] and [5,6,7]. The proposed SENN model performs a better result on all experiments. The default filter sizes used are set of 3, 4 and 5 in the compar- ison with other methods. The tradition CNN models give a more comparative result with SENN model than the other model.

In all comparison of models, RNN models such as LSTM and GRU is highly sensitive because of batch size on GPUs. Performance at lower batches is especially important when using data parallelism to distribute the computation across GPUs. When batch size is big enough, this will provide a stable enough estimate of average gradient of the full dataset.

VI. CONCLUSION In the era of the rapid development of social network and internet of things, it is very meaningful to explore the emo- tional recognition of user-generated data through artificial intelligence technology. This paper explored an emotion recognition method from text based on the combined network which consists of CNN based emotion encoder and BiLSTM based semantic encoder called SENN, a novel model is pro- posed and applied on ten real-world datasets. For the SENN model, BiLSTM is designed to capture contextual informa- tion and CNN is designed to extract emotional information effectively. In the experimental work, we have conducted the ten datasets and then analyzed using the proposed SENN model the other baseline models including state-of-the-art machine learning and deep learning models. Experiments on emotionally related datasets show that our method can

achieve better performances compared with state-of-the-art baseline methods. Our proposed framework is general enough to be applied to more scenarios. In the future works, we will extend proposed CNN based emotion encoding and BiLSTM based semantic encoding to other tasks such as affective computing and sentiment analysis. It is possible to improve the performance of SENN model by using the larger emotion word embeddings.

REFERENCES [1] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias,

W. Fellenz, and J. G. Taylor, ‘‘Emotion recognition in human-computer interaction,’’ IEEE Signal Process. Mag., vol. 18, no. 1, pp. 32–80, Jan. 2001.

[2] M. F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, ‘‘The first facial expression recognition and analysis challenge,’’ in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit. Workshops (FG), Mar. 2011, pp. 921–926.

[3] M. Karg, A.-A. Samadani, R. Gorbet, K. Kuhnlenz, J. Hoey, and D. Kulic, ‘‘Body movements for affective expression: A survey of auto- matic recognition and generation,’’ IEEE Trans. Affective Comput., vol. 4, no. 4, pp. 341–359, Oct. 2013.

[4] M. Soleymani, S. Koelstra, I. Patras, and T. Pun, ‘‘Continuous emotion detection in response to music videos,’’ in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit. Workshops (FG), Mar. 2011, pp. 803–808.

[5] F. Agrafioti, D. Hatzinakos, and A. K. Anderson, ‘‘ECG pattern analysis for emotion detection,’’ IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 102–115, Jan./Mar. 2012.

[6] P. Rodriguez, A. Ortigosa, and R. M. Carro, ‘‘Extracting emotions from texts in e-learning environments,’’ in Proc. 6th Int. Conf. Complex, Intell., Softw. Intensive Syst., Jul. 2012, pp. 887–892.

[7] A. Jamshidnejad and A. Jamshidined, ‘‘Facial emotion recognition for human computer interaction using a fuzzy model in the e-business,’’ in Proc. Innov. Technol. Intell. Syst. Ind. Appl. (CITISIA), Jul. 2009, pp. 202–204.

[8] S. Leek and G. Christodoulides, ‘‘A framework of brand value in B2B markets: The contributing role of functional and emotional components,’’ Ind. Marketing Manage., vol. 41, no. 1, pp. 106–114, Jan. 2012.

[9] P. Ekman, ‘‘An argument for basic emotions,’’ Cognition Emotion, vol. 6, nos. 3–4, pp. 169–200, May 1992.

[10] R. Plutchik, ‘‘A general psychoevolutionary theory of emotion,’’ Theories Emotion, vol. 1 no. 3, pp. 3–33, 1980.

[11] L. Zhang, S. Wang, and B. Liu, ‘‘Deep learning for sentiment analysis: A survey,’’ Wiley Interdiscipl. Rev., Data Mining Knowl. Discovery, vol. 8, no. 4, Jul./Aug. 2018, Art. no. e1253.

[12] Y. Hao, Y. Zhang, K. Liu, S. He, Z. Liu, H. Wu, and J. Zhao, ‘‘An end- to-end model for question answering over knowledge base with cross- attention combining global knowledge,’’ in Proc. 55th Annu. Meeting Assoc. Comput. Linguistics, Vol. 1, Jul. 2017, pp. 221–231.

[13] Y. Y. H. Ding Liu Luan and M. Sun, ‘‘Visualizing and understanding neural machine translation,’’ in Proc. 55th Annu. Meeting Assoc. Comput. Linguistics, Jul. 2017, pp. 1150–1159.

[14] T. Mikolov, I. C. K. Sutskever, G. S. Corrado, and J. Dean, ‘‘Distributed representations of words and phrases and their compositionality,’’ in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 3111–3119.

[15] J. Pennington, R. Socher, and C. Manning, ‘‘Glove: Global vectors for word representation,’’ in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2014, pp. 1532–1543.

[16] P. Xu, A. Madotto, C. S. Wu, J. H. Park, and P. Fung, ‘‘Emo2vec: Learning generalized emotion representation by multi-task training,’’ Sep. 2018, arXiv:1809.04505. [Online]. Available: https://arxiv.org/abs/1809.04505

[17] A. Agrawal, A. An, and M. Papagelis, ‘‘Learning emotion-enriched word representations,’’ in Proc. 27th Int. Conf. Comput. Linguistics, Aug. 2018, pp. 950–961.

[18] S. Buechel and U. Hahn, ‘‘Emotion representation mapping for auto- matic lexicon construction (mostly) performs on human level,’’ Jun. 2018, arXiv:1806.08890. [Online]. Available: https://arxiv.org/abs/1806.08890

111876 VOLUME 7, 2019

E. Batbaatar et al.: SENN for Emotion Recognition From Text

[19] B. Felbo, A. Mislove, A. Søgaard, I. Rahwan, and S. Lehmann, ‘‘Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm,’’ Aug. 2017, arXiv:1708.00524. [Online]. Available: https://arxiv.org/abs/1708.00524

[20] M. Abdul-Mageed and L. Ungar, ‘‘Emonet: Fine-grained emotion detec- tion with gated recurrent neural networks,’’ in Proc. 55th Annu. Meeting Assoc. Comput. Linguistics, vol. 4, Jul. 2017, pp. 718–728.

[21] N. Colneriĉ and J. Demsar, ‘‘Emotion recognition on Twitter: Comparative study and training a unison model,’’ IEEE Trans. Affective Comput., to be published.

[22] B. Kratzwald, S. Ilić, M. Kraus, S. Feuerriegel, and H. Prendinger, ‘‘Deep learning for affective computing: Text-based emotion recognition in deci- sion support,’’ Decis. Support Syst., vol. 115, pp. 24–35, Nov. 2018.

[23] S. M. Mohammad, ‘‘From once upon a time to happily ever after: Track- ing emotions in mail and books,’’ Decis. Support Syst., vol. 53, no. 4, pp. 730–741, Nov. 2012.

[24] S. M. Mohammad and P. D. Turney, ‘‘Crowdsourcing a word–emotion association lexicon,’’ Comput. Intell., vol. 29, no. 3, pp. 436–465, Aug. 2013.

[25] C. Strapparava and A. Valitutti, ‘‘Wordnet affect: An affective extension of WordNet,’’ in Proc. LREC, vol. 4, May 2004, p. 40.

[26] G. Miller, WordNet: An Electronic Lexical Database. Cambridge, MA, USA: MIT Press. 1998.

[27] J. Staiano and M. Guerini, ‘‘Depechemood: A lexicon for emotion analy- sis from crowd-annotated news,’’ May 2014, arXiv:1405.1605. [Online]. Available: https://arxiv.org/abs/1405.1605

[28] B. Liu and L. Zhang, ‘‘A survey of opinion mining and sentiment analysis,’’ in Mining Text Data. Boston, MA, USA: Springer, 2012, pp. 415–463.

[29] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, ‘‘Recursive deep models for semantic compositionality over a sentiment treebank,’’ in Proc. Conf. Empirical Methods Natural Lang. Process., Oct. 2013, pp. 1631–1642.

[30] J. W. Pennebaker, M. E. Francis, and R. J. Booth, ‘‘Linguistic inquiry and word count: LIWC 2001,’’ Mahway: Lawrence Erlbaum Associates, vol. 71, no. 2001, p. 2001. 2001.

[31] M. M. Bradley and P. J. Lang, ‘‘Affective norms for english words (ANEW): Instruction manual and affective ratings,’’Center Res. Psy- chophysiol., Univ. Florida, Gainesville, FL, USA, Tech. Rep. C-1, 1999. Vol. 30, no. 1, pp. 25–36.

[32] A. Agrawal and A. An, ‘‘Unsupervised emotion detection from text using semantic and syntactic relations,’’ in Proc. IEEE/WIC/ACMInt.JointConf. Web Intell. Intell. Agent Technol., vol. 1, Dec. 2012, pp. 346–353.

[33] T. Danisman and A. Alpkocak, ‘‘Feeler: Emotion classification of text using vector space model,’’ in Proc. AISB Conv. Commun., Interact. Social Intell., vol. 1, Apr. 2008, p. 53.

[34] S. Chaffar and D. Inkpen, ‘‘Using a heterogeneous dataset for emotion analysis in text,’’ in Proc. Canadian Conf. Artif. Intell. Berlin, Germany: Springer, 2011, pp. 62–67.

[35] M. Hasan, E. Rundensteiner, and E. Agu, ‘‘Emotex: Detecting emotions in Twitter messages,’’ ASE BIGDATA/SOCIALCOM/CYBERSECURITY Conf., Stanford, CA, USA, Tech. Rep., 2014.

[36] D. Gordeev, ‘‘Detecting state of aggression in sentences using CNN,’’ in Proc. Int. Conf. Speech Comput. Cham, Switzerland: Springer, 2016, pp. 240–245.

[37] A. Seyeditabari, S. Levens, C. D. Maestas, S. Shaikh, J. I. Walsh, W. Zadrozny, C. Danis, and O. P. Thompson, ‘‘Cross corpus emotion classification using survey data,’’ presented at the AISB, 2017.

[38] D. Chatzakou, A. Vakali, and K. Kafetsios, ‘‘Detecting variation of emo- tions in online activities,’’ Expert Syst. Appl., vol. 89, pp. 318–332, Dec. 2017.

[39] M. Purver and S. Battersby, ‘‘Experimenting with distant supervision for emotion classification,’’ in Proc. 13th Conf. Eur. Chapter Assoc. Comput. Linguistics, Apr. 2012, pp. 482–491.

[40] S. Wen and X. Wan, ‘‘Emotion classification in microblog texts using class sequential rules,’’ in Proc. 28th AAAI Conf. Artif. Intell., Jun. 2014, pp. 187–193.

[41] A. Bandhakavi, N. Wiratunga, D. Padmanabhan, and S. Massie, ‘‘Lexicon based feature extraction for emotion text classification,’’ Pattern Recognit. Lett., vol. 93, pp. 133–142, Jul. 2017.

[42] C. O. Alm, D. Roth, and R. Sproat, ‘‘Emotions from text: Machine learning for text-based emotion prediction,’’ in Proc. Conf. Hum. Lang. Technol. Empirical Methods Natural Lang. Process. Oct. 2005, pp. 579–586.

[43] C. Strapparava and R. Mihalcea, ‘‘Semeval-2007 task 14: Affective text,’’ in Proc. 4th Int. Workshop Semantic Eval. (SemEva), Jun. 2007, pp. 70–74.

[44] B. Schölkopf, J. Weston, E. Eskin, C. Leslie, and W. S. Noble, ‘‘A kernel approach for learning from almost orthogonal patterns,’’ in Proc.Eur.Conf. Mach. Learn. Berlin, Germany: Springer, 2002, pp. 511–528.

[45] D. Greene and P. Cunningham, ‘‘Practical solutions to the problem of diagonal dominance in kernel document clustering,’’ in Proc. 23rd Int. Conf. Mach. Learn., Jun. 2006, pp. 377–384.

[46] U. Gupta, A. Chatterjee, R. Srikanth, and P. Agrawal, ‘‘A sentiment- and-semantics-based approach for emotion detection in textual conversa- tions,’’ Jul. 2017, arXiv:1707.06996. [Online]. Available: https://arxiv.org/ abs/1707.06996

[47] H. Ren, J. Wan, and Y. Ren, ‘‘Emotion detection in cross-lingual text based on bidirectional LSTM,’’ in Proc. Int. Conf. Secur. Intell. Comput. Big- Data Services. Cham, Switzerland: Springer, 2018, pp. 838–845.

[48] A. Illendula and A. Sheth, ‘‘Multimodal emotion classification,’’ Mar. 2019, arXiv:1903.12520. [Online]. Available: https://arxiv. org/abs/1903.12520

[49] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, ‘‘Enriching word vectors with subword information,’’ Trans. Assoc. Comput. Linguistics, vol. 5, pp. 135–146, Dec. 2017.

[50] T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Černocký, ‘‘Empirical evaluation and combination of advanced language modeling techniques,’’ in Proc. 12th Annu. Conf. Int. Speech Commun. Assoc., Aug. 2011, pp. 27–31.

[51] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, ‘‘Deep contextualized word representations,’’ Feb. 2018, arXiv:1802.05365. [Online]. Available: https://arxiv.org/abs/ 1802.05365

[52] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, ‘‘Bert: Pre-training of deep bidirectional transformers for language understanding,’’ Oct. 2018, arXiv:1810.04805. [Online]. Available: https://arxiv.org/abs/1810.04805

[53] P. Zhong and C. Miao, ‘‘Ntuer at SemEval-2019 Task 3: Emotion clas- sification with word and sentence representations in RCNN,’’ Feb. 2019, arXiv:1902.07867. [Online]. Available: https://arxiv.org/abs/1902.07867

[54] L.-A.-M. Bostan and R. Klinger, ‘‘An analysis of annotated corpora for emotion classification in text,’’ in Proc. 27th Int. Conf. Comput. Linguis- tics, Aug. 2018, pp. 2104–2119.

[55] S. Hochreiter and J. Schmidhuber, ‘‘Long short-term memory,’’ Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.

[56] Kim, Y., ‘‘Convolutional neural networks for sentence classifica- tion,’’ Aug. 2014, arXiv:1408.5882. [Online]. Available: https://arxiv. org/abs/1408.5882

[57] C. Zhou, C. Sun, Z. Liu, and F. Lau, ‘‘A C-LSTM neural network for text classification,’’ Nov. 2015, arXiv:1511.08630. [Online]. Available: https://arxiv.org/abs/1511.08630

[58] Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu, ‘‘Dailydialog: A man- ually labelled multi-turn dialogue dataset,’’ Oct. 2017, arXiv:1710.03957. [Online]. Available: https://arxiv.org/abs/1710.03957

[59] S. Mohammad and S. Kiritchenko, ‘‘Understanding emotions: A dataset of tweets to study interactions between affect categories,’’ in Proc. 11th Int. Conf. Lang. Resour. Eval. (LREC), 2018, pp. 198–209.

[60] E. Agirre, J. Bos, S. Manandhar, Y. Marton, and D. Yuret, ‘‘SEM 2012: The first joint conference on lexical and computational semantics,’’ in Proc. Main Conference Shared Task, 6th International Workshop Semantic Evaluation (SemEval 2012), vols. 1–2, 2012, pp. 1–32.

[61] K. R. Scherer and H. G. Wallbott, ‘‘Evidence for universality and cultural variation of differential emotion response patterning,’’ J.PersonalitySocial Psychol., vol. 66, no. 2, pp. 310–328, 1994.

[62] S. M. Mohammad and F. Bravo-Marquez, ‘‘Emotion intensities in tweets,’’ Aug. 2017, arXiv:1708.03696. [Online]. Available: https://arxiv. org/abs/1708.03696

[63] S. M. Mohammad and S. Kiritchenko, ‘‘Using hashtags to capture fine emotion categories from tweets,’’ Comput. Intell., vol. 31, no. 2, pp. 301–326 May 2015.

[64] A. Gelbukh, ‘‘Computational linguistics and intelligent text process- ing,’’ in Proc. 16th Int. Conf., vol. 9042. Cairo, Egypt, Springer, 2015, pp. 14–20.

[65] H. Schuff, J. Barnes, J. Mohme, S. Padó, and R. Klinger, ‘‘Annotation, modelling and analysis of fine-grained emotions on a stance and sentiment detection corpus,’’ in Proc. 8th Workshop Comput. Approaches Subjectiv- ity, Sentiment Social Media Anal., Sep. 2017, pp. 13–23.

VOLUME 7, 2019 111877

E. Batbaatar et al.: SENN for Emotion Recognition From Text

ERDENEBILEG BATBAATAR received the B.S. degree in software engineering from the National University of Mongolia, in 2013, and the M.S. and Ph.D. degrees in computer science from Chungbuk National University, Cheongju, South Korea, in 2019, where he is currently holds a Postdoctoral position. His research interests include software engineering, data mining, big data analysis, bioin- formatics, machine learning, and deep learning.

MEIJING LI received the B.S. degree in com- puter science from Dalian University, Dalian, China, in 2007, and the M.S. degree in bio information technology and the Ph.D. degree in computer science from Chungbuk National Uni- versity, Cheongju, South Korea, in 2010 and 2015, respectively, where she held a Postdoctoral posi- tion. She is currently an Assistant Professor with the College of Information Engineering, Shanghai Maritime University, Shanghai, China.

Her research interests include data mining, information retrieval, database systems, bioinformatics, and biomedicine.

KEUN HO RYU (LM’83) received the Ph.D. degree in computer science and engineering from Yonsei University, South Korea, in 1988. He has served at the Reserve Officers’ Training Corps (ROTC) of the Korean Army. He was with The University of Arizona, Tucson, AZ, USA, as a Postdoctoral and a Research Scientist, and also with the Electronics and Telecommunica- tions Research Institute, South Korea, as a Senior Researcher. He is currently a Professor with the

Faculty of Information Technology, Ton Duc Thang University, Vietnam, as well as an Emeritus and Invited Professor with Chungbuk National Univer- sity, South Korea. He is also an Honorary Doctorate of the National Univer- sity of Mongolia. He has been the Leader of the Database and Bioinformatics Laboratory, South Korea, since 1986. He is the former Vice-President of the Personalized Tumor Engineering Research Center.

He has published or presented over 1000 referred technical articles in vari- ous journals and international conferences, in addition to authoring a number of books. His research interests include temporal databases, spatiotemporal databases, temporal GIS, stream data processing, knowledge-based infor- mation retrieval, data mining, biomedical informatics, and bioinformatics. He has been a member of the ACM, since 1983. He has served on numerous program committees, including roles as the Demonstration Co-Chair of the VLDB, as the Panel and Tutorial Co-Chair of the APWeb, and as the FITAT General Co-Chair.

111878 VOLUME 7, 2019