helpfn

SPECIAL SECTION ON AI-DRIVEN BIG DATA PROCESSING: THEORY, METHODOLOGY, AND APPLICATIONS

Received December 16, 2018, accepted January 18, 2019, date of publication January 29, 2019, date of current version February 14, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2895897

Time+User Dual Attention Based Sentiment Prediction for Multiple Social Network Texts With Time Series LEI LI , (Member, IEEE), YABIN WU, YUWEI ZHANG, AND TIANYUAN ZHAO Department of Intelligence Science and Technology, School of Computer, Beijing University of Posts and Telecommunications, Beijing 100876, China

Corresponding author: Lei Li ([email protected])

This work was supported in part by the Beijing Municipal Commission of Science and Technology under Grant Z181100001018035, in part by the National Social Science Foundation of China under Grant 16ZDA055, in part by the National Natural Science Foundation of China under Grant 91546121, in part by the EU FP7 IRSES MobileCloud Project under Grant 612212, in part by the 111 Project of China under Grant B08004, in part by the Engineering Research Center of Information Networks, Ministry of Education, in part by the Beijing BUPT Information Networks Industry Institute Company Ltd., in part by the Beijing Institute of Science and Technology Information, and in part by CapInfo Company Ltd.

ABSTRACT In today’s information age, the development of hot events is timely and rapid under the influence of the powerful Internet. Online social media, such as Weibo in China, has played an important role in the process of spreading public opinions and events. Sentiment analysis of social network texts can effectively reflect the development and changes of public opinions. At the same time, prediction and judgment of public opinion development can also play a key role in assisting decision-making and effective management. Therefore, sentiment analysis for hot events in online social media texts and judgment of public opinion development have become popular topics in recent years. At present, research on textual sentiment analysis is mainly aimed at a single text, and there is little-integrated analysis of multi-user and multi- document in unit time for time series. Moreover, most of the existing methods are focused on the information mined from the text itself, while the feature of identity differences and time sequence of different users and texts on social platforms are rarely studied. Hence, this paper works on the public opinion texts about some specific events on social network platforms and combines the textual information with sentiment time series to achieve multi-document sentiment prediction. Considering the related features of different social user identities and time series, we propose and implement an effective time+user dual attention mechanism model to analyze and predict the textual information of public opinion. The effectiveness of the proposed model is then verified through experiments on real data from a popular Chinese microblog platform called Sina Weibo.

INDEX TERMS Public opinion, sentiment prediction, time series, time + user dual attention mechanism.

I. INTRODUCTION With the rapid development of Internet, especially the widespread popularity of the mobile Internet, the number of Internet users has exploded. At the same time, new media based on Internet and social network platforms have become an indispensable part of people’s lives, gradually replacing traditional media as the mainstream of media form [1]. Espe- cially in recent years, Weibo has been rapidly developed and popularized in China. With its platform’s openness, terminal

The associate editor coordinating the review of this manuscript and approving it for publication was Jad Nasreddine.

scalability, content simplicity and low threshold, Sina Weibo (https://en.wikipedia.org/wiki/Sina_Weibo) has rapidly wide spread among the netizens and developed into an important social media platform [2].

More and more media and public release news or opin- ions on online social network, making them an important platform for public opinion development. In a series of hot events in recent years, network public opinion triggered by network platforms such as Weibo has played an increasingly important role. It has broken through the limitations of tradi- tional media, which is spreading faster, affecting wider and interacting in a much larger scope. Therefore, it is of great

17644 2169-3536 2019 IEEE. Translations and content mining are permitted for academic research only.

Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

VOLUME 7, 2019

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

significance to study the public opinion information on social media. Through sentiment analysis and data mining of pub- lic opinion information on network platform, we can better grasp the development trend of public opinion and provide better decision-making and scientific supervision assistance to users and relevant departments.

As an important research branch in natural language pro- cessing, sentiment analysis is of great significance in the study of public opinion information. With its help, we can better judge the public opinion tendency towards hot events in social network, thus judge the development trend of public opinion. It also plays a guiding role in the processing and development control of public opinion, which is of great social significance. Meanwhile, the public opinion is chang- ing with time. We can grasp the development trend of public opinion more accurately by analyzing the trajectory of senti- ment changes in time series.

Time series refers to sequence data in which the observa- tion data of a certain index of a system is arranged in chrono- logical order at various time points. This widely existing time series data often contains potential variation laws. Time series analysis is to explore all the information contained in the time series data, observe, estimate and study the statistical regularity of such a set of real data in the long-term change process [3]. Combining time series with sentiment analysis to construct sentiment time series can better reflect the law of sentiment development over time, and more accurately describe the development trend of public opinion.

In summary, this paper combines the sentiment analysis of social media information with time series to construct emotional time series, and analyzes sentiment situation based on Chinese text set of public opinion. Through experiments on real Chinese social media data (from July 8, 2014 to March 22, 2017), this paper mainly conducts the following three researches:

1. This paper explores the combination of time series and sentiment analysis, designs and implements a judgment model of public opinion based on social network data, and carries out experimental verification for hot events of Chinese P2P (Peer-to-Peer) network lending agencies.

2. Different from the previous research on sentiment anal- ysis of single text, this paper puts forward and implements a sentiment feature fusion method for multi-document col- lection, which can analyze and judge the overall sentiment tendency of multiple documents in a unit time period.

3. Based on the sentiment classification of the basic model, this paper proposes and implements a time series sentiment analysis model based on time-attention and user-attention, time+user dual attention mechanism to analyze the textual information and predict public opinion for hot events on social network, and its effectiveness is then verified through experiments on real data collected from Chinese Sina Weibo.

II. RELATED WORK In recent years, research on public opinion analysis of network platforms has become a hot topic, and the

applications of sentiment analysis and time series are also important research directions. Sentiment analysis of public opinion information in social networks can help people better grasp the trend of public opinion and development, which is of great significance to the monitoring and scientific control for public opinion.

Liu et al. [4] explore the possibility of Renren (http://www.renren.com/) SNS (Social Networking Services) being used for Chinese public opinion analysis in colleges and universities. By using the state, log, video as factors, and the sensitivity weight coefficient calculated by a large number of samples, they construct a public opinion quantification and judgment system for short comments/posts to judge the cur- rent degree and future trend of the public opinion and propose a public opinion response strategy in the social network era. Xian-Yi et al. [5] establish a framework for Chinese network public opinion monitoring and analysis based on semantic content recognition, which is used for online comments with short length and many emotional words. Wang [6] propose a Chinese network public opinion supervision and prediction algorithm based on semantic features of big data. The binary semantic information expression method is used to construct and match the topic vocabulary of network public opinion, and the prediction algorithm is improved by combining time series. Ma et al. [7] propose a new method for Chinese public opinion analysis of social media sites in the online social activities of the public participation in the Web2.0 era, using the Latent Dirichlet Allocation (LDA) topic model to extract the public opinions on different topics of certain events. Then they use deep learning model based on word2vec to calculate the sentiment polarity and intensity of the short text, and build a model for tracking based on time series. Shen and Xia [8] select 30 hot Chinese tourism emergencies from 2010 to 2014, combined with microblog public opinion short texts, then cluster using Self-Organizing Maps (SOM) neural net- work, and use the fitting method of exponential function. The dissemination situation of Internet public opinion is divided into six categories, and their differences in communication characteristics are analyzed.

Hutto and Gilbert [9] build a simple rule-based model for English short text sentiment classification. Tripathy et al. [10] study four different algorithms: Naive Bayes (NB), Maximum Entropy (ME), Stochastic Gradient Descent (SGD), and Support Vector Machine (SVM) to implement emotional classification of long comments on English social networks. Lin et al. [11] combine traditional text feature based sentiment classification with personality- based sentiment classification to explore the application of user personality in Chinese social media short text senti- ment classification. Lee et al. [12] propose a weak supervised learning method based on CNN and a method to identify the keywords of positive and negative sentences for English and Korean long comment texts. This model can not only cor- rectly classify the polarity of sentences, but also successfully identify corresponding words with high polarity scores. In the short text sentiment analysis of Chinese Weibo, Li and Ji [13]

VOLUME 7, 2019 17645

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

use a variety of text features, including words, parts of speech (POS), sentiment words, negative words, degree adverbs and special symbols, and select different combinations of features to examine their relationships with different machine models. It is found that the precision of SVM model can reach 88.72% when using the combination of POS, sentiment words and negative words. Benamara et al. [14] propose a long text sentiment analysis technique based on adverb-adjective com- bination, and through experiments in English news articles, it is proved that the classification result of this combination method is better than the method of only using adjectives. Aiming at the characteristics of short text with less context information, deep Convolutional Neural Network(CNN) is used by dos Santos and Gatti [15] to analyze sentiment of English short texts, and has achieved good results on different data sets. Nguyen et al. [16] focus on the overall English public sentiment for short texts with time changes, rather than the sentiment analysis of a single Weibo content. Then a public opinion analysis model is established by learning the parameters such as time window, reaction time and duration. Ambiguity and irony in the text are often hard to be accurately identified. There may be errors in the judgment of sentiment in some individual Weibo, but this article is focused on the overall trend of large data. Under such conditions, the impact on the sentiment judgment of a single Weibo will be reduced to a certain extent or even neglected.

From the above, we can see that online public opinion judgment, text sentiment analysis and time series applica- tion are all very important research fields. Although the predecessors have done a lot of related work, most of the existing researches use sentiment classification for individual text without the fusion analysis of the overall collection of multiple documents. In addition, more attention has been paid to the information extracted from the text, while less atten- tion has been paid to the multiple features associated with the text in social network scenarios, especially the different users and their identity differences, publishing time and their sequence. Therefore, in this paper we work on the public opinion texts of the social network platforms, combine these textual information with the sentiment time series and focus on the multi-document sentiment fusion analysis created by different users (also called user-generated content, with size range of 0-140 Chinese characters) in the same time period of the same topic. And considering the influence of different user identity, time series and other textual association fea- tures, we propose and implement an effective time+user dual attention mechanism model to analyze the sentiment situation of public opinion.

III. SYSTEM DESIGN The research in this paper is based on the experimental system framework shown in Figure 1.

A. RAW DATA AND PREPROCESSING We use the public opinion data of the well-known P2P related malignant events in recent years collected from Sina Weibo

FIGURE 1. Framework of system.

for experiments in this paper. P2P lending platform is a new format of financial model developed by the financial indus- try under the Internet environment. It has quickly formed a financing lending platform with low threshold, low cost and wide coverage, and has become an important form of financial reform in China. At the same time, the problems of P2P companies themselves are increasingly exposed, and even a number of malignant events, such as the Ezubao (https://en.wikipedia.org/wiki/Ezubao, a wholly-owned sub- sidiary of Yucheng Group, https://baike.baidu.com/item/ %E9%92%B0%E8%AF%9A%E7%B3%BB/19328549?fr= aladdin) incident, which have brought serious harm to China’s social economy and people’s lives. Therefore, effec- tive supervision for P2P companies has become an extremely urgent task. This paper is to study how to analyze the public opinion texts related to P2P malignant events in Sina Weibo for prediction of public opinion sentiment. We verify our pro- posed model through experiments on these real data, so that we can effectively provide useful assistance and support for risk supervision of relevant financial enterprises in the future.

Specifically, the raw data we collected mainly include 11913 microblog textual contents with related informa- tion fields (such as microblog ID, author, publishing time, number of fans and user category) about three enterprises related to the typical P2P malignant events, i.e., Yucheng Group (Yucheng), Kuailu Group (Kuailu, https://baike.baidu.com/item/%E4%B8%8A%E6%B5%B7% E5%BF%AB%E9%B9%BF%E6%8A%95%E8%B5%84% EF%BC%88%E9%9B%86%E5%9B%A2%EF%BC%89% E6%9C%89%E9%99%90%E5%85%AC%E5%8F%B8/151 40911?fr=aladdin) and Zhongjin Group (Zhongjin, https:// baike.baidu.com/item/%E4%B8%AD%E6%99%8B%E7% B3%BB/19501898?fr=aladdin). We preprocess these raw data for subsequent processing, including cleaning, removing invalid and duplicate data, retaining the required informa- tion fields, and sorting the data according to time. Finally, in our preprocessed data, Yucheng contains 574 microblogs, the time span is from July 8, 2014 to February 1, 2017; Kuailu contains 1079 ones, the time span is from March 9, 2016 to March 14, 2017; Zhongjin contains 5784 ones, the time span is from March 2, 2015 to March 22, 2017. We have used all these data in our experiments. One microblog message sample about Kuailu is shown in Table 1.

17646 VOLUME 7, 2019

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

TABLE 1. One microblog message sample about Kuailu.

B. FEATURE EXTRACTION This paper draws on a method with good results proposed by Li and Ji [13]. Based on the method of POS and senti- ment score, we select five types of Weibo textual features, which are POS, sentiment words, negative words, degree adverbs and special punctuations, to form a 12-dimension vector as the feature representation. Among them, the first to the twelfth respectively represent: the number of adjectives, verbs, nouns, adverbs, prepositions, interjections, positive sentiment words, negative sentiment words, the sentiment score, whether negative words and degree adverbs appearing before the sentiment word, the number of question marks and exclamation marks. In this paper, the basic sentiment scores of positive and negative sentiment words are set to 1 and−1 respectively, and the different degrees of adverbs are placed in different weights of 0.5, 1.0, 1.5, and 2.0 [13]. The weight of negative words is set to−1. Then the weights of the negative words and the degree adverbs appearing before the different sentiment words are multiplied by the basic scores of the sentiment words. Finally all the sentiment scores of emotional words are summed up to obtain the overall senti- ment score of a microblog. The sentiment score calculation formula is as follows:

Scoresentiment = N∑ i=1

ni ·wi · si

where si indicates the polarity of an emotional word, positive and negative sentiment words are set to 1 and−1 respectively, neutral words are not covered in calculation; wi indicates the weight of sentiment adverbs before the current sentiment word, there are four kinds, namely 0.5, 1.0, 1.5, 2.0; ni indicates whether there is a negative word before the current sentiment word, and then is set to −1, otherwise set to 1; N is the number of emotional words in a Weibo text.

C. TIME SERIES Based on the sentiment features extracted above, we select the appropriate unit time and time window through experiments to construct sentiment time series for prediction. Assuming that the time window is T , it means that we use T unit times of Weibo data to construct a time series, and use it to predict the sentiment situation of the T + 1 unit time, which represents the results of public opinion classification

for P2P events in this experiment. In one unit time, if all users only publish one Weibo text, then its 12-dimension vector is directly used as the public opinion sentiment feature vector of this unit time. Else if multiple users publish multiple Weibo texts, then their 12-dimension vectors are added together as the public opinion sentiment feature vector of the unit time. The intuition is that, in the case of multiple documents per unit time, we believe that all documents have an influence on emotional prediction, hence we choose to add them as a whole multi-document feature. But using average of vectors is more like choosing a single document, which cannot reflect the characteristics of multi-document very well. And if there is no Weibo text published in the unit time, then all the public opinion sentiment features are set to zero.

D. MODEL This is the key part of this paper. Based on the existing public opinion sentiment data, we hope to propose and implement an efficient public opinion prediction model of Weibo texts. As can be seen from the above content, deep learning has been more and more applied to sentiment analysis and time series in recent years, showing good results, thus we will also focus on the application of deep learning models.

RNN (Recurrent Neural Network) is a popular deep learn- ing model in recent years. Compared with the traditional neural network, RNN has added a loop structure to maintain the persistent memory of information. Therefore, RNN is a deep learning model for sequence data. LSTM is a variant based on RNN that adds forget gate, input gate and output gate to each neuron. The forget gate is used to selectively retain the previous information, to keep the cell state retaining part of the information and forgetting part of the information; the input gate is used to determine which information needs to be newly added to the cell state; while the output gate is used to determine the output information and generate new cell state. It is through the control of the ‘‘gates’’ inside the neurons that LSTM enhances the advantages of memorizing sequence information, and maintains the information persis- tence. At the same time, it avoids the gradient explosion and dispersion in RNN, and the dependency issues of long-term memory information. At the same time, in order to com- pare the experimental results of different algorithms, we also added SVM and CNN to conduct comparative experiments and investigate the effectiveness of our proposed model.

As to LSTM, the attention mechanism has been proved to be very useful in some applications. For instance, Bahdanau et al. [17] introduce the attention mechanism in the seq-to-seq machine translation model, which effectively solves the problem of serious information loss when the sentence is too long, and improves the translation effect. Here the attention mechanism can be understood as the expression of ‘‘word alignment’’. By increasing the attention during each translation, the model can find out which words in the input text are more closely related to the current translation, so as to avoid the impact of some interference words. Enlightened by this idea, we propose a time+user dual attention mechanism,

VOLUME 7, 2019 17647

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

FIGURE 2. Time Attention.

which combines the time characteristics and different user identities of social network texts such as Weibo.

First, we design and implement a time-attention mecha- nism in time series, as shown in Figure 2. We input a time series, each unit time’s text set of Weibo contains different characteristics, the importance of each unit time and its influ- ence to the public opinion prediction result are also different. By adding time-attention, we may make the sequence better grasp the impact of different time points, so as to improve the prediction effect. Yet our time-attention mechanism is different from the one in the usual seq-to-seq translation model. The latter implements the ‘‘word alignment’’ function from the encoder side to the decoder side, finding the words in the input sequence that are more closely related to the current translated words. However, in our time-attention mechanism shown in Figure 2, instead of directly using the encoder structure, the commonly used LSTM model is selected to perform high-dimension feature extraction on the sequence data. At the same time, we do not directly output the result on the decoder side, which is the y(y1−yn) in the figure. According to the value h(h1−hn) of the hidden layer output, the weighting factor of input for different unit time is obtained through the intermediate output result with the LSTM. After the weighting operation of weighting factor, the output result Yt(Yt1−Ytn) of the time-attention layer is finally obtained. Formula: Ot(Ot1−Otn) is the encoder’s LSTM output.

State in decoder is the encoder’s LSTM hidden layer state, and h(h1−hn) is the output of the decoder hidden layer.

Wt = softmax (h×Ot)

Yt = Ot ×Wt

Next, we design and implement a user-attention mechanism for different user identities. Here, we refer to the work of Chen et al. [18], which introduces the attention mechanism of user and product when making sentiment classification for user commodity evaluations, so as to augment the model

with measurement of the relationship between users and commodities. The two-layer LSTM acts on the word repre- sentation and sentence representation, and adds user-product attention to each layer to enhance the effect of sentiment classification. According to our analysis of the social text characteristics of Weibo, we also should consider the char- acteristics related to the user identity. For the development of social network public opinion, the influences of different user identities are also different. The influences of media and individuals are different, and the influences of ‘‘big V’’(https://baike.baidu.com/item/%E5%A4%A7%E2%85% A4/9663742?fr=aladdin) and ordinary users are different. Therefore, we propose the user-attention mechanism as shown in Figure 3. We add user’s influencing factors to the model to examine its influence on the prediction results. Please note that the user-attention we propose is different from the work of Chen et al. [18]. The vector represents user and product in the latter, and it is applied to the word representation and sentence representation of the text. While in our user-attention model, we design the representation of user attribute matrix which is represented by ‘‘User’’ in Figure 3, including user category, big ‘‘V’’ category and number of fans. ‘‘user category’’ refers to the user iden- tities, 1 means ordinary users, 2 means network media, and 3 means official media. ‘‘big V’’ means whether the user has a ‘‘V’’ tag, 1 means yes, 0 means no. ‘‘number of fans’’ refers to the number of fans of a user. Thus this matrix fully combines the user characteristics of social net- work platforms like Weibo. At the same time, our user- attention considers the different user roles corresponding to different time points in each input, and realizes the fusion of multiple documents of microblog messages (one document refers to one microblog message) and multiple users. We apply it to the input selected by time-attention, and integrate different user features of multiple dimensions into the model features to improve the effect of sentiment analysis.

17648 VOLUME 7, 2019

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

FIGURE 3. User attention.

FIGURE 4. Model structure.

Formula: Ou(Ou1−Oun) is the hidden layer result of the LSTM output. ‘‘U’’ is the user feature matrix obtained according to the corresponding user, and is composed of information such as the user’s identity category, the category of the big ‘‘V’’, and the number of fans.

E = Ou ×W1 +U ×W2 Yu = sigmoid (E ×Ou)

We will discuss the above two attention mechanisms and their fusion effects in later experiments. The overall model structure is shown in Figure 4.

E. MULTI-GRANULARITY SCANNING In order to further enhance the effect of predictions, we hope to obtain better feature representation besides the aforemen- tioned sentiment features via deeply mining the existing data with multi-granularity scanning, which is inspired by the work of Zhou and Feng [19]. They propose multi-granularity scanning of sample features to solve the problem of requiring a lot of training data and parameters in deep learning for the

task of classification. In fact, prediction is a specific kind of classification. Hence, we also try to adopt this method into the input sequence feature representation in our work.

IV. EXPERIMENTAL DESIGN AND RESULT ANALYSIS A. EXPERIMENTAL DATA According to the distribution characteristics of the real pub- lic opinion data we collected from Weibo, Yucheng and Zhongjin’s public opinion are more explosive, mostly con- centrated in a few days, almost no data in other time periods. The development of Kuailu’s public opinion lasts longer and is more stable. Therefore, we mainly use the outbreak period’s data for Yucheng and Zhongjin in the following experiments. For Yucheng, we choose the public opinion data from Weibo in 3 days (January 31, 2016 - February 2, 2016). And for Zhongjin, we choose the data in 10 days (April 6, 2016 - April 15, 2016). As for Kuailu’s data, it doesn’t erupt in a few days and is stable over a long period of time. Therefore, we choose all the public opinion data from Weibo in March 31, 2016 - September 20, 2016.

In the following experiments, we choose the LSTM model as the baseline system. And we measure the experimental results using the F1 score as the performance index in all experiments. We have carried out four sets of experiments, their methods and result analyses are detailed below.

B. EXPERIMENT 1: PARAMETER SELECTION To build a time series, we first need to decide the time unit of the data. For the three groups’ data, experiments are performed with the baseline system using four different time units, i.e., one hour, half an hour, ten minutes and five minutes, to find the best one for later experiments. Less minutes as a unit will result in too many empty samples, which will influence the result greatly. And we also need to determine the size of the time window. That is to determine how many time units we need to predict the outcome of the next time unit. Based on our experience, we do experiments with three different time windows, which are 5, 7 and 10 to determine the best time window. For the characteristics of multiple Weibo texts per unit time, we add the emotional features of each text, in order to achieve the overall emotional tendency judgment of multiple texts per unit time.

We first select Yucheng’s three-day data for experiments using one hour as the unit time. Table 2 shows the experimen- tal results with 3 different time windows, where ‘‘−1’’ means negative emotion, and ‘‘0’’ means neutral emotion, similarly hereinafter. It should be noted that, since the experimental data are mainly about P2P malignant events, most of the emotional tendencies of Weibo data are negative and neutral, and there is almost no positive Weibo.

From the results we can see that when choosing one hour as the unit time, the number of samples is too small, and it is difficult to balance between different categories. None of the samples in category 0 can be effectively predicted. Thus, one hour is not suitable for the experiments.

VOLUME 7, 2019 17649

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

TABLE 2. The results for one hour as a unit.

TABLE 3. The results for half an hour as a unit.

TABLE 4. The results for ten minutes as a unit.

TABLE 5. The results for five minutes as a unit.

Similarly, we choose half an hour as the unit time and the results are shown in Table 3.

As we can see, when we choose half an hour as the unit time, it also has the problem of a small sample size, and the experimental results are not satisfied.

Next, we choose ten minutes as the unit time, and the results are shown in Table 4.

Compared with the previous experiments, the results with ten minutes have been improved. But the smaller −1 class also has some unpredictable situations.

Finally, we choose five minutes as the unit time, and the results are shown in Table 5.

As can be seen from all the above experimental results, the performance with 5 minutes as the unit time are improved, which is better than those of the previous three groups. And, when we select 10 as the time window, we can obtain the best experimental results. Both classes can be effectively predicted, and the F1 value reaches the highest. Therefore, in the following experiments, we use the same parameters for Yucheng’s data.

TABLE 6. Baseline experiment results.

TABLE 7. SVM’s results for Kuailu.

Similarly, we continue to determine the parameters for the data of Zhongjin and Kuailu. Both are selected to use 5 minutes as the unit time, and 10 as the time window for their following experiments.

C. EXPERIMENT 2: BASELINE AND COMPARISON SYSTEMS Considering the characteristics of the sequence data, we choose LSTM as the baseline for our experiments. SVM and CNN are also used for comparison experiments. On this basis, we also try to use the two-layer LSTM model for experiments to see if it can further extract the sequence features and have a certain effect on the prediction results. At the same time, based on our previous research about time series [20], where we add the SVM layer to LSTM model, and the prediction effect can be improved. There- fore, we also consider adding the SVM layer to the LSTM model in this paper to see if it can improve the prediction results. Table 6 shows the prediction results of different models.

As we can see, the experimental results of CNN are very poor, it predicts all the test samples as −1. For time-series features, CNN’s convolution does not work effectively, and information cannot be learned at timing. It is not suitable for timing prediction. For the Yucheng and Zhongjin, the LSTM results are better than that of the SVM, but the SVM F1 value of the Kuailu has reached an abnormal 0.99. We ana- lyze Kuailu’s experimental results in detail, as shown in Table 7. We can see that SVM also predicts all samples as 0, resulting in abnormal prediction results, and cannot predict effectively.

The results of the two-layer LSTM are really better than those of the single-layer LSTM (baseline), which verifies that the two-layer LSTM model can further extract the sequence features and improve the prediction results. For the results of LSTM-LSTM-SVM, the F1 value of Yucheng’s data is improved a little, while the F1 values of Zhongjin and Kuailu’s data are slightly decreased. We specifically analyze

17650 VOLUME 7, 2019

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

TABLE 8. LSTM’s results.

TABLE 9. 2-LSTM’s results.

TABLE 10. LSTM-LSTM-SVM’s results.

TABLE 11. SVM’s results.

TABLE 12. CNN’s results.

the prediction results of the five algorithms for Yucheng’s data, as shown in Tables 8–12. Comparing these results, it can be seen that the addition of SVM layer can greatly improve the results of the small class (−1). In fact, this conclusion is similar for Zhongjin and Kuailu’s data and is also consistent with our former work [20]. It can be verified again that SVM has a good effect on the small class.

In the follow-up experiments, we will add our attention mechanisms to improve the performance.

D. EXPERIMENT 3: TIME-ATTENTION AND USER-ATTENTION We have proposed two attention mechanisms, time-attention and user-attention. Here we need to examine their impact on the prediction results. The experimental results are shown in Table 13, where l-t-l-s represents the model in which time- attention is added to the LSTM-LSTM-SVM (l-l-s in short) in Experiment 2. Similarly, l-l-u-s represents the model in which

TABLE 13. Dual attention model’s results.

user-attention is added, and l-t-l-u-s represents the model in which both time-attention and user-attention mechanisms are added.

It can be seen from Table 13 that when only the time- attention is added, which is the l-t-l-s in the table, the results of the three different data’s prediction are improved compared with that of the LSTM-LSTM-SVM in Experiment 2. This indicates that our time-attention mechanism can improve the prediction results. By adding the time-attention mechanism, our model can better capture the importance of different points in time. When predicting the emotional tendency of the next time unit, it can select the time points that have greater influence on the prediction in the input time series, thereby improving the prediction results.

When the user-attention mechanism is added, which is the l-l-u-s in the table, the result is also improved com- pared to that of the LSTM-LSTM-SVM in Experiment 2. It can be seen that our user-attention mechanism also plays a certain role in improvement, and the user factor has a certain positive effect on the result of prediction. Of course, this also verifies our understanding. The impacts of differ- ent users are not the same. Users with stronger ‘‘impor- tance’’ will have more voices on the network than ordinary users. Therefore, it will play a greater role in the devel- opment of online public opinion. Our user-attention mech- anism captures this very well, allowing different users to play different roles in the predictive model and improve the predictions.

And then, we combine the time-attention and user- attention, add the two together to the model, and modify the model to l-t-l-u-s in the table. As we can see from the results, our time-attention and user-attention can work together, and we obtain the best experimental results. Our time+user dual attention mechanisms can capture both features in time series and user identity. And these characteristics can better ‘‘pay attention’’ to the existing strong correlation information in the prediction, weaken the impact of the noisy data, and play a role in improving the prediction results. Through these experiments, we can verify the effectiveness of our proposed time+user dual attention mechanisms in sentiment prediction for public opinion.

E. EXPERIMENT 4: MULTI-GRANULARITY SCANNING Since the sample size of the experimental data for each group is not large enough, there may be some small fluctu- ations in our experimental results. Zhou and Feng [19] adopt a multi-granularity scanning method to solve the problem

VOLUME 7, 2019 17651

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

TABLE 14. Multi-granularity scanning’s results.

that deep learning requires a lot of training data. Hence we also try this method. Considering that our original sample feature is a 12-dimension vector for one unit time, we try to set sliding=12, which is to slide the sample feature by length 12. Thus we can build 109 samples with one original 120-dimension feature vector for a time series when the time window is set to 10. With this method, our sample size can be expanded by 109 times. We use the expanded data to train and predict again, and then, the sample classification probabilities are added and averaged. Finally, the category which has the highest probability is used as the final prediction result corresponding to the original sample. The results are shown in Table 14.

Comparing with former results, we can see that both Yucheng and Zhongjin’s predictions have been improved. Using multi-granularity scanning on Yucheng and Zhongjin’s data, we can expand the number of samples. Therefore, the model training is more sufficient, the prediction result is improved, and also the stability is improved. But for Kuailu’s data, things are different, the forecasting effect has not been improved. We think that the reason may be that Kuailu’s pub- lic opinion development is relatively stable. In other words, Kuailu’s data is not mainly published during the concentrated time of the outbreak period, but all the time, thus the overall time span is large. In the case of a large time span, the number of samples obtained is inherently large, and multi-granularity scanning does not improve the model from increasing the number of samples. At the same time, multi-granularity slid- ing of features may produce some incomplete noise samples, so it has a certain negative impact on the training of the model, resulting in a decrease in Kuailu’s results. In summary, through experiments we find out that the multi-granularity scanning method can improve the prediction to a certain extent when the training samples are limited in a short period of time, the effect and stability of the prediction can be improved. But, it does not help for Kuailu’s data which has already enough samples in a long time.

V. CONCLUSION In this paper, we have implemented the emotional analysis for Chinese public opinion texts about P2P network lending on Sina Weibo social networking platform. Based on the previous research on the single text, we achieve the sentiment analysis with multi-user and multi-documents per unit time. And we consider factors such as the identity and influence of different users on the social network platform, and combine

them with the order of the public opinion information in time series. Then we propose a time+user dual attention mechanism model which integrates user and time related features. Experimental results have verified its effectiveness. At the same time, we also adopt multi-granularity scanning and achieve good application results on the appropriate data sets. For future work, we will extend it to other languages and implement more experiments on more public opinion data collected from various social networks with more emo- tions and improve the sentiment features for deeper semantic understanding.

REFERENCES [1] R. Xia, ‘‘New media status and future development trends,’’ Northern

Literature, no. 8, p. 200, Aug. 2017. [2] Z. Ding, Y. Jia, and B. Zhou, ‘‘A survey of microblog data mining

research,’’ Comput. Res. Develop., vol. 51, no. 4, pp. 691–706, 2014. [3] Zhang S. Time Series Analysis Simple Course. Beijing, China: Tsinghua

Univ. Press, 1999. [4] X. Liu, X. Sun, and Q. Zheng, ‘‘Research and judgment of college pub-

lic opinion situation based on renren,’’ China Media Technol., vol 4, pp. 117–118, Feb. 2012.

[5] Y. Xian et al., ‘‘The framework of network public opinion monitoring and analyzing system based on semantic content identification,’’ J. Converg. Inf. Technol., vol. 5, no. 10, pp. 1–5, Dec. 2010.

[6] G. Wang, ‘‘Research on network public opinion supervision and prediction based on big data analysis,’’ Comput. Modernization, vol. 12, pp. 62–66, Jan. 2016.

[7] B. Ma et al., ‘‘Public opinion analysis Based on probabilistic topic model- ing,’’ in Proc. Public Opinion, 2016, p. 171.

[8] Y. Shen and R. Xia, ‘‘The spreading situation of network public opinion in tourism emergency events based on SOM neural network,’’ J. Yichun Univ., vol. 4, pp. 92–95, Sep. 2015.

[9] C. J. H. E. Gilbert, ‘‘Vader: A parsimonious rule-based model for sentiment analysis of social media text,’’ in Proc. 8th Int. Conf. Weblogs Social Media (ICWSM), Apr. 2016, p. 14.

[10] A. Tripathy, A. Agrawal, and S. K. Rath, ‘‘Classification of sentiment reviews using n-gram machine learning approach,’’ Expert Syst. Appl., vol. 57, pp. 117–126, Sep. 2016.

[11] J. Lin, W. Mao, and D. D. Zeng, ‘‘Personality-based refinement for sentiment classification in microblog,’’ Knowl.-Based Syst., vol. 132, pp. 204–214, Sep. 2017.

[12] G. Lee, J. Jeong, S. Seo, C. Kim, and P. Kang, ‘‘Sentiment classifica- tion with word localization based on weakly supervised learning with a convolutional neural network,’’ Knowl.-Based Syst., vol. 152, pp. 70–82, Jul. 2018.

[13] T. Li and D. Ji, ‘‘Analysis of Weibo sentiment based on multi-feature combination of SVM and CRF,’’ Appl. Res. Comput., vol. 32, no. 4, pp. 978–981, 2015.

[14] F. Benamara et al., ‘‘Sentiment analysis: Adjectives and adverbs are better than adjectives alone,’’ in Proc. ICWSM, 2007, pp. 1–4.

[15] S. C. dos and M. Gatti, ‘‘Deep convolutional neural networks for sentiment analysis of short texts,’’ in Proc. 25th Int. Conf. Comput. Linguistics, Tech. Papers. 2014, pp. 69–78.

[16] L. T. Nguyen et al., ‘‘Predicting collective sentiment dynamics from time- series social media,’’ in Proc.1st Int.WorkshopIssuesSentimentDiscovery Opinion Mining, Aug. 2012, p. 6.

[17] D. Bahdanau, K. Cho, and Y. Bengio. (2014). ‘‘Neural machine trans- lation by jointly learning to align and translate.’’ [Online]. Available: https://arxiv.org/abs/1409.0473

[18] H. Chen et al., ‘‘Neural sentiment classification with user and product attention,’’ in Proc. Conf. Empirical Methods Natural Language Process., 2016, pp. 1650–1659.

[19] Z. H. Zhou and J. Feng. (2017). ‘‘Deep forest: Towards an alternative to deep neural networks.’’ [Online]. Available: https://arxiv.org/abs/1702.08835

[20] Li L, Wu Y, Ou Y, ‘‘Research on machine learning algorithms and feature extraction for time series,’’ in Proc. 28th Annu. Int. Symp. Personal, Indoor, Mobile Radio Commun. (PIMRC), Oct. 2017, pp. 1–5.

17652 VOLUME 7, 2019

L. Li et al.: Time+User Dual Attention-Based Sentiment Prediction for Multiple Social Network Texts With Time Series

LEI LI (M’17) received the B.S. degree in infor- mation engineering and the Ph.D. degree in signal and information processing from the Beijing Uni- versity of Posts and Telecommunications, Beijing, China, in 1996 and 2001, respectively.

From 2000 to 2003, she was a Project Man- ager with CapInfo Company Ltd., Beijing. From 2003 to 2004, she was a Postdoctoral Researcher with Macquarie University, Sydney, NSW, Australia. From 2004 to 2006, she was a

Lecturer with the Beijing University of Posts and Telecommunications, where she has been an Associate Professor, since 2006. Her research inter- ests include natural language processing, public opinion analysis, machine learning, deep learning, social network analysis, information extraction, and data mining.

Dr. Li is a member of the Chinese Association for Artificial Intelli- gence (CAAI) and the Secretary-General for the Natural Language Under- standing Committee, CAAI.

YABIN WU received the B.S. degree in com- puter science from Hangzhou Dianzi University, Hangzhou, China, in 2016. He is currently pur- suing the M.S. degree in intelligence science and technology with the Beijing University of Posts and Telecommunications, Beijing, China. His research interests include public opinion anal- ysis, machine learning, deep learning, social net- work analysis, and data mining.

YUWEI ZHANG received the B.S. degree in accounting from Tongling University, Anhui, China, in 2017. She is currently pursuing the M.S. degree in computer technology with the Beijing University of Posts and Telecommunications, Bei- jing, China. Her research interests include public opinion analysis, machine learning, deep learning, social network analysis, and data mining.

TIANYUAN ZHAO received the B.S. degree in measurement and control technology and instru- ment from the Beijing University of Posts and Telecommunications, Beijing, China, in 2017, where he is currently pursuing the M.S. degree in intelligence science and technology. His research interests include public opinion analysis, machine learning, deep learning, social network analysis, and data mining.

VOLUME 7, 2019 17653