help

978-1-7281-4242-5/19/$31.00 ©2019 IEEE 153

A RESEARCH ON THE METHOD OF CHINESE SENTIMENT ANALYSIS

BASED ON TRANSFER LEARNING

NA LI1, WEN-HONG TIAN1, RONG FU1, TING-YING ZHENG1, TING ZHANG1, XIAO- FENG GU1

1School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu

611731

E-MAIL: [email protected], [email protected]

Abstract: Sentiment analysis has great commercial value, which is

widely used in product analysis and product recommendation

in the industry. This paper presents the potential application of

transfer learning in Chinese text categorization and have a

binary classification of the automobile comment dataset,

demonstrating that the model presented in this paper is

effective and the classification accuracy is improved to 96.35%.

Keywords: Transfer learning; Sentiment analysis; Discriminative

fine-tuning; Gradual unfreezing

1. Introduction

With the development of media and social platforms,

social focus issues and emergency discussion, e-commerce

product comment and other information spread widely on the

Internet, generating a large amount of short text information.

Short-text sentiment analysis mainly focuses on

emotional classification of subjective short texts such as

Weibo, e-commerce buyer comment, and current affairs

news commentary. With the massive increase of text

information, the existing samples are not enough to train a

reliable model, and the labeling of new sample data is time-

consuming and laborious, which leads to the traditional

method not achieving good results. Real data often has data

imbalance, so transfer learning is used to introduce auxiliary

domain data into the training data to help train the

classification model.

Considering that most of the data or tasks are related,

transfer learning migrates the trained model parameters to

the new model to help the new model train. Through transfer

learning, we can share the learned model parameters to the

new model to accelerate and optimize the learning efficiency.

Although transfer learning has made significant progress in

the classification of English texts, there are few applications

in Chinese sentiment analysis.

In this paper, we classify Chinese texts based on transfer

learning. Chinese Wikipedia data was used for pre-training,

and the automobile comment dataset was used to achieve the

two-category problem of Chinese sentiment analysis, with

the classification accuracy rate improving to 96.35%.

2. Related Work

Long Short-Term Memory (LSTM, Hochreiter et

al.,1997) [1] is an improved model for the gradient

disappearance and gradient explosion problems of RNN. It

adds a "gate" to the original RNN (Recurrent Neural

Network) model to control the transmission of information.

It can avoid gradient disappearance and explosion problems

to a certain extent, and obtain long-distance dependence

information of text semantics. The model mainly includes

input gate, forget gate, output gate and memory unit. The

specific structure is shown in fig.1.

Fig.1 Schematic diagram of the LSTM

First, the LSTM must decide which information in the

previous unit was discarded by the Forgot Gate, which is

done by sigmoid, and a number (0 to 1) is calculated by the

weighted sum of the output of the previous moment and the

input of the present moment, 0 means complete discarding

and 1 means all reserved. The calculation process is shown

in formula (1).

(1) )],[( 1 fttft bxhWf +•= −

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 17,2021 at 14:55:10 UTC from IEEE Xplore. Restrictions apply.

154

The input gate controls the information to be added. The

calculation process is shown in formula (2) and (3).

(2)

(3)

The output gate controls what information is used for

task output at this moment. The calculation process is

shown in formula (4) and (5).

(4)

(5)

, , and c are the weight matrix of the input

gate, the forget gate, and the output gate, respectively. ,

, are the Bias matrix of the input gate, the forget gate,

and the output gate, respectively. and tanh are acti -vate

functions Word2vec uses skip-gram/CBOW to construct a

word vector by considering the relationship between

words and the words within the specified window. It not

only reduces the dimension of the word, but also implies

the relationship between the current word and other words

in the context within the vector.

ELMO (Embeddings from Language Models, Peters

et al., 2018) [2] uses a deep Bi-LSTM to build a language

model on a large corpus and generates a word vector. Then

it combines the above word vector into the existing word

vector of the current task, and finally the model is fine-

tuned and the obtained word vector is used for further

supervised learning.

The bert [3] model proposed by the google AI team

in 2018 is a transfer learning model. It performs multi-

task learning based on the bi-directional deep networks

Transformer and obtains a pre-trained model, and then

adds a layer to the specific task at the end of the model.

Howard and Ruder (2018) proposed a single-model

architecture ULMFIT [4], similar to the fine-tuning of the

ImageNet model. The authors propose a variety of pre-

trained methods of arranging, including Discriminative

Fine-tuning, Slanted triangular learning rates, Concat

pooling, and Gradual unfreezing. ULMFIT includes a

unidirectional language model based on LSTM, and

divides the whole training process into three parts: pre-

training, fine-tuning and training classifier.

3. The proposed method

Although the ULMFIT model mentioned above has

made great progress in English sentiment analysis, it has not

been applied in Chinese corpus. In this paper, we first use the

Chinese Wikipedia data to pre-train the language model, and

then use this model to perform downstream tasks. We used

the automobile comment dataset to fine-tune the model and

trained a classifier for sentiment tendency classification,

which divided the automobile comment into positive and

negative categories. In the pre-training phase, we take an

unsupervised approach. The entire training process is shown

in fig.2.

Fig.2 Training process

3.1. Pre-Training Language Model

Pre-training is the most time-consuming and expensive

part of the entire model. In the pre-training process, we use

the AWD-LSTM language model (Merity et al., 2017)[5].

The pre-trained model can be used in all Chinese nlp tasks.

Before the pre-training, the corpus was preprocessed,

including word segmentation and vectorization.

In the word segmentation, we use the jieba word

segmentation algorithm, which uses the prefix dictionary to

achieve efficient word graph scanning, generating a directed

acyclic graph composed of all possible words in the sentence,

and then uses dynamic programming to find the maximum

probability path.

We use Tokenizer to vectorize the text. In order to

enable language models to capture information that may be

relevant to classification, we convert the traditional Chinese

characters in the Chinese Wikipedia dataset and the

automobile comment dataset into simplified characters.

3.2. Fine-Tuning

Because the pre-trained language model is easy to

overfit on small datasets, and it is easy to forget the

knowledge learned by the language model during pre-

training when training the classifier, we use the fine- tuning

skills of the language model proposed by Howard and Ruder

[4] et al.

(1) Discriminative fine-tuning

Because different layers of the language model capture

)],[( 1 ittit

bxhWi +•= −



)],[tanh( 11 cttftttt

bxhWiCfC +•+= −−

)],[( 1 otto

t bxhWo +•=

− 

)tanh( tt

t Coh •=

i W fW oW

i b

f b

o b



Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 17,2021 at 14:55:10 UTC from IEEE Xplore. Restrictions apply.

155

different types of information, when fine-tuning the language

model, we assign different learning rates to each layer of the

model. When the parameters are updated, each layer is

updated according to its own learning rate. Howard and

Ruder et al. found it to work well to first choose the learning

rate of the last layer by fine-tuning only the last layer and

using as the learning rate for lower layers.

We set the learning rates of the four layers to 0.004/6, 0.004/3,

0.004 and 0.004/2 respectively.

(2) Slanted triangular learning rates

This is a new way to adjust the learning rate. In order to

quickly converge the parameters to a suitable area and then

slowly adjust to refine the model parameters, we use a

slanted triangle learning rate. The learning rate rises rapidly

at the beginning of a cycle, and then slowly declines. In our

experiments, the ratio between the maximum and minimum

of the learning rate is 32, and the ratio of the peak position

of the learning rate to the rising period is 10.

3.3. Train classifier

When training the classifier, first we load the pre-

trained and fine-tuned language model parameters for

initialization, and then add two fully connected layers to the

model. Both the fully connected layers are normalized and

dropout, and the activate function is Relu. Finally, the two

fully connected layers are connected to a Softmax output

layer to output the probability of each category. The network

structure of the training classifier is shown in fig.3.

Fig.3 The network structure of the training classifier

Finally, we need to adjust parameters of the two fully

connected layers. The method we use is Gradual unfreezing,

which is to gradually unfreeze the previous layer when fine-

tuning. Because if we fine-tune all layers at once, there may

be catastrophic forgetting, So when we start the tuning, we

will freeze the parameters of all the layers except the last

layer (the parameters can not be updated), and then , we

unfreeze the next lower frozen layer step by step.

4. Experiments

4.1. Experiments data

In the process of pre-training the universal language

model, we used Chinese Wikipedia data for pre-training. The

Chinese Wikipedia data contains 4.4 million articles. The

huge pre-training dataset is the basis of transfer learning.

In the fine-tuning and training classifier module, we

choose professional websites such as Auto House[6] and

Easy Car Network[7] as our original data source(These two

websites are open websites like taobao, and all users can

check the automobile comments on the website to help them

choose a car.). We crawled their automobile comments and

built a binary categorization dataset CARD based on

automobile comments as the target corpus. The CARD

dataset included a total of 75,000 comments, of which 37,500

were positive comments and 37,500 negative comments. The

maximum length of the comment is 150 words. The dataset

samples are shown in table 1.

Table 2 Dataset CARD samples

Positive comments Negative comments

空间巨大，动力满意，外观时

尚年轻，造型犀利，配置很

高，驾驶质感好。性价比超

高。

提速慢，动力一般。车的避

震较硬。颠簸感很强。这么

大个车连个后感应都没有。

动力十足，空间大。操控也没

问题。很轻松就上手了！

储物空间太少，夏天发动机

热气都传到驾驶室，受不了

没有明显的短板就是最满意

的一点！

油耗是有点高了，小部件还

是不够精密。

In the fine-tuning and training classifier module, we

selected 63,000 comments as the training set, 7000

comments as the validation set, and 5000 comments as the

test set.

4.2. Experimental parameters

We use the AWD-LSTM language model with an

embedding size of 400, 3 layers, 1150 hidden activations per

layer, and a BPTT batch size of 70. In the ULMFIT model

[5], the three phases use the same dropout. However, in our

experiment, we used different dropouts in the pre-training,

fine-tuning, and training classifiers, as shown in table 2.

l 

6.2/ 1 ll

 = −

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 17,2021 at 14:55:10 UTC from IEEE Xplore. Restrictions apply.

156

Table 3 dropout for each layer

Pre-training Fintuning Train

classifier

input layers 0.125 0.25 0.4

layers 0.05 0.1 0.5

RNN layers 0.1 0.2 0.05

embedding layers 0.01 0.02 0.3

RNN hidden-to-

hidden matrix 0.075 0.15 0.4

We use Adam with 00.8 and 00.99 to solve

large-scale data and parameter optimization problems. We

use a batch size of 64, And we use a base learning rate of

0.004 for finetuning the language model and of 0.01 for

training the classifier. When training the classifier, we

specified a learning rate reset period of 15.

4.3. Results and analysis

We compared the method applied in this paper with

other Chinese sentiment analysis models in the same

environment. We compare with three models, including

LSTM, Bi-LSTM, Bi-LSTM + Attention.

The experimental results of the algorithm applied in this

paper and other Chinese sentiment analysis models are

shown in table 4.

Table 4 experimental results

MODEL Accuracy/%

LSTM 84.42

Bi-LSTM 85.96

Bi-LSTM + Attention 88.79

OUR PAPER 96.35

The entire training takes about 35 hours, of which pre-

training takes about 30 hours. This is approximately 15 times

the LSTM training time. Although the method proposed in

this paper has far more training time than traditional deep

learning methods, the most expensive pre-training module

can be used for all text classification problems as long as it

is trained once.

Through comparison experiments, Bi-LSTM adds

backward propagation, which makes the context information

more closely linked, so the effect is better than LSTM, and

the accuracy is improved by 1.54%. Compared with Bi-

LSTM, the model after the attention mechanism can obtain

the probability distribution of attention by calculating the

correlation between the intermediate state and the final state.

It can solve the problem of information redundancy to the

greatest extent while retaining effective information, and

further improve the accuracy of text classification by

optimizing the text feature vector.

Compared with Bi-LSTM + Attention, the accuracy

of the model presented in this paper is improved by 7.56%,

which is enough to show the effectiveness of the method

in Chinese sentiment analysis.

5. Conclusion and future directions

The Chinese sentiment analysis method based on

transfer learning presented in this paper classifies the

automobile comment dataset into two categories by pre-

training, fine-tuning and train classifier. Under the same

conditions, compared with the other three sentiment analysis

models, the method presented in this paper is effective, and

the classification accuracy is improved to 96.35%. We think

this method can also be used in other nlp tasks. The model

used in this paper is only a unidirectional LSTM, which can

be modified to improve the accuracy of text classification.

Acknowledgements

This work supported by Sichuan Science and

Technology Program（2018CC0136、2018GZDZX0039、

2018GFW0209、2018GZ0008）.

References

[1] Hochreiter S , Schmidhuber, Jürgen. Long Short-Term Memory[J]. Neural Computation, 1997, 9(8):1735-

1780.

[2] Peters M E, Neumann M, Iyyer M, et al. Deep contextualized word representations[J]. 2018.

[3] Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language

Understanding[J]. 2018

[4] Howard J, Ruder S. Universal Language Model Fine- tuning for Text Classification[C]// Proceedings of the

56th Annual Meeting of the Association for

Computational Linguistics (Volume 1: Long Papers).

2018.

[5] Merity S, Keskar N S, Socher R. Regularizing and Optimizing LSTM Language Models[J]. 2017.

[6] https://www.autohome.com.cn/ [7] https://dianping.bitauto.com/

1 

2 

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 17,2021 at 14:55:10 UTC from IEEE Xplore. Restrictions apply.