helpfn

Received December 25, 2019, accepted January 15, 2020, date of publication January 24, 2020, date of current version February 6, 2020.

Digital Object Identifier 10.1109/ACCESS.2020.2969205

Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning DONG ZHANG , SHOUSHAN LI, QIAOMING ZHU, AND GUODONG ZHOU School of Computer Science and Technology, Soochow University, Suzhou 215006, China

Corresponding author: Shoushan Li ([email protected])

This work was supported in part by the NSFC under Grant 61672366, and in part by the Two Key Project of NSFC under Grant 61836007 and Grant 61702149.

ABSTRACT Multi-modal sentiment analysis extends conventional text-based definition of sentiment analysis to a multi-modal setup where multiple relevant modalities are leveraged to perform sentiment analysis. In real applications, however, acquiring annotated multi-modal data is normally labor expensive and time-consuming. In this paper, we aim to reduce the annotation effort for multi-modal sentiment classification via semi-supervised learning. The key idea is to leverage the semi-supervised variational autoencoders to mine more information from unlabeled data for multi-modal sentiment analysis. Specifically, the mined information includes both the independent knowledge within single modality and the interactive knowledge among different modalities. Empirical evaluation demonstrates the great effectiveness of the proposed semi-supervised approach to multi-modal sentiment classification.

INDEX TERMS Natural language processing, sentiment analysis, multimedia computing.

I. INTRODUCTION As an increasingly popular area in affective computing [5], multi-modal sentiment analysis [2], [4] focuses on general- izing text-based sentiment analysis to a multi-modal setup, where various communicative modalities i.e. text (spoken language), vision (gestures), and audio (voice) are present. This generalization is particularly vital to the part of the multimedia community which deals with opinion mining and sentiment analysis [6], [7] due to the growing trend of sharing opinions in videos instead of pure text, specially in social media (Facebook, YouTube, etc.) [33].

Previous studies in the literature of multi-modal senti- ment analysis mainly focuses on supervised learning [34]. However, it is normally hard to obtain a sufficient amount of labeled data which integrate text, vision and audio since manual annotation of multi-modal data is labor expen- sive and time-consuming. To well address this challenge, semi-supervised learning becomes crucial to successful application of multi-modal sentiment analysis. However, to the best of our knowledge, there are no previous studies

The associate editor coordinating the review of this manuscript and

approving it for publication was Mehedi Masud .

to date on semi-supervised learning to sentiment analysis of multi-modal language.

In this paper, we propose to perform semi-supervised learn- ing with proper exploitation of both independent knowledge within single modality and interactive knowledge among dif- ferent modalities to multi-modal sentiment analysis, moti- vated by following two factors.

On the one hand, with regard to multi-modal data, no matter whether for supervised or semi-supervised learn- ing, we should effectively capture the independent knowl- edge within each modality [35]. It is worthwhile to stress that independent knowledge from uni-modality is partic- ularly challenging for this task since multi-modal senti- ment analysis is performed on spoken language. A spoken opinion is normally full of spontaneous language phe- nomena such as insertion and repetition, e.g., ‘‘I think it was alright . . .Hmmm . . . let me think . . . yeah . . . no . . . ok yeah’’, which almost never happens in written text. This volatile nature of spoken opinions, where proper linguis- tic structure is often ignored, much complicates sentiment analysis. Besides, vision and audio modalities contain their own independent knowledge which are expressed through both space and time [4]. Although we can employ LSTM

VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ 22945

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

FIGURE 1. Examples of the interactions among text, vision and audio behaviors.

or bidirectional LSTM to model the independent knowledge within uni-modality in supervised learning, it may be hard for semi-supervised learning [36]. This is mainly because that, besides of capturing independent knowledge within labeled data of uni-modality, semi-supervised learning needs to take care of unlabeled samples. In this paper, instead of using some conventional semi-supervised approaches such as self- training or label propagation, we employ a semi-supervised variational autoencoder for each modality, namely Uni-modal SVAE. The variational autoencoder can not only abstract the commonality of both labeled and unlabeled samples but also learn the distribution of the entire sample space [20], [21], which effectively captures the independent knowledge of uni- modality.

On the other hand, the interactive knowledge among different modalities is crucial to multi-modal sentiment anal- ysis. However, modeling the interactions among multiple modalities are challenging [1], [4] in both supervised and semi-supervised learning. This is mainly because various kinds of interactions among text, vision and audio which greatly change the perception of the expressed sentiment. For example, in uni-modal part of Figure 1, the utterance ‘‘a few moments are original’’ may be classified as positive by the independent uni-modal SVAE of text, while the frown expression may be predicted as negative at the same time with independent uni-modal SVAE of vision. As a whole, it maybe perceived as ambiguous (either positive or nega- tive) by either early fusion [8] or late fusion [14]. However, in terms of the bi-modal example in Figure 1, the sentiment of the bi-modality could be accurately perceived as negative. In this paper, we believe that the sentiment expressed by the interactive text, vision, and audio from the same utterance should be as close as possible. To address this phenomenon, we propose a bi-modal SVAE approach by adding a loss term to measure the distance between the output sentiment vector

representations from two independent uni-modal SVAEs. Returning to the bi-modal example in Figure 1, after cap- turing independent and interactive knowledge, the sentiments of two modalities generated by bi-modal interactions should become as close as possible. Although the text modality is perceived as positive primarily, the sentiment of the same utterance with interactive knowledge tends to be negative and the sentiment of vision modality is unchanged. In this way, the sentiment of bi-modality is accurately predicted as negative. Furthermore, since the tri-modal interactions are practically three different combinations of bi-modal inter- action, we propose a tri-modal SVAE approach to model- ing the interactions of tri-modality on the top of bi-modal SVAEs. Figure 1 illustrates an interactive example on tri- modality. Similar to bi-modality, the overall interactions from tri-modality successfully predict the sample as a negative one.

In summary, our main contributions are: • To the best of our knowledge, we are the first to perform semi-supervised learning to multi-modal sentiment anal- ysis at the utterance-level which includes text, vision and audio modalities.

• We propose a multi-modal semi-supervised variational autoencoder approach to alleviate manual annotation and improve the performance of multi-modal sentiment classification with both the independent and interactive knowledge. With proper modeling of both independent and interactive knowledge in different modalities, our approach is divided into three types, i.e., uni-modal SVAE, bi-modal SVAE and tri-modal SVAE.

• Our approach much advances the state-of-the-art on two popular multi-modal sentiment analysis datasets, i.e., CMU-MOSI and CMU-MOSEI.

II. RELATED WORK While conventional studies on semi-supervised sentiment classification in the literature focus on text modality, more and more recent studies focus on multi-modality.

A. TEXT-BASED SEMI-SUPERVISED SENTIMENT CLASSIFICATION The main stream of the research in textual sentiment clas- sification focused on supervised and unsupervised learning tasks. In comparison, semi- supervised sentiment classifica- tion has much less related studies.

Li et al. [12] propose a semi-stacking approach to semisu- pervised sentiment classification, which integrates two or more semi-supervised learning algorithms from an ensemble learning perspective. Xia et al. [10] propose a dual-view co-training algorithm based on dual-view BOW representa- tion for semi-supervised sentiment classification, which can also be viewed as a variation of co-training. The innova- tion of this approach is the dual-view construction technique which incorporates antonymous reviews and the bootstrap- ping mechanism by observing two opposite sides of one review.

22946 VOLUME 8, 2020

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

In addition to the above conventional approaches, the semi- supervised approach based on the deep generative networks is also applied to sequential semi-supervised language model- ing. Xu et al. [21] propose semi-supervised sequential varia- tional autoencoder (SSVAE) for semi-supervised sequential text classification, which increases the capability by feed- ing the label into its decoder RNN at each time-step. It is the state of the art for semi-supervised text-based sentiment classification.

In this paper, we draw on the SSVAE approach and propose the multi-modal semi- supervised variational autoencoder approaches by joint learning with independent knowledge and interactive knowledge.

B. AUDIO AND VIDEO-BASED SEMI-SUPERVISED SENTIMENT CLASSIFICATION Early multi-modal semi-supervised sentiment classification is based on multiple sources of images [11], [26]. These studies consider an image as a sample. Since, recently, audio and video are widely used on lots of social plat- forms, Zhang et al. [25] focus on semi-supervised learning of bi-modal information, i.e., audio and vision. This study considers a segment of speech and the corresponding video (temporal images) as a sample. However, they all ignore the text modality information. In addition, they do not conduct sequential modeling so that these approaches are difficult to extend to multi-modal language data studied in this paper.

Different from the above studies, our approach leverages the sequential variational autoencoding and considers three kinds of modality information, i.e., text, vision and audio, for semi-supervised utterance-level sentiment classification.

III. PRELIMINARIES Kingma et al. [20] propose a semi-supervised learning approach based on generative models. The objective for the entire dataset is given as follows:

J = ∑

(x,y)∈Cl

L(x,y)+ ∑ x∈Cu

U(x)

+αE(x,y)∈Cl [− logqφ(y|x)] (1)

where Cl and Cu are labeled and unlabeled datasets respec- tively, α is a hyper-parameter of additional classification loss of labeled data. qφ(y|x) is a distribution which could be learned from the labeled data. L(x,y) and U(x) are the variational bound of a single data point in the labeled data and the unlabeled data respectively.

Specifically, L(x,y) is defined as follows:

L(x,y) = DKL(qφ(z|x,y)||p(z))− logpθ(y) −Eqφ(z|x,y)[logpθ(x|y,z)] (2)

where the first term is Kullback-Leibler divergence between the prior distribution p(z) and the learned latent posterior qφ(z|x,y), and the last term is the expectation of the condi- tional log-likelihood on latent variable z.

FIGURE 2. Three components in Uni-modal SVAE. Dashed line means that the label distribution vector y is provided by the classifier when using unlabeled data, while when using the labeled data, we employ the real label vector y .

And U(x) is defined as follows:

U(x) = ∑ y

qφ(y|x)(L(x,y)−H(qφ(y|x)) (3)

where H(qφ(y|x) denotes the information entropy of the learned classifier qφ(y|x). From the loss function, it is not difficult to find that three

distribution functions, i.e., qφ(y|x), qφ(z|x,y) and pθ(x|y,z) are the key components in the generative model-based semi- supervised learning approach.

IV. SEMI-SUPERVISED CLASSIFICATION WITH VARIATIONAL AUTOENCODING In this section, we describe the three main components of uni-modal SVAE for sentiment classification in detail by neu- ral networks modeling. They consist of a classifier network qφ(y|x), an encoder network qφ(z|x,y) and a decoder network pθ(x|y,z). Figure 2 shows the architecture of the uni-modal SVAE approach. These networks are presented as follows.

(a) Classifier: For the classifier qφ(y|x) as shown in Figure 2(a), we employ the LSTM network to capture the semantic representation of each modality. Then, a fully- connected layer with a softmax activation function is applied to obtain the classification probability, i.e.,

y = softmax(WTLSTM(x)) (4)

where x denotes the input embedding of each modality. (b) Encoder: In the encoder network as shown

in Figure 2(b), each data pair (x,y) is encoded into a soft ellip- soidal region in the latent space. Specifically, the distribution of z is parameterized by a diagonal Gaussian distribution qφ(z|x,y):

x̂ = fenc(x), (5)

qφ(z|x,y) = N(µ(x̂,y),diag(σ 2(x̂,y))), (6)

z ∼ qφ(z|x,y) (7)

VOLUME 8, 2020 22947

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

where fenc(·) is the encoder function. In our approach, it is implemented by a LSTM layer. N(·) denotes a Gaussian distribution with the reparameterization trick [20], [22].

(c) Decoder: The decoder as shown in Figure 2(c), is a conditional generative model that estimates the probability of generating x given latent variable z and real or predicted label y:

pθ(x|y,z) = D(x|fdec(y,z)) (8)

where fdec(y,z) is used to parameterize a distribution D, typically a Gaussian distribution for input data.

To implement the decoder network, this paper tries a poten- tial conditional LSTM structure. This structure concatenates word embedding and label vector at each time step, which is widely used in [15], [16], [23]. We call this structure C-LSTM, which takes both words and labels as input, and modify the equations representing the operations of the LSTM cell to add the label vector y to the input gate, forget gate, cell and output gate. In each of the following equations, the term in bold is the modification made to the original LSTM equation.

it = σ(Wixt +Uiht−1 +Viy) (9)

ft = σ(Wf xt +Uf ht−1 +Vf y) (10)

ĉt = tanh(Wcxt +Ucht−1 +Vcy) (11)

ot = σ(Woxt +Uoht−1 +Voy) (12)

ht = ot ⊗ tanh(ft ⊗ ct−1 + it ⊗ ĉt) (13)

V. SEMI-SUPERVISED LEARNING ON MULTI-MODAL SENTIMENT CLASSIFICATION In this section, we extend the semi-supervised classification approach above to multi-modal sentiment analysis with joint optimization.

A. LOW-LEVEL FEATURES EXTRACTION First, we extract the low-level handcrafted features to be able to identify and interpret the factors that have the most impact on sentiment. The chosen sequences are the three modali- ties: textual, visual and acoustic. To get the exact utterance timestamp of each word, we perform forced alignment using P2FA [27], which allows us to align the three modalities together. Since words are considered as the basic units of language, we use the interval duration of each word utterance as a time-step. We calculate the expected video and audio features by taking the expectation of their view feature values over the word utterance time interval [6]. For each of the three modalities, we process the information from videos as follows.

1) TEXT MODALITY The text is presented as manual transcription of the spo- ken language, and GloVe word embeddings [28] are used to extract word vectors from transcripts. This results in a sequence of dimension n × dt after alignment, where n denotes the number of words in a video and dt denotes the

dimension of a word embedding. The timing of word utter- ances is extracted using P2FA forced aligner. This extraction enables alignment between text, vision and audio. During the training stage, we allow fine-tuning all word embeddings with the embedding layer via back propagation. Unknown words are randomly initialized.

2) VISION MODALITY The library Facet1 is used to extract a set of visual features including facial action units, facial landmarks, head pose, gaze tracking and HOG features. These visual features are extracted from the full video segment at 30Hz to form a sequence of facial gesture measures throughout time, result- ing in a sequence of dimension n×dv.

3) AUDIO MODALITY The software COVAREP [29] is used to extract acoustic features including 12 Mel-frequency cepstral coefficients (MFCCs), pitch tracking and voiced/unvoiced segmenting features [30], glottal source parameters [31], peak slope parameters and maxima dispersion quotients [32]. These acoustic features are extracted from the full audio clip of each segment at 100Hz to form a sequence that represents variations in tone of voice over an audio segment, resulting in a sequence of dimension n×da after alignment.

B. JOINT OPTIMIZATION OF MULTI-MODALITY With proper capturing of both independent and interactive knowledge in different modalities, our model could be imple- mented in three versions, as following,

1) UNI-MODAL SVAE (UNI-SVAE) This model is applied on three modalities respectively. The objective function of this uni-modal approach is defined as follows:

J = ∑

(xm,y)∈Ml

L(xm,y)+ ∑ xm∈Mu

U(xm)

+αE(xm,y)∈Ml [− logqφ(y|xm)] (14)

where xm is the input embedding of the uni-modality fea- ture. Ml denotes all labeled samples of the uni-modality, Mu denotes all unlabeled samples of the uni-modality. The symbol representing modality m ∈ {Text,Vision,Audio}.

2) BI-MODAL SVAE (BI-SVAE) Text, vision and audio are different kinds of features to the same utterance and simply mixing them together does not seem a good way. Instead, we propose the Bi-modal SVAE to perform semi-supervised learning with two kinds of these features. In this model, we first train two uni-modal SVAEs for two modalities respectively to capture the independent knowledge. In order to capture the interactive knowledge between the different modalities, we then limit the two out- puts from the two classifiers of two modalities to be as similar

1https://imotions.com/emotient/

22948 VOLUME 8, 2020

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

FIGURE 3. The architecture of Bi-modal SVAE.

as possible. Figure 3 shows the architecture of the Bi-modal SVAE approach. The objective function of this model is defined as follows:

J = Jm +Jn +γExm∈M,xn∈N[log(||qφm(ym|xm)−qφn(yn|xn)||

2 F)]

(15)

where Jm denotes the uni-modal SVAE objective func- tion of the first modality m. Jn denotes the uni-modal SVAE objective function of the second modality n. M denotes all labeled and unlabeled samples of the first modality. N denotes all labeled and unlabeled samples of the second modality. And the modality pair (m,n) ∈ {(Text,Vision), (Text,Audio), (Vision,Audio)}.

3) TRI-MODAL SVAE (TRI-SVAE) This model is the extension of Bi-modal SVAE, which utilizes three modalities. Figure 4 shows the architecture of the Tri- modal SVAE model. The objective function of this model is defined as follows:

J = Jt +Jv +Ja +αExt∈T,xv∈V [log(||qφt (yt|xt)−qφv(yv|xv)||

2 F)]

+βExt∈T,xa∈A[log(||qφt (yt|xt)−qφa(ya|xa)|| 2 F)]

+γExv∈V,xa∈A[log(||qφv(yv|xv)−qφa(ya|xa)|| 2 F)] (16)

where the symbols containing subscript t are related to the text modality, the symbols containing subscript v are related to the vision modality and likewise, the symbols containing subscript a are related to the audio modality.

VI. EXPERIMENTATION A. EXPERIMENTAL SETTINGS • Datasets: We use two mainstream open-source multi- modal datasets2 for multi-modal sentiment classifica- tion. Table 1 shows the data split of our used two datasets. CMU-MOSI: Multi-modal Opinion Sentiment Inten- sity dataset is an annotated dataset of video opinions

2https://github.com/A2Zadeh/CMU-MultimodalDataSDK

TABLE 1. Data splits to ensure speaker independent learning. Arrows indicate the number of annotated utterances in all video segments.

from YouTube movie reviews [13]. Annotation of sen- timent has closely followed the annotation scheme of the Stanford Sentiment Treebank [17], where senti- ment is annotated on a seven-step Likert scale from very negative to very positive. However, whereas the Stanford Sentiment Treebank is segmented by sentence, the CMU-MOSI dataset is segmented by opinion utter- ances to accommodate spoken language where sentence boundaries are not as clear as written text. CMU-MOSEI: Multi-modal Opinion Sentiment and Emotion Intensity [9] is the largest to date with both sentiment and emotions annotated. The set of videos are tokenized into sentences using punctuation markers manually provided by transcripts. Due to the high qual- ity of the transcripts, using punctuation markers showed better sentence quality than using the Stanford CoreNLP tokenizer [18]. After tokenization, a set of 23,453 sen- tences were chosen as the final sentences in the dataset. Table 2 shows the statistics of the two datasets.

• Data Settings: For the two datasets, we randomly select 20% of the samples from the training set as labeled, leaving 80% of the samples as unlabeled, and the given validation set and test set are unchanged. All of our experiments are performed independent of speaker iden- tity, as no speaker is shared between training and test sets for generalizability of the model to unseen speakers in real-world.

• Hyperparameters and Training: We perform binary classification for multimodal sentiment analysis and train the model with the following hyperparameters. The encoder and decoder has a hidden size of 128. The latent variable z has a size of 64. All the initial weights are sampled from a uniform distribution [-0.08, 0.08]. The mini-batch size is 40. Hyper-parameters α, β and γ are scaled from 0.5 to 0.9. The models are trained end-to- end using the Adamax optimizer [37] with a learning rate of 0.001. In all loss functions, F is the Euclidean norm. The best-performed model on the validation data is selected for testing.

• Evaluation Metrics and Significance Test: The perfor- mance is evaluated using accuracy (Acc) and F1 measure as [9]. Finally, the paired t-test is performed to test the significance of the difference between two approaches, with a default significant level of 0.05.

B. EXPERIMENTAL RESULTS: UNI-MODALITY For thorough comparison, we implement following base- line approaches to supervised and semi-supervised sentiment classification when the features of uni-modality (text, vision

VOLUME 8, 2020 22949

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

FIGURE 4. The architecture of Tri-modal SVAE.

TABLE 2. MOSI and MOSEI datasets summary of statistics.

or audio) are employed, including LSTM as a supervised uni- modal benchmark. • LSTM: A supervised approach, which is the widely used supervised baseline to sentiment analysis. We use the implementation by [19]. And we also implement the Stacked LSTM, Bidirectional LSTM and Stacked Bidi- rectional LSTMs for stronger baselines. The best per- forming result we report is stacked bidirectional LSTM.

• Label Propagation (LP): A graph-based semi- supervised learning approach, which can predict the information of unlabeled nodes by a few labeled nodes.

• Self-training (ST): A semi-supervised learning approach which extends the supervised LSTM approach based on self-training mechanism. Specifically, it first trains a supervised LSTM method based on only labeled instances, and then predicts the label of unla- beled instances. After that, by adding the predicted labels on the unlabeled data as ‘‘ground-truth’’, another supervised LSTM approach is trained. This process is repeated until the maximum number of iterations achieves.

• Enhanced Self-training (eST): A semi-supervised learning approach [25], which extends the above tra- ditional self-training approach. Specifically, to over- come the noise accumulation problem, this approach maintains the previously selected data in the original unlabeled data set at all learning iterations. By doing this, the previously selected data will be re-evaluated by

the following enhanced model. Therefore, it is possible to correct mislabeled data in future iterations with an improved model.

• Uni-SVAE: A semi-supervised sentiment classification approach, which is introduced in the above section. We apply this approach in text, vision and audio modal- ities respectively.

Table 3 shows the experimental results of uni-modality on CMU-MOSI and CMU-MOSEI datasets. From this table, we can see that LP approach can hardly exploit unlabeled uni- modal data to improve classification performance in some cases. For example, in the vision modality of MOSI and the text modality of MOSEI, LP performs much worse than the supervised approach LSTM in terms of both metrics. In general, the ST and eST approaches perform better than LSTM and LP approaches. This indicates that a large num- ber of unlabeled samples can be leveraged to help a small number of labeled samples to build a more accurate clas- sifier. Among all baselines performed on uni-modaltiy, our Uni-SVAE approach can significantly outperform the other approaches (p-value < 0.05). This suggests that it is a good choice to design the VAE-based semi-supervised approach.

C. EXPERIMENTAL RESULTS: BI-MODALITY AND TRI-MODALITY For thorough comparison, we implement following baseline approaches to supervised and semi-supervised multi-modal

22950 VOLUME 8, 2020

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

TABLE 3. Experimental results of uni-modality on two datasets.

TABLE 4. Experimental results of bi-modality and tri-modality on CMU-MOSI.

sentiment classification, including MFN as a supervised multi-modal benchmark.

• MFN: A supervised multi-modal approach, namely memory fusion network [6], which explicitly accounts for both interactions in a neural architecture and contin- uously models them through time. This approach is con- sidered as the state of the art to multi-modal sentiment classification.

• Co-training (CT): A classic dual-view semi-supervised approach, which is widely applied in sentiment clas- sification and enhances the semi-supervised sentiment classification efficiency [10]. In our implementation, two different modalities are treated as two views.

• Enhanced Co-training (eCT): A semi-supervised approach [25], which extends the above traditional co-training approach to recognize the audio and video sentiment. Specifically, to overcome the noise accumu- lation problem, this approach maintains the previously selected data in the original unlabeled data set at all learning iterations. This approach is proposed to not always trust the machine labeled data, which is possible to correct mislabeled data in future iterations with an improved model. In our implementation, two different modalities are treated as two views.

• Tri-training: A widely used semi-supervised classifi- cation approach for multi-view learning [24], which addresses the problem of lacking of labeled samples effectively. In our implementation, the triplet modalities build three classifiers with LSTM respectively to per- form tri-training algorithm.

• DGFLP: A semi-supervised multi-modality classifi- cation approach [11] on multi-modal images, namely dynamic graph fusion label propagation. We apply

this approach to multi-modal language sentiment classification.

• EF-SVAE: A semi-supervised approach, namely early fusion SVAE, which concatenates the inputs from different modalities at each time-step as a joint feature representation and uses that as the input to a unimodal SVAE.

• DV-SVAE: A semi-supervised approach, namely deci- sion voting SVAE, which builds the multiple uni-modal SVAEs individually and calculates a weighted sum of the predicted results from multiple uni-modal SVAEs’ classifiers as the final prediction.

• Ours: Our proposed Bi-SVAE and Tri-SVAE intro- duced in above section in detail.

1) FOR CMU-MOSI Table 4 shows the experimental results of bi-modality and tri-modaltiy on CMU-MOSI in detail. We report the results with 20% initial labeled from original training set. From this table, we can see that the performance of MFN is indeed much higher than the LSTM thanks to the multi-modal fusion technique. Four semi-supervised approaches including CT, eCT, Tri-training and DV-SVAE perform better than MFN due to the fact that the unlabelled data are effec- tively mined with those four semi-supervised approaches. Especially, DV-SVAE achieves the apparent improvements compared with the supervise baselines. However, two semi- supervised approaches including DGFLP and EF-SVAE per- form worse than MFN. It is somehow surprising that DGFLP approach does not outperform the supervised baseline. This might be because the graph-based label propagation approach is not able to well model sequential language, after all, it is designed for multi-modal image data. Moreover, the most

VOLUME 8, 2020 22951

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

TABLE 5. Experimental results of bi-modality and tri-modality on CMU-MOSEI.

FIGURE 5. (a) Influence of the size of initial labeled data on CMU-MOSI. (b) Influence of the size of initial labeled data on CMU-MOSEI.

basic label propagation approach performs poorly on uni- modal data as Table 3 shown. It can be seen from the side that this graph-fusion label propagation approach (the state- of-the-art for multi-modal semi-supervised learning) may be not suitable for multi-modal language analysis.

Among all baselines performed on CMU-MOSI, our proposed approach Bi-SVAE and Tri-SVAE perform best. And both significantly outperform all the other approaches (p-value < 0.05). For example, Tri-SVAE gains the improvement over MFN, Tri-training, EF-SVAE and DV-SVAE by 3.6%, 2.1%, 4.2% and 2.2% in terms of accu- racy, respectively.

2) FOR CMU-MOSEI Table 5 shows the experimental results of semi-supervised multi-modality sentiment classification on CMU-MOSEI. From this table, we can see that MFN outperforms LSTM. CT, eCT, Tri-training and DGFLP approaches in all the scenarios, except that it performs a little worse than EF-SVAE in terms of F1. However, one semi- supervised baseline DV-SVAE could achieve better per- formance, compared to MFN. This indicates that it is necessary to some extent for us to explore semi-supervised approaches.

Among all approaches performed on CMU-MOSEI, our proposed Tri-SVAE performs best. Specifically, Bi-SVAE outperforms MFN by 1.4%, 2.1%, 2.3% on three types of bi-modality respectively, in terms of accuracy metric.

Tri-SVAE outperforms MFN by 1.9% in terms of accuracy on tri-modality. Note that the similar conclusions can be obtained when the F1 metric is used. All of the improvements are significant according to the paired t-test.

VII. ANALYSIS AND DISCUSSION In this section, we introduce three types of analysis from different views, to further demonstrate the effectiveness of our approaches.

A. INFLUENCE OF THE SIZE OF INITIAL LABELED DATA We tune the sizes of the initial labeled set (from 10% to 40%), and report its influence in Figure 5(a) and Figure 5(b). For all the settings, we fix the size of validation and test sets. The x-axis denotes the percentage of initial labeled set. The best-performed results on bi-modality and tri-modality are selected to report. We can observe that our approach performs consistently the best across different sizes of the ini- tial labeled data, although the improvements become limited when the size becomes larger. For example, our approach out- performs MFN by about 4% when the initial labeled size is 10% on CMU-MOSI. However, when the initial labeled size becomes 20% or 30%, the improvements become 3% or 2%. The similar conclusions could be obtained from the results on CMU-MOSEI. Note that as in Figure 5(b) shown, MFN performs a sudden worse with the initial labeled size 20%. This suggests that supervised multi-modal fusion approach presents non-stable and more data should be well processed

22952 VOLUME 8, 2020

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

FIGURE 6. The example of multi-modal interactions.

TABLE 6. Two examples of generated utterances conditioned on different categorical label y and same latent state z .

by a proper approach such as our VAE-based semi-supervised approaches.

B. CASE STUDY From Figure 6, we can see that EF-SVAE directly determines the sample as positive in that there is a word ’like’ and a smile expression in the mixed features. DV-SVAE obtains the final result as positive according to the result of the independent uni-modal SVAE via decision voting. Only our proposed Tri-SVAE could capture both independent and interactive knowledge by joint learning, so that the three modalities are simultaneously predicted as negative, and the final result tends to be negative. This case further demonstrates that it is necessary to capture both the independent and dependent knowledge for multi-modal sentiment classification.

C. FURTHER ANALYSIS A good explorative evaluation of the model’s ability to comprehend the data manifold is to evaluate the genera- tive model. We selected a z on the modality of text and generate an utterance using trained conditional generative model pφ(x|y,z). Table 6 demonstrates two cases using the same latent variable z but with opposite sentimental labels. Utterances generated by the same z share a similar syntac- tic structure and words, but their sentimental implications are much different from each other. From these two cases, we can qualitatively explain why our approach can effectively employ the generated model to improve semi-supervised classification performance. This is mainly because if the predicted y is wrong, the reconstruction loss between gen- erated utterance and original input becomes bigger, so that our approach works hard to force the classifier to make predictions as accurate as possible.

VIII. CONCLUSION In this paper, to reduce the annotation of multi-modal senti- ment data, we propose a semi-supervised learning approach to multi-modal sentiment classification. Specifically, we pro- pose a multi-modal semi-supervised variational autoencoder approach. First, our approach establishes the uni-modal SVAEs for different modalities to capture the independent knowledge. Second, our approach adds the loss term that makes the predicted label vectors from different modalities be as close as possible, so that it can capture both the inde- pendent knowledge within single modality and the interactive knowledge among different modalities. Empirical studies on two benchmark datasets of multi-modal language analysis show that our approach greatly advances the state-of-the-art of multi-modal sentiment classification by leveraging unla- beled data.

In our future work, we will apply our proposed approach to perform other multi-modal tasks, such as sarcasm detection and personality recognition. Moreover, to further advance the interactive process, we are going to design the time- dependent interactions when performing semi-supervised learning.

REFERENCES

[1] Y. Gu, K. Yang, S. Fu, S. Chen, X. Li, and I. Marsic, ‘‘Multimodal affective analysis using hierarchical attention strategy with word-level alignment,’’ in Proc. 56th Annu. Meeting Assoc. Comput. Linguistics, vol. 1, 2018, pp. 2225–2235.

[2] S. Poria, E. Cambria, and A. Gelbukh, ‘‘Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis,’’ in Proc. Conf. Empirical Methods Natural Lang. Process., Lisbon, Portugal, Sep. 2015, pp. 2539–2544.

[3] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, ‘‘Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages,’’ IEEE Intell. Syst., vol. 31, no. 6, pp. 82–88, Nov. 2016.

[4] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L. P. Morency, ‘‘Tensor fusion network for multimodal sentiment analysis,’’ in Proc. EMNLP, Copenhagen, Denmark, Sep. 2017, pp. 1103–1114.

[5] S. Poria, E. Cambria, R. Bajpai, and A. Hussain, ‘‘A review of affective computing: From unimodal analysis to multimodal fusion,’’ Inf. Fusion, vol. 37, pp. 98–125, Sep. 2017.

[6] A. Zadeh, P. P. Liang, N. Mazumder, S. Poria, E. Cambria, and L. P. Morency, ‘‘Memory fusion network for multi-view sequential learn- ing,’’ in Proc. AAAI, New Orleans, LA, USA, Feb. 2018, pp. 5634–5641.

VOLUME 8, 2020 22953

D. Zhang et al.: Multi-Modal Sentiment Classification With Independent and Interactive Knowledge via Semi-Supervised Learning

[7] A. Zadeh, P. P. Liang, S. Poria, P. Vij, E. Cambria, and L. P. Morency, ‘‘Multi-attention recurrent network for human communication comprehen- sion,’’ in Proc. AAAI, New Orleans, LA, USA, Feb. 2018, pp. 5642–5649.

[8] S. Poria, I. C. E. Chaturvedi, and A. Hussain, ‘‘Convolutional MKL based multimodal emotion recognition and sentiment analysis,’’ in Proc. ICDM, Barcelona, Spain, Dec. 2016, pp. 439–448.

[9] A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L. P. Morency, ‘‘Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,’’ in Proc. ACL, Melbourne, VIC, Australia, Jul. 2018, pp. 2236–2246.

[10] R. Xia, C. Wang, X. Y. Dai, and T. Li, ‘‘Co-training for semi-supervised sentiment classification based on dual-view bags-of-words representa- tion,’’ in Proc. ACL, Beijing, China, Jul. 2015, pp. 1054–1063.

[11] G. Lin, K. Liao, B. Sun, Y. Chen, and F. Zhao, ‘‘Dynamic graph fusion label propagation for semi-supervised multi-modality classification,’’ Pattern Recognit., vol. 68, pp. 14–23, Aug. 2017.

[12] S. Li, L. Huang, J. Wang, and G. Zhou, ‘‘Semi-stacking for semi- supervised sentiment classification,’’ in Proc. ACL, Beijing, China, Jul. 2015, pp. 27–31.

[13] A. Zadeh, R. Zellers, E. Pincus, and L. P. Morency, ‘‘MOSI: Multi- modal corpus of sentiment intensity and subjectivity analysis in online opinion videos,’’ 2016, arXiv:1606.06259. [Online]. Available: https:// arxiv.org/abs/1606.06259

[14] H. Wang, A. Meghawat, L. P. Morency, and E. P. Xing, ‘‘Select-additive learning: Improving generalization in multimodal sentiment analysis,’’ in Proc. ICME, Hong Kong, Jul. 2017, pp. 949–954.

[15] I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau, ‘‘Building end-to-end dialogue systems using generative hierarchical neural network models,’’ in Proc. AAAI, Phoenix, AZ, USA, Feb. 2016, pp. 3776–3784.

[16] T. H. Wen, M. Gasic, N. Mrksic, P. H. Su, D. Vandyke, and S. Young, ‘‘Semantically conditioned LSTM-based natural language generation for spoken dialogue systems,’’ in Proc. EMNLP, Lisbon, Portugal, Sep. 2015, pp. 1711–1721.

[17] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, ‘‘Recursive deep models for semantic compositionality over a sentiment treebank,’’ in Proc. EMNLP, Seattle, WA, USA, 2013, pp. 1631–1642.

[18] C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky, ‘‘The Stanford CoreNLP natural language processing toolkit,’’ in Proc. ACL, Baltimore, MD, USA, Jun. 2014, pp. 55–60.

[19] A. Graves, ‘‘Generating sequences with recurrent neural networks,’’ 2013, arXiv:1308.0850. [Online]. Available: https://arxiv.org/abs/1308.0850

[20] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, ‘‘Semi- supervised learning with deep generative models,’’ in Proc. Adv. Neural Inf. Process. Syst., Montreal, QC, Canada, Dec. 2014, pp. 3581–3589.

[21] W. Xu, H. Sun, C. Deng, and Y. Tan, ‘‘Variational autoencoder for semi- supervised text classification,’’ in Proc. AAAI, San Francisco, CA, USA, Feb. 2017, pp. 3358–3364.

[22] D. J. Rezende, S. Mohamed, and D. Wierstra, ‘‘Stochastic backpropagation and approximate inference in deep generative models,’’ in Proc. 31th Int. Conf. Mach. Learn. (ICML), Beijing, China, Jun. 2014, pp. 1278–1286.

[23] S. Ghosh, O. Vinyals, B. Strope, S. Roy, T. Dean, and L. Heck, ‘‘Con- textual LSTM (CLSTM) models for large scale NLP tasks,’’ 2016, arXiv:1602.06291. [Online]. Available: https://arxiv.org/abs/1602.06291

[24] Z.-H. Zhou, EnsembleMethods:FoundationsandAlgorithms. Boca Raton, FL, USA: CRC Press, 2012.

[25] Z. Zhang, F. Ringeval, B. Dong, E. Coutinho, E. Marchi, and B. Schüller, ‘‘Enhanced semi-supervised learning for multimodal emotion recogni- tion,’’ in Proc. ICASSP, Shanghai, China, Mar. 2016, pp. 5185–5189.

[26] C. Gong, D. Tao, S. J. Maybank, W. Liu, G. Kang, and J. Yang, ‘‘Multi- modal curriculum learning for semi-supervised image classification,’’ IEEE Trans. Image Process., vol. 25, no. 7, pp. 3249–3260, Jul. 2016.

[27] J. Yuan and M. Liberman, ‘‘Speaker identification on the SCOTUS cor- pus,’’ J. Acoust. Soc. Amer., vol. 123, no. 5, p. 3878, 2018.

[28] J. Pennington, R. Socher, and C. Manning, ‘‘Glove: Global vectors for word representation,’’ in Proc. EMNLP, Doha, Qatar, Oct. 2014, pp. 1532–1543.

[29] G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer, ‘‘COVAREP—A collaborative voice analysis repository for speech tech- nologies,’’ in Proc. ICASSP, Florence, Italy, May 2014, pp. 960–964.

[30] T. Drugman and A. Alwan, ‘‘Joint robust voicing detection and pitch esti- mation based on residual harmonics,’’ in Proc. INTERSPEECH, Florence, Italy, Aug. 2011, pp. 1973–1976.

[31] T. Drugman, M. Thomas, J. Gudnason, P. Naylor, and T. Dutoit, ‘‘Detec- tion of glottal closure instants from speech signals: A quantitative review,’’ IEEE Trans. Audio, Speech, Language Process., vol. 20, no. 3, pp. 994–1006, Nov. 2011.

[32] J. Kane and C. Gobl, ‘‘Wavelet maxima dispersion for breathy to tense voice discrimination,’’ IEEE Trans. Audio, Speech, Language Process., vol. 21, no. 6, pp. 1170–1179, Feb. 2013.

[33] W. Guo, J. Wang, and S. Wang, ‘‘Deep multimodal representation learning: A survey,’’ IEEE Access, vol. 7, pp. 63373–63394, May 2019.

[34] J. K. P. Seng and K. L. M. Ang, ‘‘Multimodal emotion and sentiment mod- eling from unstructured big data: Challenges, architecture, & techniques,’’ IEEE Access, vol. 7, pp. 90982–90998, Jul. 2019.

[35] S. Nemati, R. Rohani, M. E. Basiri, M. Abdar, N. Y. Yen, and V. Makarenkov, ‘‘A hybrid latent space data fusion method for multi- modal emotion recognition,’’ IEEE Access, vol. 7, pp. 172948–172964, Nov. 2019.

[36] J. Ma, N. Wang, and B. Xiao, ‘‘Semi-supervised classification with graph structure similarity and extended label propagation,’’ IEEE Access, vol. 7, pp. 58010–58022, Apr. 2019.

[37] D. P. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimization,’’ in Proc. 3rd Int. Conf. Learn. Represent. (ICLR), San Diego, CA, USA, May 2015. [Online]. Available: http://arxiv.org/abs/1412.6980

DONG ZHANG received the B.S. degree in soft- ware engineering and the M.S. degree in computer science and technology from Soochow University, Jiangsu, China, in 2013 and 2017, respectively, where he is currently pursuing the Ph.D. degree in computer science and technology.

His research interests include the natural lan- guage processing, sentiment analysis, and multi- modal communication comprehension. His awards and honors include the Excellent Graduate Student

in 2017 (Soochow University), the Special Academic Scholarship in 2018 (Soochow University), the Excellent Ph.D. Student in 2018 (Soochow Uni- versity), the Best Chinese Paper Award (NLPCC 2019), and the National Scholarship (Top 1 in the School of Computer Science and Technology).

SHOUSHAN LI was born in Yangzhou, Jiangsu, China, in 1980. He received the Ph.D. degree from the National Laboratory of Pattern Recognition, CASIA, Beijing, China, in 2008. He is currently a Full Professor with the School of Computer Science and Technology, Soochow University. His current research interests include natural lan- guage processing, social computing, and sentiment analysis.

QIAOMING ZHU was born in Suzhou, Jiangsu, China, in 1963. He received the Ph.D. degree from the School of Computer Science and Technol- ogy, Soochow University, Suzhou, China, in 2006. He is currently a Full Professor with the School of Computer Science and Technology. His research interests include natural language processing, sen- timent analysis, and discourse analysis.

GUODONG ZHOU was born in Changzhou, Jiangsu, China, in 1967. He received the Ph.D. degree from the National University of Singapore, Singapore, in 1999. He is currently a Full Professor with the School of Computer Science and Tech- nology, and the Director of the Natural Language Processing Laboratory, Soochow University. His research interests include information retrieval and natural language processing.

22954 VOLUME 8, 2020