helpfn

2974 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 4, APRIL 2021

Social Image Sentiment Analysis by Exploiting Multimodal Content and

Heterogeneous Relations Jie Xu , Zhoujun Li , Feiran Huang , Member, IEEE, Chaozhuo Li , and Philip S. Yu, Fellow, IEEE

Abstract—In the circumstance of social big data, senti- ment analysis is attracting increasing attention for its ca- pacity in understanding individuals’ attitudes and feelings. Traditional sentiment analysis methods focus on single modality and become ineffective as enormous data are emerging on the social websites with multiple manifesta- tions. In this article, multimodal learning approaches are proposed to capture the relations between image and text, which only stay at the region level and ignore the fact that the channels are also closely correlated with the semantic information. In addition, social images in the social plat- forms are closely connected by various types of relations, which are also conducice to sentiment classification but neglected by most existing works. In this article, we pro- pose an attention-based heterogeneous relational model to improve the multimodal sentiment analysis performance by incorporating rich social information. Specifically, we propose a progressive dual attention module to capture the correlations between image and text, and then learn the joint image-text representation from the perspective of content information. A channel attention schema is pro- posed here to highlight semantically rich image channels and a region attention schema is further designed to high- light the emotional regions based on the attended chan- nels. After that, we construct a heterogeneous relation net- work and extend graph convolutional network to aggregate

Manuscript received April 28, 2020; revised June 14, 2020; accepted June 19, 2020. Date of publication June 29, 2020; date of current version January 4, 2021. This work was supported in part by the National Natural Science Foundation of China under Grant U1636211, Grant 61906075, Grant 61932010, Grant 61672081, and Grant 61370126, in part by the Beijing Advanced Innovation Center for Imaging Tech- nology under Grant BAICIT-2016001, in part by the Fund of the State Key Laboratory of Software Development Environment under Grant SKLSDE-2019ZX-17, and in part by the Natural Science Foundation of Guangdong Province, China under Grant 2019A1515011920.\penalty -\@M Paper no. TII-20-2129. (Corresponding author: Feiran Huang.)

Jie Xu and Chaozhuo Li are with the State Key Laboratory of Software Development Environment, Beihang University, Beijing 100191, China (e-mail: [email protected]; [email protected]).

Zhoujun Li is with the State Key Laboratory of Software Development Environment, Beihang University, Beijing 100191, China, and also with the College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China (e-mail: [email protected]).

Feiran Huang is with the College of Cyber Security/College of Information Science and Technology, Jinan University, Guangzhou 510632, China, and also with Guangdong Key Laboratory of Data Security and Privacy Preserving, Guangzhou 510632, China (e-mail: [email protected]).

Philip S. Yu is with the Department of Computer Science, University of Illinois at Chicago, Chicago, IL 60607 USA (e-mail: [email protected]).

Color versions of one or more of the figures in this article are available online at https://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TII.2020.3005405

the content information from social contexts as comple- ments to learn high-quality representations of social im- ages. Our proposal is thoroughly evaluated on two bench- mark datasets, and experimental results demonstrate the superiority of the proposed model.

Index Terms—Attention mechanism, multimodal senti- ment analysis, network information, social image.

I. INTRODUCTION

W ITH the continuous prosperity of social networks, peo-ple tend to increasingly share their daily lives and ex- press personal opinions online. The automatic identification of sentiment is essential to understand the individual behavior and many applications can benefit from it, such as commercial analysis [1] and mental healthcare [2]. However, traditional sentiment analysis methods mainly focus on one single modality like text [3] or image [4], which is no longer applicable to the current situations that the prevalence of mobile Internet has fueled more and more diverse content with both image and text. Therefore, combining different modalities to perform sentiment analysis for social media data are of vital practical significance.

Though multimodal data is increasingly appearing on the so- cial website, multimodal sentiment analysis is still in its infancy, which can be roughly classified into two categories. Most early works are feature selection models [5]–[8]. For example, Wang et al. [5] utilize an uniform approach to encode the texts and images into bag-of-words representations, based on which a logistic regression approach is utilized to analyze the sentiment of Weibo tweets. Though the feature selection-based models have made some progresses, it is difficult for them to bridge the semantic gaps between different types of modalities. Therefore, the performance achieved is quite limited. With the development of neural networks, the deep learning-based models emerge at the right time [9]–[11]. For example, a recurrent model is proposed in [10] to learn the contextual information of utter- ances, where the attention network is also introduced. The deep learning-based models show their superiority on multimodal sentiment analysis. However, it is still nontrival to excavate the intricate emotional correlations among different modalities.

In addition, with the rise of social websites, the social images cease to be independent ones. Instead, they are closely connected by various types of social relations. For example, images may share same tags or posed by same users. These co-occurrence information can be extracted as relations and further contributes

1551-3203 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

XU et al.: SOCIAL IMAGE SENTIMENT ANALYSIS BY EXPLOITING MULTIMODAL CONTENT AND HETEROGENEOUS RELATIONS 2975

to building the relational network, in which images with similar topological contexts tend to convey similar sentiment tenden- cies. For example, images tagged with “Wedding” have a larger chance to be annotated as positive, which can be viewed as the first-order proximity in the constructed relational network. Thus, the social context of an image can provide abundant sentiment- related information as complements. However, this significant supplementary information has not yet been considered by most existing works.

Although current multimodal sentiment analysis has achieved promising performance, there still exist two major challenges to be addressed. First, it is intractable to excavate image-text correlations to bridge the heterogeneity gap between image and text. Existing works usually exploit attention mechanism to locate the semantically meaningful parts. However, most of these works focus on region attentions, while ignoring the fact that different visual channels detect the unique semantic patterns from the input images [12], which also contribute to classifying the sentiment tendency. Therefore, how to incorporate the chan- nel information to better learn emotional correlations between image and text is of significance. Second, integrating social relations into the sentiment analysis task is nontrivial. There exist various types of relations among social images, which may have different influences on the image sentiment classification [13]. In addition, social relations cannot be employed directly to deduce the sentiment as it is intractable to formulate the highly nonlinear sophisticated social topology information. Hence, how to utilize the rich heterogeneous relational information to facilitate the image sentiment classification still needs to be addressed.

In this article, we propose an attention-based heterogeneous relational model (AHRM) to tackle the two mentioned chal- lenges, in which both the content information and the social rela- tions are effectively incorporated. Specifically, to better capture the complementary emotional information from the perspective of content, we introduce a progressive dual image-text attention mechanism to learn the joint comprehensive representation for image and text. First, a channel-based attention is proposed to highlight the semantic important channels in visual features based on the relation with text description. After that, the learned channel-attended visual features will be fed into the next atten- tion module, i.e., region attention, to locate semantic important spatial regions relevant with the text description. Through this dual attention mechanism, both the significant visual regions and the text-related channels will be further emphasized, which will ensure the final attended visual feature contain richer semantic information compared with the original visual features. Next, we introduce a heterogeneous module to incorporate the com- plementary information from social contexts into the content representation learned by the progressive dual attention mech- anism. A heterogeneous network is constructed based on the shared social attributes (e.g., tags, groups, and users) and an extended graph convolutional network (GCN) is applied on the constructed network to improve the image sentiment classifica- tion performance. Several aspects of our work are worthwhile to highlight here.

1) To better capture the emotional and discriminative fea- tures for the content of image and text, we propose a progressive dual attention mechanism to highlight

channel-level and region-level semantic information, which can effectively fuse the text and visual features to learn high-quality representations from the perspective of content.

2) In order to incorporate the social attributes of images, we introduce social relations as auxiliary to aggregate the complementary information from social contexts to learn more comprehensive representations. Network features are not isolated with the content features, instead, we construct a heterogeneous network and extend the GCN to incorporate the structural information into the content features.

3) To evaluate the effectiveness of our model, we conduct experiments on two real-life datasets, i.e., Flickr dataset and Getty image dataset, which verify the superiority of our approach.

The rest of this article is organized as follows. Section II reviews the related work. Then, Section III elaborates on the details of AHRM model. Next, Section IV presents experimen- tal results on two benchmark datasets and comparisons with baselines. Finally, Section V concludes this article.

II. RELATED WORK

A. Multimodal Learning

Multimodal learning combines data of various modalities to conduct comprehensive analysis. There exist three major types of cross-model integration strategies.

Early fusion fuses the input features of diverse types directly by the point sum or dot product operation, or even just the concatenation. It is easy to implement, and all features can be seen at one time. Therefore, it is widely used by many works [14], [15]. However, early fusion cannot effectively cap- ture the time-synchronicity of different sources, and the feature vector generated may be redundant and high dimensional.

Late fusion analyzes and classifies the the features of each modality first, after which the decisions obtained respectively will be integrated to generate the final result [16]–[18]. Based on the different characters of different features, each modality can choose its most suitable classifier, nevertheless, it is difficult to ensemble all classifiers effectively, and the correlations among modalities are ignored.

With the development of deep model, most deep multimodal fusion works employ intermediate fusion strategy [19]–[21], which fuses features of different modalities into a single hid- den layer, known as the shared representation layer. The vari- ous representations can be fused gradually at different depths, and dimensionality reduction techniques such as stacked auto- encoders and principal component analysis (PCA) may be ap- plied after that. Intermediate fusion is indeed powerful and flexible; however, it still needs to carefully design the whole architecture.

B. Multimodal Sentiment Analysis

With the emerging of social media, sentiment analysis on multimodalities has been an popular research topic, which can be roughly classified into the following two categories.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

2976 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 4, APRIL 2021

Most early works conduct multimodal sentiment analysis with feature selection models. Chen et al. [6] employ a hyper-graph structure to capture the similarities between different modalities and the sentiment prediction is conducted by learning the rele- vance score among tweets with the transductive inference. Pang et al. [22] propose a multipathway deep Bolzmann machine to learn joint features for multimodal inputs, which can capture the nonlinear correlations among different modalities for sentiment classification. Poria et al. [8] extract the corresponding features for each modality and determine the corresponding tension levels to merge effective information for emotion recognition. Based on the ontology of different modalities, Cao et al. [7] propose a cross-media sentiment analysis framework, which can fuse the cross-media sentiment effectively. Though the feature selection models flourished in the early days, the performance achieved is undesirable due to the semantic gap among different modalities.

With the advance of deep neural networks, deep learning models have achieved remarkable results on multimodal senti- ment analysis. You et al. [9] propose a cross-modality consistent regression model to make the image sentiment predicted by CNN and textual sentiment predicted by paragraph vector tend to be more consistent. By modeling the intramodality and intermodal- ity dynamics, Zadeh et al. [23] introduce a tensor fusion network to conduct sentiment analysis end-by-end. Huang et al. [24] exploit the discriminative features with two separate uni-modal attention models and then a fusion-based multimodal attention is utilized to excavate the correlations for sentiment classifica- tion. Xu et al. [25] learn the joint image-text feature through a three-level hierarchical long short-term memory (LSTMs), which will be combined with the network feature learned from social relations to conduct the sentiment prediction.

Though the existing multimodal sentiment analysis works have achieved significant achievements, there are still several critical factors ignored by them. First, most existing works neglect the channel dimension in the visual feature, which are closely related to the semantic dependencies. Second, the images are connected with each other on the social network platform. Images with similar social context tend to share similar sentiment tendencies. However, these cross-modal correlations are not captured effectively in most existing works. Though [25] takes these into consideration, the network features are learned isolated with the content features, which may impact the final results.

III. ATTENTION-BASED HETEROGENEOUS RELATIONAL MODEL

In this section, we elaborate on the details of the proposed AHRM. First, the framework of our proposal is briefly intro- duced. Then, we present the details of three major components.

1) Single-modal representation learning, which learns the single-modal image representations from the visual view and text view, respectively.

2) Progressive dual image-text attention, which embeds the correlations between images and texts into the joint image-text representations with two novel cross-modal attentions (channel attention and region attention).

3) Heterogeneous relation fusion, which utilizes the social relations to construct a heterogeneous relation network and extend GCN to aggregate the content information from social contexts as complements to learn high-quality image representations.

4) Sentiment prediction, which conducts the final sentiment classification.

A. Overview

In social websites, images usually show up with the corre- sponding text descriptions. Besides, images are usually associ- ated with some social attributes, e.g., tags, users, and groups, which can help the social images interconnect with each other. Based on these two observations, we design our AHRM to conduct multimodal sentiment analysis (positive, negative) for social images. The framework of AHRM is presented in Fig. 1. As illustrated in the left part of Fig. 1, we first embed the image and its corresponding text into a low-dimensional representa- tion vector, respectively. Then, we introduce a progressive dual image-text attention to learn the emotional semantic correla- tion between image and text, in which a channel attention is utilized to highlight semantically rich channels and a region attention is utilized to progressively highlight the emotional regions based on the attended channels. Through the progressive dual attention, we obtain the joint image-text representation from the perspective of content. Next, we construct a heterogeneous relation network based on the social relations among images and extend GCN to incorporate the structural information into the content information learned in the previous phase. Finally, the relation incorporated representations of social images are utilized to conduct the sentiment classification.

B. Single-Modal Representation Learning

Visual Modal: Similar with [26], we choose VGG-19 net- works to extract visual features for the input image. Specifically, each image is feed into VGG-19 model and feature maps of the “conv5_4′′ layer are extracted. Thus, the representation of the image is denoted by V = {v1, v2, . . ., vR}, where vr ∈ RD represents the D-dimensional feature vector of the spatial region r.

Textual modal: Similar with [27], we first encode each word in the corresponding text description into a 300-dimensional GloVe feature. After that, we choose the bi-directional gated recurrent unit (bi-GRU) with H hidden units to conduct the textual feature encoding process. The final hidden states of bi-GRU are viewed as the H-dimensional text representation T ∈ RH .

C. Progressive Dual Image-Text Attention

Image and text are two different manifestations of a social image, which both imply rich semantic information and are complement to each other. Therefore, it is crucial to explore the inner correlations between the two modalities to learn a joint discriminative representation. Abundant related researches have been proposed, in which visual attention mechanism is among the most prevalent approaches to capture the text-image corre- lations [26]. However, most existing attention-based works only

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

XU et al.: SOCIAL IMAGE SENTIMENT ANALYSIS BY EXPLOITING MULTIMODAL CONTENT AND HETEROGENEOUS RELATIONS 2977

Fig. 1. Framework of AHRM.

put emphasis on the interplay between text and image regions, whereas neglecting the correlations hidden in the channel dimen- sions. Each channel is associated with a specific convolutional filter, which detects a unique semantic pattern from the input images. To better capture the semantic correlations between image and text, we introduce a channel-based attention module to locate significant channels consistent with the text, on which basis a region attention module is employed to obtain the final attended visual feature.

Channel-based attention: To capture the correlations between text and channel-level visual features, we first transform the region-based visual feature V = {v1, v2, . . ., vR}, vr ∈ RD into the channel-based visual feature C = {c1, c2, . . ., cD}, cd ∈ RR with a transpose operation

C = V T (1)

in which cd represents the d-channel of the region feature matrix V and D is the number of channels. After that, average pooling is applied on each channel, in which the mean value of channel cd is regarded as the dth channel feature cd. Thus, the final channel feature is represented as C = [c1, c2, . . ., cD].

To measure the semantic closeness of channel feature C ∈ RD to the textual feature T ∈ RH , we first map them to a joint embedding space with the dimension of S. Then, Hadamard product is employed to combine the transformed features C̃ ∈ RS and T̃ ∈ RS into a joint feature h1 ∈ RS. The process is summarized as follows:

C̃ = tanh(WvcC + bvc) (2)

T̃ = tanh(WtcT + bqc) (3)

h1 = tanh(C̃ � T̃) (4) where Wvc ∈ RS×D, Wtc ∈ RS×H , bvc ∈ RS, bqc ∈ RS are trainable parameters, and � represents the Hadamard product. Next, we introduce a softmax function to obtain the attention weight for each channel

α = softmax(Wch1 + bc) (5)

where Wc ∈ RD×S is the weight matrix and bc ∈ RD is the bias term.

Based on the obtained channel attention weights, the channel- attended visual features can be obtained by the channel-wise multiplication operation

V att_c = (α · C)T

= {vatt_c1 , vatt_c2 , . . ., vatt_cR }, vatt_cr ∈ RD (6)

Region attention: Intuitively, some spatial regions in the im- age will be more relevant to the text description, while some other regions may not be so significant. Through channel atten- tion, we have acquired the semantic importance of each channel. The next step is to recognize the semantic-related regions ac- cording to the learned channel-attended visual features and the corresponding text descriptions. Similar with channel attention, we first project the features into a S-dimensional common space and learn the joint visual-textual features

Ṽ att_c = tanh(WvrV att_c + bvr) (7)

h2 = tanh(Ṽ att_c ⊕ T̃) (8)

where Wvr ∈ RS×D and bvr ∈ RS×R are parameters, and ⊕ represents the addition of matrix and a vector. Next, we calculate the attention weight of each region

β = softmax(Wrh2 + br) (9)

Thus, the final attended visual features can be calculated as

vatt =

R∑ i=1

βi · vatt_ci (10)

By this means, we incorporate the text content into the visual representations and obtain the final attended visual features, in which both channel and region information are considered to learn more discriminative cross-modal features. The whole process of progressive dual image-text attention is illustrated in Fig. 2. Through the progressive dual image-text attention, we have achieved the attended visual features vatt, which will

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

2978 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 4, APRIL 2021

Fig. 2. Progressive dual attention network.

be concatenated with textual features T to generate the content feature V _CONT = (vatt, T).

D. Heterogeneous Relation Fusion

Content features learned in previous section only synthesize information from the social image itself, while a set of social attributes (e.g., tags, groups, and users) are affiliated with images on social websites, which can be regarded as the bridge to extract relations between images. As stated in the introduction section, the interrelations among images have important implications for sentiment analysis, but they are ignored by most existing works. GCN has made tremendous progress on encoding both graph structure and node features effectively for its superiority in structural information extraction. Therefore, to fully exploit the social connections for sentiment analysis, we regard the content feature V _CONT learned before as the initial node feature of the image and extend GCN to refine them.

Single relation: Similar with [28], we first construct image– image relations based on their social attributes. One social image may be associated with multiattributes, and thus there exist multiple types of relations between two connected images (e.g., published by the same user and annotated with the same tag). For the sake of clarity, here only one type of relation is considered each time. Taking the rth relation type as an example, we construct a simplified network Gr = {V, Er}, where V denotes the image set and Er denotes the edges set based on the rth relation type. Each er(i, j) ∈ Er represents the similarity weight of image Ii and Ij , which is calculated on the basis of the number of shared attributes of type r. The neighbors of image Ii in type r is defined as Nr(i), which includes the P most similar images.

Given the structural neighborhood Nr, GCN can aggregate neighborhood information iteratively to update the representa- tion of nodes. Specifically, the aggregation process incorporating neighbor features can be formulated as

hl−1Nr(i) = Aggr(h l−1 j ), ∀j ∈ Nr(i) (11)

where Aggr() represents the pooling function and hl−1j is in- terpreted as the hidden feature of image Ij at the (l − 1)th convolutional layer.

Coming up next, the features of self node should be combined with the generated neighborhood features through the convo- lution operation. To achieve this, a common method is con- catenating the node hidden feature hl−1i and the neighborhood feature hl−1Nr(i) first, which will make all dimensions of the two

features interact in the convolution process. However, as each dimension has its own indications, the interactions of different dimensions in a bit-wise manner are less informative under some circumstances. Therefore, a straightforward concatenation may make the learning process more complicated while some additional costs are unnecessary.

Targeting at this problem, we introduce a vector-wise convolu- tion operation to conduct the combination process. Specifically, for each local filter, different weights are learned for the node feature hl−1i and the neighborhood feature h

l−1 Nr(i), respectively.

Then, the two transformed vector interact in a vector-wise man- ner to generate the local filter-based node feature gl−1i (k) at the (l − 1)th convolutional layer, in which k represents the kth local filter. The process can be formulated as

gl−1i (k) = σ(w l−1 i (k, 1) · hl−1i + wl−1i (k, r + 1) · hl−1Nr(i))

(12) where σ is the nonlinear activation function, wl−1i (k, 1) and wl−1i (k, r + 1) represent the weights of the kth local feature for relation r. After that, the hidden feature for image Ii in relation r at layer l can be represented with another vector-wise operation on all K local filters

hli = σ

( K∑ k=1

θl−1k · gl−1i (k) )

(13)

in which θ is the trainable weight parameter. Heterogeneous relations: Assume there exist R types of re-

lations among images, we generalize (12) to capture heteroge- neous relationships as follows:

gl−1i (k) = σ

( wl−1i (k, 1) · hl−1i +

R∑ r=1

wl−1i (k, r + 1) · hl−1Nr(i) )

(14)

Based on this, (13) is utilized to obtain the final node represen- tation hLi for each image i.

E. Sentiment Prediction

Through the abovementioned descriptions, we achieve rep- resentative node features which incorporates both the content information and the structural information of the social image. Next, the multilayer perceptions (MLP) is utilized to conduct the sentiment classification with the softmax and negative log- likelihood operation

p(hLi ) = softmax(Wh L i ) (15)

L = −log(p(hLi ), y) (16) where W is the parameters to be learned and y is the real sentiment annotation.

IV. EXPERIMENTS

In this section, we evaluate the performance of the proposed AHRM model. First, we present the details of data preparation, model implementation, and baselines. Then, evaluations are conducted on two datasets to verify our approach, combined

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

XU et al.: SOCIAL IMAGE SENTIMENT ANALYSIS BY EXPLOITING MULTIMODAL CONTENT AND HETEROGENEOUS RELATIONS 2979

with the thorough analysis. Finally, additional assessments like ablation study and parameter sensitive analysis are investigated.

A. Datasets

Our model targets at the sentiment analysis of social images, which include not only the image features, but also the associated text descriptions and the social attributes. Therefore, the datasets require the following three conditions: First, the image should be come along with the text description when posted. Second, the image should be associated with several social attributes, which can be utilized to construct the relation network. Third, the image should be with one specific sentiment tendency, positive or negative. Based on the three conditions, we construct two datasets as follows.

1) Flickr: Flickr is a well known photo-sharing site which provides the image hosting service. It is characterized by the classification of hashtags and the authorization to personal photos, which can be accessed even by nonreg- istered users. Users can upload images, text description and define some metadata, such as tags and sets, through which the images can be connected with each other. Therefore, Flickr is ideal for our task. Similar with [29], the 1200 emotional ANPs (adjective-noun pairs, the com- bination of an adjective with rich emotion and a noun decorated by the emotional adjective) defined in the visual sentiment ontology [30] are utilized to retrieve images. The sentiment tendency of the retrieved image is in accor- dance with the corresponding ANP. Apart from the image itself, we also crawl its description and social attributes (group, tag, album, and gallery). After removing images without descriptions or social attributes, the Flickr dataset with 163 281 images is constructed, in which 80 924 instances are positive.

2) Getty Images: Getty Image is a supplier of images, video, and music where users can conveniently search and browse for the content they are interested. Usually, the images posted here are along with relatively formal de- scriptions. Besides, Getty Image will also list the related keywords, which can be utilized to construct the relation network. Thus, Getty Image is also appropriate for our model. Similar with [9], we retrieve images with the 101 sentimental keywords defined in the balanced affective word list project. Except for the retrieve images, we also record the corresponding descriptions and keywords. Fi- nally, we construct the Getty Image dataset with 401 502 images, and 198 564 of them are positive.

B. Implementation Details

Before the model training, we preprocess the crawled images and texts. For the images, we first crop and resize them into the size of 224 × 224 pixels. After that, the resized images are fed into the VGG-19 network to extract the visual features from the last convolutional layer. Hence, we obtain 14 × 14 region features with the dimension of 512 for each image. For the text descriptions, we convert all words into to lowercase and remove those words whose occurrence number is no more than

5. Besides, some high-frequency words about social images are sentiment meaningless and irrelevant, such as “24–200 mm” and “canon,” which are also excluded in the preprocessing stage. After that, the remaining words are embedded into the 300-D GloVe vectors. Finally, through the embedding of bi-GRU with 512 hidden units, we obtain a 512-D representation for each text description.

Our model is implemented with PyTorch on 2 × NVIDIA GeForce GTX 1080. The split training, validation, and test set account for 70%, 10%, and 20%, respectively in both two datasets and all hyperparameters are tuned on the validation set. To make the results more reliable, each model is run five times and the average scores are presented as the final results. The whole model is trained in an end-to-end manner with the ADAM optimizer whose learning rate is set to 0.0001. The training batch size is 256 and the dropout rate is 0.5. To reduce the internal covariate shift, batch normalization is also utilized in our work.

C. Baselines

In this section, we introduce several state-of-the-art models on sentiment analysis as the baselines to be compared with.

1) Textual model: In this method, the paragraph features of text description are directly fed into a logistic regression classifier to conduct the sentiment analysis.

2) Visual model: In this method, the visual features extracted from VGG-19 are directly fed into a logistic regression classifier to conduct the sentiment analysis.

3) CCR [9]: By imposing consistent constraints across the two modalities of image and text, CCR proposes a multi- modality regression model to make the sentiment predic- tion of each modality tend to be more consistent.

4) DMAF [24]: This method first proposes two separate uni- modal attention models to extract discriminative visual and textual features, respectively. Then, an intermediate fusion-based multimodal attention mechanism is utilized to exploit the internal correlation for sentiment classifi- cation.

5) HDF [25]: By taking social relations into considera- tion, this method utilizes a weighted network embedding method to obtain the network feature for each social image, which will be combined with the content features learned by a three-level hierarchical LSTMs to conduct the sentiment prediction.

D. Results and Analysis

In this section, we perform experiments on the two real-life datasets. The experimental results of AHRM and the baselines are presented for comparison. Following [9], F1-score and Ac- curacy are selected as the measurements.

Table I presents the results on the two datasets. From the results, we have the following observations.

First, our AHRM is superior to all baselines on both two datasets, whose primary factors are as follows. Different from previous attention-based sentiment analysis methods who only consider the interplay between the text and image regions, AHRM introduces a channel attention to locate the significant

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

2980 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 4, APRIL 2021

TABLE I RESULTS ON FLICKR DATASET

emotional channels first, which can help excavate the semantic information better. Besides, the introduction of GCN helps extract sentiment-related structural information and combines graph structure with node features effectively, which is another contributor of the performance improvement.

Second, neither the visual content nor the textual content is sufficient to infer the sentiment. There is a significant perfor- mance gap between the unimodal-based methods (textual model, visual model) and multimodal-based methods (CCR, DMAF, HDF, AHRM) on both two datasets, which verifies the necessity of multimodal data in the image sentiment classification task. Uni-modal is insufficient to depict the sentiment hidden in the social data. However, the performance between textual and visual models are of little difference on Flickr dataset, whereas textual model performs much better than visual model on Getty Image dataset. This may be because the text descriptions on Getty Image are more formal and with less irrelevant content, which can provide more accurate emotion-related information.

Third, the attention mechanism contributes to fine-grained features for sentiment classification. Both DMAF and CCR utilize image and text to conduct the sentiment classification. However, DMAF outperforms CCR on both Flickr and Getty Image significantly. The reason may be that CCR views different parts of image/text equally and extracts overall features, based on which a consistent regression model is proposed to make the sentiment result predicted by each modality as consistent as possible. Nevertheless, whether in the image or in the text, some parts are more related to the sentiment while others are relatively less important. Thus, the highlight of emotional parts is important for sentiment analysis, which is ignored by CCR. Comparatively, DMAF proposes a multimodal attention mechanism to learn the sentiment-attended features for the two modalities and exploit the internal correlation between them, which can help excavate the complementary and nonredundant sentiment information.

Finally, the import of social context is indeed conductive to the sentiment inference. Compared with CCR and DMAF, which only consider the image and the corresponding text description, the methods considering social information perform better on Flickr dataset. Similar with AHRM, HDF also brings in social relations. On this basis, it learns network features as a new modality for the social image. The success verifies that two connected images are more likely to have the same sentiment tendency. However, compared with AHRM, the network feature and the content feature are learned individually in HDF. Except

TABLE II ABLATION EXPERIMENTS OF AHRM

for the final feature fusion part, there exist no interactions between the two items. But in AHRM, the content feature is updated consistently by the extended GCN, which conduces to better performance.

E. Ablation Study

To evaluate the contribution of each module to AHRM, we conduct the following ablation studies on the two datasets.

1) AHRM w/o (CA + NT): This ablation model only utilizes region attention to study the correlation between text and image region, based on which the attended visual features are obtained and fed into MLP directly to deduce the sentiment.

2) AHRM w/o NT: This ablation model utilizes channel attention and region attention to study the correlation between image and text. However, the leaned attended features will be directly fed into MLP to deduce the sentiment without the extended GCN.

The results are shown in Table II, from which we can see that the removal of the channel attention and network embedding indeed reduce the final performance.

By comparing the results of “AHRM w/o NT” with “AHRM w/o (CA + NT),” we have reason to believe that the channel- based attention is beneficial to the extraction of sentiment- related content features. Different channel dimensions corre- spond to different semantic patterns, some of which are closely related to the sentiment. With the channel attention mechanism, we can highlight the meaningful semantic information, which will be incorporated into the region representation to generated more expressive attended visual features.

There is also a performance gap between “AHRM w/o NT” and “AHRM,” which demonstrates the validity of social network relations and our heterogeneous relation fusion method. With the extended GCN, the joint image-text attended features are refined through the network structure, which verifies the importance of network information. However, with the comparison between our “AHRM w/o NT” model and the baseline results listed in Table I, it is worth noting that compared with the baselines who only consider the content information, our model is still com- petitive. This indicates that our proposed attention mechanism can fully exploit the complicated correlations between image and text, based on which the joint image-text features obtained perform relatively well on sentiment analysis.

Based on the abovementioned, we can conclude that both the channel attention and the network embedding are crucial in our model to the multimodal sentiment analysis of social images.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

XU et al.: SOCIAL IMAGE SENTIMENT ANALYSIS BY EXPLOITING MULTIMODAL CONTENT AND HETEROGENEOUS RELATIONS 2981

Fig. 3. Results of different parameter settings on two datasets. (a) Training size. (b) Number of GCN layers.

F. Parameter Sensitivity Study

To further analyze the parameter sensitivity of our model, we compare AHRM with different amount of training data. To be more specific, we randomly sample different propor- tions (20%–100%) of data from the original training set and utilize them to retrain the model for sentiment classification. Fig. 3(a) shows the evolution of performance in terms of Ac- curacy. One can see that the accuracy value has been grow- ing with the increasing of training size, which verifies that more data can provide more helpful information. In addition, the number of GCN layers is also investigated. As shown in Fig. 3(b), with the increase of GCN layers, the classification performance first increases and then decreases. Our proposal achieves the best performance given GCN with three layers on both datasets. This may be because adding too many convolu- tional layers will result in the output features over-smoothed and make them indistinguishable, leading to the poor classification performance.

V. CONCLUSION

In this article, we proposed an AHRM to conduct multimodal sentiment classification incorporating both the content informa- tion and the social relations. In order to exploit the emotional correlations between image and text, we introduced a novel pro- gressive dual attention (channel attention and region attention) to highlight the emotional semantic-significant parts and learn a joint image-text representation. After that, we constructed a heterogeneous relation network from extracted social relations and extend the GCN to aggregate the content information from social contexts as complementary to learn high-quality represen- tations. Experiments were conducted on two benchmark datasets and the results demonstrated our superiority compared with the state-of-the-art baselines. However, our model mainly focused on the samples with fine-grained correlations between image and text, whereas some pairs are not in reality. Besides, some images may not connect with others tensely, which is also not considered well in our model. Targeting at these issues, we will design a more reasonable model in the future to boost the performance further.

ACKNOWLEDGMENT

This work was done when the first author was a visiting student researcher with the University of Illinois at Chicago.

REFERENCES

[1] B. Wang, Y. Huang, Z. Yuan, and X. Li, “A multi-granularity fuzzy computing model for sentiment classification of chinese reviews,” J. Intell. Fuzzy Syst., vol. 30, no. 3, pp. 1445–1460, 2016.

[2] S. Yadav, A. Ekbal, S. Saha, and P. Bhattacharyya, “Medical sentiment analysis using social media: towards building a patient assisted system,” in Proc. 11th Int. Conf. Lang. Resour. Eval., 2018.

[3] P. Chen, Z. Sun, L. Bing, and W. Yang, “Recurrent attention network on memory for aspect sentiment analysis,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2017, pp. 452–461.

[4] J. Yang, D. She, and M. Sun, “Joint image emotion classification and distribution learning via deep convolutional neural network,” in Proc. Int. Joint Conf. Artif. Intell., 2017, pp. 3266–3272.

[5] M. Wang, D. Cao, L. Li, S. Li, and R. Ji, “Microblog sentiment analysis based on cross-media bag-of-words model,” in Proc. Int. Conf. Internet Multimedia Comput. Service, 2014, pp. 76–80.

[6] F. Chen, Y. Gao, D. Cao, and R. Ji, “Multimodal hypergraph learning for microblog sentiment prediction,” in Proc. IEEE Int. Conf. Multimedia Expo., 2015, pp. 1–6.

[7] D. Cao, R. Ji, D. Lin, and S. Li, “A cross-media public sentiment analysis system for microblog,” Multimedia Syst., vol. 22, pp. 479–486, 2016.

[8] S. Poria, E. Cambria, N. Howard, G.-B. Huang, and A. Hussain, “Fusing audio, visual and textual clues for sentiment analysis from multimodal content,” Neurocomputing, vol. 174, pp. 50–59, 2016.

[9] Q. You, J. Luo, H. Jin, and J. Yang, “Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia,” in Proc. 9th ACM Int. Conf. Web Search Data Mining, 2016, pp. 13–22.

[10] S. Poria, E. Cambria, D. Hazarika, N. Mazumder, A. Zadeh, and L.-P. Morency, “Multi-level multiple attentions for contextual multi- modal sentiment analysis,” in Proc. IEEE Int. Conf. Data Mining, 2017, pp. 1033–1038.

[11] R. L. Rosa, G. M. Schwartz, W. V. Ruggiero, and D. Z. Rodríguez, “A knowledge-based recommendation system that includes sentiment analysis and deep learning,” IEEE Trans. Ind. Informat., vol. 15, no. 4, pp. 2124–2135, Apr. 2019.

[12] J. Song, P. Zeng, L. Gao, and H. T. Shen, “From pixels to objects: Cubic visual attention for visual question answering.” in Proc. Int. Joint Conf. Artif. Intell., 2018, pp. 906–912.

[13] J. Zhao et al., “Intentgc: A scalable graph convolution framework fusing heterogeneous information for recommendation,” in Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2019, pp. 2347–2357.

[14] L.-P. Morency, R. Mihalcea, and P. Doshi, “Towards multimodal sentiment analysis: Harvesting opinions from the web,” in Proc. 13th Int. Conf. Multimodal Interfaces, 2011, pp. 169–176.

[15] A. Valada, G. L. Oliveira, T. Brox, and W. Burgard, “Deep multispectral semantic scene understanding of forested environments using multimodal fusion,” in Proc. Int. Symp. Exp. Robot., 2016, pp. 465–477.

[16] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 568–576.

[17] S. Poria, E. Cambria, and A. Gelbukh, “Deep convolutional neural network textual features and multiple kernel learning for utterance-level multi- modal sentiment analysis,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2015, pp. 2539–2544.

[18] D. Wu et al., “Deep dynamic neural networks for multimodal gesture segmentation and recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 8, pp. 1583–1597, Aug. 2016.

[19] N. Neverova, C. Wolf, G. Taylor, and F. Nebout, “ModDrop: Adaptive multi-modal gesture recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 8, pp. 1692–1706, Aug. 2016.

[20] N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, and S. Poria, “Multimodal sentiment analysis using hierarchical fusion with context modeling,” Knowl. Based Syst., vol. 161, pp. 124–133, 2018.

[21] C. Xi, G. Lu, and J. Yan, “Multimodal sentiment analysis based on multi-head attention mechanism,” in Proc. 4th Int. Conf. Mach. Learn. Soft Comput., 2020, pp. 34–39.

[22] L. Pang, S. Zhu, and C.-W. Ngo, “Deep multimodal learning for affec- tive analysis and retrieval,” IEEE Trans. Multimedia, vol. 17, no. 11, pp. 2008–2020, Nov. 2015.

[23] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fu- sion network for multimodal sentiment analysis,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2017, pp. 1103–1114.

[24] F. Huang, X. Zhang, Z. Zhao, J. Xu, and Z. Li, “Image–text sentiment anal- ysis via deep multimodal attentive fusion,” Knowl.-Based Syst., vol. 167, pp. 26–37, 2019.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.

2982 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 4, APRIL 2021

[25] J. Xu et al., “Sentiment analysis of social images via hierarchical deep fusion of content and links,” Appl. Soft Comput., vol. 80, pp. 387–399, 2019.

[26] K. Xu et al., “Show, attend and tell: Neural image caption generation with visual attention,” in Proc. 32nd Int. Conf. Mach. Learn., 2015, pp. 2048–2057.

[27] Y. Song and M. Soleymani, “Polysemous visual-semantic embedding for cross-modal retrieval,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 1979–1988.

[28] Y. Zhang, Y. Xiong, X. Kong, S. Li, J. Mi, and Y. Zhu, “Deep collective classification in heterogeneous information networks,” in Proc. World Wide Web Conf., 2018, pp. 399–408.

[29] Q. You, J. Luo, H. Jin, and J. Yang, “Robust image sentiment analysis using progressively trained and domain transferred deep networks,” in Proc. 29th AAAI Conf. Artif. Intell., 2015, pp. 381–388.

[30] D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang, “Large-scale visual sentiment ontology and detectors using adjective noun pairs,” in Proc. 21st ACM Int. Conf. Multimedia, 2013, pp. 223–232.

Jie Xu received the M.Sc. degree in computer science from the School of Computer Science, Beijing University of Posts and Telecommunica- tions, Beijing, China, in 2014. She is currently working toward the Ph.D. degree in informa- tion security with the State Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University. She is currently an exchange stu- dent with the University of Illinois at Chicago, Chicago, IL, USA, for two years.

She has authored for journals like the IEEE TRANSACTIONS ON INTEL- LIGENT TRANSPORTATION SYSTEMS, and IEEE TRANSACTIONS ON CYBER- NETICS AND KNOWLEDGE-BASED SYSTEMS. Her current research interests include data mining (e.g., route planning) and social media analysis (e.g., image sentiment classification and multimodal learning).

Zhoujun Li received the M.Sc and Ph.D de- grees in computer science from the National University of Defence Technology, Changsha, China, in 1984 and 1999, respectively.

He is currently with the School of Com- puter Science and Engineering, Beihang Uni- versity, Beijing, China, where he has been a Professor since 2001. His research interests in- clude the data mining, information retrieval, and database.

Prof. Li has been the PC Member of many international conferences, such as SIAM International Conference on Data Mining (SDM) 2015, ACM International Conference on Informa- tion and Knowledge Management 2013, International Conference on Web-Age Information Management 2012, and Pacific Rim International Conference on Artificial Intelligence 2012, and so on. He has authored more than 150 papers on international journals, such as TKDE, Informa- tion Science, Information Processing and Management, etc, and inter- national conferences such as SIGKDD, Association for Computational Linguistics, Special Interest Group on Information Retrieval (SIGIR), Association for the Advancement of Artificial Intelligence (AAAI), Inter- national Joint Conference on Artificial Intelligence, SDM, Conference on Information and Knowledge Management (CIKM), International Confer- ence on Web Search and Data Mining, etc.

Feiran Huang (Member, IEEE) received the B.Sc. degree in electronic information science and technology from Central South University, Changsha, China, in 2011, and the Ph.D. de- gree in computer software and theory from School of Computer Science and Engineering, Beihang University, Beijing, China, in 2019.

He is currently a Lecturer with the School of Information Science and Technology and College of Cyber Security, Jinan University, Guangzhou. He has authored more than ten

papers, such as IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE TRANSACTIONS ON CYBERNETICS, IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, ACM Transactions on Multimedia Com- puting, Communications, and Applications, KBS, ACM Multimedia, CIKM, and International Conference on Multimedia Retrieval. His cur- rent research interests include social media analysis and multimodal learning.

Chaozhuo Li received the B.Sc. degree in computer science from the School of Com- puter Science, Beijing University of Posts and Telecommunications, Beijing, China, in 2011. He is currently working toward the Ph.D. degree in computer software and theory with State Key Laboratory of Software Development Environ- ment, School of Computer Science and Engi- neering, Beihang University, Beijing.

He has authored more than ten papers on top conferences (e.g., AAAI, SIGIR, CIKM, and

International Conference on Data Mining) and journals in these areas. Mr. Li served as a Reviewer for multiple top international conferences

and journals in the areas of graph mining and social network analysis. His research interests include graph mining (e.g., network representa- tion learning), social network analysis (e.g., social spammer detection), and recommender systems.

Philip S. Yu (Fellow, IEEE) received the Ph.D. degree in electrical engineering from Stanford University, Stanford, CA, USA.

He is a Distinguished Professor in computer science with the University of Illinois, Chicago, IL and is the Wexler Chair in Information Technology. His research interests include big data, data mining, data stream, database, and privacy.

Prof. Yu was the Editor-in-Chief for the IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGI-

NEERING and the ACM Transactions on Knowledge Discovery from Data. He was the recipient of ACM SIGKDD 2016 Innovation Award, a Re- search Contributions Award from the IEEE International Conference on Data Mining (2003), and a Technical Achievement Award from the IEEE Computer Society (2013). He is a Fellow of ACM.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 04:10:49 UTC from IEEE Xplore. Restrictions apply.