Image Description Literature Review

a rX

iv :1

5 0

9 .0

4 9

4 2

v1

[c s.

C V

] 1

6 S

e p

2 0

1 5

Guiding Long-Short Term Memory for Image Caption Generation

Xu Jia KU Leuven ESAT-PSI, iMinds

[email protected]

Efstratios Gavves∗

QUVA, University of Amsterdam [email protected]

Basura Fernando ACRV, The Australian National University

[email protected]

Tinne Tuytelaars KU Leuven ESAT-PSI, iMinds

[email protected]

Abstract

In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search in order to prevent from favoring short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or even outperform the current state-of-the-art.

1. Introduction

Recent successes in visual classification have shifted the interest of the community towards higher-level, more com- plicated tasks, such as image caption generation [9, 38, 20, 28, 21, 22, 23, 27, 19, 26, 17, 36, 7, 37]. Although for a human describing a picture is natural, it is quite difficult for a computer to imitate this task. It requires the computer to have some level of semantic understanding of the content of an image, including which kinds of objects there are, how they look like, what they are doing, and so on. Last but not least, this semantic understanding has to be structured into a human-like sentence.

Inspired by recent advances in machine translation [5, 1, 32], neural machine translation models have lately been applied to the image caption generation task [26, 17, 36, 7, 37], with remarkable success. In particular, compared to template-based methods, that use a rigid sentence struc- ture, and transfer-based methods, that re-use descriptions available in the training data, methods based on neural ma-

∗This work was carried out while he was in KU Leuven ESAT-PSI.

Figure 1: Image caption generation using LSTM and the proposed gLSTM. The generation by LSTM and gLSTM and the cross-modal result that is used as guidance, are marked respectively in green, red and blue.

chine translation models stand out thanks to their capability to generate new sentences. They manage to effectively gen- eralize beyond the sentences seen at training time, which is possible thanks to the language model learnt. Most neu- ral machine translation models follow an encoder-decoder pipeline [5, 1, 32], where the sentence in the source lan- guage is first encoded into a fixed-length embedding vec- tor, which is then decoded to generate a new sentence in the target language. For machine translation, parallel corpora are typically used for learning and evaluating the model [5, 1, 32]. The pairs of sentences in the source and target languages usually share similar sentence struc- tures (often including regular phrases and the same order of words). This structural information is encoded in the fixed- length embedding vector and is helpful to the translation.

Applied to caption generation, the aim is to “translate” an image into a sentence describing it. However, it is ques-

1

tionable whether these models can cope with the large dif- ferences between the two modalities. The structure of the visual information is very different from the structure of the description to be generated. During the encoding phase, the algorithm compresses all visual information into an em- bedding vector. Yet this vector is unlikely to capture the same level of structural information needed for correctly generating the textual description in the subsequent decod- ing phase.

One of the latest state-of-the-art methods [36] uses a con- volutional neural network (CNN) for the encoding step and long-short term memory (LSTM) network for the decod- ing step. We notice that sometimes the generated sentence seems to “drift away” or “lose track” of the original im- age content, generating a description that is common in the dataset, yet only weakly coupled to the input image. We hypothesize this is because the decoding step needs to find a balance between two, sometimes contradicting, forces: on the one hand, the sentence to be generated needs to de- scribe the image content; on the other hand, the generated sentence needs to fit the language model, with more likely word combinations to be preferred. The system then may “lose track” of the original image content if the latter force starts to dominate. From an image caption generation point of view, however, staying close to the image content may be considered the most important of the two.

To overcome the limitation of the basic encoding- decoding pipeline, extended pipelines have been proposed in the context of both machine translation [1] and image caption generation [37]. They introduce an attention mech- anism to align the information in both the source and target domains, so that the model is able to attend to the most rele- vant part in the sentence from the source language or image.

Here, we propose an alternative extension of the LSTM model, that works at a more global scale. We start by ex- tracting semantic information from the image and then use it to “guide” the decoder, keeping it “on track” by adding a positive bias to words that are semantically linked to the image content. More specifically, we add semantic infor- mation as an extra input to the gate of each LSTM mem- ory cell. This extra input can take many different forms as long as they build a semantic connection between the image and its description.e.g. a semantic embedding, a classifica- tion or retrieval result. As an illustration we experiment with features either obtained from a multimodal semantic embedding using CCA or, the cross-modal retrieval results based on that semantic embedding.

Our contributions are two-folded. As our main contri- bution, we present an extension of LSTM which is guided by semantic information of image. We coin a term gLSTM for the proposed method. We show experimentally on mul- tiple datasets that such guiding is beneficial for learning to generate image captions. As an additional contribution, we

make the observation that current inference methodologies for caption generation are heavily biased towards short sen- tences. We show experimentally that this hurts the quality of the generated sentences and therefore propose sentence normalization, which further improves the results. In the experiments, we show that the proposed method is on par or even outperforms the latest published and unpublished state-of-the-art on the popular datasets.

2. Related Work

Caption generation. The literature on caption generation can be divided into three families. In the first family we have template-based methods [9, 38, 20, 28]. These ap- proaches first detect objects, actions, scenes and attributes, then fill them in a fixed sentence template,e.g. using a subject-verb-object template. These methods are intuitive and can work with out-of-the-box visual classification com- ponents. However, they require explicit annotations for each class. Given the typically small number of categories available, these methods do not generate rich enough cap- tions. Moreover, as they use rigid templates the generated sentence is less natural.

The second family follows a transfer-based caption gen- eration strategy [21, 22, 23, 27]. They are related to the im- age retrieval. These methods first retrieve visually similar images, then transfer captions of those images to the query image. The advantage of these methods is that the gener- ated captions are more human-like than the generations by template-based methods. However, because they directly rely on retrieval results among training data, there is little flexibility for them to add or remove words based on the content of an image.

Inspired by the success of neural networks in machine translation [5, 1, 32], recently people have proposed to use neural language models for caption generation. Instead of translating a sentence from a source language into a target one, the goal is to translate an image into a sentence that de- scribes it. In [19] a multimodal log-bilinear neural language model is proposed to model the probability distribution of a word conditioned on an image and previous words. Simi- larly, Maoet al. [26] and Karpathyet al. [17] have proposed to use a multimodal recurrent neural network [31] model for caption generation. Vinyalset al. [36] and Donahueet al. [7] have proposed to use LSTM [14], an advanced Recur- rent Neural Network for the same task. Very recently, Xu et al. [37] have proposed to integrate visual attention into the LSTM model in order to fix its gaze on different ob- jects during the generation of corresponding words. Neural language models have shown great prospects in generating human-like image captions. Most of these methods follow a similar encoding-decoding framework, except for the very recent method [37] which jointly learns visual attention and caption generation. However, [37] requires location sam-

pling both during training and testing, making the method more complicated. While they focus more on local infor- mation, our method rather exploits global cues. Overview. Our work belongs to the third family of caption generation methods which uses a neural language model to generate captions. Different from the above methods, how- ever, we propose to make use of the semantic information to guide the generation and propose an extension of LSTM model, coined gLSTM for the use of semantic informa- tion. The semantic information here denotes the correlation between image and its description, which is obtained in a similar manner as in transfer-based methods. Experiments illustrate that semantic information brings significant im- provement in the performance and our model outperforms recently proposed state-of-the-art methods [17, 36]. Inter- estingly, the proposed model is able to perform on par with the latest and unpublished state-of-the-art [37], despitetheir use of more complicated models that require location sam- pling during training and test stage.

3. Background

3.1. The LSTM Model

A Recurrent Neural Network (RNN) is a good choice to model temporal dynamics in sequences. However, it is difficult for traditional RNN to learn long-term dynam- ics because of the issue of vanishing and exploding gradi- ents [14]. The Long Short-Term Memory (LSTM) network is proposed in [14] to address these issues. The core of the LSTM architecture is the memory cell, which stores the state over time, and the gates, which control when and how to update the cell’s state. There are many variants with dif- ferent connections between the memory cell and the gates.

The LSTM block that our model is built on follows the LSTM with No Peepholes architecture [13], which is illus- trated in Figure 2 in black. The memory cell and gates in an LSTM block are defined as follows:

il = σ(Wixxl + Wimml−1) (1)

fl = σ(Wfxxl + Wfmml−1) (2)

ol = σ(Woxxl + Womml−1) (3)

cl = fl ⊙ cl−1 + il ⊙ h(Wcxxl + Wcmml−1) (4)

ml = ol ⊙ cl (5)

where⊙ represents the element-wise multiplication,σ(·) represents the sigmoid function andh(·) represents the hy- perbolic tangent function. The variableil stands for the in- put gate ,fl for the forget gate,ol for the output gate of the LSTM cell, cl is the state of the memory cell unit andml is the hidden state, that is the output of the block generated by the cell. The variablexl is the element of the sequence at timestepl andW[·][·] denote the parameters of the model.

Figure 2: The LSTM block in black, the proposed gLSTM network in black and red. Striped lines stand for external connections. By considering semantic information as an ex- tra input, we encourage the network to refresh its memory following a global guide.

3.2. Caption Generation with LSTM

The pipeline for caption generation with the RNN model [26, 17, 36, 7, 37] is inspired by the encoder-decoder principle in Neural Machine Translation [5, 32, 1]. An encoder is used to map a variable length sequence in the source language into a distributed vector and a decoder is used to generate a new sequence in the target language con- ditioned on this vector. During training, the goal is to maxi- mize the log-likelihood of correct translation given the sen- tence in the source language. When applying this principle to caption generation, the goal becomes to maximize the log-likelihood of the image caption given an image, namely

arg max θ

∑

i

log p(si1:Li|x i, θ), (6)

wherexi denotes an image,si 1:Li

denotes a sequence of words in a sentence of lengthLi andθ denotes the model parameters. For simplicity, in the following part we drop the superscripti whenever it is clear from the context. Since each sentence is composed of a sequence of words, it is nat- ural to use Bayes chain rule to decompose the likelihood of a sentence,

log p(s1:L|x, θ) = log p(s1|x, θ)+ L ∑

l=2

log p(sl|x, s1:l−1, θ),

(7)wheres1:l stands for the part of the sentence up to thel- th word. To maximize the objective in eq. (6) over the whole training corpus, we need to define the log-likelihood log p(sl|x, s1:l−1, θ), which can be modeled with the hid- den state of a timestep in RNN. The probability distribution

of the word at timestepl + 1 over the whole vocabulary is computed using the softmax functionz(·) based only on the outputml of the memory cell,pl+1 = z(ml) similar to [36].

To feed images and sentences to LSTM, they need to be encoded as fixed-length vectors. For the image, CNN features are first computed and then mapped to an embed- ding space via a linear transformation. For the sentence, each word is first represented as a one hot vector and then mapped to the same embedding space via a word embed- ding matrix. Finally, an image and sequence of words in a sentence are concatenated to form a new sequence, that is, the image is treated as the beginning symbol of the se- quence and the sequence of words forms the remaining part of the new sequence. This sequence is fed to the LSTM network for training by iterating the recurrence connection for l from 1 to Li. The parameters of the model include the linear transformation matrix for image features, the word embedding matrix and the parameters of LSTM.

3.3. Normalized Canonical Correlation Analysis

To build our semantic representation, we rely on nor- malized Canonical Correlation Analysis (CCA), proposed in [10] to address the cross-modal retrieval problem. Canonical correlation analysis (CCA) [16] is a popular method used to map visual and textual features into a com- mon semantic space. CCA aims at learning projection ma- tricesU1 andU2 for two viewsX1 andX2 such that their projections are maximally correlated, namely,

arg max U1,U2

U1ΣX1X2U2 √

U1ΣX1X1U1 √

U2ΣX2X2U2 , (8)

whereΣX1X2 , ΣX1X1 andΣX2X2 are the covariance ma- trices. The CCA objective function can be solved via gen- eralized eigenvalue decomposition. The normalized CCA is computed by using a power of the eigenvalues to weight the corresponding columns of the CCA projection matrices, and followed by L2 normalization, that is,

g1 = X1U1D

p

‖X1U1Dp‖ , g2 =

X2U2D p

‖X2U2Dp‖ (9)

whereD is a diagonal matrix whose elements are set to the eigenvalues of corresponding dimensions, whileg1 andg2 denote the semantic representation of the two views. Cosine similarity is used to find the nearest neighbor in the learned common semantic space [10].

4. The Proposed Methods

In this section, we describe the proposed extension of the LSTM model for the caption generation task. In the new ar- chitecture, we add semantic information to the computation of the gates and cell state. The semantic information here is extracted from images and their descriptions, serving asa guide in the process of word sequence generation.

4.1. gLSTM

The generation of a word in the LSTM model mainly depends on the word embedding at the current timstep and the previous hidden state (which includes image informa- tion at the beginning). This process goes step by step until it encounters the end token of a sentence. However, as this process continues, the role of the image information, which is only fed at the beginning, becomes weaker and weaker. Words generated at the beginning of a sequence also suffer from the same problem. Therefore, for a long sentence, it may carry out the generation almost “blindly” towards the end of the sentence. Though LSTM is able to keep long- term memory to some extent, still it poses a challenge for sentence generation [4, 1]. In the proposed model, the gen- eration of words is carried out under the guidance of global semantic information. Our extension of LSTM model is named gLSTM. The memory cell and gates in a gLSTM block are defined as follows:

i′l = σ(Wixxl + Wimml−1 + Wiqg) (10)

f ′

l = σ(Wfxxl + Wfmml−1 + Wfqg) (11)

o′l = σ(Woxxl + Womml−1 + Woqg) (12)

c ′

l = f ′

l ⊙ c ′

l−1 + i ′

l ⊙ h(Wcxxl +

+Wcmml−1 + Wcqg) (13)

ml = o ′

l ⊙ c ′

l (14)

whereg denotes the vector representation of semantic in- formation. Compared to the standard LSTM architecture, in gLSTM we add a new term to the computation of each gate and cell state. This new term represents the seman- tic information which works as a bridge between visual and textual domains. The semantic informationg does not de- pend on the timestepl, hence working as a global guide dur- ing the caption generation. The guidance term can also be timestep dependent in expense of higher complexity mod- els. We summarize with red the gLSTM network architec- ture additions in Figure 2.

4.2. Semantic Information.

In this section, we will detail several kinds of semantic information that can be used as guidance in our model. In- tuitively, there are three ways to extract the semantic infor- mation. First, we treat it as a cross-modal retrieval task and simply use the retrieved sentences as semantic information. Alternatively, semantic information can also be represented as the embedding in a semantic space where visual and tex- tual representations are equivalent. The last one is to use the image itself as guidance. Retrieval-based guidance (ret-gLSTM). The retrieval- based guidance is inspired by the transfer-based caption generation methods. Though the generated sentence given by transfer-based methods may not be totally correct, they

do have something in common with the true captions an- notated by humans. Given an image, we first do the cross- modal retrieval whose aim is to find relevant texts to the query image. Then we collect descriptions with top rank- ings. Instead of generating a sentence by making direct modification on these sentences, we treat these captions as auxiliary information and feed them to the neural language model we propose in the previous section. These sentences may not match perfectly to the image. However, they pro- vide rich semantic information for the image. Since these sentences are annotated by humans, the words in these sen- tences are very natural and have a high probability to appear in the reference captions.

The cross-modal retrieval method used here is based on the normalized CCA mentioned in Section 3.3. In this pa- per, image and text features correspond to the two views for CCA. CNN features are computed for the images and TF- IDF weighted BoW features are computed for the sentences. We project both images and sentences from their own do- main to the common semantic space. Given an image query, the closest sentences are then retrieved based on cosine sim- ilarity. We select the topT retrieved sentences from the training set (T = 15 in this paper). These sentences are represented by a bag-of-words (BoW) vector which is fed as extra input, i.e. the guide to the gLSTM model.

Semantic embedding guidance (emb-gLSTM).We can explicitly use the result of cross-modal retrieval as guid- ance as mentioned above. We can also implicitly use the intermediate result of cross-modal retrieval, that is the se- mantic representation computed using normalized CCA as the extra input. An image is mapped into the common se- mantic space by the learned projection matrix and the com- puted semantic embedding is fed to gLSTM model as the guide. It is assumed that in the common semantic space of CCA both views share equivalent embedding represen- tation. Therefore, we can treat the projected representation from image domain as equal to the one projected from text domain. Compared to ret-gLSTM model, the semantic rep- resentation has much lower dimensionality than the BoW representation and saves the computation of finding nearest neighbors. In addition, we also find it even performs better than the previous method.

Image as guidance (img-gLSTM).Finally, we experiment with the image itself as the extra input. This is motivated by the fact that CCA is a linear transformation. A natural question then is whether we can learn this projection matrix directly during the training of the gLSTM model. There- fore, we add the image itself as a third kind of extra input. We experimentally verify this by simply feeding the image feature itself to the gLSTM model, namelyg = x, and let the network learn the semantic information from scratch.

4.3. Beam Search with Length Normalization

In the generation stage, with a vocabulary of sizeK, there areKl sentences of lengthl as potential candidates for an image caption, wherel is unknown. Ideally, we want to find the sentence, which maximizes the log-likelihood of eq. (7). Considering the exponential search space, how- ever, exhaustive search is intractable. Therefore, a heuristic search strategy is employed instead.

Here we usebeam search, which is a fast and effective decoding method for RNN-based models [11, 32]. At each iteration only theT hypotheses generations with the highest log-likelihood are kept in the beam pool. The search along one beam stops once it encounters an end-of-sequence to- ken which is generated given previous words along the beam. The searching process continues until the searching along all beams in the pool stops.

It is problematic to directly use the log-likelihood of words as the criterion to select a generation. Since the log-likelihood of each single word is negative (because the probability is smaller than 1), summation over the log- likelihood of more words lead to a smaller value. There- fore, when the beam width is larger than 1, the beam search stops early is more likely to be selected as the final cap- tion, regardless of the quality of each generated word in the beam. That means this kind of beam search favors shorter sentence, which is also observed in [12, 4]

Interestingly, the bias towards short sentences tends to favor the low order of BLEU scores (BLEU@1,2), com- monly used to evaluate machine translation algorithms. Hence, short sentences not only tend to dominate the in- ference, but also obscure the evaluations and methodol- ogy comparisons. To remedy the bias towards short sen- tences during inference, we propose to normalize the log- likelihood of words by length, namely

p = 1

Ω(ℓ)

ℓ ∑

l=1

log p(sl|x, s1:l, θ) (15)

We investigate various forms forΩ to do the normalization.

Polynomial normalization. A first possibility is to set Ω(ℓ) = |ℓ|m. Notice that whenm = 1, eq. (15) becomes the definition of the perplexity. We usem = 1 in our paper. This kind of normalization punishes short sentences.

Min-hinge normalization. Intuitively we want to auto- matically generate a sentence whose length is close to the ground truth. Since in the test stage we do not know the length in advance, we use the average length of the sen- tences in the training data as a reference. We define the min- hinge length function asΩ(ℓ) = min{ℓ, µ}. This means a generated sentence is only punished when it is shorter than the average lengthµ. For sentences that are long enough, we only pay attention to their log-likelihood.

Max-hinge normalization. Similarly, we define the max- hinge length function ,Ω(ℓ) = max{ℓ, µ}. Instead of pe- nalizing short sentences, the max-hinge function favors long sentences.

Gaussian normalization. We can also employ a Gaussian function,Ω(ℓ) ∼ N(µ, σ) to normalize the loglikelihood, where theµ andσ are the mean and the standard deviation of the sentence lengths in the training corpus. The Gaussian regularization encourages the inference to select sentences that have similar lengths as the sentences in the training set.

We experimentally verify the effectiveness of these strategies in Section 5.1.

5. Experiments

Datasets and experimental setup. We perform experi- ments on the following datasets. Flickr8k [15], Flickr30k [39] and MS COCO [25]. The Flickr8k dataset is a popular dataset composed of8, 000 images in total collected from Flickr, divided into a train- ing, validation and test set of6, 000, 1, 000 and1, 000 im- ages respectively. Each image in the dataset is accompanied with 5 reference captions annotated by humans. Similar to Flickr8k, the Flickr30k dataset contains31, 000 images col- lected from Flickr, together with 5 reference sentences pro- vided by human annotators. However, it does not provide a split setting file. So we use the publicly available split setting used in [18] and [17], that is,29, 000 images for training, 1, 000 for validation and1, 000 for testing. The large scale dataset MSCOCO contains 82,783 images for training and 40504 for validation, with each image associ- ated with 5 captions. Note that we donot evaluate it on the test set used for MS COCO Image Captioning challenge but use the publicly available splits used in previous work [17], that is, all 82,783 images from training set for training and 5,000 images from validation set for validation and testing. Evaluation measures. Here we use the two most popu- lar measures in the machine translation and image caption generation literature, namely theBLEU [29] and theME- TEOR [6] measure.

BLEU is a precision-based metric. The main component of BLEU is n-gram precision of the generated caption with respect to the references. Precision is computed separately for each n-gram and then B@n is computed as a geomet- ric mean of these precisions.BLEU of high order n-gram indirectly measures the grammatical coherence.

However, BLEU is criticized to favor short sentences. It only considers precision but does not take recall into consideration. For this reasonMETEOR is also reported in recent works [3, 8, 37].METEOR evaluates a generated sentence by computing a score based on word level matches between the generation and a reference and returning the maximum score over a set of references. In the computa- tion of the matching score, it considers unigram-precision,

unigram-recall and a measure of alignment. Hence,ME- TEOR accounts for precision, recall and the importance of grammaticality. In user evaluation studiesMETEOR [24] has been shown to have a higher correlation with human judgments than any order ofBLEU.

All scores are computed using the coco-caption code1. Implementation details. In the following experiments we use the MatConvNet toolbox [35] and the 16-layer Oxford- Net [30] pretrained model to computeCNN features and extract the last fully-connected layer’s output as image rep- resentation. As for preprocessing of texts, for the neural language model, we use the publicly available data where texts are converted to lowercase, non-alphanumeric charac- ters are ignored and only words appearing at least 5 times in the training set are kept to create a vocabulary. For CCA, we use the NLTK toolbox[2] to further lemmatize words and build a vocabulary based on these words (3000 words for flickr8k and 5000 for flickr30k and MS COCO). Then tf-idf-weighted BoW vectors are computed as sentence rep- resentation for CCA. For Flickr8k and Flickr30K we set the number of dimensions for the image and word embeddings and the hidden layer of the gLSTM to 256. For MSCOCO we set the number to 512 (note that this is much smaller than the one used in other work). The gLSTM Models are trained with RMSProp [34], which is a stochastic gradient descent method using an adaptive learning rate algorithm. The learning rate is initialized with 1e-4 for Flickr8k and Flickr30k and 4e-4 for MS COCO. We use dropout and early stopping to avoid overfitting and use validation set log-likelihood for model selection. For CCA, we setp = 4 as suggested in [10] and the dimension of the common space to 200 for Flickr8k and Flickr30k, and 500 for MS COCO which we find works well in practice. At the test stage, we set the beam size to 10 for all experiments. We built our code for the proposed gLSTM model on Karpa- thy’s NeuralTalk code2, which implements the single model in Google’s paper [36]. Note that we take that model as the baseline.

5.1. Length Normalization

In this experiment we evaluate the importance of the sen- tence length normalization to caption generation. We carry out the experiment on the Flickr8k dataset and report the results in Table 1. For clarity we perform this experiment based on the LSTM baseline, not gLSTM.

We observe that compared to the baseline whose selec- tion is based on unnormalized log-likelihood, length nor- malization has a positive effect on either theBLEU metric or METEOR metric. Polynomial, min-hinge and Gaussian normalization respectively bring the largest improvement

1https://github.com/tylin/coco-caption 2https://github.com/karpathy/neuraltalk

Figure 3: Results of the LSTM and gLSTM model. We mark the generated sentence by LSTM and gLSTM respectively in green and red, the ground truth references in black and the most relevant retrieval results in blue. We observe that the retrieval results are helpful to caption generation. Notice that for the third example, the result of our model is not that accurately but still much better than the one of the LSTM model.

Normalization B@1 B@2 B@3 B@4 METEOR

Baseline 59.6 40.4 26.1 17.0 17.45 Polynomial 57.8 39.2 26.0 17.6 18.86 Min-hinge 60.4 41.4 27.6 18.6 18.53 Max-hinge 57.6 38.8 25.2 16.7 17.65 Gaussian 60.7 41.7 27.8 18.6 18.35

Table 1: The performance of different length normalization strategies on Flickr8k.

GT Refs Baseline Polynom. Min-hinge Max-hinge Gaussian

10.87(3.74) 8.75(2.44) 11.07(2.62) 9.64(1.92) 9.55(1.69) 9.57(3.30)

Table 2: The average and the standard deviation of the sen- tence length for the ground truth references, and different normalization strategies on Flickr8k.

to METEOR and BLEU. Therefore, in the following ex- periments, we only report the performance of the proposed gLSTM with these three kinds of length normalization. Be- sides, we also compute the average length of generated sen- tences and references.

5.2. gLSTM with Different Types of Guidance

In this experiment we evaluate the gLSTM model with different types of semantic information, as described in Sec- tion 4.2. For fair comparison, we also apply beam search with length normalization to the baseline. We run this ex- periment on Flickr8k and report the results in Table 3.

B@1 B@2 B@3 B@4 METEOR

Baseline, Original 59.6 40.4 26.1 17.0 17.45 Baseline, Polynomial 57.8 39.2 26.0 17.6 18.86 Baseline, Min-hinge 60.4 41.4 27.6 18.6 18.53 Baseline, Gaussian 60.7 41.7 27.8 18.6 18.35

Baseline 512, Original 61.0 42.4 28.6 18.9 18.21 Baseline 512, Polynomial 58.2 40.2 27.1 18.1 19.83 Baseline 512, Min-hinge 61.3 42.9 29.2 19.6 19.13 Baseline 512, Gaussian 61.3 42.8 29.1 19.5 19.07

ret-gLSTM, Original 63.4 43.7 29.2 19.3 18.54 ret-gLSTM, Polynomial 58.8 40.4 27.5 18.6 19.86 ret-gLSTM, Min-hinge 63.0 43.8 29.9 20.2 19.46 ret-gLSTM, Gaussian 63.5 44.2 30.2 20.6 19.38

emb-gLSTM, Original 63.7 44.7 30.2 20.2 19.10 emb-gLSTM, Polynomial 61.0 43.0 29.6 20.1 20.60 emb-gLSTM, Min-hinge 64.3 45.7 31.6 21.5 20.28 emb-gLSTM, Gaussian 64.7 45.9 31.8 21.6 20.19

img-gLSTM, Original 61.5 42.5 27.2 16.7 17.10 img-gLSTM, Polynomial 55.7 38.1 24.9 15.8 17.69 img-gLSTM, Min-hinge 60.4 41.9 27.6 17.7 17.76 img-gLSTM, Gaussian 60.1 41.4 27.2 17.3 17.69

Table 3: Comparison between gLSTM with different se- mantic information on Flickr8k. We denote the gLSTM model with retrieval-based guidance as ret-gLSTM, the one with semantic embedding guidance as emb-gLSTM, and the one with image as guidance as img-gLSTM.

The result illustrates that semantic information brings much improvement in the performance, especially emb-

Flickr8k Flickr30k

B@1 B@2 B@3 B@4 METEOR B@1 B@2 B@3 B@4 METEOR

LogBilinear [19] 65.6 42.4 27.7 17.7 17.31 60.0 38.- 25.4 17.1 16.88 multimodal RNN [17] 57.9 38.3 24.5 16.0 16.7 57.3 36.9 24.0 15.7 15.3 Google NIC [36] 63.- 41.- 27.- — — 66.3 42.3 27.7 18.3 — LRCN-CaffeNet [7] — — — — — 58.7 39.1 25.1 16.5 — m-RNN-AlexNet [26] — — — — — 54.- 36.- 23.- 15.- — m-RNN [26] — — — — — 60.- 41.- 28.- 19.- — Soft-Attention [37] 67.- 44.8 29.9 19.5 18.93 66.7 43.4 28.8 19.1 18.49 Hard-Attention [37] 67.- 45.7 31.4 21.3 20.3 66.9 43.9 29.6 19.9 18.46

emb-gLSTM, Polynomial 61.0 43.0 29.6 20.1 20.60 59.8 41.3 29.3 19.2 18.58 emb-gLSTM, Min-hinge 64.3 45.7 31.6 21.5 20.28 63.8 44.1 30.2 20.5 18.13 emb-gLSTM, Gaussian 64.7 45.9 31.8 21.6 20.19 64.6 44.6 30.5 20.6 17.91

Table 4: Comparison with state-of-the-art methods on Flickr8k and Flickr30k.

gLSTM, the gLSTM with semantic embedding guidance. We also observe that img-gLSTM, the gLSTM with the im- age itself as guidance, does not bring any improvement but even deteriorates the performance. Besides, we also con- duct an experiment for a baseline but with more parameters (512 dimension instead of 256 dimension) for each gate to emphasize the improvement mainly comes from the global guide. The total number of network parameters is 5.2M in total compared to 5.9M and 3.1M for the proposed gLSTM ret-gLSTM and B. As is shown in Table 3, we can see that increasing parameters indeed improves the performance, but still a little worse than the proposed emb-gLSTM even though it has much fewer parameters.

5.3. Comparison with State-of-the-art methods

We compare the proposed gLSTM with state-of-the-art methods for caption generation in the literature. We per- form the experiment on Flickr8k and Flickr30k and report the results in Table 4. We only evaluate emb-gLSTM in this experiment, since it is computationally efficient and ob- tains the best performance among different models in the previous experiment. For most evaluated methods, they use CNN with deeper network architecture such as Oxford- Net [30] and GoogLeNet [33]. Methods which do not use a deeper CNN include LRCN-CaffeNet [7] and m-RNN- AlexNet [26]. Note that Google’s method [36] uses an en- semble of multiple LSTM models, while ours only uses a single emb-gLSTM model. We can see from the table, the proposed emb-gLSTM model performs favorably against state-of-the-art approaches. Interestingly, it performseven on par with the latest state-of-the-art [37], which is based on more complicated and expensive attention mechanisms.

6. Conclusion

In this work we have proposed an extension of the LSTM model for image caption generation. By adding seman- tic information as extra input to each unit of the LSTM block, we have shown that the model can better stay “on

B@1 B@2 B@3 B@4 METEOR mCIDEr

multimodal RNN [17] 62.5 45.0 32.1 23.0 19.5 66 Google NIC [36] 66.6 46.1 32.9 24.6 — — LRCN-CaffeNet [7] 62.8 44.2 30.4 — — — m-RNN [26] 67 49 35 25 — — Soft-Attention [37] 70.7 49.2 34.4 24.3 23.9 — Hard-Attention [37] 71.8 50.4 35.7 25.0 23.04 —

emb-gLSTM, Polynomial 63.8 46.3 33.6 24.8 23.33 79.03 emb-gLSTM, Min-hinge 66.3 48.5 35.4 26.2 22.95 81.26 emb-gLSTM, Gaussian 67.0 49.1 35.8 26.4 22.74 81.25

Table 5: Comparison with state-of-the-art methods on MS COCO.

track”, describing the image content without drifting away to unrelated yet common phrases. In addition, we explore different types of length normalization for beam search in order to prevent a bias towards very short sentences, which further improves the results. The proposed method achieves state-of-the-art performance on various benchmark datasets. Moreover, our key contributions are, to a large ex- tent, complementary to key aspects of other methods, uch as attention mechanisms [37] or model ensembles [36], in- dicating that further improvements on performance may be obtained by integrating these schemes.

7. Acknowledgment

The authors acknowledge the support of the IWT-SBO project PARIS.

References

[1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine trans- lation by jointly learning to align and translate. InICLR, 2015.

[2] S. Bird, E. Klein, and E. Loper.Natural Language Process- ing with Python. O’Reilly, 2009.

[3] X. Chen and C. L. Zitnick. Mind’s eye:a recurrent visual representation for image caption generation. InCVPR, 2015.

[4] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-

decoder approaches. InEighth Workshop on Syntax, Seman- tics and Structure in Statistical Translation (SSST-8), 2014.

[5] K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. InEMNLP, 2014.

[6] M. Denkowski and A. Lavie. Meteor universal: Language specific translation evaluation for any target language. In EACL 2014 Workshop on Statistical Machine Translation, 2014.

[7] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recur- rent convolutional networks for visual recognition and de- scription. InCVPR, 2015.

[8] H. Fang, S. Gupta, F. N. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zit- nick, and G. Zweig. From captions to visual concepts and back. InCVPR, 2015.

[9] A. Farhadi, S. M. M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. A. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV (4), 2010.

[10] Y. Gong, Q. Ke, M. Isard, and S. Lazebnik. A multi-view em- bedding space for modeling internet images, tags, and their semantics.IJCV, 106(2):210–233, 2014.

[11] A. Graves. Sequence transduction with recurrent neural net- works. CoRR, abs/1211.3711, 2012.

[12] A. Graves. Generating sequences with recurrent neuralnet- works. CoRR, abs/1308.0850, 2013.

[13] K. Greff, R. K. Srivastava, J. Koutnı́k, B. R. Steunebrink, and J. Schmidhuber. LSTM: A search space odyssey.CoRR, abs/1503.04069, 2015.

[14] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735–1780, 1997.

[15] M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics.JAIR, 47:853–899, 2013.

[16] H. Hotelling. Relations between two sets of variates. Biometrika, pages 321–377, 1936.

[17] A. Karpathy and L. Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. InCVPR, 2015.

[18] A. Karpathy, A. Joulin, and F. Li. Deep fragment embed- dings for bidirectional image sentence mapping. InNIPS, 2014.

[19] R. Kiros, R. Salakhutdinov, and R. S. Zemel. Multimodal neural language models. InICML, 2014.

[20] G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Babytalk: Understanding and generating simple image descriptions.TPAMI, 35(12):2891– 2903, 2013.

[21] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Collective generation of natural image descriptions. In ACL, 2012.

[22] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi. Generalizing image captions for image-text parallel corpus. InACL, 2013.

[23] P. Kuznetsova, V. Ordonez, T. Berg, and Y. Choi. Treetalk: Composition and compression of trees for image descrip- tions. TACL, 2:351–362, 2014.

[24] A. Lavie and A. Agarwal. Meteor: An automatic metric for mt evaluation with high levels of correlation with hu- man judgments. InSecond Workshop on Statistical Machine Translation, 2007.

[25] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Dollár, and C. L. Zitnick. Microsoft COCO: com- mon objects in context. InECCV, 2014.

[26] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Deep captioning with multimodal recurrent neural networks (m- rnn). In ICLR, 2015.

[27] R. Mason and E. Charniak. Nonparametric method for data- driven image captioning. InACL, 2014.

[28] M. Mitchell, J. Dodge, A. Goyal, K. Yamaguchi, K. Stratos, X. Han, A. Mensch, A. C. Berg, T. L. Berg, and H. D. III. Midge: Generating image descriptions from computer vision detections. InEACL, 2012.

[29] K. Papineni, S. Roukos, T. Ward, and W. Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.

[30] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. InICLR, 2015.

[31] I. Sutskever, J. Martens, and G. Hinton. Generating text with recurrent neural networks. InICML, 2011.

[32] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. InNIPS, 2014.

[33] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. InCVPR, 2014.

[34] T. Tieleman and G. Hinton. Leccture 6.5 - rmsprop. Techni- cal Report MSU-CSE-00-2, 2000.

[35] A. Vedaldi and K. Lenc. Matconvnet – convolutional neural networks for matlab.CoRR, abs/1412.4564, 2014.

[36] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. InCVPR, 2015.

[37] K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhut- dinov, R. S. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.

[38] Y. Yang, C. L. Teo, H. D. III, and Y. Aloimonos. Corpus- guided sentence generation of natural images. InEMNLP, 2011.

[39] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From im- age descriptions to visual denotations: New similarity met- rics for semantic inference over event descriptions.TACL, 2:67–78, 2014.