Learning a Recurrent Visual Representation for Image Caption Generation

Introduction

A good image description is often said to “paint a picture in your mind’s eye.” The creation of a mental image may play a significant role in sentence comprehension in humans . In fact, it is often this mental image that is remembered long after the exact sentence is forgotten . What role should visual memory play in computer vision algorithms that comprehend and generate image descriptions?

Recently, several papers have explored learning joint feature spaces for images and their descriptions . These approaches project image features and sentence features into a common space, which may be used for image search or for ranking image captions. Various approaches were used to learn the projection, including Kernel Canonical Correlation Analysis (KCCA) , recursive neural networks , or deep neural networks . While these approaches project both semantics and visual features to a common embedding, they are not able to perform the inverse projection. That is, they cannot generate novel sentences or visual depictions from the embedding.

In this paper, we propose a bi-directional representation capable of generating both novel descriptions from images and visual representations from descriptions. Critical to both of these tasks is a novel representation that dynamically captures the visual aspects of the scene that have already been described. That is, as a word is generated or read the visual representation is updated to reflect the new information contained in the word. We accomplish this using Recurrent Neural Networks (RNNs) . One long-standing problem of RNNs is their weakness in remembering concepts after a few iterations of recurrence. For instance RNN language models often find difficultly in learning long distance relations without specialized gating units . During sentence generation, our novel dynamically updated visual representation acts as a long-term memory of the concepts that have already been mentioned. This allows the network to automatically pick salient concepts to convey that have yet to be spoken. As we demonstrate, the same representation may be used to create a visual representation of a written description.

We demonstrate our method on numerous datasets. These include the PASCAL sentence dataset , Flickr 8K , Flickr 30K , and the Microsoft COCO dataset . When generating novel image descriptions, we demonstrate state-of-the-art results as measured by both BLEU and METEOR on PASCAL 1K. Surprisingly, we achieve performance only slightly below humans as measured by BLEU and METEOR on the MS COCO dataset. Qualitative results are shown for the generation of novel image captions. We also evaluate the bi-directional ability of our algorithm on both the image and sentence retrieval tasks. Since this does not require the ability to generate novel sentences, numerous previous papers have evaluated on this task. We show results that are better or comparable to previous state-of-the-art results using similar visual features.

Related work

The task of building a visual memory lies at the heart of two long-standing AI-hard problems: grounding natural language symbols to the physical world and semantically understanding the content of an image. Whereas learning the mapping between image patches and single text labels remains a popular topic in computer vision , there is a growing interest in using entire sentence descriptions together with pixels to learn joint embeddings . Viewing corresponding text and images as correlated, KCCA is a natural option to discover the shared features spaces. However, given the highly non-linear mapping between the two, finding a generic distance metric based on shallow representations can be extremely difficult. Recent papers seek better objective functions that directly optimize the ranking , or directly adopts pre-trained representations to simplify the learning, or a combination of the two .

With a good distance metric, it is possible to perform tasks like bi-directional image-sentence retrieval. However, in many scenarios it is also desired to generate novel image descriptions and to hallucinate a scene given a sentence description. Numerous papers have explored the area of generating novel image descriptions . These papers use various approaches to generate text, such as using pre-trained object detectors with template-based sentence generation . Retrieved sentences may be combined to form novel descriptions . Recently, purely statistical models have been used to generate sentences based on sampling or recurrent neural networks . While also uses a RNN, their model is significantly different from our model. Specifically their RNN does not attempt to reconstruct the visual features, and is more similar to the contextual RNN of . For the synthesizing of images from sentences, the recent paper by Zitnick et al. uses abstract clip art images to learn the visual interpretation of sentences. Relation tuples are extracted from the sentences and a conditional random field is used to model the visual scene.

There are numerous papers using recurrent neural networks for language modeling . We build most directly on top of that use RNNs to learn word context. Several models use other sources of contextual information to help inform the language model . Despite its success, RNNs still have difficulty capturing long-range relationships in sequential modeling . One solution is Long Short-Term Memory (LSTM) networks , which use “gates” to control gradient back-propagation explicitly and allow for the learning of long-term interactions. However, the main focus of this paper is to show that the hidden layers learned by “translating” between multiple modalities can already discover rich structures in the data and learn long distance relations in an automatic, data-driven manner.

Approach

In this section we describe our approach using recurrent neural networks. Our goals are twofold. First, we want to be able to generate sentences given a set of visual observations or features. Specifically, we want to compute the probability of a word $w_{t}$ being generated at time $t$ given the set of previously generated words $W_{t-1}={w_{1},\ldots,w_{t-1}}$ and the observed visual features $V$. Second, we want to enable the capability of computing the likelihood of the visual features $V$ given a set of spoken or read words $W_{t}$ for generating visual representations of the scene or for performing image search. To accomplish both of these tasks we introduce a set of latent variables $U_{t-1}$ that encodes the visual interpretation of the previously generated or read words $W_{t-1}$. As we demonstrate later, the latent variables $U$ play the critical role of acting as a long-term visual memory of the words that have been previously generated or read.

Using $U$, our goal is to compute $P(w_{t}|V,W_{t-1},U_{t-1})$ and $P(V|W_{t-1},U_{t-1})$. Combining these two likelihoods together our global objective is to maximize,

That is, we want to maximize the likelihood of the word $w_{t}$ and the observed visual features $V$ given the previous words and their visual interpretation. Note that in previous papers the objective was only to compute $P(w_{t}|V,W_{t-1})$ and not $P(V|W_{t-1})$.

A critical property of the recurrent visual features $\mathbf{u}$ is their ability to remember visual concepts over the long term. The property arises from the model structure. Intuitively, one may expect the visual features shouldn’t be estimated until the sentence is finished. That is, $\mathbf{u}$ should not be used to estimate $\mathbf{v}$ until $\mathbf{w}_{t}$ generates the end of sentence token. However, in our model we force $\mathbf{u}$ to estimate $\mathbf{v}$ at every time step to help in remembering visual concepts. For instance, if the word “cat” is generated, $\mathbf{u}_{t}$ will increase the likelihood of the visual feature corresponding to cat. Assuming the “cat” visual feature in $\mathbf{v}$ is active, the network will receive positive reinforcement to propagate $\mathbf{u}$’s memory of “cat” from one time instance to the next. Figure 2 shows an illustrative example of the hidden units $\mathbf{s}$ and $\mathbf{u}$. As can be observed, some visual hidden units $\mathbf{u}$ exhibit longer temporal stability.

2 Implementation details

In this section we describe the details of our language model and how we learn our network.

3 Language Model

Our language model typically has between 3,000 and 20,000 words. While each word may be predicted independently, this approach is computationally expensive. Instead, we adopted the idea of word classing and factorized the distribution into a product of two terms:

$P(w_{t}|\cdot)$ is the probability of the word, $P(c_{t}|\cdot)$ is the probability of the class. The class label of the word is computed in an unsupervised manner, grouping words of similar frequencies together. Generally, this approach greatly accelerates the learning process, with little loss of perplexity. The predicted word likelihoods are computed using the standard soft-max function. After each epoch, the perplexity is evaluated on a separate validation set and the learning reduced (cut in half in our experiments) if perplexity does not decrease.

In order to further reduce the perplexity, we combine the RNN model’s output with the output from a Maximum Entropy language model , simultaneously learned from the training corpus. For all experiments we fix how many words to look back when predicting the next word used by the Maximum Entropy model to three.

For any natural language processing task, pre-processing is crucial to the final performance. For all the sentences, we did the following two steps before feeding them into the RNN model. 1) Use Stanford CoreNLP Tool to tokenize the sentences. 2) Lower case all the letters.

4 Learning

We used the open source RNN code of and the Caffe framework to implement our model. A big advantage of combining the two is that we can jointly learn the word and image representations: the error from predicting the words can be directly backpropagated to the image-level features. However, deep convolution neural networks require large amounts of data to train on, but the largest sentence-image dataset has only $\sim$80K images . Therefore, instead of training from scratch, we choose to fine-tune from the pre-trained 1000-class ImageNet model to avoid potential over-fitting. In all experiments, we used the 4096D 7th full-connected layer output as the visual input $\mathbf{v}$ to our model.

Results

In this section we evaluate the effectiveness of our bi-directional RNN model on multiple tasks. We begin by describing the datasets used for training and testing, followed by our baselines. Our first set of evaluations measure our model’s ability to generate novel descriptions of images. Since our model is bi-directional, we evaluate its performance on both the sentence retrieval and image retrieval tasks. For addition results please see the supplementary material.

For evaluation we perform experiments on several standard datasets that are used for sentence generation and the sentence-image retrieval task:

The dataset contains a subset of images from the PASCAL VOC challenge. For each of the 20 categories, it has a random sample of 50 images with 5 descriptions provided by Amazon’s Mechanical Turk (AMT).

Flickr 8K and 30K [31]

These datasets consists of 8,000 and 31,783 images collected from Flickr respectively. Most of the images depict humans participating in various activities. Each image is also paired with 5 sentences. These datasets have a standard training, validation, and testing splits.

MS COCO [22]

The Microsoft COCO dataset contains 82,783 training images and 40,504 validation images each with $\sim$5 human generated descriptions. The images are collected from Flickr by searching for common object categories, and typically contain multiple objects with significant contextual information. We downloaded the version which contains $\sim$40K annotated training images and $\sim$10K validation images for our experiments.

2 RNN Baselines

To gain insight into the various components of our model, we compared our final model with three RNN baselines. For fair comparison, the random seed initialization was fixed for all experiments. The the hidden layers $\mathbf{s}$ and $\mathbf{u}$ sizes are fixed to 100. We tried increasing the number of hidden units, but results did not improve. For small datasets, more units can lead to overfitting.

This is the basic RNN language model developed by , which has no input visual features.

RNN with Image Features (RNN+IF)

RNN with Image Features Fine-Tuned (RNN+FT)

This model has the same architecture as RNN+IF, but the error is back-propagated to the Convolution Neural Network . The CNN is initialized with the weights from the BVLC reference net. The RNN is initialized with the the pre-trained RNN language model. That is, the only randomly initialized weights are the ones from visual features $\mathbf{v}$ to hidden layers $\mathbf{s}$. If the RNN is not pre-trained we found the initial gradients to be too noisy for the CNN. If the weights from $\mathbf{v}$ to hidden layers $\mathbf{s}$ are also pre-trained the search space becomes too limited. Our current implementation takes $\sim$5 seconds to learn a mini-batch of size 128 on a Tesla K40 GPU. It is also crucial to keep track of the validation error and avoid overfitting. We observed this fine-tuning strategy is particularly helpful for MS COCO, but does not give much performance gain on Flickr Datasets before it overfits. The Flickr datasets may not provide enough training data to avoid overfitting.

After fine-tuning, we fix the image features again and retrain our model on top of it.

3 Sentence generation

Our first set of experiments evaluate our model’s ability to generate novel sentence descriptions of images. We experiment on all the image-sentence datasets described previously and compare to the RNN baselines and other previous papers . Since PASCAL 1K has a limited amount of training data, we report results trained on MS COCO and tested on PASCAL 1K. We use the standard train-test splits for the Flickr 8K and 30K datasets. For MS COCO we train and validate on the training set ($\sim$37K/$\sim$3K), and test on the validation set, since the testing set is not available. To generate a sentence, we first sample a target sentence length from the multinomial distribution of lengths learned from the training data, then for this fixed length we sample 100 random sentences, and use the one with the lowest loss (negative likelihood, and in case of our model, also reconstruction error) as output.

We choose three automatic metrics for evaluating the quality of the generated sentences, perplexity, BLEU and METEOR . Perplexity measures the likelihood of generating the testing sentence based on the number of bits it would take to encode it. The lower the value the better. BLEU and METEOR were originally designed for automatic machine translation where they rate the quality of a translated sentences given several references sentences. We can treat the sentence generation task as the “translation” of images to sentences. For BLEU, we took the geometric mean of the scores from 1-gram to 4-gram, and used the ground truth length closest to the generated sentence to penalize brevity. For METEOR, we used the latest versionhttp://www.cs.cmu.edu/~alavie/METEOR/ (v1.5). For both BLEU and METEOR higher scores are better. For reference, we also report the consistency between human annotators (using 1 sentence as query and the rest as references)We used 5 sentences as references for system evaluation, but leave out 4 sentences for human consistency. It is a bit unfair but the difference is usually around 0.01$\sim$0.02..

Results are shown in Table 1. Our approach significantly improves over both Midge and BabyTalk on the PASCAL 1K dataset as measured by BLEU and METEOR. Several qualitative results for the three algorithms are shown in Figure 4. Our approach generally provides more naturally descriptive sentences, such as mentioning an image is black and white, or a bus is a “double decker”. Midge’s descriptions are often shorter with less detail and BabyTalk provides long, but often redundant descriptions. Results on Flickr 8K and Flickr 30K are also provided.

On the MS COCO dataset that contains more images of high complexity we provide perplexity, BLEU and METEOR scores. Surprisingly our BLEU and METEOR scores (18.99 & 20.42) are just slightly lower than the human scores (20.19 & 24.94). The use of image features (RNN + IF) significantly improves performance over using just an RNN language model. Fine-tuning (FT) and our full approach provide additional improvements for all datasets. For future reference, our final model gives BLEU-1 to BLEU-4 (with penalty) as 60.4%, 26.4%, 12.6% and 6.5%, compared to human consistency 65.9%, 30.5%, 13.6% and 6.0%. Qualitative results for the MS COCO dataset are shown in Figure 3. Note that since our model is trained on MS COCO, the generated sentences are generally better on MS COCO than PASCAL 1K.

It is known that automatic measures are only roughly correlated with human judgment , so it is also important to evaluate the generated sentences using human studies. We evaluated 1000 generated sentences on MS COCO by asking human subjects to judge whether it had better, worse or same quality to a human generated ground truth caption. 5 subjects were asked to rate each image, and the majority vote was recorded. In the case of a tie (2-2-1) the two winners each got half of a vote. We find $12.6\%$ and $19.8\%$ prefer our automatically generated captions to the human captions without (Our Approach) and with fine-tuning (Our Approach + FT) respectively. Less than $1\%$ of the subjects rated the captions as the same. This is an impressive result given we only used image-level visual features for the complex images in MS COCO.

4 Bi-Directional Retrieval

Our RNN model is bi-directional. That is, it can generate image features from sentences and sentences from image features. To evaluate its ability to do both, we measure its performance on two retrieval tasks. We retrieve images given a sentence description, and we retrieve a description given an image. Since most previous methods are capable of only the retrieval task, this also helps provide experimental comparison.

Following other methods, we adopted two protocols for using multiple image descriptions. The first one is to treat each of the $\sim$5 sentences individually. In this scenario, the rank of the retrieved ground truth sentences are used for evaluation. In the second case, we treat all the sentences as a single annotation, and concatenate them together for retrieval.

The same evaluation metrics were adopted from previous papers for both the tasks of sentence retrieval and image retrieval. They used R@K (K = 1, 5, 10) as the measurements, which are the recall rates of the (first) ground truth sentences (sentence retrieval task) or images (image retrieval task). Higher R@K corresponds to better retrieval performance. We also report the median/mean rank of the (first) retrieved ground truth sentences or images (Med/Mean r). Lower Med/Mean $r$ implies better performance. For Flickr 8K and 30K several different evaluation methodologies have been proposed. We report three scores for Flickr 8K corresponding to the methodologies proposed by , and respectively, and for Flickr 30K and .

Measured by Mean r, we achieve state-of-the-art results on PASCAL 1K image and sentence retrieval (Table 2). As shown in Tables 3 and 4, for Flickr 8K and 30K our approach achieves comparable or better results than all methods except for the recently proposed DeepFE . However, DeepFE uses a different set of features based on smaller image regions. If the same features are used (DeepFE+DECAF) as our approach, we achieve better results. We believe these contributions are complementary, and by using better features our approach may also show further improvement. In general ranking based on text and visual features (T + I) outperforms just using text (T). Please see the supplementary material for retrieval results on MS COCO.

Discussion

Image captions describe both the objects in the image and their relationships. An area of future work is to examine the sequential exploration of an image and how it relates to image descriptions. Many words correspond to spatial relations that our current model has difficultly in detecting. As demonstrated by the recent paper of better feature localization in the image can greatly improve the performance of retrieval tasks and similar improvement might be seen in the description generation task.

In conclusion, we describe the first bi-directional model capable of the generating both novel image descriptions and visual features. Unlike many previous approaches using RNNs, our model is capable of learning long-term interactions. This arises from using a recurrent visual memory that learns to reconstruct the visual features as new words are read or generated. We demonstrate state-of-the-art results on the task of sentence generation, image retrieval and sentence retrieval on numerous datasets.

Acknowledgements

We thank Hao Fang, Saurabh Gupta, Meg Mitchell, Xiaodong He, Geoff Zweig, John Platt and Piotr Dollar for their thoughtful and insightful discussions in the creation of this paper.

References