Source-linked AI summary
Learning a Recurrent Visual Representation for Image Caption Generation
Xinlei Chen, C. Lawrence Zitnick
TL;DR
The paper addresses the limitation that common image–sentence embeddings do not generate novel sentences or visual depictions. It learns a bi-directional recurrent representation with visual memory, achieving state-of-the-art caption generation and strong retrieval results across datasets.
Problem
Prior common-embedding approaches support retrieval or ranking but cannot generate novel sentences or visual depictions from the embedding.
Method
A recurrent neural network uses a dynamically updated visual representation to generate sentences from images and reconstruct visual features from descriptions.
Results
The method achieves state-of-the-art sentence-generation results and better or comparable retrieval results to prior methods using similar visual features.
Takeaways & Limitations
The same recurrent visual-memory representation supports both image-to-sentence generation and sentence-to-visual-feature reconstruction.
Abstract
from arXiv · showhide
In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. We propose learning this mapping using a recurrent neural network. Unlike previous approaches that map both sentences and images to a common embedding, we enable the generation of novel sentences given an image. Using the same model, we can also reconstruct the visual features associated with an image given its visual description. We use a novel recurrent visual memory that automatically learns to remember long-term visual concepts to aid in both sentence generation and visual feature reconstruction. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are preferred by humans over $19.8\%$ of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.
1. Introduction
The paper asks how visual memory should support computer vision systems that comprehend and generate image descriptions. It contrasts this goal with prior methods that learn joint image–sentence feature spaces.
- Visual memory may support sentence comprehension because mental images can outlast exact sentence memory.
- Prior approaches project image and sentence features into a common space for image search or caption ranking.They use methods including KCCA, recursive neural networks, and deep neural networks.
C. Lawrence Zitnick Microsoft Research, Redmond
The paper proposes a bi-directional recurrent representation that generates descriptions from images and visual representations from descriptions. A dynamically updated visual representation serves as long-term memory during both directions.
- Earlier common-embedding approaches cannot generate novel sentences or visual depictions from the embedding.
- The proposed representation generates novel descriptions from images and visual representations from descriptions.It updates the visual representation as words are generated or read.
- The recurrent visual memory helps retain previously mentioned concepts and select salient concepts that have not yet been spoken.
- The method achieves state-of-the-art caption-generation results on PASCAL 1K and results better or comparable to prior methods on retrieval tasks using similar visual features.On MS COCO, performance is reported as only slightly below humans by BLEU and METEOR.
- Figure 1 separates the full training model from the components used for sentence generation and visual-feature generation.
2. Related work
Related work largely learns shared image–sentence spaces for retrieval or uses recurrent models for language generation. The paper situates its approach around the unresolved difficulty of long-range sequential relationships and inverse generation.
- Visual-memory research connects language grounding with semantic image understanding, while joint embeddings model correlated text and images.
- Joint-embedding methods use projection, ranking, pretrained representations, or combinations of these strategies.
- Retrieval methods can support bi-directional image–sentence search, but generating novel descriptions or visual scenes requires additional capabilities.
- Prior sentence-to-image work includes relation-tuple extraction and conditional random fields for interpreting abstract clip-art scenes.
- Recurrent language models capture word context but have difficulty representing long-range relationships in sequences.LSTM networks address this issue with gates that control gradient back-propagation and support long-term interactions.
3. Approach
The approach augments a recurrent language model with a visual hidden layer that reconstructs visual features and informs word prediction. The model is trained and evaluated as a shared bi-directional system, with implementation choices supporting efficient language modeling and pretrained visual features.
- Model objectives: The model predicts the next word from visual features, previous words, and latent visual variables, while also estimating visual features from words.
- Model objectives: Its objective jointly maximizes the likelihood of the next word and observed visual features conditioned on previous words and their visual interpretation.
- Model structure: The recurrent visual hidden layer reconstructs visual features and helps predict the next word by comparing remembered and currently observed visual information.
- Model structure: The visual hidden state is trained to estimate visual features at every time step, encouraging long-term retention of visual concepts.Figure 2 contrasts temporally stable visual units u with more rapidly changing language units s.
- Model structure: The same network generates sentences when visual features are known and predicts visual features when words are known.
- Language model: Word-class factorization decomposes word probability into class and within-class probabilities to reduce computational cost.Classes are formed unsupervised by grouping words with similar frequencies.
- Language model: The language model combines recurrent outputs with a Maximum Entropy language model using a three-word history.
- Training and features: Sentences are tokenized with Stanford CoreNLP and lowercased before entering the recurrent model.
4. Results
The paper evaluates its bidirectional RNN across sentence generation and image-sentence retrieval on PASCAL 1K, Flickr 8K, Flickr 30K, and MS COCO. The approach improves generation against baselines and achieves state-of-the-art or comparable retrieval performance under matched visual features.
- Evaluation setup: Experiments evaluate novel sentence generation, sentence retrieval, and image retrieval across PASCAL 1K, Flickr 8K, Flickr 30K, and MS COCO.PASCAL 1K generation is trained on MS COCO because of its limited training data; Flickr uses standard splits, while MS COCO uses approximately 37K/3K training/validation images and tests on validation.
- Baselines and training: The final model uses RNN baselines ranging from no visual input to image features and fine-tuning, with 4096D BVLC reference-net features for RNN+IF.Fine-tuning back-propagates error into the CNN and is followed by fixing image features and retraining the model on top of them.
- Sentence generation: Our approach significantly improves BLEU and METEOR over Midge and BabyTalk on PASCAL 1K, producing more naturally descriptive sentences than the shorter or redundant baselines.Examples include mentioning that an image is black and white or that a bus is a “double decker”.
- Sentence generation: 18.99 BLEU and 20.42 METEOR on MS COCO are slightly below human scores of 20.19 and 24.94, respectively.Image features improve substantially over an RNN language model alone, while fine-tuning and the full approach add improvements across datasets.
- Bidirectional retrieval: Measured by Mean r, the approach achieves state-of-the-art PASCAL 1K image and sentence retrieval and comparable or better Flickr retrieval except against DeepFE’s different features.Using DeepFE with the same DECAF features as this approach yields better results, and combining text and visual features generally outperforms text alone.
5. Discussion
The model handles bidirectional generation and retrieval, but spatial relations remain difficult to detect and better feature localization may improve performance.
- Spatial relations remain difficult for the current model to detect in image descriptions.
- Better feature localization may improve retrieval and description generation performance.
- The model generates both novel image descriptions and visual features, using recurrent visual memory to learn long-term interactions.