Source-linked AI summary
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio
TL;DR
Image captioning must identify visual content and relationships while expressing them in natural language, but static image representations can lose information needed for descriptive captions. The paper introduces soft and hard visual attention mechanisms and achieves state-of-the-art performance on Flickr8k, Flickr30k, and MS COCO while making the model’s focus interpretable.
Problem
Image captioning requires identifying objects and their relationships while compressing salient visual information into natural language, and static representations may lose details needed for richer captions.
Method
The paper introduces soft deterministic and hard stochastic attention mechanisms that dynamically weight image regions while generating captions.
Results
The models achieve state-of-the-art performance on Flickr8k, Flickr30k, and MS COCO, with attention visualizations providing interpretability.
Takeaways & Limitations
Learned visual attention can make caption generation more interpretable by aligning generated words with salient object and non-object image regions.
Takeaways & Limitations
Performance comparisons are complicated by differences in convolutional feature extractors and the lack of standardized dataset splits for Flickr30k and MS COCO.
Abstract
from arXiv · showhide
Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.
1. Introduction
Image caption generation requires identifying objects, modeling their relationships, and expressing them in natural language. This paper introduces soft and hard attention mechanisms that focus on salient image regions during caption generation and make the model’s visual focus interpretable.
- Motivation: Image caption generation is difficult because it requires identifying image objects, capturing their relationships, and expressing them naturally.The task is framed as closely connected to scene understanding and a primary goal of computer vision.
- Prior work: Recent systems improved caption quality by combining convolutional networks for image representations with recurrent networks for sentence generation.These advances were supported by neural-network training methods and large classification datasets.
- Attention motivation: Attention dynamically brings salient image features forward instead of compressing the entire image into a static representation, especially under clutter.This motivates attending to salient parts of an image while generating its caption.
- Contributions: The paper introduces two attention-based caption generators: soft deterministic attention trained by back-propagation and hard stochastic attention trained by a variational lower bound or REINFORCE.Both mechanisms are presented under a common framework.
- Interpretability: Visualizing attention reveals where and what the model focuses on while generating captions.The models are designed to attend to salient image regions during caption generation, enabling interpretation of their results.
2. Related Work
Prior image-captioning work increasingly used recurrent neural networks inspired by sequence-to-sequence machine translation, while earlier approaches relied on templates or retrieval. Existing neural methods typically encoded images as single CNN feature vectors, and this work extends prior attention-based neural-network research for vision.
- Neural caption generation: Recent image-captioning methods commonly use recurrent neural networks, drawing inspiration from sequence-to-sequence neural machine translation.The cited translation work includes Cho et al. (2014), Bahdanau et al. (2014), and Sutskever et al. (2014).
- Neural caption generation: Early neural approaches included multimodal log-bilinear, ranking-and-generation, recurrent language-model, and LSTM-based methods for image or video descriptions.Kiros et al. introduced multimodal log-bilinear and ranking-generation models; Mao et al. used a recurrent language model, while Vinyals et al. and Donahue et al. used LSTMs.
- Image representations: Most neural captioning systems represented images with a single feature vector from the top layer of a pretrained convolutional network.Karpathy and Li instead learned a joint embedding using R-CNN object detections and bidirectional RNN outputs, while Fang et al. used a multi-step detection-based pipeline.
- Pre-neural captioning: Before neural captioning, dominant strategies generated detection-based templates or retrieved and modified captions from similar images.The template approach incorporated object detections and attribute discovery; the retrieval approach began with similar captioned images from a large database.
- Attention mechanisms: The attention component builds on earlier attention methods for vision-related neural tasks, particularly work by Bahdanau, Mnih, and Ba.The paper also cites Larochelle and Hinton, Denil et al., and Tang et al. as sharing the same spirit.
3. Image Caption Generation with Attention Mechanism
The model extracts spatial image features and uses an attention-based LSTM decoder to generate captions one word at a time. A learned attention mechanism dynamically forms a context vector from weighted image locations relevant to the next word.
- Model framework: The model maps a raw image to a caption represented as a sequence of 1-of-K encoded words.K denotes vocabulary size and C denotes caption length.
- Image representation: A convolutional neural network extracts L D-dimensional annotation vectors, each representing a different image region.Features from a lower convolutional layer preserve correspondence with 2-D image locations.
- Sequence decoder: An LSTM generates one word per time step conditioned on the context vector, previous hidden state, and previously generated words.The implementation follows the LSTM formulation of Zaremba et al. (2014).
- Attention mechanism: At each time step, attention assigns positive weights to annotation vectors, representing the relevance or probability of each image location for producing the next word.The context vector is a dynamic representation of the relevant part of the image input.
- Attention mechanism: The context vector is computed from annotation vectors and their weights, which sum to one, through a function φ.The paper describes two attention-model variants that share this framework but differ in the definition of φ.
4. Learning Stochastic “Hard” vs Deterministic “Soft” Attention
The model supports stochastic hard attention, which samples discrete locations and optimizes a variational lower bound, and deterministic soft attention, which uses expected context vectors for differentiable end-to-end training. Variance reduction and doubly stochastic regularization improve learning robustness and caption quality.
- Stochastic hard attention: Stochastic attention treats each attention location as a one-hot latent variable sampled from a multinoulli distribution parameterized by α.The sampled location determines which visual feature is extracted when generating each word.
- Stochastic hard attention: The stochastic model directly optimizes Ls, a variational lower bound on the marginal log-likelihood of the target word sequence.Its gradient can be approximated by Monte Carlo sampling and is equivalent to a REINFORCE learning rule with reward proportional to sampled-trajectory log likelihood.
- Stochastic hard attention: A moving-average baseline, an entropy term, and 0.5-probability replacement of sampled locations with their expected values reduce variance and improve stochastic-learning robustness.The baseline is updated as bk = 0.9 × bk−1 + 0.1 × log p(y | ˜sk, a).
- Deterministic soft attention: Deterministic attention replaces sampled locations with the expected context vector, computed as the α-weighted sum of annotation vectors.This produces a smooth, differentiable model that can be trained end-to-end with standard backpropagation.
- Deterministic soft attention: Doubly stochastic regularization encourages each image region to receive approximately equal total attention across generation, improving BLEU and producing richer, more descriptive captions.The soft model also predicts a gating scalar β from the previous hidden state at each time step.
5. Experiments
Experiments validate the attention-based captioning model on Flickr8k, Flickr30k, and MS COCO using BLEU and METEOR, while visualizations show interpretable alignments with salient image regions. Comparisons account for feature extractors, ensembling, evaluation-code consistency, and dataset-split differences.
- Experimental setup: Experiments use Flickr8k, Flickr30k, and MS COCO, with five reference sentences per image retained consistently across datasets.The datasets contain 8,000, 30,000, and 82,783 images respectively; excess MS COCO references are discarded for consistency.
- Evaluation methodology: Results are evaluated with BLEU-1 through BLEU-4 without a brevity penalty and, where possible, with METEOR.The authors verified that their BLEU code matches implementations from several prior studies and restrict fair comparisons accordingly.
- Experimental setup: Comparisons control for convolutional feature extractors, report single-model performance, and use predefined or publicly available dataset splits.The evaluation directly compares results using comparable GoogLeNet/Oxford VGG features and notes the lack of standardized Flickr30k and COCO splits.
- Quantitative results: The model achieves state-of-the-art performance on Flickr8k, Flickr30k, and MS COCO, significantly improving METEOR on MS COCO.The authors speculate that the MS COCO METEOR improvement is connected to their regularization techniques and lower-level representation.
- Attention visualization: Visualizing learned attention adds interpretability, revealing human-intuitive alignments and allowing inspection of why mistakes occur.Unlike approaches relying on object-detection candidate targets, the model can attend to non-object salient regions.
6. Conclusion
The paper proposes an attention-based approach that achieves state-of-the-art performance on three benchmark datasets using BLEU and METEOR. Learned attention improves interpretability, aligns with human intuition, and may generalize through modular encoder-decoder designs to other domains.
- The attention-based approach achieves state-of-the-art performance on three benchmark datasets using BLEU and METEOR.
- Learned attention makes the generation process more interpretable, with alignments that correspond well to human intuition.
- The modular encoder-decoder approach combined with attention may have useful applications in other domains.
A. Appendix
The appendix visualizes where the hard and soft attention models attend during image captioning. Example captions describe a girl with a teddy bear and a person on a beach with a surfboard.
- Attention visualizations: The appendix compares visualizations from the “hard” and “soft” attention models, with white regions indicating where the model roughly attends.The visualizations are referenced in section 5.4.
- Example captions: One example describes a little girl sitting on a couch with a teddy bear.A related caption instead places the girl on a bed with the teddy bear.
- Example captions: Another example describes a person standing on a beach with a surfboard.The paired caption specifies that the person is a man.