Source-linked AI summary
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
Jiasen Lu, Caiming Xiong, Devi Parikh, Richard Socher
TL;DR
Existing image-captioning attention models generally force visual attention for every generated word, despite language context sometimes making image information unnecessary. The paper proposes adaptive attention with a visual sentinel, allowing the decoder to choose whether and where to attend to the image, and reports state-of-the-art performance across COCO and Flickr30k.
Problem
Spatial-attention decoders cannot determine when to rely on visual signals and when to rely on the language model.
Method
The model uses adaptive attention with a visual sentinel that lets the decoder choose between image regions and a latent representation of its existing knowledge.
Results
The model achieves state-of-the-art performance across standard image-captioning benchmarks and significantly outperforms prior methods on COCO and Flickr30k.
Takeaways & Limitations
Adaptive attention provides a fallback option for sequential word generation and supports analysis of visual grounding and attention localization.
Takeaways & Limitations
The model cannot distinguish truly non-visual words from visually related words whose language correlations make visual signals unnecessary.
Abstract
from arXiv · showhide
Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict non-visual words such as "the" and "of". Other words that may seem visual can often be predicted reliably just from the language model e.g., "sign" after "behind a red stop" or "phone" following "talking on a cell". In this paper, we propose a novel adaptive attention model with a visual sentinel. At each time step, our model decides whether to attend to the image (and if so, to which regions) or to the visual sentinel. The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation. We test our method on the COCO image captioning 2015 challenge dataset and Flickr30K. Our approach sets the new state-of-the-art by a significant margin.
1. Introduction
Image-captioning attention models typically inspect the image at every step, even when language context makes visual information unnecessary. The paper introduces adaptive attention that chooses both when to use image features and which region to inspect.
- Captioning systems need fine-grained visual clues, and attention mechanisms typically highlight image regions relevant to each generated word.
- Many attention models attend to the image at every time step, although words such as “a” and “of” lack canonical visual signals.
- Language correlations can make visual information unnecessary for words such as “on”, “top”, and “sign” in suitable caption contexts.
- The proposed adaptive encoder-decoder decides when to rely on visual signals versus the language model and which image region to attend to.
- A visual sentinel provides a decoder-memory fallback, while a sentinel gate controls the balance between image information and that fallback.
- The model significantly outperforms other state-of-the-art methods on COCO and Flickr30k and includes analyses of visual grounding and weakly supervised localization.
2. Method
The method extends attention-based image captioning with spatial attention and an adaptive visual sentinel, allowing the decoder to choose between image regions and stored decoder information when predicting each word.
- Encoder-Decoder Framework: The encoder-decoder framework models each conditional word probability with an LSTM hidden state and a visual context vector extracted from the image.The context vector provides visual evidence for caption generation, while the LSTM replaces a vanilla recurrent neural network.
- Spatial Attention Model: The spatial attention model uses the LSTM hidden state to compute attention over k spatial image features and combines the weighted features into the context vector.The spatial features represent different image regions, and the resulting context vector is combined with the hidden state to predict the next word.
- Spatial Attention Model: Unlike prior attention formulations, the model uses the current hidden state to determine where to look before combining visual and linguistic information for word prediction.The authors interpret the attended context as residual visual information that complements the hidden state.
- Adaptive Attention Model: The adaptive model adds a visual sentinel, a latent representation extracted from the decoder memory that serves as a fallback when image information is unnecessary.The sentinel is obtained by extending the LSTM and is updated at each time step.
- Adaptive Attention Model: The adaptive context vector mixes spatially attended image features with the visual sentinel through a sentinel gate that controls the balance between new visual information and decoder memory.The gate value ranges from 0 to 1: 1 uses only the sentinel, while 0 uses only spatial image information.
- Adaptive Attention Model: The augmented attention distribution includes the sentinel alongside image regions, and its resulting context is used to calculate the vocabulary probability for the next word.The last attention element is interpreted as the gate value, and the sentinel vector is updated at every time step.
3. Implementation Details
The model uses CNN-derived spatial and global image features with an LSTM decoder and visual sentinel, trained with specified optimization settings and beam search.
- Decoder-RNN: The decoder concatenates each word embedding with the global image feature and transforms the visual sentinel and LSTM output into dimension-d vectors.This supplies the decoder with linguistic, global visual, and sentinel representations.
- Training details: The network uses a single-layer LSTM with hidden size 512 and Adam learning rates of 5e-4 for the language model and 1e-5 for the CNN.Training uses momentum 0.8, weight decay 0.999, CNN fine-tuning after 20 epochs, batch size 80, and early stopping based on validation CIDEr.
- Training details: The model trains for up to 50 epochs, can train within 30 hours on one Titan X GPU, and uses beam size 3 for COCO and Flickr30k caption sampling.Early stopping occurs when validation CIDEr fails to improve for six epochs.
4. Related Work
Prior work spans template-based and neural image-captioning systems, including attention mechanisms for latent visual-word alignment. The paper positions its contribution as the first to decide when image attention is needed during word generation.
- Related approaches: Template-based systems fill caption slots using outputs from object detection, attribute classification, and scene recognition, while neural systems frame captioning as image-to-text sequence generation.The related work contrasts these two broad families of image-captioning methods.
- Attention mechanisms: Attention-based neural frameworks learn latent alignments or inject semantic concepts and attributes while generating corresponding caption words.The cited approaches include attention mechanisms, semantic attention, and review networks.
- Novelty: The paper claims to be the first work to reason about when a model should attend to an image while generating a sequence of words.This distinguishes its contribution from prior attention methods that focus on where to attend.
- Evaluation: Table 1 reports performance on Flickr30k and COCO test splits, with BLEU n-gram scores higher-is-better and additional ROUGE-L and SPICE values.The caption gives the authors' ROUGE-L/SPICE scores as 0.467/0.145 on Flickr30k and 0.549/0.194 on COCO.
- Evaluation: Table 2 presents a leaderboard of published state-of-the-art models on the online COCO testing server, using an ensemble of five differently initialized models for the submission.The table caption specifies the evaluation setting and submission composition.
5. Results
Experiments on Flickr30k and COCO show that adaptive attention improves captioning performance over spatial attention and prior systems, while learning context-dependent visual grounding and spatial alignments.
- 0.531/1.085 CIDEr improves over 0.493/1.029 for the spatial-attention model on Flickr30k/COCO.
- 0.332 BLEU-4, 0.266 METEOR, and 1.085 CIDEr improve over MSM† baselines of 0.325, 0.251, and 0.986 on COCO.
- The single model significantly outperforms previous methods in all reported metrics, and achieves the best performance among published systems on the COCO evaluation server.
- Attention maps generally align with human intuition, although failures remain for counting, texture, and fine-grained categories.
- Visual grounding probabilities exceed 0.9 for visual words such as “red,” “rose,” “doughnuts,” “woman,” and “snowboard,” while varying by context.
- The model attends more to object, attribute, and number words, less to non-visual words, and intermediately to abstract notions.
- The model cannot distinguish truly non-visual words from technically visual words predictable through language correlations, such as “phone” after “cell.”
- 0.373 average localization accuracy for adaptive attention exceeds 0.362 for spatial attention, indicating improved spatial localization alongside adaptive attendance.
6. Conclusion
The paper presents an adaptive attention encoder-decoder with a visual sentinel and reports state-of-the-art image-captioning performance across standard benchmarks.
- The framework provides the decoder with a fallback option through an additional visual sentinel produced by a new LSTM extension.
- The model achieves state-of-the-art performance across standard image-captioning benchmarks and receives extensive adaptive-attention evaluation.
- Although evaluated on image captioning, the framework may have useful applications in other domains.
7. Supplementary
The supplementary analysis examines how the visual sentinel suppresses noisy attention for non-visual words, compares visual grounding across COCO and Flickr30K, and visualizes attention and localization behavior.
- Adaptive attention analysis: Without a visual sentinel, attention for the non-visual word “of” spreads around image boundaries, propagating a noisy visual signal.With the sentinel, “of” has a small visual grounding probability, restricting this noise from backpropagating through the visual attention model.
- Adaptive attention analysis: The supplementary experiments further evaluate visual sentinel effectiveness through the spatial-attention versus spatial-attention-plus-sentinel ablation.This comparison is identified as the ablation used to assess the visual sentinel in Table 1.
- Cross-dataset grounding: Visual grounding trends are generally similar across COCO and Flickr30K, with a rank correlation of 0.483 for common words.Some words differ substantially: “sheep” and “railing” ground more strongly in COCO, whereas “hair” and “run” reverse the pattern.
- Attention visualization: Figures 9 and 10 visualize spatial and temporal attention alongside generated captions and word-region correspondences.Different colors indicate correspondence between attended regions and underlined words, while Figure 10 includes visual grounding probability.
- Weakly supervised localization: Figure 11 visualizes weakly supervised localization by comparing red ground-truth boxes with blue locations predicted from spatial attention maps.The visualization connects generated captions with localization results without requiring explicit localization supervision.