Source-linked AI summary
Attention Correctness in Neural Image Captioning
Chenxi Liu, Junhua Mao, Fei Sha, Alan Yuille
TL;DR
Image-captioning attention maps had been assessed mainly through qualitative visualizations, leaving their agreement with human perception uncertain. The paper defines a quantitative correctness metric using region-to-caption alignments and trains models with strong or weak attention supervision. Across Flickr30k and COCO settings, supervision improves both attention correctness and caption quality, while weak semantic labels provide a cheaper alternative when alignments are unavailable.
Problem
The paper addresses the lack of quantitative evaluation for whether implicitly learned attention maps align with human annotations.
Method
The paper defines attention correctness using region-to-caption alignments and introduces models with strong alignment supervision or weak semantic-label supervision.
Results
Supervised attention models improve both attention-map correctness and captioning performance, including when only segmentation masks and object categories are available.
Takeaways & Limitations
Attention correctness is positively correlated with captioning quality, and cheaper semantic annotations can significantly boost captioning performance without ground-truth alignments.
Takeaways & Limitations
Word2vec-based weak attention targets can confuse scene words with object classes, requiring scene words to receive uniform attention.
Abstract
from arXiv · showhide
Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision. But despite their popularity, the "correctness" of the implicitly-learned attention maps has only been assessed qualitatively by visualization of several examples. In this paper we focus on evaluating and improving the correctness of attention in neural image captioning models. Specifically, we propose a quantitative evaluation metric for the consistency between the generated attention maps and human annotations, using recently released datasets with alignment between regions in images and entities in captions. We then propose novel models with different levels of explicit supervision for learning attention maps during training. The supervision can be strong when alignment between regions and caption entities are available, or weak when only object segments and categories are provided. We show on the popular Flickr30k and COCO datasets that introducing supervision of attention maps during training solidly improves both attention correctness and caption quality, showing the promise of making machine perception more human-like.
Introduction
This work asks whether image-captioning attention maps align with human annotations and whether more human-like attention improves captioning. It introduces quantitative evaluation and explicit attention supervision, showing improvements in attention consistency and caption quality.
- Introduction: Attention-based captioning models attend to different image regions while generating words sequentially.These models use CNNs to extract image features and RNNs to decode them into captions.
- Introduction: The paper evaluates whether generated attention maps are consistent with human perception and annotations.It frames attention as alignment from language space to image space.
- Introduction: The proposed attention-correctness metric measures consistency between generated attention maps and the regions described by caption words or phrases.Flickr30k Entities region-to-phrase alignments provide the ground truth.
- Introduction: Implicit attention outperforms uniform attention but remains inconsistent with human annotations.The paper reports that the implicit model still has room for improvement in attention consistency.
- Introduction: Explicit attention supervision improves both attention-map accuracy and generated-caption quality using either detailed alignments or cheaper semantic region labels.The supervised models apply to Flickr30k Entities-style alignment annotations and MS COCO-style region labels.
Related Work
Prior image-captioning work demonstrated strong results and qualitatively visualized attention, but the extent of object recognition remained uncertain. Region-to-phrase annotations in Flickr30k provide ground truth for evaluating generated attention maps.
- Image Captioning Models: Image-captioning research has produced impressive results across several benchmark datasets and model families.The cited work includes attention-based and non-attention captioning approaches.
- Image Captioning Models: Earlier attention work qualitatively showed region selection, while the models’ recognition of image objects during caption generation remained uncertain.The paper motivates quantitative evaluation of whether attended regions correspond to described content.
- Image Description Datasets: Flickr30k Entities extends Flickr30k captions with noun-phrase annotations linked to corresponding image regions.Annotators identify noun phrases and mark their corresponding regions with bounding boxes.
Deep Attention Models for Image Captioning
The paper describes implicit attention for image captioning and introduces explicit supervision to make attention maps align with image regions and caption words. Supervision can use direct region-word alignments or cheaper object labels.
- Implicit Attention Model: Implicit attention combines a visual feature extractor, an LSTM decoder, and a spatial attention module to generate captions.The extractor produces spatial feature vectors, while attention selects relevant image information at each decoding step.
- Implicit Attention Model: At each time step, attention weights are computed from image features and the decoder’s previous hidden state.The resulting context vector is a weighted combination of spatial image features used by the decoder.
- Supervised Attention Model: Explicit supervision addresses the implicit model’s inability to exploit prior knowledge about the desired attention map.The supervised objective adds an attention loss alongside the caption-generation loss.
- Strong Supervision: With direct region-word alignments, ground-truth attention maps are treated as probability distributions and trained using cross entropy.Words without image-region alignments receive zero attention loss; maps are resized and normalized when resolutions differ.
- Weak Supervision with Semantic Labeling: When only object regions and class labels are available, word-class embedding similarity approximates region-word alignment for weak supervision.This setting uses cheaper annotations than direct ground-truth alignment and then resizes and normalizes the resulting map.
Attention Correctness: Evaluation Metric
The paper introduces attention correctness to quantitatively assess whether generated attention aligns with human-annotated image regions. It defines word- and phrase-level scores based on attention mass within corresponding regions.
- Motivation: Attention correctness evaluates whether an implicitly learned attention module is effective beyond its word-likelihood training signal.The attention module generates maps, but receives error only through the word-likelihood loss.
- Word-level metric: For a word, correctness is the sum of its normalized attention weights that fall inside the human-annotated ground-truth region.The score lies between 0 and 1; Figure 2 illustrates a value of 0.54 from four in-region weights.
- Phrase-level metric: Phrase attention correctness is the maximum correctness among the individual words sharing the same entity region.The max operation can ignore less informative words with ambiguous attention, such as “of” in “a group of people.”
- Metric robustness: Using the maximum rather than the average does not affect the experiments’ main conclusion.This robustness finding is reported for the paper’s experiments.
- Testing: Testing requires strategies for finding ground-truth regions because generated captions may differ from the ground-truth captions.The paper notes that region correspondences are available for ground-truth phrases but not necessarily for generated phrases.
Experiments
Experiments show that supervised attention improves alignment with human annotations over implicit and uniform attention, with the strongest gains for small objects. Supervision also improves captioning metrics on Flickr30k and COCO.
- Weak Supervision: Scene-category handling prevents scene words such as “kitchen” from being aligned to small objects such as “spoon” or “oven”.When a scene word appears, the method assigns uniform attention across the image.
- Attention Correctness: The supervised model outperforms both implicit and uniform attention baselines in attention correctness on Flickr30k.This holds in both ground-truth-caption and generated-caption evaluation settings.
- Attention Correctness: 11.14% average gain over uniform attention exceeds the implicit model’s 6.22% gain under ground-truth captions.The supervised distribution is visually shifted further right in the attention-correctness histogram.
- Attention Correctness: 18.19% average improvement over uniform attention exceeds the implicit model’s 12.07%, a 50% relative gain, for generated captions.The supervised attention maps are more consistent with human judgment.
- Attention Correctness: The supervised model’s improvement over the implicit model is greatest for small bounding boxes.The experiment partitions Flickr30k test images equally into small, medium, and large ground-truth boxes.
- Captioning Performance: BLEU and METEOR consistently increase with supervised attention on Flickr30k and COCO, including BLEU-4 increases of 0.9 and 0.7 percent.The paper also reports higher captioning scores for examples grouped by higher attention correctness.
Discussion
The paper frames attention correctness as consistency with human annotations and uses it to assess image-captioning attention. Supervision improves both attention-map quality and captioning performance, including when only weak object information is available.
- Attention correctness: Attention correctness is defined as consistency between generated attention maps and human annotations of image regions described by caption words or phrases.Flickr30k Entities alignment annotations provide the ground truth for this evaluation.
- Attention correctness: Implicit attention models outperform uniform attention but still leave room for improvement in human-perception consistency.The evaluation considers consistency at both word and phrase levels.
- Supervised attention: Supervising attention maps improves both attention-map quality and image-captioning performance.The supervised model’s generated attention maps are described as having higher quality in qualitative examples.
- Supervised attention: Higher attention correctness is positively associated with better captioning quality.This relationship is reported for generated captions in the implicit-attention experiment.
- Weak supervision: Segmentation masks and object categories can provide weak supervision when ground-truth attention is unavailable, significantly boosting captioning performance.This weak-supervision setting does not require region-to-caption-entity alignment.