Source-linked AI summary
Attention on Attention for Image Captioning
Lun Huang, Wenmin Wang, Jie Chen, Xiao-Yong Wei
TL;DR
Conventional attention can return results unrelated to the decoder’s query, potentially misleading image-caption generation. The paper introduces AoA, which gates attention-derived information by query-conditioned relevance, and applies it in AoANet’s encoder and decoder. AoANet outperforms previously published image-captioning methods and reaches 129.8 CIDEr-D on the MS COCO Karpathy offline test split.
Problem
Conventional attention always produces a weighted average, even when no candidate vector is relevant to the query, leaving the decoder unable to assess the result’s relevance.
Method
AoA generates a query-conditioned information vector and attention gate, applies element-wise gating, and uses the module in both AoANet’s encoder and decoder.
Results
129.8 CIDEr-D was achieved by a single AoANet model on the MS COCO Karpathy offline test split, outperforming previously published image-captioning models.
Takeaways & Limitations
AoA provides a general extension that determines attention-result relevance and retains useful information across encoder and decoder attention.
Abstract
from arXiv · showhide
Attention mechanisms are widely used in current encoder/decoder frameworks of image captioning, where a weighted average on encoded vectors is generated at each time step to guide the caption decoding process. However, the decoder has little idea of whether or how well the attended vector and the given attention query are related, which could make the decoder give misled results. In this paper, we propose an Attention on Attention (AoA) module, which extends the conventional attention mechanisms to determine the relevance between attention results and queries. AoA first generates an information vector and an attention gate using the attention result and the current context, then adds another attention by applying element-wise multiplication to them and finally obtains the attended information, the expected useful knowledge. We apply AoA to both the encoder and the decoder of our image captioning model, which we name as AoA Network (AoANet). Experiments show that AoANet outperforms all previously published methods and achieves a new state-of-the-art performance of 129.8 CIDEr-D score on MS COCO Karpathy offline test split and 129.6 CIDEr-D (C40) score on the official online testing server. Code is available at https://github.com/husthuaan/AoANet.
1. Introduction
Image captioning uses attention to guide decoding, but conventional attention always returns a weighted average even when it is irrelevant to the query. The paper proposes AoA and AoANet to assess attention relevance in both encoder and decoder, achieving state-of-the-art MS COCO results.
- Motivation: Attention guides caption decoding with weighted averages, but the decoder cannot determine whether the attended result matches its query.Irrelevant attention results can mislead decoding, including when candidate vectors contain no useful information.
- Attention on Attention: AoA extends conventional attention by generating an information vector and a sigmoid attention gate from the attention result and current query.The gate represents channel-wise relevance and is applied to the information vector through element-wise multiplication.
- Attention on Attention: AoA can determine relevance for single-head attention and filter multi-head attention results to retain useful information.The module is presented as a general extension applicable to various attention mechanisms.
- AoANet: AoANet applies AoA in the encoder to model object relationships and in the decoder to filter irrelevant or misleading attention results.The encoder uses self-attention before applying AoA to object features.
- Results: 129.8 CIDEr-D was achieved by a single AoANet model on the MS COCO offline test split, establishing state-of-the-art performance.The paper also reports 129.6 CIDEr-D (C40) for an ensemble of four models on the online testing server.
2. Related Work
Related work develops neural encoder-decoder captioning and attention-based models, including self-attention for object relationships. AoA differs by assessing attention-result relevance rather than combining modalities, while using a single gate without hidden states.
- Image Captioning: Earlier captioning systems used templates, whereas recent approaches use CNN encoders and recurrent decoders in neural encoder-decoder frameworks.These frameworks encode images into feature vectors and decode captions with LSTMs or related sequence models.
- Attention Mechanisms: Attention computes normalized importance weights over candidate vectors and combines them into a weighted-average attention result.Spatial and channel-wise variants are among the other attention mechanisms discussed.
- Self-Attention: Self-attention has achieved state-of-the-art results in machine translation and inspired its use for modeling relationships among image objects.The paper applies this idea to image captioning.
- Related Techniques: AoA uses an attention gate and information vector with element-wise gating, alongside techniques related to GLU, multimodal fusion, and recurrent gates.These related methods address long-range dependencies, multimodal interactions, or sequential inputs.
- Distinctions: AoA is a general attention extension that measures relevance between attention results and queries, unlike multimodal fusion, and uses one gate without hidden states.LSTM/GRU uses hidden states and more gates and is described as applicable only to sequence modeling.
3. Method
AoANet extends attention with an AoA module that evaluates the relevance between attention results and queries, then applies it in both encoder and decoder components.
- 3.1. Attention on Attention: AoA measures whether an attention result is relevant to the current query instead of always returning a potentially misleading weighted average.Conventional attention produces a weighted average even when candidate vectors are unrelated to the query.
- 3.1. Attention on Attention: The module generates an information vector and an attention gate from the attention result and current context, then multiplies them element-wise to obtain attended information.The gate and information vector are produced by separate linear transformations, with the gate using sigmoid activation.
- 3.2. AoANet: AoANet incorporates AoA into both the image encoder and caption decoder within an encoder/decoder framework.The encoder uses AoA to refine image features, while the decoder uses it to filter irrelevant or misleading attention results.
- 3.2.1 Encoder with AoA: The encoder first extracts feature vectors, models object interactions with self-attentive multi-head attention, and applies AoA to determine their relationships.The refining module preserves feature dimensionality and can be stacked six times.
- 3.2.2 Decoder with AoA: The decoder combines an LSTM, AoA, and word prediction to generate captions from refined image features.Its context vector combines the LSTM output with newly acquired information from the attended feature vector.
- 3.3. Training and Objectives: 132.0 CIDEr-D is achieved by a four-model AoANet ensemble on the MS COCO Karpathy test split after CIDEr-D optimization.Models are first trained with cross-entropy loss and then optimized for CIDEr-D using self-critical sequence training.
4. Experiments
AoANet is evaluated on MS COCO through offline and online benchmarks, ablations, qualitative examples, human judgments, and video-captioning transfer. Across these evaluations, AoA improves caption quality and reduces misleading attention effects.
- Quantitative Evaluation: AoANet achieves 132.0 CIDEr-D with a four-model ensemble after CIDEr-D optimization, outperforming the compared models.All models were first trained with XE loss and then optimized for CIDEr-D.
- Quantitative Evaluation: AoANet achieves the highest scores for most online COCO metrics, except for a slightly lower BLEU-1 (C40).Online results use an ensemble of four models trained on the Karpathy training split.
- Ablative Analysis: The encoder refining module adds 3.0 CIDEr-D over the base model without AoA, while adding AoA contributes a further 2.0.The ablation compares refining without AoA against refining with AoA after XE training.
- Qualitative Analysis: AoA suppresses irrelevant attention that can cause incorrect words, such as predicting “book” instead of “chair” for a teddy bear image.The decoder with AoA is less likely to be misled by irrelevant attended regions.
- Human Evaluation: In human evaluation, captions from the decoder with AoA were preferred 49.15% of the time versus 21.2% for the base model.Thirty evaluators compared captions for 100 randomly selected images.
- Generalization: On MSR-VTT, decoder AoA improves the base model from BLEU-4 33.53, CIDEr-D 38.83, ROUGE-L 56.90 to 37.22, 42.44, 58.32.The experiment uses ResNet-101 frame features, a bi-LSTM, and either a base or AoA decoder.
5. Conclusion
The paper introduces AoA to address irrelevant attention and applies it in both the encoder and decoder to form AoANet. Experiments on MS COCO report state-of-the-art performance and support the module’s broader applicability.
- The paper proposes AoA as an extension to conventional attention mechanisms for addressing irrelevant attention.
- AoANet applies AoA to both the image encoder and caption decoder.
- Experiments on MS COCO demonstrate state-of-the-art performance, superiority, and general applicability for AoA and AoANet.