PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Linear Attention Networks for Image Captioning

Yingwei Pan, Ting Yao, Yehao Li, Tao Mei

arXiv:2003.14080v1cs.CV

TL;DR

Image-captioning attention mechanisms have limited capacity for multi-modal reasoning because they mainly model first-order interactions. The paper introduces X-Linear attention, which uses bilinear spatial and channel-wise attention and integrates it into captioning encoders and decoders; X-LAN reaches a reported CIDEr score of 132.0% on the COCO Karpathy test split.

  • Problem

    Existing image-captioning attention mechanisms mainly model first-order image-language interactions, limiting multi-modal reasoning capacity.

  • Method

    X-Linear attention uses bilinear pooling with spatial and channel-wise attention, while X-LAN integrates blocks into image encoders and sentence decoders for higher-order intra- and inter-modal interactions.

  • Results

    132.0% CIDEr on the COCO Karpathy test split is reported for X-LAN, with a 2.2% absolute improvement over AoANet.

  • Takeaways & Limitations

    X-LAN achieves new state-of-the-art performance on the COCO image-captioning benchmark while exploiting higher-order feature interactions.

  • Takeaways & Limitations

    The formulation assumes a query vector and sets of keys and values as inputs to the X-Linear attention block.

Abstract

from arXiv · show

Recent progress on fine-grained visual recognition and visual question answering has featured Bilinear Pooling, which effectively models the 2$^{nd}$ order interactions across multi-modal inputs. Nevertheless, there has not been evidence in support of building such interactions concurrently with attention mechanism for image captioning. In this paper, we introduce a unified attention block -- X-Linear attention block, that fully employs bilinear pooling to selectively capitalize on visual information or perform multi-modal reasoning. Technically, X-Linear attention block simultaneously exploits both the spatial and channel-wise bilinear attention distributions to capture the 2$^{nd}$ order interactions between the input single-modal or multi-modal features. Higher and even infinity order feature interactions are readily modeled through stacking multiple X-Linear attention blocks and equipping the block with Exponential Linear Unit (ELU) in a parameter-free fashion, respectively. Furthermore, we present X-Linear Attention Networks (dubbed as X-LAN) that novelly integrates X-Linear attention block(s) into image encoder and sentence decoder of image captioning model to leverage higher order intra- and inter-modal interactions. The experiments on COCO benchmark demonstrate that our X-LAN obtains to-date the best published CIDEr performance of 132.0% on COCO Karpathy test split. When further endowing Transformer with X-Linear attention blocks, CIDEr is boosted up to 132.8%. Source code is available at \url{https://github.com/Panda-Peter/image-captioning}.

1. Introduction

Image captioning commonly uses encoder-decoder models and attention to connect visual regions with language, but conventional approaches largely model only first-order interactions. X-Linear attention addresses this limitation with bilinear spatial and channel-wise attention and integrates it into both encoder and decoder structures.

  • Image captioning uses CNN-based visual encoding and RNN-based sentence decoding within an encoder-decoder paradigm.
  • X-Linear attention uses bilinear pooling to model second-order query-key interactions while measuring both spatial and channel-wise attention distributions.
  • Conventional visual attention selects image regions conditioned on the decoder’s hidden language state, then accumulates region features to guide word generation.
  • Stacked X-Linear blocks encode region features with higher-order intra-modal interactions, producing enhanced region-level and image-level visual features.
  • The proposed X-LAN integrates X-Linear attention blocks into image encoders and sentence decoders to exploit higher-order intra- and inter-modal interactions.

2. Related Work

Prior image-captioning attention mechanisms mainly capture first-order image-language interactions, limiting multi-modal reasoning. X-Linear extends bilinear-pooling ideas from recognition and question answering to image-captioning encoders and decoders for higher-order interactions.

  • Image Captioning: Early image-captioning systems use CNN encoders and RNN decoders, while later attention methods increase interaction between visual content and language.
  • Image Captioning: Existing attention mechanisms mainly explore first-order interactions between image content and sentence representations, limiting multi-modal reasoning capacity.
  • Bilinear Pooling: Bilinear pooling computes feature-vector outer products to model all pairwise, second-order interactions and produce more discriminative representations.
  • Bilinear Pooling: Earlier bilinear-pooling techniques primarily target fine-grained visual recognition and visual question answering rather than image captioning.
  • Bilinear Pooling: X-Linear applies bilinear pooling within image encoders and sentence decoders to exploit higher-order intra-modal and inter-modal interactions for captioning.

3. X-linear Attention Networks (X-LAN)

X-LAN introduces a unified X-Linear attention block that uses bilinear pooling to model second-order spatial and channel-wise interactions, with extensions for higher and infinity-order interactions. Integrated into image encoders and sentence decoders, these blocks strengthen intra-modal visual encoding and inter-modal reasoning for image captioning.

  • X-Linear Attention Block: X-Linear attention forms bilinear query-key representations that capture second-order interactions, unlike conventional attention’s first-order feature interaction.Low-rank bilinear pooling combines each query-key pair, and the resulting representation is used to derive attention distributions.
  • X-Linear Attention Block: The block jointly applies spatial and channel-wise bilinear attention to aggregate enhanced value features.Spatial weights are normalized with softmax, while channel-wise weights are produced through squeeze-excitation and sigmoid gating.
  • Higher-Order Interactions: Stacking M X-Linear blocks captures higher-order, specifically 2M-th order, feature interactions through iterative attention and feature updates.Each block uses the previous attended feature as a query and updates keys and values with residual connections and layer normalization.
  • Infinity-Order Interactions: Applying ELU before bilinear pooling approximately models infinity-order feature interactions without stacking infinitely many blocks.The paper connects this approximation to the Taylor expansion of bilinear pooling over exponentially transformed features.
  • X-LAN for Image Captioning: X-LAN integrates X-Linear blocks into both image encoding and sentence decoding to capture higher-order intra-modal and inter-modal interactions.Faster R-CNN supplies region features; the encoder enhances region- and image-level representations, while the decoder reasons between visual features and the sentence state.
  • X-LAN for Image Captioning: Training uses cross-entropy loss for initialization before further optimization with a CIDEr-based self-critical objective.The supplied training description specifies cross-entropy training followed by CIDEr-reward optimization.

4. Experiments

Experiments on COCO evaluate X-LAN against established captioning systems, including qualitative examples, online testing, and ablations of its encoder-decoder components. X-LAN achieves stronger reported performance and produces more accurate captions in the presented comparisons.

  • Offline Evaluation: 132.0% CIDEr is achieved by X-LAN on the offline COCO Karpathy test split, improving 2.2% over AoANet without model ensembling.The comparison includes both non-attention and attention-based single models optimized with cross-entropy and CIDEr objectives.
  • Online Evaluation: X-Transformer improves Transformer performance by integrating X-Linear attention blocks into both the Transformer encoder and decoder.This tests whether the X-Linear attention block generalizes beyond the LSTM-based X-LAN structure.
  • Online Evaluation: X-LAN and X-Transformer outperform published state-of-the-art systems across most metrics on the COCO online testing server.Table 2 reports BLEU@N, METEOR, ROUGE-L, and CIDEr percentages using official testing images with five and forty reference captions.
  • Qualitative Analysis: Qualitative examples show X-LAN generating more accurate and descriptive captions than Up-Down, including correctly identifying cars stopped at a traffic light.In the cited example, Up-Down describes a truck driving, whereas X-LAN matches the visual content and ground-truth descriptions.
  • Attention Visualization: Attention visualizations compare the maximum-weight image region selected at each decoding step for Up-Down and X-LAN.The visualization examines whether attended regions align with the words generated during caption decoding.
  • Ablation Study: Ablations replace conventional decoder attention with X-Linear attention and add stacked X-Linear blocks to the image encoder to assess their separate contributions.The base model uses Faster R-CNN features and conventional LSTM attention; the study also examines stacking depth and ELU-equipped blocks.

5. Conclusions

The paper concludes that X-Linear attention models higher-order intra- and inter-modal interactions for image captioning. Integrated into X-LAN, it achieves state-of-the-art performance on COCO.

  • 5. Conclusions: X-Linear attention models 2nd-order interactions through spatial and channel-wise bilinear attention.Higher and infinity-order interactions are modeled by stacking blocks and equipping them with ELU.
  • 5. Conclusions: X-LAN integrates X-Linear attention blocks into the image encoder and sentence decoder to exploit higher-order intra- and inter-modal interactions.The paper reports extensive COCO experiments validating the block and the resulting network.
  • 5. Conclusions: X-LAN obtains new state-of-the-art performance on the COCO image-captioning dataset.The conclusion attributes the reported efficacy to the proposed X-Linear attention block and X-LAN.
Loading 2003.14080v1…