PaperlayerView exact revision on arXiv

Source-linked AI summary

Normalized and Geometry-Aware Self-Attention Network for Image Captioning

Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, Hanqing Lu

arXiv:2003.08897v1cs.CVcs.CLcs.MM

TL;DR

Image captioning self-attention struggles with internal covariate shift and with modeling geometric relations among image objects. The paper introduces NSA and GSA, combines them in NG-SAN, and reports improved performance across image captioning and three additional tasks.

  • Problem

    Self-attention does not adequately address internal covariate shift or the geometric relationships among image objects.

  • Method

    The paper introduces NSA to normalize hidden activations inside self-attention and GSA to incorporate relative geometry and object content into attention.

  • Results

    NG-SAN raises the best single-model MS-COCO CIDEr result from 125.5 to 128.6 and improves strong baselines across video captioning, machine translation, and visual question answering.

  • Takeaways & Limitations

    Combining NSA and GSA establishes a new MS-COCO image-captioning state-of-the-art and demonstrates generality across three additional tasks.

Abstract

from arXiv · show

Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we propose Normalized Self-Attention (NSA), a reparameterization of SA that brings the benefits of normalization inside SA. While normalization is previously only applied outside SA, we introduce a novel normalization method and demonstrate that it is both possible and beneficial to perform it on the hidden activations inside SA. Second, to compensate for the major limit of Transformer that it fails to model the geometry structure of the input objects, we propose a class of Geometry-aware Self-Attention (GSA) that extends SA to explicitly and efficiently consider the relative geometry relations between the objects in the image. To construct our image captioning model, we combine the two modules and apply it to the vanilla self-attention network. We extensively evaluate our proposals on MS-COCO image captioning dataset and superior results are achieved when comparing to state-of-the-art approaches. Further experiments on three challenging tasks, i.e. video captioning, machine translation, and visual question answering, show the generality of our methods.

1. Introduction

Image captioning requires recognizing objects and their relationships before organizing them into natural language. The paper addresses limitations of self-attention with normalized activations and geometry-aware object relations, achieving strong results across several tasks.

  • Image captioning must recognize objects, their relationships, and their organization in natural language.
  • Self-attention is susceptible to internal covariate shift because changing query distributions force subsequent layers to adapt during training.
  • Normalized Self-Attention (NSA) normalizes hidden activations inside self-attention to fix their distributions and improve conditioning.
  • Geometry-aware Self-Attention (GSA) adds a geometric bias computed from relative object geometry and the associated query or key content.
  • NG-SAN combines NSA and GSA, improving the best single-model CIDEr result on MS-COCO from 125.5 to 128.6.
  • Experiments report generality across video captioning, machine translation, and visual question answering, while establishing a new MS-COCO state-of-the-art.

2. Related Work

Related work covers encoder-decoder image captioning, normalization methods for deep networks, and positional or relational mechanisms for self-attention. The paper distinguishes GSA by combining relative geometry with object content.

  • Earlier image captioning methods commonly use CNN-RNN architectures, attention mechanisms, object-region features, reinforcement learning, and self-attention.
  • Normalization methods including Batch Normalization, Layer Normalization, Instance Normalization, and Group Normalization address activation-distribution or mini-batch dependencies.
  • Self-attention position and relation methods use absolute sinusoidal positions, relative sequence distances, or bounding-box relations computed from coordinates and sizes.
  • Unlike prior relation weighting based solely on bounding-box coordinates and sizes, GSA also explores the content of associated elements.

3. Preliminaries

The baseline SAN uses Transformer-style self-attention with stacked encoder and decoder layers, while image regions provide object-level inputs. Attention transforms inputs into queries, keys, and values whose weights produce the output.

  • Scaled Dot-Product Attention is introduced as a core Transformer component and forms the basis of the reviewed self-attention mechanism.
  • A self-attention layer projects input matrix X into queries Q, keys K, and values V using learned parameter matrices.
  • Energy scores form an N × N matrix, and softmax converts these scores into weights used to compute a weighted sum of values.
  • Self-attention network for image captioning: The SAN baseline contains stacked encoder and decoder layers with multi-head attention, feed-forward networks, residual connections, and layer normalization.
  • Self-attention network for image captioning: Encoder inputs are region-based visual features from Faster-RCNN, with each input element corresponding to an image object.
  • Self-attention network for image captioning: The decoder combines attended visual features with previous-word embeddings to recursively predict the next word, while positional encodings are added to decoder inputs.

4. Approach

The approach improves self-attention through normalization inside attention and geometry-aware biases that incorporate object relationships and content. The combined NG-SAN applies these modules to the encoder while accounting for autoregressive decoder constraints.

  • Normalized Self-Attention: NSA reparameterizes self-attention by normalizing hidden query activations, addressing distribution shifts caused by dynamically computed attention parameters.The normalization is motivated by viewing attention weights as a fully connected layer whose parameters depend on the input.
  • Normalized Self-Attention: The method normalizes each instance independently using per-channel feature statistics, corresponding to 1D Instance Normalization.Subtracting query means is described as highlighting differences among queries and encouraging distinctive information retrieval.
  • Geometry-Aware Self-Attention: Relative geometry is represented by a four-dimensional vector describing bounding-box position and size, then projected through a fully connected layer and ReLU.The ReLU acts as zero trimming so only object relations with certain geometric relationships are considered.
  • Geometry-Aware Self-Attention: GSA augments content-based attention with geometric bias computed from pairwise relative geometry and the content of associated queries or keys.The geometric attention function outputs an N × N score matrix, and its query/key projections are computed from the input.
  • Geometry-Aware Self-Attention: GSA supports content-independent, query-dependent, and key-dependent geometric biases, which may be used individually or combined.Dot-product matching is used for the query-dependent variant because it is more computation- and memory-efficient than concatenation followed by a fully connected operation.
  • Normalized and Geometry-aware Self-Attention Network: NG-SAN combines NSA and GSA in the SAN encoder, while omitting NSA from the autoregressive decoder because sequence-length-one statistics are unsuitable for Instance Normalization.The underlying SAN follows the Transformer-style encoder-decoder structure with multi-head attention, feed-forward networks, residual connections, and layer normalization.

5. Experiments on Image Captioning

Experiments on MS-COCO evaluate NSA, GSA, and their combination within self-attention image-captioning networks. The proposed modules improve the SAN baseline, with NG-SAN achieving the strongest reported results while adding little complexity.

  • Experimental setup: The experiments use MS-COCO Karpathy splits, Bottom-Up region features, standard captioning metrics, and Transformer-Base-style hyperparameters.The training split contains 113,287 images, while validation and test splits each contain 5,000 images; the vocabulary has 9,487 words.
  • NSA analysis: N-SAN performance improves with more attention layers, peaks at L = 6, and uses L = 4 later as a performance–complexity compromise.Model size grows linearly with the number of layers, while the gain from increasing L from 4 to 6 is limited.
  • NSA analysis: Instance normalization outperforms SAN and the other tested normalization methods, while affine transformations are unnecessary.Layer normalization slightly decreases performance, and batch normalization improves over SAN but remains inferior to instance normalization.
  • NSA analysis: Normalizing queries alone performs similarly to normalizing both queries and keys, whereas normalizing keys alone is inferior.The authors attribute the weaker key-only variant to its equivalence to normalizing Θ, which may limit self-attention capacity.
  • GSA analysis: All proposed GSA variants improve SAN, with the query-dependent variant performing best; combining variants yields no further improvement.GSA incorporates relative geometry information, while absolute geometry information is not beneficial.
  • Full model: NG-SAN significantly outperforms LSTM- and self-attention-based competitors across all metrics, improving the best CIDEr score from 130.9 to 132.1.On the official test server, the single model reaches CIDEr (C40) 128.6; NG-SAN adds about 2k parameters over SAN and incurs almost-ignorable normalization overhead.

6. Extension: Experiments on Other Tasks

The authors extend NSA and GSA beyond image captioning to video captioning, machine translation, and visual question answering. Each replacement improves its corresponding strong baseline on the reported metrics.

  • Experimental design: The extensions use Transformer+NSA for video captioning and machine translation, and replace MCAN self-attention modules with GSA for VQA.Video captioning uses VATEX features, machine translation uses WMT 2014 En-De, and VQA uses VQA-v2 with MCAN.
  • Video Captioning: 3.7 CIDEr points separate Transformer+NSA from Transformer on VATEX, while the improvement over the VATEX baseline is 11.4 points.Transformer+NSA consistently improves over Transformer on all reported video-captioning metrics.
  • Machine Translation: 0.36 BLEU points separate NSA from Transformer-Base on WMT 2014 En-De without adding parameters.The evaluation uses BLEU on newstest-2014.
  • Visual Question Answering: GSA boosts MCAN test-std accuracy from 70.83 to 71.28 on VQA-v2.The comparison reports overall accuracies on the online test-dev and test-std splits.

7. Conclusion

The paper concludes that NSA and GSA improve self-attention across image captioning, video captioning, machine translation, and visual question answering. Replacing vanilla self-attention with the proposed methods yields solid improvements over strong baselines.

  • Conclusion: NSA addresses internal covariate shift inside self-attention, while GSA dynamically incorporates geometric bias between objects.Their combination forms NG-SAN, which is evaluated extensively on MS-COCO and extended to three additional tasks.
  • Conclusion: Across all four tasks, simply replacing vanilla self-attention with NSA or GSA provides solid improvements over strong baselines.The conclusion presents this cross-task result as evidence of the methods’ generality.

A.1. Visualization of Geometric Weights

The geometric weights in GSA generally favor nearby objects and objects with compatible sizes, while reducing attention between nearly coincident boxes that likely represent the same object. Caption examples further indicate that G-SAN improves recognition of relationships between objects.

  • Relative position: Geometric weights generally decrease as the relative distance between objects increases, except near coincident centers.This pattern is observed while fixing relative width and height differences at 0.5, 1, or 2.
  • Coincident objects: When similarly sized boxes nearly coincide, their geometric weight is relatively small because they likely refer to the same object.Reducing the weight helps avoid repeating an object.
  • Relative position: Geometric weights tend to be larger for nearby objects than for objects separated by larger relative distances.For example, weights are larger at (∆x, ∆y) = (0.5, 0.5) than at (2, 2).
  • Relative size: Weights are small when the relative size difference between two boxes is either very small or very large.This behavior is evaluated while fixing ∆x = ∆y at 0.5, 1, or 2 and varying ∆w and ∆h from 0 to 3.
  • Caption examples: G-SAN generates captions that better determine relationships between objects than the SAN baseline.One example identifies pots and pans as hanging on the wall.
Loading 2003.08897v1…