PaperlayerView exact revision on arXiv

Source-linked AI summary

Bilinear Attention Networks

Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang

arXiv:1805.07932v2cs.CVcs.AIcs.CLcs.LG

TL;DR

BAN targets the computational and interaction limitations of co-attention, which uses separate modality-specific attention distributions. It learns bilinear attention over pairs of visual and language channels, combines their representations with low-rank pooling and residual integration, and achieves new state-of-the-art results on VQA 2.0 and Flickr30k Entities.

  • Problem

    Co-attention neglects interactions between visual regions and question words because it uses separate attention distributions for each modality.

  • Method

    BAN learns bilinear attention distributions over two groups of multimodal channels, uses low-rank bilinear pooling for joint representations, and applies residual learning to exploit multiple maps.

  • Results

    BAN achieves new state-of-the-art results on VQA 2.0 and Flickr30k Entities, with 69.69% Recall@1 for Flickr30k phrase localization.

  • Takeaways & Limitations

    BAN considers every pair of multimodal input channels while retaining the same order of time complexity as a one-input attention model.

  • Takeaways & Limitations

    Visualization errors remain for relatively small objects and entities requiring semantic inference.

Abstract

from arXiv · show

Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.

1 Introduction

BAN addresses the computational and representational limitations of co-attention by modeling interactions between pairs of visual and textual channels. It combines bilinear attention, low-rank pooling, and residual learning to use multiple attention maps efficiently.

  • Co-attention separately attends to visual regions and question words, neglecting interactions between the two modalities to control computational complexity.
  • BAN models bilinear attention between pairs of multimodal channels, such as question words and image regions.
  • Low-rank bilinear pooling extracts joint representations for each pair of channels, while BAN models interactions among channel groups.
  • Residual summations let the model use up to eight attention maps while keeping learning parameter-efficient and performance-effective.
  • BAN achieves a new state-of-the-art on VQA 2.0 and improves Flickr30k Entities phrase localization, with 25.37% faster inference.

2 Low-rank bilinear pooling

Low-rank bilinear pooling reduces bilinear-model complexity by factorizing weight matrices and introducing a pooling matrix for vector outputs. In attention networks, it combines multi-channel inputs into attended representations.

  • Low-rank bilinear models replace each bilinear weight matrix with two smaller matrices, limiting its rank to d ≤ min(N, M).
  • The low-rank formulation computes interactions through element-wise products and matrix operations.
  • Low-rank bilinear pooling introduces a pooling matrix P to produce vector outputs while significantly reducing parameters.
  • Unitary attention selectively combines channels of a multi-channel input into a single attended representation using attention weights.
  • With multiple glimpses, attended outputs are concatenated before a second low-rank bilinear pooling operation forms a classifier representation.

3 Bilinear attention networks

BAN extends bilinear pooling to two multi-channel inputs by learning a pairwise attention map and joint representations. Residual integration supports multiple glimpses, while matrix-chain computation preserves the single-input time-complexity order.

  • BAN introduces A ∈ R^ρ×φ to jointly reduce two multi-channel inputs X and Y through pairwise channel attention.
  • Each bilinear-attention logit is computed from low-rank representations of one channel from each input, then normalized into an attention map.
  • The resulting bilinear joint representation is f = P^T f′, with pooling matrix P mapping the intermediate representation to classifier features.
  • Parameters U and V are shared across multiple attention maps, while glimpse-specific vectors p_g remain distinct.
  • Residual attention integrates successive bilinear maps while keeping intermediate feature size constant before summing channels for classifier logits.
  • O(KMφ) is the single-glimpse time complexity under the stated channel-size assumptions, matching the order for one multi-channel input.

4 Related works

Prior co-attention methods estimate modality-specific attention distributions, while BAN-related work examines richer cross-modal interactions and their computational trade-offs.

  • Multimodal factorized bilinear pooling: Multimodal factorized bilinear pooling generalizes low-rank bilinear pooling with rank greater than one, but was not effective for BAN in the reported setting.BAN-1+MFB did not significantly improve over BAN-1 and consumed more peak GPU memory, hindering multiple-glimpse models.
  • Co-attention networks: Co-attention methods estimate correlations between image patches and sentence tokens, then derive attention distributions from those correlations.Some variants condition one modality’s attention on the other, while others use self-attention followed by question-conditioned visual attention.
  • Co-attention networks: These approaches use separate attention distributions for each modality and therefore neglect interactions between individual visual regions and words.

5 Experiments

The experiments evaluate BAN on VQA 2.0 and Flickr30k Entities using recurrent question representations, bottom-up image features, and task-specific binary cross-entropy objectives.

  • Datasets: VQA 2.0 emphasizes visual understanding by reducing answer bias, motivating evaluation of joint question-image representations.Its metric accounts for inter-human variability using Accuracy(ans) = min(#humans that said ans/3, 1).
  • Datasets: Flickr30k Entities contains 31,783 images and 244,035 phrase annotations mapped to image boxes for visual-grounding evaluation.The task localizes a corresponding box for each textual entity.
  • Input representations: Question embeddings use GloVe and GRU outputs, with up to 14 tokens for VQA and full sentence length for Flickr30k Entities.Annotated phrase-ending positions select entity-relevant GRU channels for Flickr30k Entities.
  • Input representations: Image inputs are fixed bottom-up attention features from Faster R-CNN, representing 10 to 100 detected objects with 2,048-dimensional features.Padding logits are masked with minus infinity so padded objects receive zero softmax probability.
  • Training and classifiers: BAN uses ReLU, a two-layer MLP for VQA, binary cross entropy for both tasks, and regularization through weight normalization and dropout.The VQA classifier has 3,129 outputs, while Flickr30k Entities uses the bilinear attention-map output.
  • Training and evaluation settings: For VQA testing, training and validation data are combined, Visual Genome augmentation is used, and an eight-glimpse model is evaluated; Flickr30k uses the prior test split.

6 VQA results and discussions

BAN is reported to outperform competing attention and state-of-the-art approaches on VQA 2.0 while using multiple bilinear attention maps efficiently through residual learning.

  • Comparison with state-of-the-arts: BAN significantly outperforms the 2017 VQA Challenge winner and the latest model using the same bottom-up attention features.It uses up to eight bilinear attention maps through residual learning.
  • Comparison with other attention methods: Bilinear attention is significantly better than unitary and co-attention methods on VQA 2.0 validation scores.
  • Comparison with other attention methods: BAN is the most parameter-efficient among the compared attention methods, and four-glimpse BAN uses parameters more parsimoniously than one-glimpse BAN.
  • Residual learning of attention: Residual attention learning significantly outperforms alternative integration methods, while summation and concatenation do not differ significantly in performance.Concatenation uses more parameters because it increases the classifier input size.
  • Ablation study: The first attention glimpse provides the largest accuracy gain, with additional glimpses producing smoothly diminishing gains during incremental ablation.
  • Entropy of attention: In four-glimpse BAN, attention-map entropies converge to different levels, suggesting that the maps do not contribute equally to voting.The paper presents this as an observation associated with residual learning in stacked attention networks.
  • Visualization: Flickr30k visualizations show correct localization for visually described entities but errors for a small object and an entity requiring semantic inference.
  • Visualization: Two-glimpse visualization attends to question and content words while highlighting image regions relevant to the queried pants color, producing the answer brown.

7 Flickr30k entities results and discussions

BAN outperforms prior methods on Flickr30k Entities phrase localization, achieving strong Recall@1 without additional visual features and faster inference through multi-channel processing. Its errors remain on small objects and entities requiring semantic inference.

  • 4.48% is BAN’s phrase-localization improvement over the previous state of the art, with faster inference.
  • 69.69% Recall@1 establishes a new state of the art without box size, color, segmentation, or pose-estimation features.
  • 0.67 ms/entity versus 0.84 ms/entity yields a 25.37% inference improvement by processing multiple entities as multi-channel input.
  • Visualization examples are correct for visible properties but fail on a relatively small cigarette and a semantically inferred male conductor.

8 Conclusions

BAN extends unitary attention with bilinear attention maps and low-rank bilinear pooling for multimodal multi-channel inputs. Efficient matrix-chain computation keeps the computational cost in the same magnitude, while residual learning uses up to eight maps.

  • BAN extracts joint representations from multimodal multi-channel inputs using low-rank bilinear pooling.
  • BAN considers every pair of multimodal input channels while keeping computational cost in the same magnitude through matrix chain multiplication.
  • Residual learning efficiently uses up to eight bilinear attention maps.

A.1 Enhancing glove word embedding

The appendix evaluates enhanced word embeddings, counting integration, and MFB-based variants. The embedding and counting variants improve controlled validation results, whereas MFB does not significantly improve BAN and increases GPU-memory consumption.

  • A.1 Enhancing glove word embedding: 66.03 (±0.12) versus 65.72 (±0.11) shows improved validation performance from the enhanced GloVe variant under controlled parameters.
  • A.2 Integrating counting module: The counting module produces dense representations from detected-object spatial information to improve counting-related performance.
  • A.2 Integrating counting module: 58.21 (±0.49) versus 54.92 (±0.30) improves counting performance, while overall performance rises from 65.81 (±0.09) to 66.01 (±0.14).
  • A.3 Integrating multimodal factorized bilinear (MFB) pooling: MFB pooling does not significantly improve BAN-1 and increases peak GPU-memory consumption, hindering multiple-glimpse BAN.
Loading 1805.07932v2…