PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer with Deformable Attention

Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, Gao Huang

arXiv:2201.00520v3cs.CV

TL;DR

Vision Transformers face a trade-off between costly dense attention and data-agnostic sparse patterns that may miss relevant long-range relations. The paper introduces DAT, which learns query-conditioned offsets for shared keys and values, and reports consistent improvements across classification, segmentation, and detection benchmarks.

  • Problem

    Dense attention incurs excessive computation, while hand-crafted sparse attention can drop relevant keys and values because its pattern is data-agnostic.

  • Method

    DAT uses query-conditioned offsets to shift shared keys and values toward important regions, forming a deformable self-attention backbone for vision tasks.

  • Results

    DAT consistently outperforms competitive baselines, including Swin Transformer, by 0.7 top-1 accuracy, 1.2 mIoU, and 1.1 box AP and mask AP.

  • Takeaways & Limitations

    DAT provides data-dependent sparse attention while retaining linear space complexity and supports image classification and dense prediction tasks.

Abstract

from arXiv · show

Transformers have recently shown superior performances on various vision tasks. The large, sometimes even global, receptive field endows Transformer models with higher representation power over their CNN counterparts. Nevertheless, simply enlarging receptive field also gives rise to several concerns. On the one hand, using dense attention e.g., in ViT, leads to excessive memory and computational cost, and features can be influenced by irrelevant parts which are beyond the region of interests. On the other hand, the sparse attention adopted in PVT or Swin Transformer is data agnostic and may limit the ability to model long range relations. To mitigate these issues, we propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant regions and capture more informative features. On this basis, we present Deformable Attention Transformer, a general backbone model with deformable attention for both image classification and dense prediction tasks. Extensive experiments show that our models achieve consistently improved results on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/DAT.

1. Introduction

Vision Transformers offer strong long-range modeling but dense attention is costly, while hand-crafted sparse patterns may discard relevant features. DAT addresses this trade-off with data-dependent deformable attention and reports consistent gains across classification and dense prediction.

  • Vision Transformers provide larger receptive fields and stronger long-range dependency modeling than CNN counterparts.
  • Dense attention increases computational cost, slows convergence, and raises overfitting risk by attending to excessive keys per query.
  • Hand-crafted sparse patterns such as windows or downsampled keys and values are data-agnostic, potentially retaining irrelevant features while dropping relevant ones.
  • DAT learns query-conditioned offsets that shift shared keys and values toward important regions while maintaining linear space complexity.Uniform reference points are first generated, then an offset network uses query features to produce offsets for candidate keys and values.
  • DAT is a deformable self-attention backbone for image classification and dense prediction tasks.
  • 0.7 top-1 accuracy, 1.2 mIoU, and 1.1 box AP and mask AP separate DAT from competitive baselines across reported benchmarks.The reported advantage for small and large objects reaches 2.1.

2. Related Work

Prior work uses efficient or deformable mechanisms to improve visual representations, but existing approaches do not incorporate deformable attention into general vision backbones. DAT instead learns shared global keys for visual tokens, with deformable attention illustrated through query-derived sampling.

  • Vision Transformer research has explored windowed attention, global tokens, focal attention, dynamic token sizes, and convolutional inductive biases.
  • Deformable CNNs condition flexible spatial locations on input data, while Deformable DETR selects a small number of keys for each query.
  • Existing deformable vision methods do not incorporate deformable attention into vision backbones or provide DAT's shared global keys across visual tokens.
  • DAT's deformable attention samples features at query-derived deformed points, projects them into keys and values, and applies standard multi-head attention with relative position bias.

3. Deformable Attention Transformer

DAT replaces fixed sparse attention patterns with deformable attention that samples keys and values at learned, data-dependent locations. Its hierarchical architecture combines local and deformable attention while keeping computational overhead modest.

  • Deformable Attention: Data-dependent sampling addresses information loss from fixed sparse patterns and avoids the biquadratic space complexity of independently deforming every query-key pair.The paper contrasts deformable attention with downsampling, shifted windows, and direct deformable-convolution-style attention.
  • Deformable Attention: Deformable attention learns groups of query-agnostic offsets that shift keys and values toward important regions.Sampled features are projected into deformed keys and values before standard multi-head attention aggregates them.
  • Deformable Attention: Bilinear interpolation samples differentiable features at deformed points, using nearby feature locations to construct keys and values.The interpolation reduces to a weighted average over the four nearest integral locations.
  • Computational Complexity: The offset network adds 5.08M FLOPs to a 79.63M-FLOP attention module, corresponding to only 6.0% overhead in the stated Swin-T example.A larger downsampling factor can further reduce complexity for high-resolution tasks.
  • Model Architectures: DAT uses a four-stage pyramid, with local attention followed by deformable attention in later stages to combine local aggregation with global relations.The first two stages use shift-window attention, while deformable attention is placed only in the third and fourth stages to balance capacity and computation.

4. Experiments

Experiments evaluate DAT across ImageNet classification, COCO detection and instance segmentation, ADE20K segmentation, ablations, and visualizations. DAT consistently improves over competitive baselines, while ablations examine geometric components and stage placement.

  • Experimental setup: DAT is evaluated on ImageNet-1K classification, COCO detection and instance segmentation, and ADE20K semantic segmentation.The experiments also include ablation studies and visualizations.
  • COCO object detection: DAT outperforms Swin Transformer by 1.1 and 1.2 mAP on RetinaNet for tiny and small models, respectively.Across two-stage detectors, DAT also achieves consistent improvements, with gains up to +2.1 for object-size categories.
  • ADE20K semantic segmentation: DAT improves ADE20K segmentation over Swin Transformer by +1.0, +0.7, and +1.2 mIoU at the tiny, small, and base scales.DAT-T also exceeds PVT-S by +0.5 mIoU with fewer FLOPs, while larger variants achieve +3.1 and +2.5 mIoU gains over PVT.
  • Attention placement: Replacing Swin attention with deformable attention in the last two stages yields a 0.7 gain and 82.0 overall accuracy, whereas earlier replacement slightly decreases accuracy.Replacing attention only in the last stage improves accuracy by 0.1.
  • Attention visualization: Visualizations show deformable attention placing the most important keys mostly in object foreground regions.Orange circles indicate key points with the highest propagated attention scores, and larger circles indicate higher scores.

5. Conclusion

The paper presents Deformable Attention Transformer, a hierarchical Vision Transformer for image classification and dense prediction. Its deformable attention learns sparse, data-dependent patterns and models geometric transformations, with experiments demonstrating effectiveness over competitive baselines.

  • Conclusion: Deformable Attention Transformer is a hierarchical Vision Transformer adaptable to image classification and dense prediction tasks.The model uses a deformable attention module.
  • Conclusion: DAT learns sparse-attention patterns in a data-dependent way and models geometric transformations.The authors position the approach as a flexible attention technique for visual recognition.
  • Conclusion: Extensive experiments demonstrate DAT's effectiveness over competitive baselines.

A. DAT and Deformable DETR

DAT’s deformable attention is designed as a vision-backbone feature extractor, unlike the deformable attention in Deformable DETR, and performs better under comparable memory and computation.

  • A. DAT and Deformable DETR: DAT’s deformable attention serves as a vision-backbone feature extractor, whereas Deformable DETR uses deformable attention as a detection head.The comparison concerns the role and design of deformable attention in the two systems.
  • A. DAT and Deformable DETR: Deformable DETR predicts attention weights directly from queries and applies softmax normalization across its K sampled keys.Its weights are produced by a linear projection of the query rather than by measuring query-key similarities.
  • A. DAT and Deformable DETR: Deformable DETR’s attention is not compatible with dot-product attention because of enormous memory consumption, motivating linear attention and K = 4 in one comparison.These choices reduce memory cost but differ from the dot-product attention setting discussed for DAT.
  • A. DAT and Deformable DETR: 1.4% lower performance is reported for the deformable DETR model using 16 keys under a smaller memory budget.The comparison is between the first and last rows of Table 8.
  • A. DAT and Deformable DETR: 2.6× memory and 1.3× FLOPs are required by D-DETR with the same number of keys as DAT, while its performance remains lower.This comparison is reported between the third and last rows of Table 8.

B. Adding Convolutions to DAT

DAT can incorporate convolutional patch embeddings while retaining its backbone design, and the added convolutions improve ImageNet classification performance.

  • B. Adding Convolutions to DAT: DAT readily combines with convolutional patch embedding techniques, although the main paper maintains a convolution-free architecture for fair baseline comparison.Prior work cited in the passage reports typical ImageNet gains of 0.5% ∼1.0% from convolutional patch embedding.
  • B. Adding Convolutions to DAT: 0.7% improvement is achieved when DAT uses strided and overlapped convolutional patch embeddings instead of the original patch embedding layers.The modified model also consistently outperforms the other baselines in the comparison.

C. More Visualizations

DAT visualizations show that its learned sampling locations and attention maps shift toward relevant foreground regions, including target objects and object-associated structures.

  • C. More Visualizations: DAT’s learned sampling points are visualized over object-detection boxes and instance-segmentation masks at Stages 3 and 4.Orange and yellow points represent one group of deformed points.
  • C. More Visualizations: The sampling points shift toward target objects, contracting around two giraffes and concentrating on a person’s body and surfboard.Other points remain close to a uniform grid with small offsets in the giraffe example.
  • C. More Visualizations: Compared with Swin Transformer, DAT allocates more attention to foreground objects, while Swin’s regions of interest remain comparatively local and fail to distinguish foreground from background in the surfboard example.The visualization compares key tokens with the highest attention values for specific query tokens.
Loading 2201.00520v3…