PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai

arXiv:2010.04159v4cs.CV

TL;DR

DETR is limited by slow convergence and weak small-object detection because Transformer attention struggles with image feature maps. Deformable DETR replaces this attention with sparse sampling around reference points, achieving better COCO performance, especially on small objects, with 10× fewer training epochs.

  • Problem

    DETR requires many training epochs to converge and performs relatively poorly on small objects because Transformer attention handles image feature maps inefficiently.

  • Method

    Deformable DETR replaces Transformer attention with deformable attention that samples a small set of key locations around each reference point and aggregates multi-scale features.

  • Results

    10× fewer training epochs yield better COCO performance than DETR, especially for small objects, while two-stage Deformable DETR further improves performance.

  • Takeaways & Limitations

    Deformable DETR provides an efficient, fast-converging end-to-end detector and supports practical detector variants such as iterative refinement and two-stage designs.

Abstract

from arXiv · show

DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10 times less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach. Code is released at https://github.com/fundamentalvision/Deformable-DETR.

1 INTRODUCTION

Deformable DETR addresses DETR’s slow convergence and weak small-object performance by combining sparse spatial sampling with Transformer relation modeling. It introduces deformable attention, iterative box refinement, and a two-stage variant, achieving better COCO performance with 10× fewer training epochs.

  • Motivation: DETR requires 500 COCO training epochs to converge, around 10 to 20 times slower than Faster R-CNN, and performs relatively poorly on small objects.These limitations arise from DETR’s attention processing of image feature maps and its limited exploitation of multi-scale features.
  • Approach: Deformable DETR combines deformable convolution’s sparse spatial sampling with Transformers’ element-relation modeling to mitigate DETR’s convergence and complexity issues.The approach preserves sparse spatial attention while retaining the relation modeling mechanism identified as central to DETR’s success.
  • Approach: Its deformable attention module attends to a small set of sampling locations as a pre-filter for prominent key elements among all feature-map pixels.The module naturally extends to aggregating multi-scale features.
  • Variants: The paper explores iterative bounding-box refinement and a two-stage Deformable DETR whose generated region proposals are fed into the decoder for iterative refinement.These mechanisms extend the end-to-end detector and target improved detection performance.
  • Results: 10× fewer training epochs yields better performance than DETR on COCO, especially for small objects, while the two-stage variant further improves performance.The paper reports extensive COCO experiments demonstrating the effectiveness of the approach.

2 RELATED WORK

The related work organizes efficient attention into predefined sparse patterns, data-dependent sparsity, and low-rank approximations, while noting limitations of existing image-domain designs. It also situates deformable attention among deformable convolutions and multi-scale feature methods for object detection.

  • Efficient Attention Mechanism: Transformers face high time and memory complexity when processing vast numbers of key elements, motivating three categories of efficient attention mechanisms.The categories are predefined sparse attention, data-dependent sparse attention, and low-rank self-attention.
  • Efficient Attention Mechanism: Predefined sparse attention restricts key access through fixed local windows, fixed-interval patterns, special tokens, or predefined patterns for distant keys.These designs trade unrestricted attention for structured sparsity and broader receptive fields.
  • Efficient Attention Mechanism: Data-dependent sparse attention selects keys using learned or input-dependent structure, including locality-sensitive hashing, k-means matching, and learned block permutations.Kitaev et al. use LSH, Roy et al. use k-means, and Tay et al. learn block permutations.
  • Efficient Attention Mechanism: Low-rank approaches reduce self-attention cost through linear projection along the size dimension or kernelization-based approximations.Wang et al. reduce key elements through projection, while Katharopoulos et al. and Choromanski et al. rewrite self-attention using kernelization.
  • Efficient Attention Mechanism: In image attention, existing sparse designs remain limited to predefined patterns, and some are at least 3× slower than traditional convolution under equal FLOPs.Deformable convolution is more effective and efficient for image recognition but lacks element relation modeling; deformable attention addresses this tradeoff by predicting a small fixed set of sampling points from each query feature and being only slightly slower than convolution.
  • Multi-scale Feature Representation for Object Detection: Object detectors use multi-scale features to represent objects at different scales; FPN combines scales with a top-down path, while PANet adds a bottom-up path.These methods address the challenge of representing objects with vastly different scales.

3 REVISITING TRANSFORMERS AND DETR

Transformers aggregate key contents through attention but face long convergence schedules and quadratic computational growth on image feature maps. DETR applies this encoder-decoder design with Hungarian matching for end-to-end detection, yet inherits limitations in small-object detection and attention-map learning.

  • Multi-Head Attention in Transformers: Multi-head attention adaptively aggregates key contents according to query-key compatibility weights, using content and positional embeddings to distinguish spatial positions.Attention heads combine learned projections of query and key elements, while positional embeddings disambiguate locations.
  • Multi-Head Attention in Transformers: Transformers require long training schedules because diffuse attention weights produce ambiguous gradients when the number of key elements is large.With suitable initialization, attention weights are approximately 1/Nk for large Nk.
  • Multi-Head Attention in Transformers: O(NqNkC) dominates image-domain multi-head attention, causing computational and memory complexity to grow quadratically with feature-map size.The full complexity is O(NqC^2 + NkC^2 + NqNkC), with Nq = Nk much larger than C for image pixels.
  • DETR: DETR combines a Transformer encoder-decoder with set-based Hungarian loss and bipartite matching to force unique predictions for ground-truth bounding boxes.CNN feature maps are transformed into object-query features, followed by an FFN and linear projection detection head.
  • DETR: DETR’s attention deficits on image feature maps contribute to relatively low small-object performance and require long schedules to learn substantial attention-map changes.High-resolution feature maps improve small-object detection but make attention computation prohibitively expensive, while significant attention-map changes are difficult to learn.

4 METHOD

Deformable DETR replaces dense Transformer attention with deformable attention that samples a small set of points around reference locations, extending this mechanism across feature scales and encoder-decoder attention. The design yields linear encoder complexity, spatial-size-independent decoder complexity, and supports iterative refinement and two-stage proposals.

  • Deformable Attention Module: Deformable attention attends to a small set of key sampling points around each reference point, regardless of feature-map spatial size.Sampling offsets and attention weights are predicted from the query feature.
  • Multi-scale Deformable Attention Module: Multi-scale deformable attention extends the module to sample multiple points from multiple feature levels.It uses normalized reference coordinates and re-scales them to each input feature map.
  • Deformable Transformer Encoder: The encoder replaces DETR’s feature-map attention with multi-scale deformable attention over four 256-channel feature maps derived from ResNet stages C3 through C5 and C6.The encoder preserves the input feature-map resolutions and does not use an FPN top-down structure.
  • Deformable Transformer Decoder: Decoder object queries predict normalized reference points, and the detection head predicts bounding boxes as offsets relative to those points.This uses the reference point as the initial estimate of the box center to reduce optimization difficulty.
  • Deformable DETR Variants: Each decoder layer can iteratively refine boxes, while a two-stage variant generates region proposals that become decoder object queries.These variants extend Deformable DETR’s end-to-end detection system.

5 EXPERIMENT

Experiments on COCO 2017 show that Deformable DETR improves convergence and small-object detection while maintaining competitive computational efficiency. Ablations and comparisons further demonstrate gains from multi-scale deformable attention, sampling points, refinement, two-stage design, and stronger backbones.

  • Experimental setup: Experiments use COCO 2017, training on the train set and evaluating on the val and test-dev sets.Ablations use ImageNet-pretrained ResNet-50, multi-scale feature maps without FPN, M = 8, and K = 4 by default.
  • Main results: 10× fewer training epochs enable Deformable DETR to achieve better performance than DETR, especially on small objects.Iterative bounding box refinement and the two-stage paradigm further improve detection accuracy.
  • Efficiency: 1.6× faster runtime than DETR-DC5 is achieved with on-par FLOPs, while the method is just 25% slower than Faster R-CNN + FPN.Deformable attention reduces memory-access overhead relative to Transformer attention but remains slightly slower than traditional convolution.
  • Ablation study: 1.7% AP comes from multi-scale inputs, including 2.9% APS for small objects; increasing K adds 0.9% AP, and multi-scale attention adds 1.5% AP.These ablations evaluate design choices in the deformable attention module, including cross-level information exchange.
  • State-of-the-art comparison: 52.3 AP is achieved with ResNeXt-101 plus DCN and additional test-time augmentations, compared with 48.7 AP using ResNet-101 and 49.0 AP using ResNeXt-101.The models in this comparison use iterative bounding box refinement and the two-stage mechanism.

6 CONCLUSION

Deformable DETR is an efficient, fast-converging end-to-end object detector centered on multi-scale deformable attention for processing image feature maps. The authors suggest it enables practical detector variants and new directions for end-to-end object detection.

  • Deformable DETR is an efficient and fast-converging end-to-end object detector.
  • Its core component is multi-scale deformable attention, an efficient mechanism for processing image feature maps.
  • The detector enables exploration of more interesting and practical end-to-end object-detector variants.
  • The work aims to open new possibilities for exploring end-to-end object detection.

A APPENDIX · A.1 COMPLEXITY FOR DEFORMABLE ATTENTION

The appendix derives the computational complexity of deformable attention by separating offset and weight calculation from attention computation. Under the default experimental settings, the overall complexity simplifies to O(2NqC^2 + min(HWC^2, NqKC^2)).

  • A.1 COMPLEXITY FOR DEFORMABLE ATTENTION: The complexity of calculating sampling coordinate offsets and attention weights is O(3NqCMK).This term accounts for the deformable attention module's offset and weight calculations.
  • A.1 COMPLEXITY FOR DEFORMABLE ATTENTION: Given offsets and attention weights, computing Equation 2 has complexity O(NqC^2 + NqKC^2 + 5NqKC).The 5NqKC term reflects bilinear interpolation and the weighted sum in attention.
  • A.1 COMPLEXITY FOR DEFORMABLE ATTENTION: The factor 5 in 5NqKC arises from bilinear interpolation and the weighted sum in attention.These operations contribute to the attention computation after offsets and weights are available.
  • A.1 COMPLEXITY FOR DEFORMABLE ATTENTION: Computing W′mx before sampling changes Equation 2's complexity to O(NqC^2 + HWC^2 + 5NqKC).W′mx can be calculated before sampling because it is independent of the query.
  • A.1 COMPLEXITY FOR DEFORMABLE ATTENTION: The overall deformable attention complexity is O(NqC^2 + min(HWC^2, NqKC^2) + 5NqKC + 3NqCMK).The minimum selects between the alternative costs for computing the transformed feature before or during sampling.
  • A.1 COMPLEXITY FOR DEFORMABLE ATTENTION: In the experiments, M = 8, K ≤ 4, and C = 256 by default.These are the default settings used to simplify the complexity expression.
  • A.1 COMPLEXITY FOR DEFORMABLE ATTENTION: Under the defaults, 5K + 3MK < C, yielding complexity O(2NqC^2 + min(HWC^2, NqKC^2)).The inequality makes the terms involving K and M lower-order relative to the C-dependent terms.

A.2 CONSTRUCTING MULT-SCALE FEATURE MAPS FOR DEFORMABLE DETR

Deformable DETR constructs four encoder feature maps from ResNet stages C3–C5, adds a lower-resolution map from C5, and omits FPN because multi-scale deformable attention exchanges information across scales.

  • Feature-map construction: The encoder uses L = 4 feature maps from ResNet stages C3–C5, transformed with 1×1 convolutions, plus a lowest-resolution map from C5 using a 3×3 stride 2 convolution.These maps form the input multi-scale feature maps {x_l}^L−1_{l=1}.
  • Feature-map construction: FPN is omitted because multi-scale deformable attention can exchange information among the feature maps directly.The attention mechanism itself provides cross-scale information exchange.

A.3 BOUNDING BOX PREDICTION IN DEFORMABLE DETR

Deformable DETR predicts bounding boxes as normalized offsets relative to a reference point, which serves as the initial box-center estimate. This design aligns decoder attention with predicted boxes and accelerates training convergence.

  • Bounding-box parameterization: The detection head predicts bounding-box coordinates as relative offsets with respect to the reference point.The reference point provides the initial guess for the box center.
  • Optimization effect: The relative-offset formulation creates strong correlation between learned decoder attention and predicted boxes, accelerating training convergence.This follows from extracting image features around the reference point with multi-scale deformable attention.
  • Bounding-box parameterization: Sigmoid-transformed center and size predictions constrain the bounding box to normalized coordinates in [0, 1]^4.The predicted box is expressed as {σ(b_qx), σ(b_qy), σ(b_qw), σ(b_qh)}.

A.4 MORE IMPLEMENTATION DETAILS … A.7 NOTATIONS

The paper details iterative and two-stage box refinement, initialization procedures, and attention sampling behavior. Visualizations show that Deformable DETR uses object boundaries and interiors while adapting sampling to object scale and shape.

  • A.4 MORE IMPLEMENTATION DETAILS: Each decoder layer refines bounding boxes from the preceding layer’s predictions, with separate prediction-head parameters across decoder layers.The implementation considers D decoder layers, for example D = 6.
  • A.4 MORE IMPLEMENTATION DETAILS: The initial reference box uses q_h = 0.1, while values from 0.05 to 0.5 produce similar performance.The tested values are 0.05, 0.1, 0.2, and 0.5.
  • A.4 MORE IMPLEMENTATION DETAILS: Iterative refinement makes sampling locations depend on the previous box’s center and size, using its center as the new reference point and modulating offsets by box dimensions.Gradients are back-propagated only through the predicted box offsets to stabilize training.
  • A.4 MORE IMPLEMENTATION DETAILS: In the two-stage variant, encoder pixels receive a 3-layer FFN for box regression and a linear foreground/background classifier trained with the Hungarian loss.The base object scale is s = 0.05.
  • A.4 MORE IMPLEMENTATION DETAILS: Top-scoring first-stage boxes become decoder region proposals, providing initial boxes and positional embeddings for iterative refinement.The query positional embeddings are set from the region-proposal coordinates.
  • A.5 WHAT DEFORMABLE DETR LOOKS AT?: Gradient visualizations show that bounding-box coordinates rely on object boundaries, whereas category prediction also uses pixels inside the object.The x coordinate and width use left/right boundaries; the y coordinate and height use top/bottom boundaries.
  • A.6 VISUALIZATION OF MULTI-SCALE DEFORMABLE ATTENTION: Multi-scale deformable attention adapts its sampling points and attention weights to the foreground object’s scale and shape.The visualizations combine sampling points and attention weights from feature maps at different resolutions.
  • A.6 VISUALIZATION OF MULTI-SCALE DEFORMABLE ATTENTION: Encoder representations separate instances, while decoder attention focuses on whole foreground instances rather than only extreme points.The authors attribute interior attention partly to category prediction requiring interior points.
Loading 2010.04159v4…