PaperlayerView exact revision on arXiv

Source-linked AI summary

DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, Lei Zhang

arXiv:2201.12329v4cs.CV

TL;DR

DETR’s learnable queries lead to slow convergence, while their role and positional behavior are not fully understood. DAB-DETR uses dynamically updated box coordinates as queries, with size-modulated positional attention and temperature tuning. It reports 45.7% AP on COCO with ResNet-50-DC5 after 50 epochs and the best performance among DETR-like models under the same setting.

  • Problem

    DETR typically requires 500 epochs for good performance, and the role and positional behavior of its learned queries remain insufficiently understood.

  • Method

    DAB-DETR directly uses dynamically updated 4D anchor boxes as decoder queries, using position, size, and temperature to shape positional cross-attention.

  • Results

    45.7% AP is achieved on COCO with ResNet-50-DC5 after 50 epochs, reported as the best performance among DETR-like architectures under the same setting.

  • Takeaways & Limitations

    The formulation interprets decoder queries as layer-by-layer soft ROI pooling in a cascade manner and provides a deeper understanding of their role.

  • Takeaways & Limitations

    Like other deep neural networks, the model can be vulnerable to adversarial examples and may produce training-data-induced bias.

Abstract

from arXiv · show

We present in this paper a novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer a deeper understanding of the role of queries in DETR. This new formulation directly uses box coordinates as queries in Transformer decoders and dynamically updates them layer-by-layer. Using box coordinates not only helps using explicit positional priors to improve the query-to-feature similarity and eliminate the slow training convergence issue in DETR, but also allows us to modulate the positional attention map using the box width and height information. Such a design makes it clear that queries in DETR can be implemented as performing soft ROI pooling layer-by-layer in a cascade manner. As a result, it leads to the best performance on MS-COCO benchmark among the DETR-like detection models under the same setting, e.g., AP 45.7\% using ResNet50-DC5 as backbone trained in 50 epochs. We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our methods. Code is available at \url{https://github.com/SlongLiu/DAB-DETR}.

1 INTRODUCTION

DETR uses learnable queries for set-based detection but converges slowly, and the role of those queries remains incompletely understood. DAB-DETR addresses this by using dynamically updated 4D anchor boxes as queries, improving positional priors through position and size information.

  • DETR uses 100 learnable queries to pool image features without non-maximum suppression, but typically requires 500 epochs for good performance.
  • The role of learned DETR queries remains not fully understood or utilized despite follow-up efforts to associate queries with spatial positions.
  • DAB-DETR uses 4D anchor boxes (x, y, w, h) as queries and updates them layer-by-layer.
  • Anchor-box queries provide positional priors using both location and size, supporting cross-attention that focuses on local regions and accommodates different object scales.
  • 45.7% AP is achieved on COCO with a single ResNet-50 backbone trained for 50 epochs, reported as the best performance among DETR-like architectures under the same setting.

2 RELATED WORK

Prior work improves DETR convergence and query localization through architectural changes, reference points, or Gaussian priors, but lacks a unified account of query roles. DAB-DETR instead treats all query information as box coordinates and compares models across anchor, attention, and update properties.

  • Prior approaches address DETR’s slow convergence by removing decoders, regulating cross-attention with Gaussian priors, or changing query representations.
  • Related methods make queries spatially explicit using reference points, dense top-K selection, or conditional positional embeddings.
  • DAB-DETR differs by treating all query information as box coordinates rather than assuming learnable query vectors contain box coordinates.
  • DAB-DETR directly learns anchor boxes, updates them layer-by-layer, retains standard dense cross-attention, and modulates attention for object scale.

3 WHY A POSITIONAL PRIOR COULD SPEEDUP TRAINING?

The paper attributes DETR’s slow convergence primarily to undesirable positional behavior in decoder queries rather than query optimization alone. Dynamic anchor boxes impose localized positional priors, and their improved training curves support this explanation.

  • Encoder and decoder attention share key and value components; their main difference is that decoder queries combine decoder content with learned positional queries.
  • Fixing pretrained DETR queries improves convergence only slightly during the first 25 epochs, suggesting query optimization is not the key concern.
  • Conditional DETR produces Gaussian-like priors but does not adapt them to object scale, whereas DAB-DETR modulates attention using anchor width and height.
  • Vanilla DETR queries can produce multimodal, overly broad, overly narrow, or nearly uniform positional attention, weakening localization.
  • Replacing DETR queries with dynamic anchor boxes yields better detection AP and training and testing losses than DETR.

4 DAB-DETR

DAB-DETR uses 4D dynamic anchor boxes as positional queries, updates them layer-by-layer, and modulates positional cross-attention with anchor size. This design frames decoder queries as cascade-like soft ROI pooling and adapts attention to object scale.

  • Overview: The framework combines positional anchor-box queries with content queries to probe image features for matching objects.Image features come from a CNN backbone followed by Transformer encoders, while dual queries are refined layer-by-layer toward target objects.
  • Learning Anchor Boxes Directly: DAB-DETR uses anchor boxes (x, y, w, h) as positional queries and updates them after each decoder layer.The decoder uses self-attention for query updating and cross-attention for feature probing; shared prediction heads estimate relative position updates.
  • Learning Anchor Boxes Directly: Positional queries are generated by applying sinusoidal encodings to anchor coordinates and projecting their concatenation with an MLP.The encoding covers x, y, w, and h, and the MLP parameters are shared across decoder layers.
  • Learning Anchor Boxes Directly: Cross-attention concatenates content information with position information, while keys combine image features with coordinate encodings.This separates content and positional contributions to query-to-feature similarity and uses conditional spatial-query scaling.
  • Width & Height-Modulated Gaussian Kernel: Anchor width and height modulate the positional attention map separately along x and y, replacing a fixed isotropic prior with scale-aware attention.The modulated attention is intended to extract features from objects with different widths and heights, as illustrated in Figure 6.
  • Temperature Tuning: The sinusoidal temperature controls positional-prior spread: larger T flattens the attention map, and DAB-DETR empirically uses T = 20.The paper notes that DETR coordinates are floats between 0 and 1, unlike the integer positions used in the original NLP setting with T = 10000.

5 EXPERIMENTS

Experiments evaluate DAB-DETR on COCO and isolate the contributions of its components, showing strong performance and benefits from dynamic anchor design, modulated attention, and temperature tuning.

  • Main results: DAB-DETR is evaluated against DETR and several other detection models on the COCO 2017 validation set.The comparison includes standard and pattern-embedding variants under the reported settings.
  • Main results: Pattern-embedding DAB-DETR outperforms previous DETR-like methods on all four backbones with a large margin.The paper reports that these results are even better than multiscale architectures.
  • Ablations: Removing modulated attention reduces performance to 45.0%, while removing temperature tuning reduces it to 44.4%.These comparisons use Row 1 as the reference configuration.

6 CONCLUSION

DAB-DETR uses dynamic anchor boxes as queries to provide size-aware positional attention and update anchors iteratively, clarifying queries as cascade-like soft ROI pooling.

  • Conclusion: DAB-DETR directly learns anchor boxes as queries and updates them layer-by-layer in Transformer decoders.The query formulation uses box coordinates and supports positional attention modulation by anchor size.
  • Conclusion: Anchor boxes provide a better positional prior through temperature tuning and size-modulated attention for objects at different scales.The method also uses iterative anchor updates to improve anchor estimates gradually.
  • Conclusion: The design interprets DETR queries as performing soft ROI pooling layer-by-layer in a cascade manner.The paper states that extensive experiments confirmed this analysis and verified the algorithm design.

ETHICS STATEMENT

The paper frames object detection as broadly consequential and notes risks associated with deep-learning-based detection systems, including adversarial attacks, training-data bias, and privacy or security threats.

  • Ethics statement: Object detection supports applications including autonomous vehicles, medical imaging, word recognition, and instance segmentation.The paper presents these applications as reasons improvements in detection can have broad impact.
  • Ethics statement: Because the model relies on deep neural networks, it can be attacked by adversarial examples.This is identified as a common deep-learning problem.
  • Ethics statement: Reliance on training data may produce biased results induced by the training samples.The paper identifies this as another common deep-learning problem.
  • Ethics statement: Face or human detection models may threaten people’s privacy and security if misused.The paper states this risk conditionally for harmful use.

REPRODUCIBILITY STATEMENT

The authors state that the results are reproducible and that the materials needed for reproduction, including the code, will be released after blind review.

  • Reproducibility statement: The authors confirm reproducibility and plan to release all materials needed to reproduce the results after blind review.They also state that the code will be open sourced.

A TRAINING DETAILS

DAB-DETR is compared with DETR-like architectures through query formulations, dynamic box updates, and training behavior. The experiments also inspect learned anchors and a Deformable DETR variant using the same dynamic-anchor design.

  • Training details: The implementation uses a CNN backbone, 6 Transformer encoders, 6 Transformer decoders, and prediction heads for boxes and labels.Experiments use ResNet-50, ResNet-101, and their 16×-resolution DC5 extensions on COCO train2017 and val2017 splits.
  • Model comparisons: DAB-DETR formulates queries as dynamic anchor boxes, while Anchor DETR uses anchor points and Deformable DETR couples box refinement to deformable attention.DAB-DETR is designed to let both self-attention and cross-attention leverage refined anchor boxes across decoder layers.
  • Model comparisons: Under the same R50 setting, DAB-Deformable-DETR improves Deformable-DETR by 0.5 AP, from 46.3 to 46.8.The only experimental change is the query formulation, with no more than 10 lines of code modified.
  • Training behavior: DAB-Deformable-DETR has a lower final-layer loss despite a higher total loss than Deformable DETR, and the final-layer result determines inference.The comparison also reports faster convergence for the dynamic-anchor variant.
  • Anchor analysis: Learned anchor points distribute evenly across the image, whereas learned anchor-box centers distribute randomly because centers are coupled with anchor sizes.Visualized learned boxes are mostly medium-sized, with no particular pattern reported.

E RESULTS WITH DIFFERENT TEMPERATURES

Temperature changes alter the positional attention map and shift performance across object sizes. Flattened attention maps favor larger objects, while lower temperatures perform better for small and medium objects.

  • Temperature effects: Larger temperature values flatten positional attention maps and lead to better performance for larger objects.This pattern links temperature choice to the scale of detected objects.
  • Temperature effects: T = 2 and T = 10000 achieve similar AP, but T = 2 performs better on APS and APM while T = 10000 performs better on APL.The result supports the role of positional priors in DETR.

F RESULTS WITH LESS DECODER LAYERS

The decoder-layer analysis examines models trained with different numbers of decoder layers and the stability of layer-by-layer box refinement. Dynamic anchor boxes are intended to make this refinement more stable.

  • Decoder depth: Models are compared under the standard ResNet-50-DC setting while varying only the number of decoder layers.The experiment is reported in Table 7.
  • Anchor-coordinate analysis: Fixing the initial x, y coordinates of anchor boxes with random values consistently improves performance, while later layers still update x and y.The coordinates are fixed only at the first layer to prevent learning positional information from data there.
  • Layer-wise refinement: DAB-DETR treats stacked decoder outputs as a layer-by-layer box-refinement procedure.Its dynamic anchors are updated across layers to approach target objects gradually.
  • Layer-wise refinement: Conditional DETR exhibits larger box variances and some large boundary changes between decoder layers.These changes are highlighted in yellow in the comparison figure.

I ANALYSIS OF FAILURE CASES

The failure-case and comparison analyses identify difficult object-scale and density conditions, while reporting runtime and convergence behavior against other DETR variants. DAB-DETR converges faster than Conditional DETR, especially early in training, with similar inference speed.

  • Failure cases: DAB-DETR struggles with dense scenes, very small objects, and very large objects.The authors propose introducing multi-scale techniques to improve detection on small and large objects.
  • Runtime: DAB-DETR has similar inference speed but better performance than Conditional DETR on a single Nvidia A100 GPU.Runtime comparisons include DETR, Conditional DETR, and DAB-DETR.
  • Convergence: DAB-DETR converges faster than Conditional DETR, especially during the early epochs.The convergence comparison uses the standard R50(DC5) setting, with 300 queries for DAB-DETR and Conditional DETR versus 100 for DETR.
Loading 2201.12329v4…