PaperlayerView exact revision on arXiv

Source-linked AI summary

DN-DETR: Accelerate DETR Training by Introducing Query DeNoising

Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, Lei Zhang

arXiv:2203.01305v3cs.CVcs.AI

TL;DR

DETR achieves end-to-end detection but converges prohibitively slowly because unstable bipartite matching creates inconsistent optimization goals. DN-DETR introduces denoising training to stabilize this process, accelerating convergence and improving performance, including the best results in the 12-epoch setting.

  • Problem

    DETR’s dynamic bipartite matching can change dramatically with small cost-matrix changes, causing inconsistent decoder-query optimization and prohibitively slow convergence.

  • Method

    DN-DETR adds denoising training to a DAB-DETR-based architecture, using noisy ground-truth boxes and label embeddings so the decoder reconstructs the original targets alongside Hungarian loss.

  • Results

    +1.9 AP over the DAB-DETR baseline is reported, while DN-DETR achieves the best results in the 1x (12 epochs) setting.

  • Takeaways & Limitations

    Denoising training significantly accelerates convergence, improves performance, and can be integrated into other DETR-like, segmentation, and CNN detection models.

  • Takeaways & Limitations

    The authors identify applying denoising training to weakly labeled pre-training and other denoising schemes as future work.

Abstract

from arXiv · show

We present in this paper a novel denoising training method to speedup DETR (DEtection TRansformer) training and offer a deepened understanding of the slow convergence issue of DETR-like methods. We show that the slow convergence results from the instability of bipartite graph matching which causes inconsistent optimization goals in early training stages. To address this issue, except for the Hungarian loss, our method additionally feeds ground-truth bounding boxes with noises into Transformer decoder and trains the model to reconstruct the original boxes, which effectively reduces the bipartite graph matching difficulty and leads to a faster convergence. Our method is universal and can be easily plugged into any DETR-like methods by adding dozens of lines of code to achieve a remarkable improvement. As a result, our DN-DETR results in a remarkable improvement ($+1.9$AP) under the same setting and achieves the best result (AP $43.4$ and $48.6$ with $12$ and $50$ epochs of training respectively) among DETR-like methods with ResNet-$50$ backbone. Compared with the baseline under the same setting, DN-DETR achieves comparable performance with $50\%$ training epochs. Code is available at \url{https://github.com/FengLi-ust/DN-DETR}.

1 INTRODUCTION

DETR eliminates hand-designed anchors and NMS through end-to-end set prediction, but its bipartite matching is unstable and training converges slowly. DN-DETR adds denoising training to stabilize matching and accelerate convergence.

  • DETR uses learnable queries and bipartite graph matching for end-to-end box prediction without hand-designed anchors or NMS.
  • 500 epochs are typically required for good COCO performance, compared with 12 epochs for the original Faster-RCNN training.
  • Early-training bipartite matching is unstable, so the same query can match different objects across epochs and receive inconsistent optimization goals.
  • DN-DETR feeds noised ground-truth boxes alongside learnable anchor queries into Transformer decoders to stabilize matching through query denoising.
  • The method combines reconstruction and Hungarian losses and is designed to plug into existing DETR-like methods.
  • +1.9 AP improvement over the DAB-DETR baseline and the best result among detection algorithms in the 12-epoch setting are reported.

2 RELATED WORK

Related work improves DETR architectures and queries, while this paper focuses on bipartite matching as a source of slow convergence. Its denoising loss is presented as a distinct, broadly applicable training intervention.

  • Classical CNN detectors rely on hand-crafted anchors, label assignment rules, or NMS, preventing fully end-to-end optimization.
  • DETR removes anchors but requires 500 training epochs for good performance, motivating methods that accelerate convergence.
  • Prior DETR work accelerates training by changing cross-attention, decoders, or query representations rather than treating bipartite matching as the main cause.
  • This method adds a denoising loss as an auxiliary task alongside Hungarian loss, distinguishing its training strategy from architectural changes and other noise-object augmentation.
  • Subsequent models adopt denoising training for detection and segmentation, with extensions including hard-negative rejection and mask reconstruction.

3 WHY DENOISING ACCELERATES DETR TRAINING?

DETR's discrete bipartite matching can change substantially during early stochastic training, producing inconsistent optimization goals. DN-DETR uses denoising to bypass matching for relative-offset learning, reducing matching instability and anchor-target distances.

  • Hungarian matching can change enormously after small cost-matrix changes, creating inconsistent optimization goals for decoder queries.
  • DN-DETR treats training as learning good anchors and relative offsets, using denoising to bypass bipartite matching for the latter.
  • The instability metric compares each image's matching index vectors across consecutive epochs and averages the differences over the dataset.
  • DN-DETR effectively alleviates matching instability on COCO 2017 validation, where the maximum possible IS is 14.72.
  • DN-DETR produces smaller mean l1 distances between initial anchors and matched ground-truth boxes than DAB-DETR.
  • Shorter anchor-target distances make queries focus on nearby regions, prevent potential prediction conflicts, and make training easier to converge.

4 DN-DETR

DN-DETR adds denoising queries alongside matching queries in a Transformer decoder. Attention masks isolate denoising groups and prevent information leakage, while the approach remains a plug-in training method for DETR-like models.

  • The Transformer decoder maps queries and encoded image features, together with the denoising-derived attention mask, to decoder outputs.
  • DN-DETR's decoder contains matching queries from learnable anchors and denoising queries from noised ground-truth box-label pairs.
  • The matching part uses bipartite graph matching, while the denoising part reconstructs the original ground-truth objects.
  • The method uses multiple noised ground-truth versions and attention masks to prevent leakage between matching queries and denoising groups.

4.2 Intro to DAB-DETR

DAB-DETR represents decoder queries as dynamically updated 4D anchor boxes. DN-DETR keeps this anchor formulation while making a minimal decoder-embedding modification to support label denoising.

  • DAB-DETR represents each decoder query as a 4D tuple (x, y, w, h), containing box-center coordinates and width and height.
  • Each decoder layer outputs relative offsets (∆x, ∆y, ∆w, ∆h) that update the anchor coordinates layer by layer.
  • DN-DETR specifies the decoder embedding as a label embedding, adding only minimal modifications to DAB-DETR to support label denoising.

4.3 Denoising

Denoising training creates auxiliary reconstruction examples by perturbing ground-truth boxes and labels, then removes this branch during inference.

  • Each image’s ground-truth objects receive random noise in their bounding boxes and class labels, with multiple noisy versions per object.
  • Box noise uses center shifting and box scaling, controlled by separate noise-scale hyperparameters.
  • Label noising randomly flips some ground-truth labels, encouraging prediction of the original labels from noisy boxes.
  • Reconstruction uses L1 and GIOU losses for boxes and focal loss for class labels.
  • Denoising is used only during training; inference retains only the matching branch.

4.4 Attention Mask

The attention mask isolates denoising groups and the matching queries to prevent information leakage, while adding negligible training computation.

  • Noised ground-truth objects are divided into groups before constructing the attention mask.
  • Each denoising group contains one query for every ground-truth object in the image.
  • The mask prevents the matching branch from seeing denoising queries and prevents one denoising group from seeing another.
  • The mask matrix covers denoising and matching queries, with entries indicating whether one query can see another.
  • With 5 denoising groups, training GFLOPs increase from 94.4 to 94.6 for DAB-DETR with a ResNet-50 backbone, with no testing overhead.

4.5 Label Embedding

The model uses label embeddings to support box and label denoising, and the denoising approach is evaluated across additional DETR variants.

  • Decoder embeddings combine class-label information with an indicator distinguishing denoising queries from matching queries.
  • An unknown-class embedding supplements the 80 COCO classes for semantic consistency between denoising and matching parts.
  • Denoising training is integrated into Deformable DETR by representing its queries as 4D boxes.
  • DN-Deformable-DETR is compared with a same-setting anchor-box baseline without denoising, while DN-Deformable-DETR++ also adds deformable decoder attention.

4.7 Introducing DN to Other DETR-like models with different anchor formulations

Denoising training is presented as a plug-in method for DETR-like and traditional detectors with different query or label-assignment designs. The section describes adaptations for 2D anchors, anchor-free queries, and Faster R-CNN.

  • Introducing DN to Other DETR-like models with different anchor formulations: Denoising is described as a training-only method that can be plugged into other detection models to accelerate training.
  • Introducing DN to Anchor DETR with 2D Anchors: For Anchor DETR, denoising is applied directly to 2D anchor points using center-shifting noise without other modifications.
  • Introducing DN to Vanilla DETR without Explicit Anchors: Vanilla DETR represents positional queries as high-dimensional vectors without explicit meanings.
  • Introducing DN to Vanilla DETR without Explicit Anchors: Table 1 compares DN-DETR and other detection models under the same setting, using 300 queries for DETR-like models except DETR, which uses 100.
  • Introducing DN to Vanilla DETR without Explicit Anchors: For Vanilla DETR, noised boxes are linearly embedded to match the query dimension and combined with label embeddings.
  • Introducing DN to Faster R-CNN for Traditional Detectors: The method is extended to Faster R-CNN, whose RoI pooling and IoU-threshold label assignment differ from DETR’s cross-attention and bipartite matching.
  • Introducing DN to Faster R-CNN for Traditional Detectors: For Faster R-CNN, noised boxes are added alongside RPN boxes and directly regress the ground truth; only box denoising is used.

4.9 Introducing DN to Mask2Former for Segmentation Models

The paper extends denoising training to Mask2Former-style segmentation by perturbing ground-truth masks, and a simple shifting-noise baseline demonstrates its effectiveness.

  • Introducing DN to Mask2Former for Segmentation Models: Denoising training is extended to segmentation models such as Mask2Former by adding noise to ground-truth masks.Mask2Former uses a DETR-like decoder whose predicted masks guide attention in subsequent layers.
  • Introducing DN to Mask2Former for Segmentation Models: A simple baseline shifts each ground-truth mask randomly along the x- and y-axes without changing its shape or size.The shift corresponds to the center-shifting noise described for detection.
  • Introducing DN to Mask2Former for Segmentation Models: The shifting-noise baseline already demonstrates the effectiveness of denoising training.

5 EXPERIMENT

Experiments on detection and segmentation models show that denoising training improves performance, accelerates convergence, and transfers across DETR-like architectures with small computational overhead.

  • Performance Improvement: +1.9 AP is achieved over DAB-DETR with ResNet-50 under exactly the same setting, with negligible added parameters and computation.
  • 1× Setting: +3.7 AP is achieved by DN-DETR over DAB-DETR within 12 epochs with the DC5-R50 backbone.DN-Deformable-DETR achieves 44.1 AP within 12 epochs with ResNet-101, surpassing Faster R-CNN ResNet-101 trained for 108 epochs.
  • Extending DN to Other Detection and Segmentation Models: 1.2 −2.6 AP improvements are reported for DETR-like detection models under the 12-epoch setting.
  • Extending DN to Other Detection and Segmentation Models: Denoising training is compatible with positional queries, deformable attention, semantic alignment, and query selection.The paper also reports extensions to other detection and segmentation models.
  • Compared with State-of-Art Detectors: 48.6 AP is achieved by DN-Deformable-DETR with the ResNet-50 backbone in the multi-scale comparison.Against the strong DAB-Deformable-DETR baseline, denoising training still yields 1.7 AP absolute improvement.
  • Convergence Analysis: 40 AP is reached in 20 epochs by DN-DETR before learning-rate drop, whereas DAB-DETR needs 40 epochs in the plotted comparison.
  • Effectiveness of using more denoising groups: More denoising groups improve performance, but gains become marginal as their number increases.The default experimental setting uses 5 denoising groups.
  • Acceleration Analysis: 2x acceleration is reported because DN-DETR reaches comparable baseline performance with half the training epochs.DAB-DETR-R50 takes 2555 minutes for 50 epochs, while DN-DAB-DETR-R50 takes 1443 minutes for 25 epochs; denoising adds minor per-epoch cost.

6 CONCLUSION

The paper attributes DETR’s slow convergence to unstable bipartite matching and proposes denoising training as a general, low-overhead remedy, while leaving more complex noise schemes for future work.

  • Conclusion: Unstable bipartite matching is identified as the reason for DETR’s slow convergence, motivating a denoising training method.
  • Conclusion: Denoising training is integrated into DAB-DETR and Deformable DETR, accelerating convergence and improving detection performance.
  • Conclusion: The method is presented as a general training approach for DETR-like models with only a small training cost overhead.
  • Limitations and Future Work: The added noises are sampled from a uniform distribution, while more complex noising schemes remain unexplored.
Loading 2203.01305v3…