Source-linked AI summary
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, Heung-Yeung Shum
TL;DR
DETR-like detectors needed better convergence, performance, and evidence of scalability to large models and datasets. DINO combines improved denoising training, mixed query selection, and look forward twice, achieving state-of-the-art COCO results while using smaller model and pre-training data sizes than compared leaderboard models. Its results establish an end-to-end Transformer detector as a strong detection framework.
Problem
DETR-like models were inferior to improved classical detectors and lacked reported scalability results for large backbones and datasets.
Method
DINO is an end-to-end DETR-like detector using contrastive denoising training, mixed query selection, and look forward twice for box prediction.
Results
DINO achieves the best reported results of 63.2 AP on COCO val2017 and 63.3 AP on test-dev, while improving ResNet-50 results over previous best DETR-like models.
Takeaways & Limitations
DINO establishes an end-to-end Transformer detector as a mainstream detection framework with strong performance and scalability.
Abstract
from arXiv · showhide
We present DINO (\textbf{D}ETR with \textbf{I}mproved de\textbf{N}oising anch\textbf{O}r boxes), a state-of-the-art end-to-end object detector. % in this paper. DINO improves over previous DETR-like models in performance and efficiency by using a contrastive way for denoising training, a mixed query selection method for anchor initialization, and a look forward twice scheme for box prediction. DINO achieves $49.4$AP in $12$ epochs and $51.3$AP in $24$ epochs on COCO with a ResNet-50 backbone and multi-scale features, yielding a significant improvement of $\textbf{+6.0}$\textbf{AP} and $\textbf{+2.7}$\textbf{AP}, respectively, compared to DN-DETR, the previous best DETR-like model. DINO scales well in both model size and data size. Without bells and whistles, after pre-training on the Objects365 dataset with a SwinL backbone, DINO obtains the best results on both COCO \texttt{val2017} ($\textbf{63.2}$\textbf{AP}) and \texttt{test-dev} (\textbf{$\textbf{63.3}$AP}). Compared to other models on the leaderboard, DINO significantly reduces its model size and pre-training data size while achieving better results. Our code will be available at \url{https://github.com/IDEACVR/DINO}.
1 Introduction
DINO targets the performance and scalability gaps of DETR-like detectors with improved denoising training, query initialization, and box prediction. It achieves strong COCO results at short training schedules and with larger models and datasets.
- DETR background: DETR removes hand-designed components by formulating object detection as set prediction with bipartite matching and learnable queries.Queries probe object existence and combine image features similarly to soft ROI pooling.
- Motivation: DETR-like models previously faced slow convergence and unclear query meaning, while DAB-DETR and DN-DETR addressed these issues with dynamic anchor boxes and denoising.Their combination made DETR-like models competitive with classical detectors in training efficiency and inference performance.
- Motivation: Previous DETR-like models remained below 50 AP on COCO, and their scalability to large backbones and datasets had not been reported.DINO explicitly aims to address both performance and scalability concerns.
- DINO: DINO improves denoising training, query initialization, and box prediction through contrastive denoising, mixed query selection, and look forward twice.The model builds on DN-DETR, DAB-DETR, and Deformable DETR.
- Results: 49.4 AP in 12 epochs and 51.3 AP in 24 epochs with ResNet-50 and multi-scale features improved over the previous best DETR-like model by +6.0 AP and +2.7 AP.DINO also achieved 63.2 AP on COCO val2017 and 63.3 AP on test-dev after Objects365 pre-training with SwinL.
2 Related Work
Related work spans classical convolutional detectors, end-to-end Transformer detectors, query-focused DETR variants, and large-scale pre-training. These directions motivate DINO's emphasis on end-to-end detection, query design, and scalability.
- Convolution-based detectors: Classical detectors include two-stage and one-stage convolutional models built around hand-crafted anchors or reference points.Two-stage models use region proposal networks, while one-stage models predict offsets relative to predefined anchors.
- Transformer detectors: DETR introduced end-to-end Transformer detection without hand-designed anchor design and non-maximum suppression, prompting work on faster convergence.Follow-up approaches include encoder-only designs and dynamic decoders focused on important regions.
- Decoder queries: Query-focused methods associate decoder queries with spatial information using reference points, dense encoder predictions, or dynamic anchor boxes.Deformable DETR, Efficient DETR, and DAB-DETR represent successive approaches to spatially informed queries.
- Large-scale pre-training: Large-scale pre-training has improved computer-vision models, with leading detectors using large backbones and substantial pre-training datasets.The related work contrasts these resource-intensive approaches with DINO's publicly available pre-training setting.
3 DINO: DETR with Improved DeNoising Anchor Boxes
DINO improves DETR-like detection through contrastive denoising, mixed query selection, and look forward twice box prediction. Its architecture combines dynamic anchor-box queries, deformable attention, and image-adaptive positional initialization.
- Model overview: DINO formulates decoder positional queries as dynamic 4D anchor boxes and refines them across decoder layers.The boxes use center coordinates and dimensions as query representations.
- Contrastive DeNoising Training: Contrastive DeNoising adds positive and negative anchor queries so the decoder learns to reject anchors without nearby objects.Positive and negative examples are defined by their distance from a ground-truth box, and multiple CDN groups are used during training.
- Contrastive DeNoising Training: +1.3 AP on small objects is achieved by CDN queries over DN queries after 12 epochs with ResNet-50 and multi-scale features.CDN finds better anchors for small objects, while DN is described as sufficient for selecting good anchors overall.
- Mixed Query Selection: Mixed query selection initializes anchor boxes from positional information in the selected top-K encoder features while keeping content queries static or learnable.This differs from Deformable DETR, which uses selected features to enhance both positional and content queries.
- Look Forward Twice: Look forward twice updates box predictions so later-layer box information can influence adjacent earlier-layer parameters.The method contrasts with Deformable DETR’s look forward once update, which blocks gradient backpropagation for stabilization.
4 Experiments
DINO is evaluated on COCO across training schedules, backbones, model scales, and ablations. It improves detection performance and convergence while scaling to large models and datasets.
- Experimental Setup: DINO is evaluated on COCO 2017 using ResNet-50 or SwinL backbones, with SwinL models pre-trained on Objects365 before COCO fine-tuning.The ResNet-50 setting uses train2017 without extra data, while SwinL uses ImageNet-22k pre-training followed by Objects365 pre-training.
- Main Results: DINO uses 900 queries, while most other DETR-like models use 300 queries in the 12-epoch comparison.The table compares models on COCO val2017 and reports GFLOPS and FPS for models without multi-scale features using ResNet-50-DC5.
- Main Results: +5.6 AP is obtained with ResNet-50 and 4-scale features, while +6.0 AP is obtained with 5-scale features under the same setting.The 4-scale model adds little computation and parameter overhead, and gains +7.2 AP or +7.5 AP on small objects with 4 or 5 scales, respectively.
- Main Results: +1.8 AP and +2.7 AP are achieved after 24 epochs with 4 and 5 scales, respectively; after 36 epochs, gains reach +2.3 AP and +2.6 AP.The convergence comparison is shown in Fig. 7, supporting evaluation of both training speed and final performance.
- Comparison with SOTA Models: 63.2 AP on COCO val2017 and 63.3 AP on test-dev establish DINO as an end-to-end Transformer detector with strong scalability to larger model and data sizes.The SwinL model uses Objects365 pre-training and achieves these results without test-time augmentation.
- Ablation Studies: Ablation studies evaluate the effectiveness of DINO’s proposed algorithm components against optimized DN-DETR and query-selection baselines.The components are reported as Query Selection, Contrastive De-Noising Training, and Look Forward Twice.
5 Conclusion
DINO combines several training and prediction improvements into an end-to-end Transformer detector. The paper reports improved efficiency and detection performance, while its test-time augmentation exploration remains limited.
- Conclusion: DINO combines contrastive denoising training, mixed query selection, and look forward twice to improve training efficiency and final detection performance.The model is presented as an end-to-end Transformer detector.
- Conclusion: DINO achieves 63.3 AP on COCO 2017 test-dev after training with a stronger backbone on a larger dataset.The conclusion describes this as a new state-of-the-art result and reports that DINO outperforms previous ResNet-50-based models in 12- and 36-epoch settings.
- Conclusion: DN-query training can produce duplicate boxes for one person, whereas the proposed method is shown alongside it for comparison.Fig. 8 displays only person boxes and highlights three duplicate boxes in the DN-query result.
- Test Time Augmentations: Test-time augmentation does not provide an inspiring gain for DINO compared with traditional detectors.The exploration uses the large SwinL model and is included to compare with traditional detection models.
B Training Efficiency
The paper reports training resource measurements for DINO and compares architectural depth settings. These experiments frame DINO as both effective and efficient to train.
- Training Resource Usage: DINO’s base-model GPU memory and training time are reported using 8 Nvidia A100 GPUs with a ResNet-50 backbone.The authors state that the results demonstrate training effectiveness and efficiency.
- Training Resource Usage: Table 5 compares training efficiency across models with ResNet-50 under a common 8-GPU setup.The table includes training-efficiency measurements for different models and identifies framework-specific testing for Faster R-CNN.
- Architecture Ablation: Table 6 ablates the numbers of encoder and decoder layers using ResNet-50 on COCO val2017 in the 12-epoch setting.The ablation uses 100 denoising queries without negative samples.
C Additional Analysis on our Model Components
The ablations examine decoder depth and denoising-query count, showing that both choices affect performance, with diminishing returns beyond 100 denoising queries.
- Number of Encoder and Decoder Layers: A 3.0 AP drop results when decoder layers decrease from 6 to 2 with 6 encoder layers fixed.The analysis attributes this sensitivity to dynamic box updates and refinement across decoder layers.
- Query Denoising: Increasing denoising queries improves performance significantly below 100 queries.The experiment uses the optimized dynamic denoising-group setting.
- Query Denoising: Beyond 100 denoising queries, additional queries provide only small or sometimes worse performance improvements.Table 7 reports the ablation on COCO validation with a ResNet-50 backbone.
D.1 Dynamic DN groups
The dynamic DN-group procedure improves denoising efficiency by using multiple noised versions of ground-truth groups while accounting for variable object counts across minibatches.
- Ground-Truth Grouping: DN-DETR forms one denoising ground-truth group from all labeled objects and boxes in each image.Multiple noised versions of this group are used during training.
- Group Count: DN-DETR sets the number of denoising groups to five or ten according to model size.The method uses multiple noised versions of the ground-truth group during training.
- Batching: Minibatch padding makes each image use the largest denoising-query count present in the batch.This padding is needed because DETR-like models train with minibatches and images contain different numbers of objects.
D.2 Large-Scale Model Pre-trianing
DINO is pre-trained on the large-scale Objects365 dataset before fine-tuning on COCO, using separate training schedules and GPU configurations.
- Objects365 Pre-training: Objects365 provides over 1.7M annotated training images and 80,000 annotated validation images.The authors use the first 5,000 validation images for validation and add the remainder to training.
- Pre-training Setup: DINO is pre-trained on Objects365 for 26 epochs using 64 Nvidia A100 GPUs.Each GPU uses a local batch size of one image.
- COCO Fine-tuning: The model is fine-tuned on COCO for 18 epochs using 16 Nvidia A100 GPUs.The fine-tuning configuration also uses a local batch size of one image per GPU.
D.3 Other Implementation Details
The implementation combines standard Transformer, optimization, augmentation, multi-scale, loss, and query settings inherited from or aligned with prior DETR-like models.
- Basic Hyper-parameters: The default model uses 6-layer encoder and decoder Transformers with a hidden feature dimension of 256.The initial learning rate is 1 × 10^-4.
- Basic Hyper-parameters: The learning rate drops at epochs 11, 20, and 30 for the 12-, 24-, and 36-epoch ResNet-50 schedules.Each drop multiplies the learning rate by 0.1.
- Optimization and Queries: DINO uses AdamW with weight decay 1×10^-4 and trains with batch size 16 on Nvidia A100 GPUs.The implementation uses 900 decoder queries to match the computation cost of 300 queries with 3 patterns.
- Loss Function: Box regression uses L1 and GIOU losses, while classification uses focal loss with α = 0.25 and γ = 2.Auxiliary losses are added after each decoder layer, with intermediate losses after query selection.
- Detailed Model Components: The pipeline adopts multi-scale features and deformable attention, while dynamic denoising groups improve denoising efficiency and reduce memory overhead.Prediction heads use unshared parameters across decoder layers.
- Training Augmentation: Training uses random crop and scale augmentation with shorter image sides between 480 and 800 pixels and longer sides up to 1333 pixels.SwinL fine-tuning uses a 1.5× larger scale, with shorter sides between 720 and 1200 pixels and longer sides up to 2000 pixels.
- Multi-scale Setting: Four-scale models use backbone stages 2–4 plus a downsampled stage-4 feature, while five-scale models additionally use stage 1.The configuration uses λ1 = 1.0, λ2 = 2.0, and 100 CDN pairs.
D.4 Detailed Hyper-parameters
This section points readers to Table 8 for the hyper-parameters used in the models, supporting reproduction of the reported results.
- The hyper-parameter details are provided for readers who want to reproduce the results.
- The section directs readers to Table 8 as the location of these implementation details.
- Table 8 lists the hyper-parameters used in the models.