DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

DETR background

• DETR removes hand-designed components by formulating object detection as set prediction with bipartite matching and learnable queries.Queries probe object existence and combine image features similarly to soft ROI pooling.Source: §1 · p. 2

Motivation

• DETR-like models previously faced slow convergence and unclear query meaning, while DAB-DETR and DN-DETR addressed these issues with dynamic anchor boxes and denoising.Their combination made DETR-like models competitive with classical detectors in training efficiency and inference performance.Source: §1 · p. 2
• Previous DETR-like models remained below 50 AP on COCO, and their scalability to large backbones and datasets had not been reported.DINO explicitly aims to address both performance and scalability concerns.Source: §1 · pp. 2–3

DINO

• DINO improves denoising training, query initialization, and box prediction through contrastive denoising, mixed query selection, and look forward twice.The model builds on DN-DETR, DAB-DETR, and Deformable DETR.Source: §1 · pp. 3–4

Results

• 49.4 AP in 12 epochs and 51.3 AP in 24 epochs with ResNet-50 and multi-scale features improved over the previous best DETR-like model by +6.0 AP and +2.7 AP.DINO also achieved 63.2 AP on COCO val2017 and 63.3 AP on test-dev after Objects365 pre-training with SwinL.Source: §1 · pp. 3–4

Convolution-based detectors

• Classical detectors include two-stage and one-stage convolutional models built around hand-crafted anchors or reference points.Two-stage models use region proposal networks, while one-stage models predict offsets relative to predefined anchors.Source: §2.1 · p. 4

Transformer detectors

• DETR introduced end-to-end Transformer detection without hand-designed anchor design and non-maximum suppression, prompting work on faster convergence.Follow-up approaches include encoder-only designs and dynamic decoders focused on important regions.Source: §2.2 · p. 4

Decoder queries

• Query-focused methods associate decoder queries with spatial information using reference points, dense encoder predictions, or dynamic anchor boxes.Deformable DETR, Efficient DETR, and DAB-DETR represent successive approaches to spatially informed queries.Source: §2.2 · p. 5

Large-scale pre-training

• Large-scale pre-training has improved computer-vision models, with leading detectors using large backbones and substantial pre-training datasets.The related work contrasts these resource-intensive approaches with DINO's publicly available pre-training setting.Source: §2.3 · p. 5

Model overview

• DINO formulates decoder positional queries as dynamic 4D anchor boxes and refines them across decoder layers.The boxes use center coordinates and dimensions as query representations.Source: §3.1 · p. 5, §3.2 · p. 7

Contrastive DeNoising Training

• Contrastive DeNoising adds positive and negative anchor queries so the decoder learns to reject anchors without nearby objects.Positive and negative examples are defined by their distance from a ground-truth box, and multiple CDN groups are used during training.Source: §3.3 · pp. 7–8, Figure 3 · p. 8
• +1.3 AP on small objects is achieved by CDN queries over DN queries after 12 epochs with ResNet-50 and multi-scale features.CDN finds better anchors for small objects, while DN is described as sufficient for selecting good anchors overall.Source: §3.3 · p. 9, Figure 4 · p. 9

Mixed Query Selection

• Mixed query selection initializes anchor boxes from positional information in the selected top-K encoder features while keeping content queries static or learnable.This differs from Deformable DETR, which uses selected features to enhance both positional and content queries.Source: §3.4 · pp. 9–10, Figure 5 · p. 10

Look Forward Twice

• Look forward twice updates box predictions so later-layer box information can influence adjacent earlier-layer parameters.The method contrasts with Deformable DETR’s look forward once update, which blocks gradient backpropagation for stabilization.Source: Figure 6 · p. 10, §3.5 · p. 11

Experimental Setup

• DINO is evaluated on COCO 2017 using ResNet-50 or SwinL backbones, with SwinL models pre-trained on Objects365 before COCO fine-tuning.The ResNet-50 setting uses train2017 without extra data, while SwinL uses ImageNet-22k pre-training followed by Objects365 pre-training.Source: §4.1 · p. 11

Main Results

• DINO uses 900 queries, while most other DETR-like models use 300 queries in the 12-epoch comparison.The table compares models on COCO val2017 and reports GFLOPS and FPS for models without multi-scale features using ResNet-50-DC5.Source: Table 1 · p. 12, §4.2 · p. 12
• +5.6 AP is obtained with ResNet-50 and 4-scale features, while +6.0 AP is obtained with 5-scale features under the same setting.The 4-scale model adds little computation and parameter overhead, and gains +7.2 AP or +7.5 AP on small objects with 4 or 5 scales, respectively.Source: §4.2 · p. 12
• +1.8 AP and +2.7 AP are achieved after 24 epochs with 4 and 5 scales, respectively; after 36 epochs, gains reach +2.3 AP and +2.6 AP.The convergence comparison is shown in Fig. 7, supporting evaluation of both training speed and final performance.Source: §4.2 · p. 13, Figure 7 · p. 13

Comparison with SOTA Models

• 63.2 AP on COCO val2017 and 63.3 AP on test-dev establish DINO as an end-to-end Transformer detector with strong scalability to larger model and data sizes.The SwinL model uses Objects365 pre-training and achieves these results without test-time augmentation.Source: §4.3 · p. 14, Table 3 · p. 14

Ablation Studies

• Ablation studies evaluate the effectiveness of DINO’s proposed algorithm components against optimized DN-DETR and query-selection baselines.The components are reported as Query Selection, Contrastive De-Noising Training, and Look Forward Twice.Source: §4.4 · p. 14, Table 4 · p. 15

Conclusion

• DINO combines contrastive denoising training, mixed query selection, and look forward twice to improve training efficiency and final detection performance.The model is presented as an end-to-end Transformer detector.Source: §5 · p. 15
• DINO achieves 63.3 AP on COCO 2017 test-dev after training with a stronger backbone on a larger dataset.The conclusion describes this as a new state-of-the-art result and reports that DINO outperforms previous ResNet-50-based models in 12- and 36-epoch settings.Source: §5 · p. 15
• DN-query training can produce duplicate boxes for one person, whereas the proposed method is shown alongside it for comparison.Fig. 8 displays only person boxes and highlights three duplicate boxes in the DN-query result.Source: Figure 8 · p. 15

Test Time Augmentations

• Test-time augmentation does not provide an inspiring gain for DINO compared with traditional detectors.The exploration uses the large SwinL model and is included to compare with traditional detection models.Source: A Test Time Augmentations (TTA) · p. 19

Training Resource Usage

• DINO’s base-model GPU memory and training time are reported using 8 Nvidia A100 GPUs with a ResNet-50 backbone.The authors state that the results demonstrate training effectiveness and efficiency.Source: B Training Efficiency · p. 19
• Table 5 compares training efficiency across models with ResNet-50 under a common 8-GPU setup.The table includes training-efficiency measurements for different models and identifies framework-specific testing for Faster R-CNN.Source: Table 5 · p. 20

Architecture Ablation

• Table 6 ablates the numbers of encoder and decoder layers using ResNet-50 on COCO val2017 in the 12-epoch setting.The ablation uses 100 denoising queries without negative samples.Source: Table 6 · p. 20

Number of Encoder and Decoder Layers

• A 3.0 AP drop results when decoder layers decrease from 6 to 2 with 6 encoder layers fixed.The analysis attributes this sensitivity to dynamic box updates and refinement across decoder layers.Source: C Additional Analysis on our Model… · p. 20

Query Denoising

• Increasing denoising queries improves performance significantly below 100 queries.The experiment uses the optimized dynamic denoising-group setting.Source: C Additional Analysis on our Model… · p. 21
• Beyond 100 denoising queries, additional queries provide only small or sometimes worse performance improvements.Table 7 reports the ablation on COCO validation with a ResNet-50 backbone.Source: Table 7 · p. 20, C Additional Analysis on our Model… · p. 21

Ground-Truth Grouping

• DN-DETR forms one denoising ground-truth group from all labeled objects and boxes in each image.Multiple noised versions of this group are used during training.Source: §D.1 · p. 21

Group Count

• DN-DETR sets the number of denoising groups to five or ten according to model size.The method uses multiple noised versions of the ground-truth group during training.Source: §D.1 · p. 21

Batching

• Minibatch padding makes each image use the largest denoising-query count present in the batch.This padding is needed because DETR-like models train with minibatches and images contain different numbers of objects.Source: §D.1 · p. 21

Objects365 Pre-training

• Objects365 provides over 1.7M annotated training images and 80,000 annotated validation images.The authors use the first 5,000 validation images for validation and add the remainder to training.Source: §D.2 · p. 21

Pre-training Setup

• DINO is pre-trained on Objects365 for 26 epochs using 64 Nvidia A100 GPUs.Each GPU uses a local batch size of one image.Source: §D.2 · p. 21

COCO Fine-tuning

• The model is fine-tuned on COCO for 18 epochs using 16 Nvidia A100 GPUs.The fine-tuning configuration also uses a local batch size of one image per GPU.Source: §D.2 · p. 21

Basic Hyper-parameters

• The default model uses 6-layer encoder and decoder Transformers with a hidden feature dimension of 256.The initial learning rate is 1 × 10^-4.Source: Basic hyper-parameters · p. 21
• The learning rate drops at epochs 11, 20, and 30 for the 12-, 24-, and 36-epoch ResNet-50 schedules.Each drop multiplies the learning rate by 0.1.Source: Basic hyper-parameters · p. 21

Optimization and Queries

• DINO uses AdamW with weight decay 1×10^-4 and trains with batch size 16 on Nvidia A100 GPUs.The implementation uses 900 decoder queries to match the computation cost of 300 queries with 3 patterns.Source: Basic hyper-parameters · pp. 21–22

Loss Function

• Box regression uses L1 and GIOU losses, while classification uses focal loss with α = 0.25 and γ = 2.Auxiliary losses are added after each decoder layer, with intermediate losses after query selection.Source: Loss function · p. 22

Detailed Model Components

• The pipeline adopts multi-scale features and deformable attention, while dynamic denoising groups improve denoising efficiency and reduce memory overhead.Prediction heads use unshared parameters across decoder layers.Source: Detailed model components · p. 22

Training Augmentation

• Training uses random crop and scale augmentation with shorter image sides between 480 and 800 pixels and longer sides up to 1333 pixels.SwinL fine-tuning uses a 1.5× larger scale, with shorter sides between 720 and 1200 pixels and longer sides up to 2000 pixels.Source: Training augmentation · p. 22

Multi-scale Setting

• Four-scale models use backbone stages 2–4 plus a downsampled stage-4 feature, while five-scale models additionally use stage 1.The configuration uses λ1 = 1.0, λ2 = 2.0, and 100 CDN pairs.Source: Multi-scale setting · p. 22

• The hyper-parameter details are provided for readers who want to reproduce the results.Source: §D.4 · p. 22
• The section directs readers to Table 8 as the location of these implementation details.Source: §D.4 · p. 22, Table 8 · p. 23
• Table 8 lists the hyper-parameters used in the models.Source: Table 8 · p. 23