Source-linked AI summary
D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement
Yansong Peng, Hebei Li, Peixi Wu, Yueyi Zhang, Xiaoyan Sun, Feng Wu
TL;DR
Real-time DETR detectors need more precise and efficient bounding-box regression because fixed-coordinate methods poorly model localization uncertainty and are difficult to optimize. D-FINE addresses this with iterative distribution refinement and global optimal localization self-distillation, achieving strong COCO accuracy-efficiency results and improving other DETR models with negligible overhead.
Problem
Fixed-coordinate bounding-box regression does not model localization uncertainty well, while real-time detectors face constrained computation and parameter budgets.
Method
D-FINE combines Fine-grained Distribution Refinement, which iteratively refines probability distributions, with Global Optimal Localization Self-Distillation, which transfers localization knowledge across decoder layers.
Results
D-FINE improves a variety of DETR models by up to 5.3% AP and reaches 54.0% AP for D-FINE-L and 55.8% AP for D-FINE-X on COCO val2017.
Takeaways & Limitations
D-FINE provides a real-time DETR design that balances localization accuracy, efficiency, and low additional training cost across models.
Takeaways & Limitations
The performance gap between lighter D-FINE models and other compact models remains small, potentially because shallow decoder layers provide less accurate predictions for distillation.
Abstract
from arXiv · showhide
We introduce D-FINE, a powerful real-time object detector that achieves outstanding localization precision by redefining the bounding box regression task in DETR models. D-FINE comprises two key components: Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). FDR transforms the regression process from predicting fixed coordinates to iteratively refining probability distributions, providing a fine-grained intermediate representation that significantly enhances localization accuracy. GO-LSD is a bidirectional optimization strategy that transfers localization knowledge from refined distributions to shallower layers through self-distillation, while also simplifying the residual prediction tasks for deeper layers. Additionally, D-FINE incorporates lightweight optimizations in computationally intensive modules and operations, achieving a better balance between speed and accuracy. Specifically, D-FINE-L / X achieves 54.0% / 55.8% AP on the COCO dataset at 124 / 78 FPS on an NVIDIA T4 GPU. When pretrained on Objects365, D-FINE-L / X attains 57.1% / 59.3% AP, surpassing all existing real-time detectors. Furthermore, our method significantly enhances the performance of a wide range of DETR models by up to 5.3% AP with negligible extra parameters and training costs. Our code and pretrained models: https://github.com/Peterande/D-FINE.
1 INTRODUCTION
D-FINE targets the latency and localization limitations of real-time DETR detectors by reformulating box regression and adding self-distillation. It reports strong accuracy-efficiency results on COCO and improves other DETR models with little overhead.
- DETR offers global context modeling and direct set prediction without NMS or anchor boxes, but traditional variants often incur high latency and computational demands.
- Fixed-coordinate regression fails to model localization uncertainty and provides insufficient independent guidance for adjusting box edges, slowing convergence and limiting performance.
- D-FINE introduces FDR, which represents boxes as probability distributions and iteratively refines them through residual adjustments for finer localization.
- GO-LSD transfers localization knowledge from deeper refined predictions to shallower layers while simplifying deeper residual prediction tasks.
- 54.0% and 55.8% AP are achieved by D-FINE-L and D-FINE-X on COCO at 124 FPS and 78 FPS, respectively, while the method improves DETR models by up to 5.3% AP.
2 RELATED WORK
Related work spans efficient real-time detectors, distribution-based box representations, and knowledge distillation. D-FINE builds on these directions while targeting their compatibility, refinement, and training-cost limitations.
- Real-Time / End-to-End Object Detectors: YOLO detectors emphasize efficiency but typically rely on NMS, whereas DETR removes NMS and anchors through end-to-end set prediction.
- Distribution-Based Object Detection: Distribution-based detectors model localization uncertainty with Gaussian or discrete distributions, but prior approaches rely on anchor-based frameworks.
- Knowledge Distillation: Knowledge distillation transfers information through logits, features, or localization outputs, while self-distillation lets earlier layers learn from refined predictions without separately training a teacher.
3 PRELIMINARIES
The preliminaries identify limitations of fixed Dirac-delta box representations and explain how distribution-based regression and localization distillation address uncertainty while retaining important constraints.
- Traditional regression models box edges as precise Dirac delta distributions, making uncertainty difficult to represent and small prediction shifts prone to localization errors.
- GFocal represents four edge distances with discretized probability distributions, providing a more flexible representation of ambiguous bounding boxes.
- GFocal remains anchor-dependent and predicts in one shot without iterative refinement, limiting prediction diversity, anchor-free compatibility, and regression robustness.
- Fixed distance ranges and uniform bin intervals can cause coarse localization, particularly for small objects.
- Localization Distillation transfers localization knowledge rather than only logits or features, but still depends on anchor-based architectures and adds training cost.
4 METHOD
D-FINE combines Fine-grained Distribution Refinement with Global Optimal Localization Self-Distillation to improve DETR localization while retaining real-time efficiency. FDR iteratively refines edge distributions, and GO-LSD transfers refined localization knowledge to shallower layers.
- D-FINE combines FDR and GO-LSD to improve localization with negligible additional parameters and training-time cost.The method also streamlines computationally intensive modules and operations to make real-time DETR architectures faster and more lightweight.
- Fine-Grained Distribution Refinement: FDR replaces fixed-coordinate regression with iterative probability-distribution refinement for independently modeling each bounding-box edge.Initial boxes provide references, while subsequent decoder layers adjust four edge distributions and progressively update the box.
- Fine-Grained Distribution Refinement: FDR uses non-uniform weighting functions to support fine adjustments near accurate predictions and larger corrections when predictions are inaccurate.The weighting function’s curvature changes across candidate offsets, providing flexibility for both incremental and substantial corrections.
- Fine-Grained Distribution Refinement: FGL Loss uses weighted cross-entropy interpolation and IoU-based weighting to concentrate lower-uncertainty distributions around more precise edge offsets.The weights align adjacent bins with the ground-truth relative offset, while IoU weighting encourages more concentrated distributions.
- Global Optimal Localization Self-Distillation: GO-LSD aggregates Hungarian matches across decoder layers and distills final-layer refined distributions into shallower layers through DDF Loss.DDF applies decoupled weighting to account for high-IoU low-confidence predictions and balance matched with unmatched predictions.
5 EXPERIMENTS
Experiments evaluate D-FINE’s efficiency, accuracy, modular enhancements, ablations, distillation strategy, and localization refinement across COCO-based comparisons. Results show strong real-time detection performance, broad DETR compatibility, and improved localization through FDR and GO-LSD.
- Comparison with Real-Time Detectors: D-FINE-L attains 54.0% AP with 31M parameters, 91 GFLOPs, and 8.07 ms latency, while D-FINE-X reaches 55.8% AP with 62M parameters, 202 GFLOPs, and 12.89 ms latency.
- Comparison with Real-Time Detectors: 54.0% AP for D-FINE-L exceeds YOLOv10-L, RT-DETR-R50, and LW-DETR-X while using 91 GFLOPs versus 120, 136, and 174 GFLOPs, respectively.
- Comparison with Real-Time Detectors: 57.1% and 59.3% AP are achieved by D-FINE-L and D-FINE-X after Objects365 pretraining, exceeding YOLOv10 counterparts by 3.1% and 4.4% AP.D-FINE reaches these gains with 21 pretraining epochs, compared with 300 epochs for YOLOv10 under the cited protocol.
- Effectiveness on Various DETR Models: FDR and GO-LSD improve Deformable DETR, DAD-DETR, DN-DETR, and DINO by 2.0% to 5.3% AP without increasing parameters or computational burden.The methods are described as flexibly integrable across DETR architectures.
- The Roadmap to D-FINE: The roadmap combines projection removal, Target Gating, GELAN substitution, reduced hidden dimensions, uneven sampling, and RT-DETRv2 training to balance accuracy and efficiency.Removing decoder projections lowers GFLOPs from 110 to 97 and latency from 9.25 ms to 8.02 ms, while Target Gating recovers AP from 52.4% to 52.8%.
- Comparison of Distillation Methods: GO-LSD achieves 54.5% AP with only a 6% training-time increase and 2% memory increase over baseline, outperforming alternative distillation methods in the comparison.The baseline records 53.0% AP, 29 minutes per epoch, and 8552 MB memory on four NVIDIA RTX 4090 GPUs.
6 CONCLUSION
D-FINE redefines DETR bounding-box regression through FDR and GO-LSD, achieving strong real-time accuracy and efficiency while identifying a remaining challenge for lighter models.
- D-FINE combines Fine-grained Distribution Refinement with Global Optimal Localization Self-Distillation to redefine bounding-box regression in DETR models.
- D-FINE achieves state-of-the-art accuracy and efficiency on COCO, surpassing existing real-time detectors.
- Limitation and Future Work: Lighter D-FINE models remain close to other compact models in performance.The paper attributes this partly to less accurate shallow decoder predictions, which limit localization-knowledge distillation into earlier layers.
- Limitation and Future Work: Future work could improve lightweight-model localization by training with additional decoder layers and discarding them during inference.
A.1.1 HYPERPARAMETER CONFIGURATIONS
The D-FINE variants use pretrained HGNetV2 backbones and AdamW, with larger models configured using wider embeddings and more decoder layers.
- All D-FINE variants use HGNetV2 backbones pretrained on ImageNet and the AdamW optimizer.
- D-FINE-X uses embedding dimension 384 and feedforward dimension 2048, while the other variants use 256 and 1024.
- D-FINE-X and D-FINE-L use 6 decoder layers, whereas D-FINE-M and D-FINE-S use 4 and 3 layers.
A.1.2 DATASETS SETTINGS
The experiments combine Objects365 pretraining data with standard COCO2017 training and validation splits, while resizing larger images for efficiency.
- Objects365 pretraining combines its train and validation images while excluding the first 5k validation images.
- Images exceeding 640 × 640 are resized to 640 × 640 before training to improve efficiency.
- The experiments train on COCO train2017 and evaluate on COCO val2017 using the standard COCO2017 split.
A.2 VISUALIZATION OF D-FINE PREDICTIONS
The appendix presents D-FINE-X predictions in challenging visual conditions and reports lightweight-detector comparisons, alongside clarification of the initial-layer refinement formulation.
- Visualization: D-FINE-X predictions are visualized under occlusion, low light, motion blur, depth-of-field effects, rotation, and densely populated scenes.The visualization uses a confidence threshold of 0.5.
- Comparison with Lighter Detectors: 48.5% AP is achieved by D-FINE-S on COCO val2017 with 3.49 ms latency, 10.2M parameters, and 25.2 GFLOPs.Pretraining on Objects365 raises D-FINE-S to 50.7% AP, a +2.2% improvement.
- Comparison with Lighter Detectors: 52.3% AP is achieved by D-FINE-M with 19.2M parameters, 56.6 GFLOPs, and 5.62 ms latency.Pretraining on Objects365 yields a +2.8% gain for D-FINE-M.
- Comparison with Lighter Detectors: Table 7 compares S- and M-sized real-time detectors on COCO val2017.
- Initial Layer Refinement: For the initial refinement layer, the formulation uses logits predicted by the first layer because no previous layer exists.The clarification states that this makes the formulation consistent across layers.