Source-linked AI summary
Real-Time Object Detection Meets DINOv3
Shihua Huang, Yongjie Hou, Longfei Liu, Xuanlong Yu, Xi Shen
TL;DR
Real-time object detection must balance detection performance with computational efficiency, and integrating DINOv3 into DETR-based detectors remains insufficiently investigated. DEIMv2 combines DINOv3 features, STA, scale-specific backbones, a simplified decoder, and enhanced Dense O2O across eight model sizes. It reports state-of-the-art COCO performance across resource settings, including 57.8 AP for DEIMv2-X and compact models suited to constrained deployment.
Problem
Real-time object detection needs to balance detection performance and computational efficiency, while DINOv3 integration into DETR-based real-time detectors remains insufficiently investigated.
Method
DEIMv2 combines DINOv3-pretrained or distilled backbones with STA, pruned HGNetv2 variants, a simplified decoder, and enhanced Dense O2O across eight model sizes.
Results
DEIMv2 achieves state-of-the-art performance across model sizes; DEIMv2-X reaches 57.8 AP with 50.3M parameters, while DEIMv2-S surpasses 50 AP and DEIMv2-Pico matches YOLOv10-N with over 50% fewer parameters.
Takeaways & Limitations
DEIMv2 provides a scalable real-time detection framework spanning resource-constrained edge devices to high-performance detection systems.
Takeaways & Limitations
Further work is needed to better integrate DINOv3 features into real-time detectors, including improving fine-grained detail representation and latency performance with proper optimization.
Abstract
from arXiv · showhide
Driven by the simple and effective Dense O2O, DEIM demonstrates faster convergence and enhanced performance. In this work, we extend it with DINOv3 features, resulting in DEIMv2. DEIMv2 spans eight model sizes from X to Atto, covering GPU, edge, and mobile deployment. For the X, L, M, and S variants, we adopt DINOv3-pretrained or distilled backbones and introduce a Spatial Tuning Adapter (STA), which efficiently converts DINOv3's single-scale output into multi-scale features and complements strong semantics with fine-grained details to enhance detection. For ultra-lightweight models (Nano, Pico, Femto, and Atto), we employ HGNetv2 with depth and width pruning to meet strict resource budgets. Together with a simplified decoder and an upgraded Dense O2O, this unified design enables DEIMv2 to achieve a superior performance-cost trade-off across diverse scenarios, establishing new state-of-the-art results. Notably, our largest model, DEIMv2-X, achieves 57.8 AP with only 50.3 million parameters, surpassing prior X-scale models that require over 60 million parameters for just 56.5 AP. On the compact side, DEIMv2-S is the first sub-10 million model (9.71 million) to exceed the 50 AP milestone on COCO, reaching 50.9 AP. Even the ultra-lightweight DEIMv2-Pico, with just 1.5 million parameters, delivers 38.5 AP, matching YOLOv10-Nano (2.3 million) with around 50 percent fewer parameters. Our code and pre-trained models are available at https://github.com/Intellindust-AI-Lab/DEIMv2
1. Introduction
DEIMv2 extends real-time DETR-based detection with DINOv3 features and a scalable design spanning GPU, edge, and mobile deployment. Across COCO model scales, it reports state-of-the-art performance while targeting the balance between accuracy and computational efficiency.
- Real-time object detection supports autonomous driving, robotics, and industrial defect detection, but lightweight deployment requires balancing detection performance with computational efficiency.
- DEIMv2 addresses the incomplete investigation of integrating DINOv3’s robust feature representations into DETR-based real-time detectors.
- DEIMv2 provides eight model sizes covering GPU, edge, and mobile deployment, using DINOv3-pretrained or distilled backbones for larger variants and specialized ultra-lightweight variants.
- The Spatial Tuning Adapter converts DINOv3’s single-scale outputs into multi-scale features while adding fine-grained detail features with small receptive fields.
- 57.6 AP on COCO with 50.3M parameters lets DEIMv2-X surpass DEIM-X, which requires over 60M parameters for 56.5 AP.
- DEIMv2-Pico reaches 38.5 AP with 1.5M parameters, matching YOLOv10-Nano at 2.3M parameters while reducing parameter count by approximately 50%.
- The work presents DINOv3 adaptation and a framework spanning ultra-lightweight to high-performance real-time detectors, with COCO experiments reporting state-of-the-art results across resource settings.
2. Method
DEIMv2 combines scale-specific backbones with STA, an efficient decoder, and enhanced Dense O2O. DINOv3-based variants target mainstream scales, while pruned HGNetv2 variants address strict ultra-lightweight budgets.
- Overall architecture: The RT-DETR-style architecture comprises a backbone, hybrid encoder, and decoder that produces candidate boxes before iterative refinement.
- Backbones: DINOv3-based backbones serve X, L, M, and S variants, while Nano, Pico, Femto, and Atto use progressively pruned HGNetv2-B0 backbones.
- Spatial Tuning Adapter: STA uses a fully convolutional design, lightweight feedforward processing, and Bi-Fusion to extract fine-grained multi-scale details and strengthen DINOv3 representations.
- Spatial Tuning Adapter: Parameter-free bilinear interpolation resizes 1/16-scale features from selected ViT blocks into multiple scales for object detection.
- Efficient Decoder: The decoder combines SwiGLUFFN, RMSNorm, and shared query position embeddings to improve representation, stabilize training, and eliminate redundant computation.
- Enhanced Dense O2O: Object-level Copy-Blend extends Dense O2O by adding objects without their backgrounds to increase effective supervision.
- Training setting and loss: The training objective is a weighted sum of MAL, FGL, DDF, L1 BBox, and GIoU losses, with fixed component weights across experiments.
- Training setting and loss: FGL and DDF losses are excluded for Pico, Femto, and Atto because applying them degrades performance for these limited-capacity models.
3. Experiments
Experiments show that DEIMv2 improves the accuracy–efficiency trade-off across standard and ultra-light model scales, while gains are strongest for medium and large objects. The authors also identify latency optimization and fine-grained detail representation as remaining boundaries.
- Standard-scale detectors: 57.8 AP with approximately 50M parameters and 151 GFLOPs makes DEIMv2-X outperform DEIM-X at 56.5 AP, 62M parameters, and 202 GFLOPs.The reported latency has not yet been optimized.
- Limitations: Latency remains an evaluation boundary because the proposed methods have not been optimized with techniques such as Flash Attention.The paper states that proper optimization could improve latency performance.
- Object-size analysis: DEIMv2 improves medium- and large-object accuracy more than small-object accuracy relative to comparable DEIM models.For DEIMv2-S, APM rises from 52.6 to 55.3 and APL from 65.7 to 70.3, while APS changes from 30.4 to 31.4; DEIMv2-X improves APM from 61.4 to 62.8 and APL from 74.2 to 75.9, with APS at 39.2 versus 38.8 for DEIM-M.
- Limitations: DINOv3 provides strong global semantics but limited fine-grained detail representation, leaving small-object detection comparatively challenging.The paper identifies better integration of DINOv3 features as future work.
- Ultra-light detectors: DEIMv2-Atto achieves performance comparable to NanoDet-M with 0.49M parameters, while DEIMv2-Pico matches YOLOv10-N with less than half the parameters.These results support deployment in resource-constrained edge settings.
4. Conclusion
DEIMv2 combines DINOv3 semantics with a lightweight STA in a scalable family spanning ultra-lightweight edge models to high-performance detectors. Its reported results advance the accuracy–efficiency frontier across diverse deployment scenarios.
- DEIMv2 combines DINOv3 semantic representations with a lightweight STA across a unified family of real-time detectors.
- 57.8 AP with significantly fewer parameters makes DEIMv2-X more efficient than previous large-scale detectors.
- DEIMv2-S surpasses 50 AP at its compact scale, while DEIMv2-Pico matches YOLOv10-N with over 50% fewer parameters.
- The model family spans resource-constrained edge devices and high-performance detection systems, supporting deployment across diverse scenarios.