PaperlayerView exact revision on arXiv

Source-linked AI summary

DETRs Beat YOLOs on Real-time Object Detection

Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, Jie Chen

arXiv:2304.08069v3cs.CV

TL;DR

Real-time YOLO detectors rely on NMS, which slows inference and introduces threshold-related instability, while DETRs avoid NMS but are computationally costly. RT-DETR combines an efficient hybrid encoder with uncertainty-minimal query selection and outperforms advanced YOLO detectors in speed and accuracy.

  • Problem

    YOLO detectors require NMS post-processing, which slows inference and introduces threshold-related instability in speed and accuracy.

  • Method

    RT-DETR uses an efficient hybrid encoder and uncertainty-minimal query selection to process multi-scale features and initialize high-quality decoder queries.

  • Results

    53.1% AP and 108 FPS on T4 GPU let RT-DETR-R50 outperform advanced YOLO detectors in both speed and accuracy.

  • Takeaways & Limitations

    RT-DETR extends DETR to real-time object detection while eliminating NMS and supporting flexible speed tuning without retraining.

  • Takeaways & Limitations

    RT-DETR’s performance on small objects remains inferior to that of strong real-time detectors.

Abstract

from arXiv · show

The YOLO series has become the most popular framework for real-time object detection due to its reasonable trade-off between speed and accuracy. However, we observe that the speed and accuracy of YOLOs are negatively affected by the NMS. Recently, end-to-end Transformer-based detectors (DETRs) have provided an alternative to eliminating NMS. Nevertheless, the high computational cost limits their practicality and hinders them from fully exploiting the advantage of excluding NMS. In this paper, we propose the Real-Time DEtection TRansformer (RT-DETR), the first real-time end-to-end object detector to our best knowledge that addresses the above dilemma. We build RT-DETR in two steps, drawing on the advanced DETR: first we focus on maintaining accuracy while improving speed, followed by maintaining speed while improving accuracy. Specifically, we design an efficient hybrid encoder to expeditiously process multi-scale features by decoupling intra-scale interaction and cross-scale fusion to improve speed. Then, we propose the uncertainty-minimal query selection to provide high-quality initial queries to the decoder, thereby improving accuracy. In addition, RT-DETR supports flexible speed tuning by adjusting the number of decoder layers to adapt to various scenarios without retraining. Our RT-DETR-R50 / R101 achieves 53.1% / 54.3% AP on COCO and 108 / 74 FPS on T4 GPU, outperforming previously advanced YOLOs in both speed and accuracy. We also develop scaled RT-DETRs that outperform the lighter YOLO detectors (S and M models). Furthermore, RT-DETR-R50 outperforms DINO-R50 by 2.2% AP in accuracy and about 21 times in FPS. After pre-training with Objects365, RT-DETR-R50 / R101 achieves 55.3% / 56.2% AP. The project page: https://zhao-yian.github.io/RTDETR.

1. Introduction

The introduction identifies NMS and Transformer encoder cost as barriers to real-time detection, then presents RT-DETR as an end-to-end detector designed to improve speed and accuracy without NMS. RT-DETR uses an efficient hybrid encoder, achieves strong COCO/T4 results, and supports speed tuning without retraining.

  • Motivation: YOLO detectors offer a reasonable speed–accuracy trade-off but require NMS, which slows inference and introduces hyperparameters that destabilize speed and accuracy.Different scenarios also require different NMS thresholds for recall and accuracy.
  • Motivation: DETRs eliminate hand-crafted components and NMS, but their high computational cost prevents them from meeting real-time detection requirements.Multi-scale feature interaction substantially lengthens encoder sequences and makes the Transformer encoder a computational bottleneck.
  • Method: RT-DETR is proposed as the first real-time end-to-end object detector, using an efficient hybrid encoder that decouples intra-scale interaction from cross-scale feature fusion.This redesign targets faster processing of multi-scale features and improves inference speed.
  • Results: 53.1% AP and 108 FPS are achieved by RT-DETR-R50 on COCO val2017 and a T4 GPU, while RT-DETR-R101 achieves 54.3% AP and 74 FPS.These models outperform the L and X models of previously advanced YOLO detectors in both speed and accuracy; scaled RT-DETRs also outperform lighter S and M YOLO detectors.
  • Contributions: RT-DETR supports flexible speed tuning by adjusting the number of decoder layers to accommodate different scenarios without retraining.The contributions also include quantitative analysis of NMS effects and an end-to-end speed benchmark for real-time detectors.

2. Related Work

Related work spans YOLO detectors, which established practical real-time one-stage detection, and DETR-based end-to-end detectors, which remove anchors and NMS but remain computationally intensive. Prior DETR variants improve query optimization or initialization, while RT-DETR targets computational-cost reduction and real-time detection.

  • YOLO Detectors: YOLO detectors became synonymous with real-time object detection by offering a practical speed–accuracy trade-off and include anchor-based and anchor-free designs.YOLOv1 introduced CNN-based one-stage detection with true real-time performance; subsequent development led YOLO detectors to outperform other one-stage detectors.
  • End-to-End DETRs: DETR eliminates hand-crafted anchors and NMS through bipartite matching and direct one-to-one object-set prediction, but suffers from slow training convergence and high computational demands.Its streamlined end-to-end Transformer pipeline provides distinctive advantages, while the passage identifies convergence and computational limitations.
  • DETR Query Design: Conditional DETR and Anchor DETR reduce query-optimization difficulty, while two-stage DETR and DINO improve query initialization through query-selection strategies.The cited methods represent successive efforts to address query optimization and initialization in DETR architectures.
  • RT-DETR Motivation: Current DETRs remain computationally intensive and are not designed for real-time detection, motivating RT-DETR’s focus on computational-cost reduction and query initialization.The passage states that RT-DETR explores both computational efficiency and query initialization while targeting real-time detection.

3. End-to-end Speed of Detectors

This section examines how NMS affects detector speed and establishes an end-to-end benchmark for fair comparison. It finds that anchor-free detectors are faster than anchor-based YOLO detectors at equivalent accuracy because they require less NMS time.

  • NMS overhead: NMS filters overlapping boxes using confidence and IoU thresholds through an iterative per-category process.Boxes below the confidence threshold are removed, while lower-scoring boxes are discarded when pairwise IoU exceeds the IoU threshold.
  • NMS overhead: NMS is sensitive to its confidence-threshold hyperparameter because higher thresholds filter out more prediction boxes and reduce subsequent processing.The study samples confidence thresholds from 0.001 to 0.25 and counts remaining boxes on the same input.
  • End-to-end benchmark: The study measures end-to-end detector speed on COCO val2017 using TensorRT FP16 on a T4 GPU across multiple images.The benchmark includes anchor-based YOLOv5 and YOLOv7 and anchor-free PP-YOLOE, YOLOv6, and YOLOv8.
  • End-to-end benchmark: Three times more prediction boxes are produced by the tested anchor-based detectors than by the anchor-free detectors.This difference explains why anchor-based detectors require more NMS time.
  • End-to-end benchmark: At equivalent accuracy, anchor-free YOLO detectors outperform anchor-based detectors because they require less NMS time.The NMS timing uses TensorRT’s efficientNMSPlugin, while the reported comparison focuses on the EfficientNMS kernel.

4. The Real-time DETR

RT-DETR combines a backbone, efficient hybrid encoder, and Transformer decoder to process multi-scale features without relying on simultaneous intra-scale and cross-scale interaction. Its encoder improves efficiency through AIFI and CCFF, while uncertainty-minimal query selection supplies higher-quality decoder queries and supports flexible scaling.

  • Architecture: RT-DETR feeds the backbone’s last three stages into an efficient hybrid encoder, which uses AIFI for intra-scale interaction and CCFF for cross-scale fusion before decoder query selection.The decoder uses selected encoder features as initial object queries and includes auxiliary prediction heads.
  • Efficient hybrid encoder: The encoder becomes a computational bottleneck because multi-scale features sharply increase sequence length, motivating the separation of intra-scale interaction from cross-scale fusion.The cited analysis reports that the encoder accounts for 49% of GFLOPs in prior work.
  • Efficient hybrid encoder: 35% faster and 0.4% AP higher: the DS5 variant with CCFF reduces latency while improving accuracy over its comparison variant.CCFF uses convolutional fusion blocks to combine adjacent-scale features into a new feature.
  • Query selection: Uncertainty-minimal query selection models joint classification-localization uncertainty to provide high-quality encoder features as decoder queries.It defines feature uncertainty as the discrepancy between predicted localization and classification distributions.
  • Query selection: 138% more purple dots than green dots: uncertainty-minimal query selection produces more high-quality encoder features than vanilla query selection.Purple dots represent uncertainty-minimal selection and green dots represent vanilla selection; higher-quality features lie nearer the plot’s top right.
  • Flexible scaling: RT-DETR supports flexible scaling by adjusting encoder width and depth, decoder queries and layers, and inference speed through decoder-layer removal.Removing a few decoder layers at the end has minimal effect on accuracy while greatly increasing inference speed.

5. Experiments

RT-DETR outperforms comparable YOLO and DETR detectors in speed and accuracy under the reported benchmarks. Ablations support its encoder and query-selection designs, while decoder depth enables flexible accuracy–latency tuning.

  • Overall comparison: 53.1% AP and 108 FPS are achieved by RT-DETR-R50, while RT-DETR-R101 reaches 54.3% AP and 74 FPS, outperforming comparable YOLOs and DETRs.Measurements use 640 × 640 inputs for RT-DETR and YOLO detectors, with FPS reported on a T4 GPU using TensorRT FP16.
  • Comparison with real-time detectors: 4.1% / 1.7% / 0.3% AP accuracy gains are achieved by RT-DETR-R50 over YOLOv5-L / PP-YOLOE-L / YOLOv6-L, alongside FPS increases of 100.0% / 14.9% / 9.1%.RT-DETR-R101 also improves accuracy and FPS over YOLOv5-X and PP-YOLOE-X, while reducing parameters.
  • Comparison with end-to-end detectors: 2.2% AP and 21 times speed improvements are obtained by RT-DETR-R50 over DINO-Deformable-DETR-R50, reaching 108 FPS versus 5 FPS.The DETR comparison uses TensorRT FP16 and an input size of 800 × 1333 for the DINO-Deformable-DETR speed test.
  • Ablation studies: 0.8% AP improvement and 8% lower latency from variant D over C indicate that decoupling intra-scale interaction and cross-scale fusion reduces computation while improving accuracy.DS5 further reduces latency by 35% and improves AP by 0.4% over variant D, while variant E improves AP by 1.5% over D.
  • Ablation studies: 0.82% vs 0.35% high-classification-score proportions and 0.67% vs 0.30% high-quality-feature proportions favor uncertainty-minimal query selection.The scheme selects K = 300 encoder features using classification scores and uses their corresponding prediction boxes as initial position queries.
  • Decoder-layer tuning: 53.1% AP is achieved with six decoder layers, while using the fifth layer loses only 0.1% AP, supporting flexible inference-depth tuning.The accuracy difference between adjacent decoder layers gradually decreases as decoder-layer index increases.

6. Limitation and Discussion

RT-DETR remains inferior to strong real-time detectors on small objects, despite its overall speed and accuracy advantages. Its homogeneous decoders also enable distillation from large, high-accuracy DETR models, suggesting a future research direction.

  • Limitation: RT-DETR shares other DETRs’ limitation of inferior performance on small objects compared with strong real-time detectors.RT-DETR-R50 is 0.5% AP lower, while RTDETR-R101 is 0.9% AP lower than the highest APval reported for the referenced real-time detectors.
  • Discussion: RT-DETR’s decoders remain homogeneous across scales and compatible with other DETRs.This compatibility makes it possible to distill lightweight RT-DETR models using large DETRs pre-trained for high accuracy.
  • Discussion: The authors identify distillation from large, high-accuracy DETRs as an advantage over other real-time detectors and a direction for future exploration.Existing large DETR models have demonstrated impressive performance on the COCO test-dev leaderboard.

7. Conclusion

RT-DETR extends DETR to real-time end-to-end object detection and achieves state-of-the-art performance. Its design combines an efficient hybrid encoder, uncertainty-minimal query selection, and flexible speed tuning without retraining.

  • RT-DETR extends DETR to the real-time detection scenario and achieves state-of-the-art performance.
  • The efficient hybrid encoder expeditiously processes multi-scale features.
  • Uncertainty-minimal query selection improves the quality of initial object queries.
  • RT-DETR supports flexible speed tuning without retraining.

Appendix of “DETRs Beat YOLOs on Real-time Object Detection” · 1. Experimental Settings

Experiments use COCO and Objects365, with RT-DETR trained on COCO train2017 and validated on COCO val2017. Evaluation reports standard COCO AP metrics across IoU thresholds and object scales, using ImageNet-pretrained ResNet backbones.

  • 1. Experimental Settings: RT-DETR is evaluated on COCO and Objects365, with training on COCO train2017 and validation on COCO val2017.
  • 1. Experimental Settings: Evaluation includes AP averaged over IoU thresholds from 0.50–0.95 in 0.05 increments, plus AP50, AP75, APS, APM, and APL.
  • 1. Experimental Settings: The backbone uses ResNet pretrained on ImageNet.

2. Comparison with Lighter YOLO Detectors

The section introduces lighter RT-DETR variants for diverse real-time detection scenarios by scaling the encoder and decoder, then compares them with YOLO S and M models. The variants halve RepBlock channels and adjust decoder layers during inference.

  • Model scaling: Lighter RT-DETRs are developed by scaling the encoder and decoder with ResNet50, ResNet34, and ResNet18.The variants target diverse real-time detection scenarios.
  • Model scaling: The scaled variants halve RepBlock channels while leaving the other components unchanged.This scaling produces a set of RT-DETR models using ResNet50/34/18 backbones.
  • Inference flexibility: Decoder-layer counts are adjusted during inference to obtain different scaled RT-DETR configurations.The passage states that this adjustment is applied to scaled RT-DETR-R50/34/18 models.
  • Comparison with YOLO: The scaled RT-DETRs are compared with the S and M models of YOLO detectors in Table B.The supplied passage introduces the comparison but does not provide its quantitative results.

3. Large-scale Pre-training for RT-DETR · 4. Visualization of Predictions with Different Post-processing Thresholds · 5. Visualization of RT-DETR Predictions

Large-scale pre-training on Objects365 improves RT-DETR’s COCO performance across R18, R50, and R101 models. Visualizations show that RT-DETR avoids NMS-related trade-offs and maintains strong detection in complex and difficult conditions.

  • 3. Large-scale Pre-training for RT-DETR: Objects365 pre-training followed by COCO fine-tuning demonstrates RT-DETR’s potential as a real-time object detector for industry scenarios.The experiments cover RT-DETR-R18, RT-DETR-R50, and RT-DETR-R101.
  • 3. Large-scale Pre-training for RT-DETR: 2.7%/2.2%/1.9% AP improvements are achieved on COCO val2017 for RT-DETR-R18/50/101 after Objects365 pre-training.All models are pre-trained for 12 epochs; R18, R50, and R101 are fine-tuned on COCO for 60, 24, and 24 epochs, respectively.
  • 4. Visualization of Predictions with Different Post-processing Thresholds: YOLOv8-L predictions are visualized under different NMS thresholds, while RT-DETR-R50 predictions are visualized under different score thresholds on two COCO val2017 samples.The visualizations use different post-processing thresholds for the two detectors.
  • 4. Visualization of Predictions with Different Post-processing Thresholds: Higher confidence thresholds filter more prediction boxes and increase false negatives, whereas a confidence threshold of 0.001 produces redundant boxes and increases false positives.IoU thresholds also affect overlapping-box filtering and false-positive counts.
  • 4. Visualization of Predictions with Different Post-processing Thresholds: RT-DETR uses bipartite matching to produce one-to-one object sets, eliminating overlapping-box suppression and filtering low-confidence boxes with a score threshold.The score threshold can be adjusted according to scenario-specific emphasis.
  • 4. Visualization of Predictions with Different Post-processing Thresholds: RT-DETR post-processing threshold selection is straightforward and does not affect inference speed, improving adaptability across real-time scenarios.By contrast, YOLO NMS requires careful selection of confidence and IoU thresholds for different scenarios.
  • 5. Visualization of RT-DETR Predictions: RT-DETR-R101 detects diverse objects, including small or densely packed cups, wine glasses, and individuals, in complex scenes.These examples are shown with score threshold=0.5.
  • 5. Visualization of RT-DETR Predictions: RT-DETR-R101 successfully detects objects under motion blur, rotation, and occlusion, supporting its detection performance under difficult conditions.The difficult-condition visualizations use score threshold=0.5.
Loading 2304.08069v3…