Source-linked AI summary
RT-DETRv3: Real-time End-to-End Object Detection with Hierarchical Dense Positive Supervision
Shuo Wang, Chunlong Xia, Feng Lv, Yifeng Shi
TL;DR
Sparse Hungarian-matching supervision limits training in real-time end-to-end RT-DETR. RT-DETRv3 adds training-only dense-supervision branches, and the authors report better results than other real-time detectors, including a matched-latency R18 gain over RT-DETR.
Problem
RT-DETR’s Hungarian matching provides sparser supervision than dense detectors, leaving encoder and decoder training insufficient and limiting optimal performance.
Source: §1 · p. 2Method
RT-DETRv3 adds CNN-based encoder supervision and decoder-side query perturbation and one-to-many supervision through training-only branches.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
The authors report improved performance and faster convergence without additional inference latency from the proposed approaches.
Source: §1 · p. 2The authors note that RT-DETRv3 tends to overfit as model size increases and suggest a mismatch between training-data and model size may contribute.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
RT-DETR is the first real-time end-to-end transformer-based object detector. Its efficiency comes from the framework design and the Hungarian matching. However, compared to dense supervision detectors like the YOLO series, the Hungarian matching provides much sparser supervision, leading to insufficient model training and difficult to achieve optimal results. To address these issues, we proposed a hierarchical dense positive supervision method based on RT-DETR, named RT-DETRv3. Firstly, we introduce a CNN-based auxiliary branch that provides dense supervision that collaborates with the original decoder to enhance the encoder feature representation. Secondly, to address insufficient decoder training, we propose a novel learning strategy involving self-attention perturbation. This strategy diversifies label assignment for positive samples across multiple query groups, thereby enriching positive supervisions. Additionally, we introduce a shared-weight decoder branch for dense positive supervision to ensure more high-quality queries matching each ground truth. Notably, all aforementioned modules are training-only. We conduct extensive experiments to demonstrate the effectiveness of our approach on COCO val2017. RT-DETRv3 significantly outperforms existing real-time detectors, including the RT-DETR series and the YOLO series. For example, RT-DETRv3-R18 achieves 48.1% AP (+1.6%/+1.4%) compared to RT-DETR-R18/RT-DETRv2-R18, while maintaining the same latency. Furthermore, RT-DETRv3-R101 can attain an impressive 54.6% AP outperforming YOLOv10-X. The code will be released at https://github.com/clxia12/RT-DETRv3.
1. Introduction
RT-DETR’s sparse Hungarian-matching supervision limits encoder and decoder training, motivating dense positive supervision. RT-DETRv3 adds training-time auxiliary branches and reports stronger COCO results, including an R18 gain over RT-DETR at the same latency.
1. Introduction
- The paper situates RT-DETR among end-to-end DETR models, whose set prediction and Hungarian matching remove NMS but whose computational complexity limits practical use.Later DETR variants introduced refinement and denoising training to improve convergence and performance.Source: §1 · p. 2
- RT-DETR’s Hungarian matching provides sparse supervision, which the authors say leaves both encoder and decoder training insufficient.RT-DETRv2 improves performance without sacrificing speed but requires longer training time.Source: §1 · p. 2
- A CNN-based one-to-many auxiliary head collaborates with the original detection branch to strengthen encoder representations.This adds dense supervision alongside RT-DETR’s original detection branch.Source: §1 · p. 2
- Self-attention perturbations diversify positive assignments across query groups, while a shared-weight decoder branch supplies more high-quality queries per ground truth.The authors report improved performance and faster convergence without additional inference latency.Source: §1 · p. 2
- 48.1% AP (+1.6%) for RT-DETRv3-R18 over RT-DETR-R18 at the same latency; the paper also reports outperforming real-time detectors including the RT-DETR and YOLO series.Figure 1 presents the speed–accuracy comparison.Source: Figure 1 · p. 1, §1 · p. 2
2. Related Work
The related work contrasts CNN-based real-time detectors with transformer-based RT-DETR models, then positions RT-DETRv3 among auxiliary-supervision approaches. Its training-only branches extend dense supervision to RT-DETR while preserving inference latency.
2.1. CNN-based real-time object detection.
- The YOLO series is presented as the main CNN-based real-time detector family, with successive versions refining architecture, feature fusion, and training strategies.The section also notes the PP-YOLO series as a PaddlePaddle-based real-time solution derived from YOLO.Source: §2.1 · p. 2
2.2. Transformer-based real-time object detection.
- RT-DETR combines an efficient hybrid encoder and IoU-aware query selection, while RT-DETRv2 improves its training strategy; both retain one-to-one supervision that limits convergence and final performance.The section identifies one-to-many assignment as a potential way to improve performance.Source: §2.2 · p. 2
2.3. Auxiliary training strategy.
- Prior auxiliary-branch methods add one-to-many supervision to encoder or decoder training but are not real-time detectors; RT-DETRv3 applies such branches to both while discarding them at inference.Its architecture includes a CNN-based branch, self-attention perturbation for query groups, and a supplementary one-to-many decoder branch.Source: §2.3 · pp. 2–3, Figure 2 · p. 3
3. Method
RT-DETRv3 adds dense supervision to RT-DETR’s encoder and decoder while retaining its overall framework. Training-only branches support encoder learning, diversify decoder query assignments, and add one-to-many decoder supervision.
3.1. Overall Architecture.
- RT-DETRv3 retains RT-DETR’s backbone, efficient hybrid encoder, query selection, and decoder while adding parallel CNN and transformer dense-supervision branches.The branches provide encoder supervision, perturb decoder query attention, and add one-to-many decoder matching.Source: §3.1 · p. 3
3.2. Overview of RT-DETR.
- RT-DETR uses Hungarian one-to-one matching, L1 and GIoU box losses, and VFL classification loss to train its selected object queries.Its decoder uses self-attention, cross-attention, and feed-forward blocks to refine queries and predict classes and boxes.Source: §3.2 · p. 3
3.3. One-to-Many Auxiliary Branch Based on CNN.
- A CNN auxiliary head applies one-to-many assignment to encoder features C3, C4, and C5 to strengthen encoder supervision.It uses ATSS early in training, then TaskAlign, with VFL for classification and DFL for localization.Source: §3.3 · p. 4
3.4. Multi-Group Self-Attention Perturbation Branches Based on Transformer.
- Multiple query sets receive random attention masks, diversifying query features so related queries can become positive samples.The query sets are concatenated into one decoder branch with shared parameters, while loss computation and label assignment remain consistent with RT-DETR.Source: §3.4 · p. 4, Figure 3 · p. 4
3.5. One-to-Many Dense Supervision Branch Based on Transformer.
- A shared-weight decoder branch matches queries against a target set replicated by a default factor of four, providing one-to-many dense supervision.Its loss computation follows the original detection loss.Source: §3.5 · pp. 4–5
3.6. Total Loss.
- The total loss combines CNN auxiliary supervision, multi-group one-to-one decoder supervision, and one-to-many decoder supervision, with all three weights defaulting to 1.The corresponding terms are Laux, Lo2o, and Lo2m.Source: §3.6 · p. 5
4. Experiments
On COCO, RT-DETRv3 outperforms the compared real-time detectors, including RT-DETR variants and CNN-based models, while its auxiliary branches are training-only. Experiments also examine convergence, overfitting, and the contributions of individual modules.
4.1. Datasets and Evaluation Metrics.
- Evaluation uses MS COCO 2017, with 115k training images and 5k test images, and reports AP, convergence efficiency, inference speed, and effectiveness.The comparisons include transformer-based and CNN-based real-time detectors.Source: §4.1 · p. 5
4.2. Implementation Details.
- The CNN auxiliary branch uses the PP-YOLOE head, while three parameter-shared self-attention perturbation groups and a shared one-to-many branch augment the RT-DETR decoder.The default one-to-many branch matches each ground truth with four queries; smaller backbones use 120 epochs and larger backbones 72 epochs.Source: §4.2 · pp. 5–6
4.4. Comparison with CNN-Based Real-time Object Detectors.
- 54.6 AP: RT-DETRv3-R101 exceeds YOLOv10-X; at similar inference speeds, R18 beats listed small CNN detectors by 2.0–4.4%.RT-DETRv3 also reduces training epochs to 60% or less compared with CNN-based real-time detectors while achieving superior performance.Source: Table 2 · p. 6, §4.4 · p. 7
4.3. Comparison with Transformer-based Real-time Object Detectors.
- With 6x training, RT-DETRv3 improves over RT-DETR by 1.6%, 1.0%, 0.3%, and 0.3% for R18, R34, R50, and R101, with unchanged inference speed.It also outperforms RT-DETR and RT-DETRv2 in convergence speed across schedules, reaching comparable performance in half the epochs. Larger models tend to overfit; the authors suspect a training-data/model-size mismatch, while added data supports continued improvement with more epochs.Source: §4.3 · pp. 6–7, Table 3 · p. 7
4.5. Ablation Study.
- Adding O2M-T to RT-DETR improves performance by 1.0%, and integrating all proposed modules improves it by 1.6%.With three self-attention perturbation branches, RT-DETRv3-R18 achieves its best reported AP of 48.1; fewer branches reduce supervision richness, while more increase learning difficulty without significant gains.Source: §4.5 · pp. 7–8, Table 4 · p. 8, Table 5 · p. 8
5. Conclusion
RT-DETRv3 adds dense positive-sample auxiliary supervision to RT-DETR. On COCO, it achieves better results than other real-time detectors, with the added modules used only during training.
5. Conclusion
- RT-DETRv3 adds multiple dense positive-sample auxiliary supervision modules to RT-DETR’s encoder and decoder.The modules apply one-to-many object supervision to specific features.Source: §5 · p. 8
- The auxiliary modules are training-only, and experiments on COCO show better results than other real-time object detectors.The authors state that the approach accelerates convergence and improves performance.Source: §5 · p. 8
- The authors hope the work inspires research and development on real-time transformer-based object detection.Source: §5 · p. 8