Source-linked AI summary
RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer
Wenyu Lv, Yian Zhao, Qinyao Chang, Kui Huang, Guanzhong Wang, Yi Liu
TL;DR
Real-time detection needs detectors that balance performance with flexibility and deployment practicality. RT-DETRv2 adds scale-aware sampling, optional discrete sampling, and adaptive training strategies to RT-DETR, reporting improved performance without loss of speed. The result is an improved RT-DETR baseline intended to broaden its applications.
Problem
Real-time object detection requires a balance of performance, flexibility, and practicality, while RT-DETR's deployment-specific grid_sample operator limits broad applicability.
Method
RT-DETRv2 adds distinct scale-specific sampling points, optional discrete sampling, dynamic augmentation, and scale-adaptive hyperparameters to RT-DETR.
Results
RT-DETRv2 outperforms RT-DETR at different detector scales without loss of speed.
Takeaways & Limitations
RT-DETRv2 provides a more flexible and practical RT-DETR baseline and is intended to broaden RT-DETR applications.
Abstract
from arXiv · showhide
In this report, we present RT-DETRv2, an improved Real-Time DEtection TRansformer (RT-DETR). RT-DETRv2 builds upon the previous state-of-the-art real-time detector, RT-DETR, and opens up a set of bag-of-freebies for flexibility and practicality, as well as optimizing the training strategy to achieve enhanced performance. To improve the flexibility, we suggest setting a distinct number of sampling points for features at different scales in the deformable attention to achieve selective multi-scale feature extraction by the decoder. To enhance practicality, we propose an optional discrete sampling operator to replace the grid_sample operator that is specific to RT-DETR compared to YOLOs. This removes the deployment constraints typically associated with DETRs. For the training strategy, we propose dynamic data augmentation and scale-adaptive hyperparameters customization to improve performance without loss of speed. Source code and pre-trained models will be available at https://github.com/lyuwenyu/RT-DETR.
1 Introduction
Real-time detection balances broad application needs against deployment and performance demands. RT-DETRv2 builds on RT-DETR with flexibility, practicality, and training improvements.
- Real-time object detection supports applications such as autonomous driving and requires a reasonable balance of capabilities.
- RT-DETR introduced an end-to-end real-time DETR that reduces dependence on YOLO detectors.It uses an efficient hybrid encoder, uncertainty-minimal query selection, flexible speed tuning, and multiple detector sizes.
- RT-DETRv2 extends RT-DETR with bag-of-freebies targeting flexibility, practicality, and enhanced performance.Its stated improvements include selective multi-scale extraction, deployment-oriented sampling, and optimized training.
2 Method
RT-DETRv2 retains the RT-DETR framework while modifying decoder deformable attention and refining deployment and training strategies. These changes provide scale-aware feature extraction, optional discrete sampling, and adaptive optimization.
- Framework: RT-DETRv2 keeps the RT-DETR framework and modifies the decoder's deformable attention module.
- Framework: Distinct sampling-point counts across feature scales enable more flexible and efficient multi-scale feature extraction.The fixed count used at every scale is described as ignoring intrinsic scale differences.
- Framework: The optional discrete_sample operator replaces grid_sample by rounding predicted offsets and omitting bilinear interpolation.Training first uses grid_sample, followed by fine-tuning with discrete_sample because rounding is non-differentiable.
- Training Scheme: Dynamic augmentation uses stronger transformations early and disables four listed augmentations during the last two training epochs.The schedule adapts augmentation to the target domain later in training.
- Training Scheme: Scale-adaptive hyperparameters increase learning rates for light detectors and decrease them for large detectors.The adjustment reflects lower feature quality in light pretrained backbones and higher feature quality in large ones.
3 Experiment
Experiments train and evaluate RT-DETRv2 on COCO using standard AP metrics and compare it with RT-DETR across detector scales. Ablations examine sampling-point reduction and discrete sampling.
- Evaluation: The reported implementation uses an ImageNet-pretrained ResNet backbone, AdamW, batch size 16, and EMA with ema_decay = 0.9999.Optional discrete sampling uses 6× grid_sample pretraining followed by 1× discrete_sample fine-tuning.
- Evaluation: RT-DETRv2 is trained on COCO train2017 and validated on COCO val2017 using standard AP metrics.AP averages uniformly sampled IoU thresholds from 0.50 to 0.95, while APval50 is also reported.
- Results: RT-DETRv2 outperforms RT-DETR at different detector scales without loss of speed.The comparison uses fixed 640 × 640 inputs, with FPS measured on a T4 GPU using TensorRT FP16.
- Ablations: Reducing the total number of sampling points does not significantly degrade performance.The total is calculated as num_head×num_point×num_query×num_decoder.
- Ablations: Replacing grid_sample with discrete_sample causes no noticeable reduction in APval50 while removing DETR deployment constraints.
4 Conclusion
RT-DETRv2 improves RT-DETR's flexibility and practicality through bag-of-freebies while optimizing training for enhanced performance without loss of speed.
- RT-DETRv2 increases RT-DETR's flexibility and practicality and optimizes training to enhance performance without loss of speed.The authors position the report as providing insights for the DETR family and broadening RT-DETR applications.