Source-linked AI summary
RTMDet: An Empirical Study of Designing Real-Time Object Detectors
Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, Kai Chen
TL;DR
Real-time object detection needs efficient models that can also support related recognition tasks. RTMDet addresses this with compatible backbone and neck designs, large-kernel depth-wise convolutions, soft-label dynamic assignment, and improved training, achieving strong speed–accuracy trade-offs and real-time extensions.
Problem
RTMDet addresses the need for efficient real-time object detectors that extend beyond the YOLO series to instance segmentation and rotated object detection.
Method
RTMDet combines large-kernel depth-wise convolution blocks with compatible backbone and neck capacities, soft labels in dynamic assignment, and improved augmentation, optimization, and training strategies.
Results
RTMDet achieves a favorable parameter–accuracy trade-off across five model sizes, including 52.8% AP at 300+ FPS for RTMDet-x and new real-time results for instance segmentation and rotated detection.
Takeaways & Limitations
RTMDet provides a versatile real-time object-recognition framework with model sizes and task extensions for varied application scenarios.
Abstract
from arXiv · showhide
In this paper, we aim to design an efficient real-time object detector that exceeds the YOLO series and is easily extensible for many object recognition tasks such as instance segmentation and rotated object detection. To obtain a more efficient model architecture, we explore an architecture that has compatible capacities in the backbone and neck, constructed by a basic building block that consists of large-kernel depth-wise convolutions. We further introduce soft labels when calculating matching costs in the dynamic label assignment to improve accuracy. Together with better training techniques, the resulting object detector, named RTMDet, achieves 52.8% AP on COCO with 300+ FPS on an NVIDIA 3090 GPU, outperforming the current mainstream industrial detectors. RTMDet achieves the best parameter-accuracy trade-off with tiny/small/medium/large/extra-large model sizes for various application scenarios, and obtains new state-of-the-art performance on real-time instance segmentation and rotated object detection. We hope the experimental results can provide new insights into designing versatile real-time object detectors for many object recognition tasks. Code and models are released at https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet.
1. Introduction
RTMDet is introduced as an efficient real-time detector family that extends beyond YOLO-style detection to instance segmentation and rotated object detection. Its design combines large-kernel depth-wise convolutions, soft labels in dynamic assignment, and improved training strategies, achieving strong speed–accuracy results across model sizes and tasks.
- RTMDet targets efficient real-time object detection while supporting instance segmentation and rotated object detection.
- Large-kernel depth-wise convolutions improve representation, while soft labels in dynamic label assignment improve optimization and model accuracy.
- RTMDet uses compatible backbone and neck capacities, reducing building-block depth and increasing width to balance inference speed and model capacity.
- 41.1% AP at 1020 FPS is achieved by RTMDet-tiny with 4.8M parameters, while RTMDet-s reaches 44.6% AP at 819 FPS.
- 44.6% mask AP at 180 FPS on COCO val and 81.33% AP on DOTA v1.0 establish new real-time results for instance segmentation and rotated object detection.
2. Related Work
Related work addresses efficient one-stage detection, label assignment, instance segmentation, and rotated object detection. RTMDet builds on these directions with a shared backbone–neck architecture and extensions for segmentation and rotated boxes.
- Efficient detection research explores anchor-based and anchor-free one-stage detectors, efficient backbones, model scaling, multi-scale features, and re-parameterization.
- Dynamic label assignment improves convergence speed and model accuracy beyond earlier IoU-, center-, and auxiliary-head-based strategies.
- One-stage instance segmentation methods include mask classification, top-down, bottom-up, and dynamic-kernel approaches.
- RTMDet uses large-kernel depth-wise convolution blocks in both backbone and neck, then supports instance segmentation and rotated detection with few modifications.
- RTMDet extends instance segmentation by adding kernel prediction and mask feature heads.
- Rotated object detection predicts object orientation alongside location and category, motivating specialized feature extraction and box representations.
3. Methodology
RTMDet combines a one-stage detector architecture with large-kernel depth-wise convolutions, balanced backbone–neck capacity, soft-label dynamic assignment, and revised training strategies. The design targets efficient inference while supporting extensible real-time detection tasks.
- Architecture: RTMDet is a one-stage detector whose methodology balances backbone, neck, and head design for efficient object detection.The architecture decomposes into backbone, neck, and head, then studies building blocks and computation allocation across components.
- Architecture: Large-kernel depth-wise convolutions enlarge the receptive field with lower computational cost than earlier large-context mechanisms.The paper motivates them as a practical alternative to dilated convolutions and non-local blocks for real-time detection.
- Architecture: Reducing blocks and increasing block width offsets the added depth of large-kernel depth-wise convolution while improving inference parallelization.The adjustment aims to maintain model capacity while addressing slower inference caused by additional point-wise convolutions.
- Architecture: RTMDet allocates more capacity to the neck, shares detection-head parameters across scales with separate Batch Normalization layers, and preserves multi-scale features.These choices target parameter efficiency while retaining accuracy and feature-pyramid functionality.
- Training Strategy: Soft labels make dynamic assignment costs more discriminative and reduce noisy or unstable matching compared with binary labels.The classification cost uses soft targets related to prediction–ground-truth quality, while logarithmic IoU regression cost amplifies differences among lower-quality matches.
- Training Strategy: The training strategy combines augmentation changes, cached cross-sample mixing, a two-stage schedule, and AdamW optimization to improve efficiency and accuracy.Caching reduces image-mixing data-loading cost, while the two-stage schedule transitions from strong training augmentation to Large Scale Jittering.
4. Experiments
Experiments evaluate RTMDet on object detection, instance segmentation, rotated detection, and component ablations. Across tasks and scales, RTMDet improves speed-accuracy trade-offs and benefits from balanced architecture, soft label assignment, and revised augmentation.
- Object detection: 52.8% AP at 300+ FPS is achieved by RTMDet-x, outperforming mainstream detectors without extra information beyond box annotations.
- Instance segmentation: 44.6% mask AP is achieved by RTMDet-Ins-x, surpassing YOLOv5-seg-x by 3.2% AP while retaining real-time speed.
- Rotated object detection: 81.33% mAP is achieved by RTMDet-R-l with COCO pretraining on DOTA-v1.0, setting a new record.
- Model architecture: Reducing stage blocks from 9 to 6 cuts latency by 20% but lowers AP by 0.5%; Channel Attention limits the loss to 0.1% while improving latency by 7%.
- Label assignment: Dynamic soft label assignment surpasses SimOTA by 1.3% AP on RTMDet-s with longer training and stronger augmentation.
- Data augmentation: Using LSJ in the second stage improves RTMDet-s and RTMDet-l by 2% AP and 1.5%, respectively, over Mosaic and MixUp.
- Step-by-step results: Successive architecture, optimization, label-assignment, augmentation, and pretraining changes yield a 4.3% AP improvement over the YOLOX-s baseline.
5. Conclusion
The paper studies critical components of real-time object detectors and develops RTMDet derivatives for instance segmentation and rotated object detection. These models use minimal adaptations while targeting high precision and real-time operation.
- RTMDet empirically studies model architectures, label assignment, data augmentation, and optimization in real-time object detectors.
- RTMDet derivatives extend a high-precision real-time detector to instance segmentation and rotated object detection with minimal adaptations.
A.1. Benchmark Results
RTMDet-R is competitive across rotated object detection benchmarks, combining strong accuracy with faster inference and state-of-the-art results on challenging datasets.
- 18.5% and 20.8% faster, RTMDet-R-m and RTMDet-R-l surpass larger PPYOLOE-R-l and PPYOLOE-R-x models, respectively.
- 1.32% mAP, RTMDet-R-l surpasses the previous best ReDet method on the more challenging DOTA-v1.5 dataset.
- RTMDet-R is evaluated on HRSC2016, which contains 1K images and 2.9K ships collected from Google Earth.
- 90.6% mAP07, RTMDet-R achieves new state-of-the-art performance on the HRSC2016 ship detection dataset.