Source-linked AI summary
YOLO-MS: Rethinking Multi-Scale Representation Learning for Real-time Object Detection
Yuming Chen, Xinbin Yuan, Jiabao Wang, Ruiqi Wu, Xiang Li, Qibin Hou, Ming-Ming Cheng
TL;DR
Real-time object detectors need stronger multi-scale representations while preserving speed and accuracy, but existing designs overlook representation learning inside basic blocks. YOLO-MS combines hierarchical multi-branch blocks, Global Query Learning, and heterogeneous kernel selection, achieving stronger computation-performance trade-offs and outperforming recent real-time detectors on MS COCO.
Problem
Existing real-time detectors improve multi-scale aggregation mainly through macro-structure while overlooking multi-scale representation learning in basic building blocks.
Method
YOLO-MS combines an MS-Block with Global Query Learning for dynamic cross-branch guidance and HKS for progressively larger kernels in deeper stages.
Results
YOLO-MS-XS achieves 42.8% AP on MS COCO with 5.1M parameters and 8.7G MACs, while outperforming recent real-time detectors in computation-performance trade-off.
Takeaways & Limitations
The proposed MS-Block with GQL and HKS significantly enhances the speed-accuracy trade-off of real-time object detection.
Abstract
from arXiv · showhide
We aim at providing the object detection community with an efficient and performant object detector, termed YOLO-MS. The core design is based on a series of investigations on how multi-branch features of the basic block and convolutions with different kernel sizes affect the detection performance of objects at different scales. The outcome is a new strategy that can significantly enhance multi-scale feature representations of real-time object detectors. To verify the effectiveness of our work, we train our YOLO-MS on the MS COCO dataset from scratch without relying on any other large-scale datasets, like ImageNet or pre-trained weights. Without bells and whistles, our YOLO-MS outperforms the recent state-of-the-art real-time object detectors, including YOLO-v7, RTMDet, and YOLO-v8. Taking the XS version of YOLO-MS as an example, it can achieve an AP score of 42+% on MS COCO, which is about 2% higher than RTMDet with the same model size. Furthermore, our work can also serve as a plug-and-play module for other YOLO models. Typically, our method significantly advances the APs, APl, and AP of YOLOv8-N from 18%+, 52%+, and 37%+ to 20%+, 55%+, and 40%+, respectively, with even fewer parameters and MACs. Code and trained models are publicly available at https://github.com/FishAndWasabi/YOLO-MS. We also provide the Jittor version at https://github.com/NK-JittorCV/nk-yolo.
1 INTRODUCTION
YOLO-MS addresses limited multi-scale representation in real-time detectors by redesigning multi-branch blocks and assigning kernel sizes across stages. Its MS-Block, GQL, and HKS designs improve the performance-computation trade-off on MS COCO.
- Real-time detectors must balance speed and accuracy, while recognizing objects at different scales remains a fundamental challenge.
- Existing neck mechanisms improve multi-scale feature aggregation but overlook multi-scale representation learning inside basic building blocks.
- Inter-branch features in popular YOLO blocks are homogenized, whereas Res2Net enriches diversity but introduces contaminative spatial information.
- YOLO-MS uses Global Query Learning to dynamically balance information granularities across branches and stages.
- HKS gradually increases convolution kernel sizes with depth, using small kernels for high-resolution shallow features and large kernels for deep semantic features.
- 42.8% AP is achieved by YOLO-MS-XS on MS COCO with 5.1M parameters and 8.7G MACs, while YOLO-MS-S and YOLO-MS reach 45.4% and 52.1% AP.
2 RELATED WORK
Real-time object detectors typically use one-stage YOLO-style architectures and multi-scale fusion in the neck. Related work also explores multi-branch blocks and large-kernel convolutions to improve receptive fields and multi-scale representations.
- Real-Time Object Detection: Real-time object detection prioritizes an optimal trade-off between speed and accuracy for practical applications.
- Real-Time Object Detection: YOLO architectures evolved from CSP-based designs, while YOLOv7 introduced E-ELAN and RTMDet introduced 5×5 large-kernel convolutions.
- Real-Time Object Detection: Figure 3 contrasts prior CSP or ELAN blocks with the proposed MS-Block architecture.
- Multi-Scale Feature Learning: FPN, PAFPN, and Gather-Distribute mechanisms fuse features from different levels to support multi-scale feature learning in real-time detectors.
- Large-Kernel Convolutions: Large-kernel convolutions provide wider receptive fields for multi-scale representations, but homogeneous block designs and speed limitations constrain their use.
3 METHODOLOGY
YOLO-MS builds multi-scale representations through hierarchical multi-branch blocks, Global Query Learning, and stage-specific convolution kernels. MS-Block enhances branch diversity and dynamically aggregates information, while HKS expands receptive fields efficiently in deeper stages.
- MS-Block and Global Query Learning: MS-Block combines hierarchical multi-branch processing with Global Query Learning to improve multi-scale feature representations.The hierarchical structure increases inter-branch diversity, while GQL provides cross-stage guidance and reduces harmful spatial information.
- MS-Block and Global Query Learning: Hierarchical branches provide different receptive fields, enriching information at different scales compared with CSP and ELAN structures.Each branch processes features with a distinct receptive field rather than using the more redundant branch arrangements described for CSP and ELAN.
- MS-Block and Global Query Learning: MS-Block transforms input features with a 1 × 1 convolution, splits them into three groups, processes later groups with inverted bottleneck branches, and concatenates the outputs.The block is used in the backbone and neck; small-scale models may use a simplified inverted bottleneck module for faster speed.
- MS-Block and Global Query Learning: GQL applies a lightweight global query to multi-scale features, using global average pooling and a linear layer to extract spatial information.The authors state that global average pooling keeps the query lightweight and computational cost almost negligible.
- Heterogeneous Kernel Size Selection Protocol: HKS gradually increases convolution kernel sizes from shallow to deep stages, using small kernels for high-resolution features and large kernels for deeper semantic features.The protocol uses kernel sizes 3, 5, 7, and 9 from the shallowest toward deeper stages.
- Heterogeneous Kernel Size Selection Protocol: HKS enlarges receptive fields in deep stages without affecting shallow stages and achieves nearly the same inference speed as using only 3 × 3 convolutions.This design targets richer fine-grained and coarse-grained semantic information while preserving efficient inference.
- YOLO-MS Architecture: The YOLO-MS encoder uses four stages, stride-2 3 × 3 convolutions for downsampling, an SPP block after stage three, and PAFPN as the neck.RTMDet is used as the baseline, with SiLU activation and batch normalization in the encoder.
4 EXPERIMENTS
Experiments show that GQL and HKS improve multi-scale feature representations and detection performance, while YOLO-MS achieves strong speed–accuracy trade-offs across tasks and conditions.
- Analysis of GQL: GQL improves multi-scale performance, including average precision across objects at different scales.The analysis also links GQL to more influential activations within ground-truth object regions.
- Analysis of GQL: GQL aligns branch attention with inter-branch feature-distance trends during training and improves target-branch feature distributions.The target branch is selected according to receptive field and network stage for small or large objects.
- Analysis of HKS Protocol: HKS uses increasing kernel sizes in deeper stages, where larger receptive fields capture coarse-grained information more effectively.Compared with HKS, one alternative drops by 0.7% AP, while HKS incurs the least computation overhead.
- Application in other Tasks: YOLO-MS improves performance beyond object detection, raising MS COCO instance-segmentation AP from 40.5% to 42.8%.The method also improves arbitrary-oriented detection and crowded-scene detection under the reported training settings.
- Application in other Tasks: YOLO-MS achieves state-of-the-art results on underwater, foggy, and RAW-image detection, with AP gains of 0.5%, 0.3%, and 1.1%.These experiments cover challenging aquatic, hazy, indoor, outdoor, lighting, and weather conditions.
5 CONCLUSIONS AND DISCUSSIONS
The paper proposes a computationally reasonable real-time object detector by investigating feature distributions and varying convolutional kernel sizes to extract multi-scale representations. Experiments show that combining MS-Block, GQL, and HKS improves the detector’s speed-accuracy trade-off over other real-time detectors.
- The detector investigates feature distribution and varying convolutional kernel sizes to extract multi-scale feature representations.
- Combining MS-Block with GQL and HKS significantly improves the detector’s speed-accuracy trade-off.
- The resulting detector surpasses other real-time detectors in the reported experimental studies.