YOLO-MS: Rethinking Multi-Scale Representation Learning for Real-time Object Detection

• Real-time detectors must balance speed and accuracy, while recognizing objects at different scales remains a fundamental challenge.Source: §1 · p. 1
• Existing neck mechanisms improve multi-scale feature aggregation but overlook multi-scale representation learning inside basic building blocks.Source: §1 · p. 1
• Inter-branch features in popular YOLO blocks are homogenized, whereas Res2Net enriches diversity but introduces contaminative spatial information.Source: §1 · pp. 1–2
• YOLO-MS uses Global Query Learning to dynamically balance information granularities across branches and stages.Source: §1 · p. 2
• HKS gradually increases convolution kernel sizes with depth, using small kernels for high-resolution shallow features and large kernels for deep semantic features.Source: §1 · p. 2
• 42.8% AP is achieved by YOLO-MS-XS on MS COCO with 5.1M parameters and 8.7G MACs, while YOLO-MS-S and YOLO-MS reach 45.4% and 52.1% AP.Source: §1 · p. 2

Real-Time Object Detection

• Real-time object detection prioritizes an optimal trade-off between speed and accuracy for practical applications.Source: §2.1 · p. 2
• YOLO architectures evolved from CSP-based designs, while YOLOv7 introduced E-ELAN and RTMDet introduced 5×5 large-kernel convolutions.Source: §2.1 · pp. 2–3
• Figure 3 contrasts prior CSP or ELAN blocks with the proposed MS-Block architecture.Source: Figure 3 · p. 3

Multi-Scale Feature Learning

• FPN, PAFPN, and Gather-Distribute mechanisms fuse features from different levels to support multi-scale feature learning in real-time detectors.Source: §2.2 · p. 3

Large-Kernel Convolutions

• Large-kernel convolutions provide wider receptive fields for multi-scale representations, but homogeneous block designs and speed limitations constrain their use.Source: §2.2 · p. 3

MS-Block and Global Query Learning

• MS-Block combines hierarchical multi-branch processing with Global Query Learning to improve multi-scale feature representations.The hierarchical structure increases inter-branch diversity, while GQL provides cross-stage guidance and reduces harmful spatial information.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4
• Hierarchical branches provide different receptive fields, enriching information at different scales compared with CSP and ELAN structures.Each branch processes features with a distinct receptive field rather than using the more redundant branch arrangements described for CSP and ELAN.Source: §3.1 · p. 3
• MS-Block transforms input features with a 1 × 1 convolution, splits them into three groups, processes later groups with inverted bottleneck branches, and concatenates the outputs.The block is used in the backbone and neck; small-scale models may use a simplified inverted bottleneck module for faster speed.Source: §3.1 · p. 4
• GQL applies a lightweight global query to multi-scale features, using global average pooling and a linear layer to extract spatial information.The authors state that global average pooling keeps the query lightweight and computational cost almost negligible.Source: §3.2 · p. 4

Heterogeneous Kernel Size Selection Protocol

• HKS gradually increases convolution kernel sizes from shallow to deep stages, using small kernels for high-resolution features and large kernels for deeper semantic features.The protocol uses kernel sizes 3, 5, 7, and 9 from the shallowest toward deeper stages.Source: §3.3 · pp. 4–5
• HKS enlarges receptive fields in deep stages without affecting shallow stages and achieves nearly the same inference speed as using only 3 × 3 convolutions.This design targets richer fine-grained and coarse-grained semantic information while preserving efficient inference.Source: §3.3 · p. 5

YOLO-MS Architecture

• The YOLO-MS encoder uses four stages, stride-2 3 × 3 convolutions for downsampling, an SPP block after stage three, and PAFPN as the neck.RTMDet is used as the baseline, with SiLU activation and batch normalization in the encoder.Source: §3.4 · p. 5

Analysis of GQL

• GQL improves multi-scale performance, including average precision across objects at different scales.The analysis also links GQL to more influential activations within ground-truth object regions.Source: §4.2 · p. 6
• GQL aligns branch attention with inter-branch feature-distance trends during training and improves target-branch feature distributions.The target branch is selected according to receptive field and network stage for small or large objects.Source: §4.2 · p. 6

Analysis of HKS Protocol

• HKS uses increasing kernel sizes in deeper stages, where larger receptive fields capture coarse-grained information more effectively.Compared with HKS, one alternative drops by 0.7% AP, while HKS incurs the least computation overhead.Source: §4.3 · p. 7, §4.4 · p. 8

Application in other Tasks

• YOLO-MS improves performance beyond object detection, raising MS COCO instance-segmentation AP from 40.5% to 42.8%.The method also improves arbitrary-oriented detection and crowded-scene detection under the reported training settings.Source: §4.6 · p. 10
• YOLO-MS achieves state-of-the-art results on underwater, foggy, and RAW-image detection, with AP gains of 0.5%, 0.3%, and 1.1%.These experiments cover challenging aquatic, hazy, indoor, outdoor, lighting, and weather conditions.Source: §4.6 · pp. 10–11

• The detector investigates feature distribution and varying convolutional kernel sizes to extract multi-scale feature representations.Source: §5 · p. 11
• Combining MS-Block with GQL and HKS significantly improves the detector’s speed-accuracy trade-off.Source: §5 · p. 11
• The resulting detector surpasses other real-time detectors in the reported experimental studies.Source: §5 · p. 11