PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLO-Master: MOE-Accelerated with Specialized Transformers for Enhanced Real-time Detection

Xu Lin, Jinlong Peng, Zhenye Gan, Jiawen Zhu, Jun Liu

arXiv:2512.23273v2cs.CV

TL;DR

YOLO-Master targets the resource mismatch created by static dense computation in real-time object detectors. It introduces ES-MoE with instance-conditional routing and sparse expert activation, achieving broad benchmark improvements while preserving real-time efficiency.

  • Problem

    Static dense computation gives all inputs the same processing, misallocating capacity between simple and complex scenes and limiting the accuracy–efficiency trade-off.

  • Method

    YOLO-Master integrates an ES-MoE module with dynamic routing, specialized multi-scale experts, soft Top-K training, hard Top-K inference, and load-balancing supervision.

  • Results

    Across five benchmarks, YOLO-Master-N outperforms YOLOv13-N by +0.8% mAP on COCO, +1.4% on VOC, +2.1% on VisDrone, +1.5% on KITTI, and +0.7% on SKU-110K while being 18% faster.

  • Takeaways & Limitations

    Adaptive expert activation supports improved detection across object densities and visual domains while maintaining real-time inference speed.

Abstract

from arXiv · show

Existing Real-Time Object Detection (RTOD) methods commonly adopt YOLO-like architectures for their favorable trade-off between accuracy and speed. However, these models rely on static dense computation that applies uniform processing to all inputs, misallocating representational capacity and computational resources such as over-allocating on trivial scenes while under-serving complex ones. This mismatch results in both computational redundancy and suboptimal detection performance. To overcome this limitation, we propose YOLO-Master, a novel YOLO-like framework that introduces instance-conditional adaptive computation for RTOD. This is achieved through a Efficient Sparse Mixture-of-Experts (ES-MoE) block that dynamically allocates computational resources to each input according to its scene complexity. At its core, a lightweight dynamic routing network guides expert specialization during training through a diversity enhancing objective, encouraging complementary expertise among experts. Additionally, the routing network adaptively learns to activate only the most relevant experts, thereby improving detection performance while minimizing computational overhead during inference. Comprehensive experiments on five large-scale benchmarks demonstrate the superiority of YOLO-Master. On MS COCO, our model achieves 42.4% AP with 1.62ms latency, outperforming YOLOv13-N by +0.8% mAP and 17.8% faster inference. Notably, the gains are most pronounced on challenging dense scenes, while the model preserves efficiency on typical inputs and maintains real-time inference speed. Code will be available.

1. Introduction

YOLO-Master addresses the static computation and accuracy–speed limitations of YOLO-like detectors by adapting expert computation to input complexity. Its sparse MoE design combines dynamic routing, specialized experts, and load balancing, with experiments showing broad benchmark gains and efficiency.

  • Motivation: YOLO-like detectors apply fixed computation to simple and complex scenes alike, wasting resources and producing suboptimal feature representations.Static designs can over-allocate computation to sparse scenes and under-serve densely populated or difficult scenarios.
  • Proposed approach: YOLO-Master introduces conditional computation that dynamically activates experts according to input content, breaking the static capacity–cost trade-off.The framework integrates a Mixture-of-Experts design into the YOLO pipeline for adaptive resource allocation.
  • Proposed approach: Its ES-MoE design uses soft Top-K routing during training, hard Top-K routing during inference, multi-scale expert groups, and load-balancing supervision.These mechanisms support gradient flow, deployment sparsity, distinct receptive fields, and balanced expert utilization.

2. Related Work

Real-time detectors largely rely on static computation, while YOLO-Master introduces conditional expert activation for adaptive resource allocation in lightweight CNN-based detection. Its feature-pyramid routing and decoupled training-inference strategy target both optimization stability and deployment efficiency.

  • Static Computation in Existing Detectors: Static computation remains a limitation of YOLO, RT-DETR, and attention-based detectors because their architectures apply uniform processing despite differing input complexity.The cited methods improve architecture, feature fusion, or attention efficiency but do not dynamically allocate computation across inputs.
  • Mixture-of-Experts for Detection: MoE extends conditional computation from model-capacity scaling and vision tasks toward dense object detection, where routing must handle multi-scale spatial features with varying object densities and scales.Unlike classification, detection requires routing across spatial feature hierarchies rather than only global image representations.
  • Mixture-of-Experts for Detection: YOLO-Master introduces a feature-pyramid MoE framework for lightweight CNN-based real-time detectors, enabling adaptive expert activation based on spatial characteristics.The approach is presented as the first MoE framework tailored to this detector setting.
  • YOLO-Master Framework: Training uses soft Top-K routing for gradient flow, whereas inference uses hard Top-K routing to provide genuine sparse activation and deployment efficiency.The framework applies ES-MoE modules in the Backbone and Neck, with routing across P3, P4, and P5 prediction layers.
  • Adaptive Computation versus Attention: Unlike attention, which recalibrates features through static architectures, YOLO-Master allocates capacity conditionally: simple regions activate fewer experts while complex regions access more.This distinction is framed as adaptive computation through expert activation rather than adaptive feature weighting.

3. Methodology

YOLO-Master integrates an Efficient Sparse Mixture-of-Experts module into the YOLO backbone and neck to enable instance-conditional computation. Dynamic routing selects and aggregates specialized experts, using soft Top-K training and hard Top-K inference to balance learning, accuracy, and efficiency.

  • 3.1. Overview of YOLO-Master: YOLO-Master inserts ES-MoE modules into the Backbone and Neck, enabling sparse, instance-conditional adaptive computation within a standard YOLO pipeline.The framework retains the Backbone, Neck, and Detection Head while dynamically enhancing feature extraction and feature processing across varying object scales and scene complexity.
  • 3.1. Overview of YOLO-Master: ES-MoE uses dynamic routing, Softmax gating, and weighted aggregation to select top-K experts and produce an enhanced feature map.The routing network generates instance-dependent signals, the gating mechanism assigns expert weights, and selected expert outputs are combined and normalized.
  • 3.4. Phased Routing Strategy: During training, soft Top-K routing preserves gradient flow, whereas hard Top-K inference activates only K experts with K ≪ E for practical computational sparsity.The phased strategy emphasizes high-weight experts during training and invokes only selected expert modules during sparse forward propagation at inference.
  • 3.2. Dynamic Routing Network: Each expert uses depthwise separable convolution and different kernel sizes, providing efficient transformations with diverse receptive fields for varied spatial contexts.Depthwise separable convolution reduces parameters and FLOPs, while kernel sizes such as 3, 5, and 7 cover multiple receptive-field scales.
  • 3.3. Gating Network Design: The lightweight gating network applies global average pooling and two 1 × 1 convolutions, with logits complexity independent of spatial dimensions H × W.Channel reduction constrains routing overhead, allowing efficient operation on high-resolution feature maps in both the Backbone and Neck.
  • 3.5. Loss Function Design: The combined YOLO detection loss and load-balancing loss promotes accurate detection while encouraging uniform expert utilization and preventing expert collapse.Minimizing the load-balancing loss helps leverage all E experts during training, supporting generalization and robustness without sacrificing inference sparsity.

4.1. Experimental Setup

Experiments evaluate YOLO-Master across five diverse detection benchmarks using a YOLOv12-Nano baseline with integrated MoE modules. Training uses standardized 640 × 640 settings, while evaluation reports accuracy and real-time efficiency under hard-sparse inference.

  • Datasets: Five benchmarks cover general, autonomous-driving, aerial, and crowded-scene detection: MS COCO, PASCAL VOC, VisDrone, KITTI, and SKU-110K.The datasets span 80, 20, 10, 3, and 1 categories, respectively.
  • Implementation: YOLOv12-Nano with width scaling factor 0.5 serves as the baseline with integrated MoE modules.Models are trained for 600 epochs at 640 × 640 resolution using SGD and cosine learning-rate scheduling.
  • Metrics: Evaluation reports mAP50:95, mAP50, parameter count with K activated experts, latency, and FPS.Efficiency measurements follow FP16, batch-size-1 deployment conditions on a dedicated inference accelerator.

4.2. Main Results

YOLO-Master-N achieves state-of-the-art performance across five benchmarks while maintaining real-time inference speed. It improves accuracy over YOLOv13-N across all benchmarks, with especially strong gains on challenging detection settings.

  • Main Results: +0.8% mAP on COCO, +1.4% on VOC, +2.1% on VisDrone, +1.5% on KITTI, and +0.7% on SKU-110K versus YOLOv13-N.The largest gains occur on VisDrone and KITTI.
  • Main Results: 18% faster than YOLOv13-N and only 8% slower than YOLOv11-N, YOLO-Master-N balances accuracy improvements with real-time efficiency.On SKU-110K, containing 147 objects per image, it achieves 58.2% mAP.

4.3. Ablation Studies

Ablations identify backbone-only ES-MoE with four experts, Top-2 routing, and MoE-only loss as effective design choices. The resulting architecture also improves detection, classification, and segmentation performance across reported tasks.

  • ES-MoE Placement: Backbone-only ES-MoE reaches 62.1% mAP with 2.66M parameters, improving 1.3% over the 60.8% baseline.
  • Expert Count: Four experts provide the best balance at 62.3% mAP and 82.2% mAP50 with 2.76M parameters, whereas eight add parameters without improving mAP.
  • Top-K Routing: Top-2 routing achieves 61.8% mAP with 50% sparsity, while Top-1 routing loses 0.5% mAP.
  • Generalization to Downstream Tasks: 76.6% Top-1 accuracy on ImageNet exceeds YOLOv11 and YOLOv12 by 6.6% and 4.9%, respectively.
  • Generalization to Downstream Tasks: 35.6% mAP mask surpasses YOLOv12-seg-N by 2.8% on MS COCO segmentation at 640×640.
  • Generalization to Downstream Tasks: 49.1% mAP box establishes the detection variant as a new state-of-the-art for small-scale models.

4.4. Qualitative Analysis

Qualitative comparisons show YOLO-Master-N producing more confident, complete, and localized detections across small-object, occlusion, complex, and dense scenarios.

  • Small Object Detection: 0.65-0.82 confidence enables accurate small-animal detection, compared with 0.47 for YOLOv12-N and 0.53 for YOLOv13-N.
  • Category Disambiguation: Accurate localization recovers an occluded person that YOLOv10-N through YOLOv12-N miss and YOLOv13-N detects only marginally.
  • Complex Scene: 0.85 average confidence versus 0.77 for v13 accompanies clean localization in the overlapping sheep-shearing scene.
  • Dense Scene: 0.87-0.97 confidence accompanies comprehensive detection in the dense dining scene with overlapping objects.
  • Overall Comparison: Across all scenarios, YOLO-Master-N shows higher average confidence and more complete detection coverage.

5. Conclusion

YOLO-Master introduces ES-MoE conditional computation into YOLO, using dynamic expert selection to address the capacity-efficiency trade-off. Experiments report state-of-the-art performance and efficiency, with potential extension to other vision tasks.

  • YOLO-Master integrates Efficient Sparse Mixture-of-Experts into YOLO for real-time object detection.
  • Soft top-K routing during training preserves gradient flow, while hard top-K routing during inference provides computational sparsity.
  • Five large-scale benchmarks support state-of-the-art performance with superior efficiency and simultaneous accuracy-efficiency gains.
  • The framework is presented as extensible to vision tasks beyond detection and resource-constrained real-time vision systems.
Loading 2512.23273v2…