PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLOv10: Real-Time End-to-End Object Detection

Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, Guiguang Ding

arXiv:2405.14458v2cs.CV

TL;DR

YOLO detectors remain hindered by NMS-dependent inference and insufficiently optimized architectures. YOLOv10 addresses both with NMS-free consistent dual assignments and holistic efficiency-accuracy model design, achieving state-of-the-art performance-latency trade-offs across model scales, including 46% lower latency than YOLOv9-C at the same performance.

  • Problem

    YOLOs rely on NMS, which slows inference and complicates end-to-end deployment, while their architectures lack comprehensive efficiency-accuracy optimization.

  • Method

    YOLOv10 combines consistent dual assignments for NMS-free training with holistic efficiency-accuracy-driven optimization of YOLO architecture components.

  • Results

    46% lower latency than YOLOv9-C at the same performance, with state-of-the-art computation-accuracy trade-offs across model scales.

  • Takeaways & Limitations

    YOLOv10 provides a family of real-time end-to-end detectors with improved performance-efficiency trade-offs across model scales.

  • Takeaways & Limitations

    Under NMS-free training, YOLOv10 still has a performance gap relative to one-to-many training with NMS, especially for smaller models.

Abstract

from arXiv · show

Over the past years, YOLOs have emerged as the predominant paradigm in the field of real-time object detection owing to their effective balance between computational cost and detection performance. Researchers have explored the architectural designs, optimization objectives, data augmentation strategies, and others for YOLOs, achieving notable progress. However, the reliance on the non-maximum suppression (NMS) for post-processing hampers the end-to-end deployment of YOLOs and adversely impacts the inference latency. Besides, the design of various components in YOLOs lacks the comprehensive and thorough inspection, resulting in noticeable computational redundancy and limiting the model's capability. It renders the suboptimal efficiency, along with considerable potential for performance improvements. In this work, we aim to further advance the performance-efficiency boundary of YOLOs from both the post-processing and model architecture. To this end, we first present the consistent dual assignments for NMS-free training of YOLOs, which brings competitive performance and low inference latency simultaneously. Moreover, we introduce the holistic efficiency-accuracy driven model design strategy for YOLOs. We comprehensively optimize various components of YOLOs from both efficiency and accuracy perspectives, which greatly reduces the computational overhead and enhances the capability. The outcome of our effort is a new generation of YOLO series for real-time end-to-end object detection, dubbed YOLOv10. Extensive experiments show that YOLOv10 achieves state-of-the-art performance and efficiency across various model scales. For example, our YOLOv10-S is 1.8$\times$ faster than RT-DETR-R18 under the similar AP on COCO, meanwhile enjoying 2.8$\times$ smaller number of parameters and FLOPs. Compared with YOLOv9-C, YOLOv10-B has 46\% less latency and 25\% fewer parameters for the same performance.

1 Introduction

The introduction identifies NMS-dependent post-processing and insufficiently optimized YOLO architectures as barriers to accuracy–speed efficiency. YOLOv10 addresses both through consistent dual assignments and holistic efficiency–accuracy-driven design, yielding NMS-free real-time detectors across model scales.

  • Motivation: One-to-many label assignment produces redundant predictions, requiring NMS that slows inference and makes performance sensitive to NMS hyperparameters.This prevents optimal end-to-end deployment of YOLOs.
  • Motivation: YOLO architecture design remains a fundamental challenge because it strongly affects both detection accuracy and speed.The introduction notes extensive exploration of backbone and neck design strategies.
  • Contributions: The work proposes a holistic efficiency-accuracy-driven model design strategy that optimizes the detection pipeline’s architecture from both efficiency and accuracy perspectives.The stated goal is to advance YOLO accuracy–speed boundaries through joint post-processing and architectural improvements.
  • Contributions: YOLOv10 introduces consistent dual assignments with dual label assignments and a consistent matching metric for NMS-free training.The strategy provides rich, harmonious supervision during training while eliminating NMS during inference.
  • Results: YOLOv10 comprises N / S / M / B / L / X real-time end-to-end detector scales and achieves improved computation–accuracy trade-offs across model scales on COCO.The introduction reports that YOLOv10 significantly outperforms previous state-of-the-art models on standard object-detection benchmarks.

2 Related Work

Related work covers efficient real-time object detectors, with YOLOs emerging as mainstream systems, and end-to-end detectors that streamline architectures by eliminating hand-crafted components and post-processing.

  • Real-time object detectors: Real-time object detection targets low-latency classification and localization for real-world applications, motivating substantial efforts to develop efficient detectors.The YOLO series is identified as the mainstream family of real-time object detectors.
  • Real-time object detectors: YOLO detectors commonly use a backbone, neck, and head, while successive versions introduce architectural, augmentation, training, and scaling improvements.YOLOv4 and YOLOv5 adopt CSPNet and enhanced PAN; YOLOv6 adds BiC, SimCSPSPPF, anchor-aided training, and self-distillation.
  • End-to-end object detectors: End-to-end object detection streamlines traditional pipelines by removing hand-crafted components and post-processing through one-to-one matching prediction.DETR introduces transformers and Hungarian loss, while subsequent DETR variants improve performance and efficiency.

3 Methodology

YOLOv10 combines dual label assignments for NMS-free training with a holistic efficiency-accuracy-driven redesign of YOLO components. The methodology aligns one-to-one and one-to-many supervision while reducing architectural redundancy and computational cost.

  • Dual label assignments: YOLOv10 adds a one-to-one head alongside the one-to-many branch, jointly optimizing both to combine rich supervision with NMS-free matching.The one-to-many branch supplies richer supervisory signals, while one-to-one matching avoids NMS post-processing.
  • Consistent matching metric: The uniform matching metric is m(α, β) = s · p^α · IoU(ˆb, b)^β, balancing semantic prediction and localization through α and β.Here, p is the classification score, IoU measures prediction-instance box overlap, and s indicates whether the prediction anchor lies within the instance.
  • Consistent matching metric: The consistent matching metric sets αo2o=r·αo2m and βo2o=r·βo2m, making the best one-to-many positive sample also best for one-to-one matching.The default uses r=1, so αo2o=αo2m and βo2o=βo2m; alignment improves across the top-1, top-5, and top-10 results.
  • Efficiency driven model design: The architecture redesign targets the stem, downsampling layers, stages, and head, addressing computational redundancy and constrained capability through efficiency- and accuracy-driven analysis.The stem incurs little computational cost, so efficiency-driven design focuses on the other three parts.
  • Lightweight classification head: 2.5× and 2.4×: YOLOv8-S classification-head FLOPs and parameters exceed the regression head, despite regression contributing more significantly to performance.The classification head uses 5.95G FLOPs and 1.51M parameters, versus 2.34G and 0.64M for regression.
  • Spatial-channel decoupled downsampling: The spatial-channel decoupled downsampling replaces standard stride-2 convolutions with pointwise channel modulation followed by depthwise spatial reduction.This reduces cost to O(2HWC^2 + 9/2HWC) and parameters to O(2C^2 + 18C), while retaining information and reducing latency.

4 Experiments

Experiments on COCO show that YOLOv10 delivers state-of-the-art accuracy-efficiency trade-offs and end-to-end latency across model scales. Ablations attribute these gains to consistent dual assignments and efficiency-driven architectural design, while identifying a remaining performance gap versus one-to-many NMS training.

  • Main results: YOLOv10 improves over YOLOv8 by 1.2% / 1.4% / 0.5% / 0.3% / 0.5% AP across N / S / M / L / X, respectively.It also uses 28% / 36% / 41% / 44% / 57% fewer parameters, 23% / 24% / 25% / 27% / 38% less calculations, and 70% / 65% / 50% / 41% / 37% lower latencies.
  • Main results: 1.5 AP and 2.0 AP gains over YOLOv6-3.0-N / S come with 51% / 61% fewer parameters and 41% / 52% less computations.For medium models, YOLOv10-B / M achieves 46% / 62% latency reduction compared with YOLOv9-C / YOLO-MS under the same or better performance.
  • Main results: 68% fewer parameters, 32% lower latency, and 1.4% AP improvement distinguish YOLOv10-L from Gold-YOLO-L.YOLOv10 also shows state-of-the-art performance and efficiency across model scales when evaluated with the original one-to-many training approach.
  • Ablation study: 4.63ms lower end-to-end latency from consistent dual assignments preserves 44.3% AP for YOLOv10-S.The dual assignments provide one-to-many supervision during training and one-to-one efficiency during inference, achieving the best AP-latency trade-off.
  • Ablation study: 11.8 M fewer parameters, 20.8 GFlOPs fewer computations, and 0.65ms lower latency result from efficiency-driven model design for YOLOv10-M.Spatial-channel decoupled downsampling, CIB, rank-guided block design, and large-kernel components are examined as architectural efficiency mechanisms.
  • Analysis: The performance gap versus original one-to-many training with NMS diminishes as model size increases, and regression errors form the larger performance bottleneck.The analysis concludes that a lightweight classification head can improve efficiency without compromising performance.

5 Conclusion

YOLOv10 advances real-time end-to-end object detection by addressing both post-processing and model architecture. It combines NMS-free training with holistic efficiency-accuracy-driven design to improve detection efficiency and performance-efficiency trade-offs.

  • YOLOv10 targets both post-processing and model architecture throughout the YOLO detection pipeline.
  • Consistent dual assignments enable NMS-free training and efficient end-to-end detection.
  • Holistic efficiency-accuracy-driven model design improves YOLO's performance-efficiency trade-offs.

A Appendix · A.1 Implementation Details

YOLOv10 models are trained from scratch with specified SGD settings and standard augmentations on eight NVIDIA 3090 GPUs. End-to-end speed is benchmarked on COCO validation images using batch size 1, TensorRT efficientNMSPlugin, and averaged latency with I/O overhead omitted.

  • A.1 Implementation Details: 500 epochs of from-scratch training use the SGD optimizer.All models are trained on 8 NVIDIA 3090 GPUs.
  • A.1 Implementation Details: SGD uses momentum 0.937 and weight decay 5×10−4.
  • A.1 Implementation Details: The initial learning rate is 1×10−2 and decays linearly to 1×10−4.
  • A.1 Implementation Details: Training applies Mosaic, Mixup, and copy-paste data augmentation.
  • A.1 Implementation Details: Latency is measured on the COCO val set with batch size 1 because NMS execution time depends on the input.
  • A.1 Implementation Details: The benchmark uses each detector's validation NMS hyperparameters and appends TensorRT efficientNMSPlugin for post-processing.
  • A.1 Implementation Details: Reported latency averages execution time across all images while omitting I/O overhead.
  • A.1 Implementation Details: Table 14 provides the detailed YOLOv10 hyper-parameters.

A.2 Details of Consistent Matching Metric

The consistent matching metric aligns classification targets between one-to-many and one-to-one branches, minimizing their supervision gap when the selected prediction is the best positive sample. It is obtained by matching the metric coefficients across branches, with the simplest choice setting them equal.

  • Target construction: The one-to-many branch defines positive samples Ω, while the one-to-one branch selects the i-th prediction and derives classification targets from normalized matching metrics.For j ∈ Ω, to2m,j = u* · mo2m,j; for the selected prediction, to2o,i = u* · mo2o,i / m*o2o.
  • Supervision gap: The supervision gap between branches is derived as the 1-Wasserstein distance between their classification targets.The one-to-many targets are ordered as t̂1 ≥ t̂2 ≥ ... ≥ t̂|Ω|, and the one-to-one target is replaced with u* in the derivation.
  • Supervision-gap minimization: The supervision gap is minimized when the selected prediction is the best positive sample in Ω, with mo2m,i = m*.As the lowest ordered target t̂n increases, A decreases and the rank n of i within Ω improves; the minimum occurs at t̂n = t̂1.
  • Metric derivation: The consistent matching metric makes mo2o monotonically increase with mo2m by setting r1 = r2 = r, yielding αo2o = r · αo2m and βo2o = r · βo2m.Taking r = 1 gives αo2o = αo2m and βo2o = βo2m.

A.3 Details of Rank-Guided Block Design

This section details the rank-guided block design algorithm and computes convolutional numerical rank from a reshaped weight matrix.

  • A.3 Details of Rank-Guided Block Design: The rank-guided block design algorithm is presented in Algorithm 1, with convolutional weights reshaped to (C_o, K^2 × C_i) for numerical-rank calculation.C_o and C_i denote output and input channels, respectively, while K denotes kernel size.

A.4 Training Cost Analyses

YOLOv10 maintains affordable training cost despite 500 training epochs, achieving high throughput on 8 NVIDIA 3090 GPUs. Its one-to-many head has similar training time but inferior performance relative to YOLOv10-M.

  • Training cost: 500 training epochs still yield affordable training cost for YOLOv10, which achieves high training throughput on 8 NVIDIA 3090 GPUs.The comparison uses official codebases and medium-scale models.
  • Training cost: 29.3 vs. 29.1 hours of training time is similar, but the one-to-many head reaches 48.9% vs. 51.1% AP compared with YOLOv10-M.The one-to-many head therefore has inferior performance despite comparable training time.

A.5 More Results on COCO

Additional COCO experiments evaluate YOLOv10 under original one-to-many training with NMS, across detailed scale- and IoU-specific metrics, lightweight-detector comparisons, and different training durations. YOLOv10 maintains favorable performance-efficiency trade-offs, including better performance and inference latency than YOLOv6 in 300-epoch experiments and lower training cost than compared models despite 500-epoch training.

  • COCO evaluation: YOLOv10 latency is measured on COCO using original one-to-many training with NMS, with detailed performance reported in Tabs. 16 and 17.Reported metrics include APval 50, APval 75, and APval small and APval large across different scales and IoU thresholds.
  • Lightweight-detector comparisons: YOLOv10 achieves superior performance-efficiency trade-offs compared with lightweight detectors DAMO-YOLO, YOLOv7, and DEYO.These comparisons are presented in Tab. 18.
  • Training-duration comparisons: YOLOv10 exhibits better performance and inference latency than YOLOv6 when models are trained for 300 epochs.The experiments follow prior work using 500 epochs and additionally report 300-epoch comparisons with YOLOv6.
  • Training cost: Despite 500-epoch training, YOLOv10 has lower training cost than the compared models.The training-cost comparison is presented in Tab. 15.

A.6 Inference Efficiency Comparison on CPU … A.9 Contribution, Limitation, and Broader Impact

YOLOv10 shows strong CPU performance-efficiency trade-offs, with holistic design reducing latency while preserving or improving accuracy. The paper also reports precise difficult-scene detection, introduces NMS-free training and architectural advances, and acknowledges limitations and potential applications.

  • A.6 Inference Efficiency Comparison on CPU: YOLOv10 achieves state-of-the-art performance-efficiency trade-offs on Intel Xeon Skylake CPU using OpenVINO.The comparison is presented in Fig. 5.
  • A.7 More Analyses for Holistic Efficiency-Accuracy Driven Model Design: 5.3% latency reduction is achieved for YOLOv10-S without compromising performance despite its small model scale.The result supports further accuracy-driven model design.
  • A.7 More Analyses for Holistic Efficiency-Accuracy Driven Model Design: 12.5% latency reduction is achieved for YOLOv10-M through efficiency-driven design, while combined design yields 0.8% AP improvement and 0.48ms latency reduction.YOLOv10-M has a larger model scale and more redundancy.
  • A.8 Visualization Results: YOLOv10 performs precise detection in difficult conditions, including low light and rotation, while detecting diverse, densely packed objects.The visualization includes bottles, cups, and people.
  • A.9 Contribution, Limitation, and Broader Impact: Consistent dual assignments enable rich one-to-many training supervision and efficient one-to-one inference for NMS-free YOLOs.The consistent matching metric reduces the theoretical supervision gap and improves performance.
  • A.9 Contribution, Limitation, and Broader Impact: The holistic architecture strategy reduces redundancy through lightweight heads, decoupled downsampling, and rank-guided blocks, while improving performance with large-kernel convolution and partial self-attention.These designs target high efficiency and low-cost performance enhancement.
  • A.9 Contribution, Limitation, and Broader Impact: The authors identify limited large-scale pretraining investigation and a remaining NMS-free performance gap, especially in small models, as limitations.They also note potential applications in medical image analysis and autonomous driving, alongside risks of malicious use.
Loading 2405.14458v2…