PaperlayerView exact revision on arXiv

Source-linked AI summary

Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models

Glenn Jocher, Jing Qiu, Mengyu Liu, Shuai Lyu, Fatih Cagatay Akyon, Muhammet Esat Kalfaoglu

arXiv:2606.03748v1cs.CVcs.AI

TL;DR

Real-time YOLO detectors still face inference, head-complexity, training, and small-object supervision limitations that complicate efficient deployment. YOLO26 addresses these with a unified architecture and training pipeline, improving accuracy–latency trade-offs across tasks and scales, including up to +7.2 AP for COCO pose estimation.

  • Problem

    Current YOLO-family detectors commonly depend on NMS, use heavy DFL-based heads, require long training, and may lack positive assignments for the smallest objects.

  • Method

    YOLO26 combines dual-head NMS-free inference, DFL removal, MuSGD, Progressive Loss, STAL, and task-specific heads and losses across vision tasks.

  • Results

    Across tasks and scales, YOLO26 improves over YOLO11 by up to +7.2 pose AP on COCO, +3.7 mask AP on COCO, and +3.4 mAP on DOTA-v1.0 OBB detection.

  • Takeaways & Limitations

    YOLO26 advances the accuracy–latency Pareto front while supporting detection, segmentation, classification, pose, and oriented detection in one model family.

Abstract

from arXiv · show

Real-time vision demands models that are accurate, efficient, and simple to deploy across diverse hardware. The YOLO family has become widely deployed for this reason, yet most YOLO detectors still rely on non-maximum suppression at inference, carry heavy detection heads due to Distribution Focal Loss, require long training schedules, and can leave the smallest objects without positive label assignments. We present Ultralytics YOLO26, a unified real-time vision model family that addresses these limitations through coordinated architecture and training advances. YOLO26 uses a dual-head design for native NMS-free end-to-end inference and removes DFL entirely, yielding a lighter head with unconstrained regression range. Its training pipeline combines MuSGD, a hybrid Muon-SGD optimizer adapted from large language model training; Progressive Loss, which shifts supervision toward the inference-time head; and STAL, a label assignment strategy that guarantees positive coverage for small objects. Beyond detection, YOLO26 introduces task-specific head and loss designs for instance segmentation, pose estimation, and oriented detection, producing consistent gains across tasks and scales. The family spans five scales (n/s/m/l/x) and supports detection, instance segmentation, pose estimation, classification, and oriented detection in a single pipeline, with an open-vocabulary extension, YOLOE-26, for text-, visual-, and prompt-free inference. Across all scales, YOLO26 achieves 40.9-57.5 mAP on COCO at 1.7-11.8 ms T4 TensorRT latency, advancing the accuracy-latency Pareto front over prior real-time detectors, while YOLOE-26x reaches 40.6 AP on LVIS minival under text prompting. Code and models are available at https://github.com/ultralytics/ultralytics.

1. Introduction

Ultralytics YOLO26 addresses persistent real-time detection limitations with a DFL-free dual-head architecture and coordinated training designed for native NMS-free inference, efficient regression, and small-object supervision. The unified family extends these improvements across multiple vision tasks and open-vocabulary inference while advancing accuracy–latency trade-offs over YOLO11.

  • Detector architecture: YOLO26 uses a dual-head design for native NMS-free inference and removes DFL, producing a lighter regression head with unconstrained range.The architecture retains an optional dense branch for accuracy-critical deployment.
  • Unified multi-task family: The family spans five size variants—n/s/m/l/x—and supports detection, instance segmentation, classification, pose estimation, and OBB detection.Task-specific extensions include a multi-scale proto pathway with auxiliary semantic supervision, an RLE-based uncertainty-aware keypoint objective, and revised long-edge angle supervision.
  • Multi-task results: +3.7 mask AP, +7.2 pose AP, and +3.4 mAP are the reported improvements over YOLO11 for COCO instance segmentation, COCO keypoints, and DOTA-v1.0 OBB detection, respectively.These gains are reported across scales for the task-specific YOLO26 extensions combined with shared detector improvements.
  • Open-vocabulary extension: YOLOE-26 retains text-prompted, visual-prompted, and prompt-free inference, and YOLOE-26x reaches 40.6 AP on LVIS minival under text prompting.The extension adds a stronger detector backbone, MobileCLIP2 text encoder, pseudo-label data engine, and decoupled segmentation training.
  • Accuracy–latency trade-off: At matched scales, YOLO26 improves COCO AP over YOLO11 by 1.6–2.8 points and sits on or advances the accuracy–latency Pareto front across model scales.The strongest AP–latency trade-off is reported at the medium, large, and extra-large scales.
  • Training pipeline: MuSGD, Progressive Loss, and STAL jointly improve optimization, align supervision with the inference-time head, and guarantee positive coverage for the smallest objects.Progressive Loss shifts supervision toward the inference-time head, while STAL addresses missing positive assignments for small objects.

2. Related Work

Related work progresses from two-stage and one-stage CNN detectors toward anchor-free, NMS-free, and transformer-based end-to-end designs, while extending real-time vision to segmentation, pose, oriented detection, and open-vocabulary inference. YOLO26 builds on these lines with task-specific advances, including aspect-ratio-aware rotated-box modeling and an enhanced unified open-vocabulary formulation.

  • Object detection: Two-stage detectors use region proposals followed by classification and refinement, whereas one-stage methods remove proposals for faster dense multi-scale prediction.R-CNN, Fast R-CNN, Faster R-CNN, SSD, RetinaNet, and the YOLO family represent successive accuracy–efficiency design shifts.
  • End-to-end detection: Anchor-free detectors remove hand-designed anchors, while YOLOv10 and transformer detectors reduce or eliminate NMS through one-to-one matching and direct set prediction.FCOS and CenterNet exemplify anchor-free prediction; YOLOv10 uses dual assignments, while DETR-style methods use Hungarian matching.
  • Real-time transformers: Real-time transformer detectors improve speed–accuracy trade-offs through efficient encoders, multi-scale processing, distillation, and refined localization or optimization.RT-DETR targets deployment efficiency, RT-DETRv4 explores distillation, and D-FINE and DEIM improve localization and training efficacy.
  • Instance segmentation: Instance segmentation spans RoI-aligned two-stage masks, fully convolutional or prototype-based methods, and transformer set-based decoding with denoising and masked attention.Mask R-CNN, CondInst, YOLACT, Mask2Former, and MaskDINO illustrate the main accuracy–efficiency and global-context trade-offs.
  • Pose and oriented detection: YOLO26 extends prior YOLO pose and oriented-detection lineages with RLE-enhanced direct keypoint regression and an aspect-ratio-aware angle loss with an optimized decoder.YOLO-Pose established heatmap-free joint detection and keypoint regression, while YOLOv8 and YOLO11 introduced ProbIoU-trained OBB heads; YOLO26 addresses rotated-box ambiguity and boundary discontinuities.
  • Open-vocabulary detection: YOLOE unifies text-prompted, visual-prompted, and prompt-free inference, and YOLOE-26 advances this formulation with stronger language, data, backbone, and segmentation components.The cited formulation uses RepRTA, SAVPE, and LRPCHead, while YOLOE-26 adds an upgraded text encoder, pseudo-label data engine, and decoupled segmentation training.

3. Methodology

YOLO26 combines a dual-head, DFL-free detector with MuSGD, Progressive Loss, and STAL to support efficient end-to-end inference and stronger training. It extends the shared architecture with task-specific designs for segmentation, pose, oriented detection, and open-vocabulary inference.

  • Shared architecture and training: YOLO26 targets end-to-end simplicity, deployment efficiency, and stronger optimization through NMS-free one-to-one inference, DFL-free regression, MuSGD, Progressive Loss, and STAL.The shared backbone and neck feed one-to-many and one-to-one heads while STAL preserves tiny-object assignments and Progressive Loss shifts branch emphasis during training.
  • Shared architecture and training: The dual-head design offers an accuracy–latency knob: the one-to-one head is NMS-free and fixed-output, whereas the one-to-many head uses dense supervision and NMS.The one-to-many head uses TAL with topk = 10 and produces outputs of shape (N, nc + 4, 8400).
  • Shared architecture and training: YOLO26 removes DFL from the detection head, producing unconstrained box regression and improving compatibility with runtimes and accelerators favoring standard operators and minimal decoding.The methodology frames this as an accuracy–efficiency trade-off for resource-constrained deployment, supported by compensating training enhancements.
  • Shared architecture and training: MuSGD combines Muon with SGD-momentum, while Progressive Loss shifts optimization emphasis from the dense branch early to the one-to-one branch later.Muon applies momentum updates followed by lightweight orthogonalization, and early dense-branch supervision stabilizes feature learning.
  • Shared architecture and training: STAL decouples candidate-selection geometry from regression geometry to address small-object assignment failures.During candidate filtering, STAL constructs an assignment surrogate using the smallest feature-pyramid stride smin.
  • Task-specific extensions: Task-specific extensions preserve the common backbone and neck while adapting heads and supervision for segmentation, pose estimation, and oriented bounding-box detection.Segmentation adds multi-scale prototypes and training-only semantic supervision; pose adds RLE-based per-joint uncertainty; oriented detection addresses angle-boundary ambiguity.

4. Experiments

Experiments show that YOLO26’s coordinated architecture and training changes improve accuracy, simplify end-to-end deployment, and transfer across detection, segmentation, and open-vocabulary tasks. Ablations and benchmark comparisons validate DFL removal, STAL, MuSGD, scheduled branch weighting, task-specific designs, and YOLOE-26 refinements.

  • Detection ablations: Removing DFL costs 0.6 AP on YOLO11s, but L1 supervision, STAL, and backbone/neck refinement recover the gap while producing a lighter head.The baseline changes 47.0 to 46.4 AP before the three successive +0.2 improvements.
  • Optimization: MuSGD improves mAP by +0.4 while shortening YOLO26 training from 600 to 500 epochs.The comparison trains YOLO26 from scratch on COCO against standard SGD, with the same optimizer advantage also observed in controlled ImageNet classification experiments.
  • Training dynamics: The best branch-weight schedule, (0.8, 0.2) →(0.1, 0.9), raises E2E AP from 46.4 to 46.7 by retaining one-to-one supervision early and emphasizing it later.Starting with a fully suppressed one-to-one branch or an excessively one-to-many-dominated schedule performs worse.
  • Small-object assignment: STAL with sref = 16 reaches 46.8 AP, improving the YOLO11s baseline by +0.2 AP and raising APS from 29.0 to 29.6.Using sref = 8 fails to improve overall AP and reduces APS, while sref = 32 also degrades performance.
  • Deployment comparison: The one-to-one head trails the NMS-based variant by only 0.6–0.8 AP across model scales, while YOLO26 provides the strongest overall AP–latency trade-off at the standard NMS operating point.The one-to-one path enables simpler end-to-end deployment, whereas the one-to-many head remains available when maximum AP is prioritized.
  • Task-specific and open-vocabulary results: YOLO26’s task-specific designs improve segmentation and open-vocabulary performance, including 32.0% to 32.4% mAP from the multi-scale Proto Module and final YOLOE-26 results of 29.9/31.0 AP.YOLOE-26 improves prompted detection, zero-shot segmentation, and prompt-free detection, reaching up to 31.1 AP on LVIS minival.

5. Conclusion

YOLO26 is presented as a unified real-time vision model family combining a dual-head NMS-free architecture with MuSGD, Progressive Loss, and STAL to improve the accuracy–latency trade-off across five scales. Removing DFL yields a lighter detection head while preserving localization quality, alongside task-specific refinements beyond detection.

  • Conclusion: YOLO26 combines dual-head NMS-free architecture, MuSGD, Progressive Loss, and STAL across five model scales.The coordinated design targets the accuracy–latency trade-off in real-time vision.
  • Conclusion: Removing DFL produces a lighter detection head while preserving localization quality.The conclusion attributes this result to DFL removal together with stronger optimization and label assignment.
  • Conclusion: Beyond detection, YOLO26 incorporates task-specific refinements including multi-scale prototype fusion for segmentation.The supplied passage introduces additional task-specific designs beyond detection, though the excerpt ends before listing them all.

S. Supplementary Materials · S1. MuSGD Transfer to Classification

The supplementary S1 section tests whether MuSGD’s advantage transfers beyond detection through a controlled ImageNet classification comparison that isolates the optimizer.

  • S1. MuSGD Transfer to Classification: The study evaluates MuSGD transfer beyond detection using a controlled ImageNet classification comparison summarized in Table S1.The comparison is organized by model scale.
  • S1. MuSGD Transfer to Classification: For each scale, paired models use the same backbone architecture and training recipe, differing only in optimizer.This design isolates optimizer effects between the paired models.
  • S1. MuSGD Transfer to Classification: Table S1 therefore compares classification models across scales under matched architecture and training conditions.The reported setup is intended to assess whether the MuSGD advantage transfers beyond detection.

S2. Architecture Visualizations

This section supplements the methodology with visualizations of YOLO26’s overall architecture and the building blocks composing the network. Together, Figure S1 and Figure S2 clarify the model’s macro-architecture and module composition.

  • Architecture Visualizations: Figure S1 presents the overall YOLO26 model architecture, while Figure S2 details the building blocks used to assemble the network.Together, the figures make the macro-architecture and underlying module composition explicit.

S3. Training Recipes

YOLO26’s training recipes document how released configurations implement regression, pretraining, and COCO fine-tuning. The recipes clarify that YOLO26 replaces DFL with direct L1 regression while retaining the legacy `dfl` field for compatibility.

  • Loss configuration: YOLO26 removes Distribution Focal Loss and uses direct L1 regression; the retained `dfl` field scales the L1 loss gain for backward compatibility.A nonzero `dfl` value therefore does not mean DFL is active.
  • Objects365-v1 Pretraining: Objects365-v1 pretraining runs for 150 epochs at 640 × 640 resolution with batch size 128 and size-specific augmentation and internal settings.Tables S2–S4 are transcribed from released pretraining checkpoint train arguments, with most floating-point values rounded to two decimals.
  • COCO Fine-Tuning: Released COCO detection checkpoints use end-to-end training at 640×640 resolution with batch size 128.Tables S5–S7 are transcribed from released COCO checkpoint train arguments; most floating-point values are rounded to two decimals, while lr0 and weight decay retain original precision.

S4. Comparison with Recent Real-Time Detectors

Table S8 presents the complete grouped comparison of YOLO26 with recent real-time detectors across the s, m, l, and x scales, complementing Section 4.4’s positioning in the main paper.

  • Comparison with Recent Real-Time Detectors: Table S8 provides the full grouped s/m/l/x comparison with recent real-time detectors.The table supplies the complete per-scale breakdown used to position YOLO26 in the main paper.

S5. Additional Task Benchmarks

This subsection presents supplementary full-model benchmark comparisons between YOLO11 and YOLO26 for instance segmentation, pose estimation, and oriented bounding-box detection. These tables extend the additional-task results while keeping the main paper focused on task-specific ablations and methodological takeaways.

  • Instance Segmentation: Table S9 reports the full COCO instance segmentation benchmark comparison between YOLO11 and YOLO26.The comparison is provided in the supplementary materials as part of the additional-task benchmark results.
  • Pose Estimation: Table S10 reports the full COCO pose estimation benchmark comparison between YOLO11 and YOLO26.This full model-family comparison is included in the supplementary materials.
  • Oriented Bounding Box Detection: Table S11 reports the full DOTA-v1.0 OBB benchmark comparison between YOLO11 and YOLO26.The supplementary table covers oriented bounding-box detection among the additional task results.

S6. YOLOE-26 Implementation Details

YOLOE-26 training uses a four-stage, multi-GPU pipeline that initializes text-prompt training from Objects365-v1-pretrained YOLO26 checkpoints, then branches into visual-prompt, prompt-free, and segmentation fine-tuning. The stages use selective parameter unfreezing, shared grounding data, and stage-specific optimizers and schedules.

  • Training pipeline: A total batch size of 256 is used across multiple GPUs, with text-prompt training preceding three parallel downstream branches.The downstream branches are visual prompt, prompt-free, and segmentation, each initialized from the best text-prompt checkpoint.
  • TP stage: The text-prompt stage initializes from YOLO26 Objects365-v1 checkpoints pretrained for 150 epochs and uses MuSGD with an initial learning rate of 1.25 × 10−3.The schedule uses lrf = 0.5, momentum 0.9, weight decay 5 × 10−4, or 7 × 10−4 for the S scale; epochs vary by scale.
  • VP and PF stages: 10 epochs are used for both visual-prompt and prompt-free fine-tuning with AdamW, while selectively unfreezing SAVPE and cv4 for visual prompting and cv3 for prompt-free training.Both branches use an initial learning rate of 2×10−3, lrf = 0.01, momentum 0.9, and weight decay 0.025.
  • SEG stage: The segmentation branch fine-tunes for 10 epochs with MuSGD and the text-prompt learning-rate schedule, unfreezing only cv5 and proto.Its schedule uses lr0 = 1.25×10−3 and lrf = 0.5 across all scales.
  • Training data: All four stages share Objects365-v1, GQA, and Flickr30k training data refined with the YOLOE data engine.The refined annotations exhibit fewer missing instances in representative examples.

S7. Additional YOLOE-26 Benchmarks

Additional YOLOE-26 benchmarks show consistent gains in text-prompted, zero-shot segmentation, and prompt-free open-vocabulary settings across model scales. The strongest results include 40.6 AP for text prompting, 27.4 / 26.7 APm for zero-shot segmentation, and 31.1 AP for prompt-free detection.

  • Prompt-based Detection: YOLOE-26 improves text-prompted detection over prior YOLOE variants at matched s/m/l scales, while YOLOE-26x reaches 40.6 AP.Gains are +3.3/+2.4/+2.6 AP over YOLOE-11 and +2.9/+2.8/+1.9 AP over YOLOE-v8 at s/m/l.
  • Prompt-based Detection: YOLOE-26n reaches 24.7 AP with 3.9M parameters, extending text-prompted gains to resource-constrained settings.The result is reported at the lightweight end of the model family.
  • Zero-shot Segmentation: YOLOE-26 consistently outperforms prior variants in zero-shot segmentation, with YOLOE-26x achieving 27.4 / 26.7 APm (T / V).YOLOE-26s achieves 20.5 / 19.1 APm (T / V) at less than half the computational cost of YOLOE-v8m.
  • Zero-shot Segmentation: YOLOE-26x achieves 24.9 / 23.3 APm_r (T / V) on rare categories in zero-shot segmentation.These rare-category results accompany the model’s best overall segmentation performance.
  • Prompt-free Detection: YOLOE-26x attains 31.1 AP in prompt-free detection, while YOLOE-26l reaches 28.0 AP with 23.6M parameters.YOLOE-26l is comparable to GenerateU-L’s 27.9 AP with 467M parameters, using nearly 20× fewer parameters.
Loading 2606.03748v1…