PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLOE-26: Integrating YOLO26 with YOLOE for Real-Time Open-Vocabulary Instance Segmentation

Ranjan Sapkota, Manoj Karkee

arXiv:2602.00168v1cs.CV

TL;DR

YOLOE-26 addresses the closed-vocabulary limits of YOLO detectors by combining YOLOv26’s deployment-oriented, NMS-free pipeline with YOLOE’s open-vocabulary learning. It supports text-prompted, visual-prompted, and prompt-free real-time instance segmentation while preserving YOLO-family speed and determinism, though prompt-free performance and fine-grained generalization remain limited.

  • Problem

    YOLOv26 remains constrained by fixed training-time categories, motivating open-vocabulary vision systems for evolving real-world environments where retraining and redeployment are impractical.

  • Method

    YOLOE-26 integrates YOLOv26’s NMS-free, end-to-end detection pipeline with YOLOE’s embedding-based support for text prompts, visual prompts, and prompt-free inference.

  • Results

    YOLOE-26 enables real-time instance segmentation across three prompting modes while preserving YOLO-family speed and determinism through embedding-based similarity matching and re-parameterizable prompting components.

  • Takeaways & Limitations

    YOLOE-26 provides a practical balance between open-world semantic flexibility and edge-ready real-time segmentation for diverse deployment domains.

  • Takeaways & Limitations

    Prompt-free performance remains below prompted settings, while multi-source supervision, pseudo-masks, prompt sensitivity, dataset bias, and long-tail semantics constrain precision and generalization.

Abstract

from arXiv · show

This paper presents YOLOE-26, a unified framework that integrates the deployment-optimized YOLO26(or YOLOv26) architecture with the open-vocabulary learning paradigm of YOLOE for real-time open-vocabulary instance segmentation. Building on the NMS-free, end-to-end design of YOLOv26, the proposed approach preserves the hallmark efficiency and determinism of the YOLO family while extending its capabilities beyond closed-set recognition. YOLOE-26 employs a convolutional backbone with PAN/FPN-style multi-scale feature aggregation, followed by end-to-end regression and instance segmentation heads. A key architectural contribution is the replacement of fixed class logits with an object embedding head, which formulates classification as similarity matching against prompt embeddings derived from text descriptions, visual examples, or a built-in vocabulary. To enable efficient open-vocabulary reasoning, the framework incorporates Re-Parameterizable Region-Text Alignment (RepRTA) for zero-overhead text prompting, a Semantic-Activated Visual Prompt Encoder (SAVPE) for example-guided segmentation, and Lazy Region Prompt Contrast for prompt-free inference. All prompting modalities operate within a unified object embedding space, allowing seamless switching between text-prompted, visual-prompted, and fully autonomous segmentation. Extensive experiments demonstrate consistent scaling behavior and favorable accuracy-efficiency trade-offs across model sizes in both prompted and prompt-free settings. The training strategy leverages large-scale detection and grounding datasets with multi-task optimization and remains fully compatible with the Ultralytics ecosystem for training, validation, and deployment. Overall, YOLOE-26 provides a practical and scalable solution for real-time open-vocabulary instance segmentation in dynamic, real-world environments.

1 INTRODUCTION

YOLOE-26 addresses the closed-vocabulary limits of efficient YOLO detectors by combining YOLOv26’s deployment-oriented, NMS-free pipeline with YOLOE’s open-vocabulary prompting. The framework targets real-time instance segmentation across text-prompted, visual-prompted, and prompt-free settings while balancing accuracy, efficiency, and deployability.

  • 1 INTRODUCTION: YOLOv26’s development emphasizes deployment robustness and pipeline simplification through NMS-free prediction, DFL removal, and the MuSGD optimizer.These choices are described as reducing latency and improving performance on low-power CPUs and edge devices.
  • 1 INTRODUCTION: YOLOE-26 combines YOLOv26’s NMS-free, end-to-end detection with YOLOE’s embedding-based open-vocabulary learning for real-time instance segmentation.The resulting framework supports text prompts, visual prompts, and prompt-free inference within a unified approach.
  • 1 INTRODUCTION: YOLO26 comparisons use COCO mAP(50–95) against inference and end-to-end latency, including comparisons with earlier YOLO versions, YOLOv10, and RT-DETR variants.The figure focuses on accuracy–speed trade-offs and overall pipeline efficiency under TensorRT FP16 on an NVIDIA T4 GPU.
  • 1.1 Background and Motivation: Closed-vocabulary YOLO detectors cannot adapt to unseen categories at inference, making retraining and redeployment impractical as real-world object categories evolve.This limitation motivates open-vocabulary vision systems for open-world environments.
  • 1.1 Background and Motivation: Open-vocabulary methods use semantic embeddings and vision–language pretraining, but many rely on transformer-heavy architectures and face deployment-efficiency challenges.YOLOE improves this direction but still has limitations in prompt handling, scalability, training cost, and inference efficiency.

2 YOLOE-26 Architecture Overview

YOLOE-26 combines YOLOv26’s deployment-efficient, NMS-free pipeline with open-vocabulary instance segmentation through semantic object embeddings and similarity-based prompting. Its unified design supports prompted and prompt-free inference while scaling across model sizes with varied accuracy–efficiency trade-offs.

  • Core Architecture: YOLOE-26 integrates a YOLOv26 backbone, PAN/FPN-style multi-scale neck, and end-to-end regression and segmentation heads.The multi-scale neck fuses fine-grained localization cues with global semantic information for instance segmentation.
  • Open-Vocabulary Representation: Replacing fixed class logits, the object embedding head represents each anchor point with a semantic vector for open-vocabulary recognition.Category prediction uses similarity between object embeddings and prompt embeddings rather than fixed-category classification.
  • Prompting Components: RepRTA preserves the standard YOLO inference path by re-parameterizing its auxiliary alignment network into the embedding head with zero additional runtime cost.This design retains the deployment advantages of the YOLOv26 pipeline while enabling text–object alignment.
  • Prompt-Free Inference: LRPC reduces prompt-free computation by filtering relevant object regions before matching them against a large built-in vocabulary.The prompt-free setting achieves 29.9 mAP50-95 and 38.7 mAP50 for YOLOE-26x-seg-pf on the minival benchmark.
  • Unified Inference Modes: A shared embedding space supports text prompts, visual prompts, and prompt-free vocabulary retrieval within one architecture.The three modes share the same object embedding head, enabling flexible open-vocabulary instance segmentation without retraining for new categories.
  • Performance and Scaling: Prompted and prompt-free evaluations show monotonic scaling across model sizes, while model variants trade segmentation accuracy against computational cost.YOLOE-26x-seg reaches 39.5 mAP50–95 with text prompts and 36.2 with visual prompts; YOLOE-26l-seg exceeds 36 mAP50–95 with fewer than 90 FLOPs (B).

3 OPEN-VOCABULARY PROMPTING MECHANISMS

YOLOE-26 supports text-prompted, visual-prompted, and prompt-free open-vocabulary instance segmentation within a unified, real-time architecture. These mechanisms use semantic matching to connect user prompts or built-in categories with predicted object embeddings and masks.

  • Overview: YOLOE-26 formulates open-vocabulary recognition as similarity matching between learned object embeddings and prompt embeddings.The framework replaces fixed class-logit reasoning with a shared embedding-based formulation.
  • Overview: Together, the three prompting mechanisms unify text-guided, example-guided, and autonomous perception in one deployment-oriented architecture.The paper positions this versatility for robotics, autonomous systems, surveillance, industrial inspection, and precision agriculture.
  • 3.1 Text-Prompted Instance Segmentation: Text prompts encode natural-language object descriptions into semantic embeddings that determine category assignment and instance-mask generation.Text prompts can specify targets such as “person,” “bus,” or “red apple.”
  • 3.1 Text-Prompted Instance Segmentation: RepRTA refines text embeddings during training and is re-parameterized into the object embedding head, adding zero inference cost.This preserves the speed and determinism of the YOLOv26 design while supporting open-vocabulary alignment.
  • 3.2 Visual-Prompted Instance Segmentation: Visual prompting uses example boxes or masks to identify visually similar objects when text descriptions are insufficiently precise.SAVPE produces a compact visual prompt embedding aligned with object embeddings through semantic and activation branches.
  • 3.2 Visual-Prompted Instance Segmentation: SAVPE matches encoded visual-prompt embeddings with object embeddings to predict instance masks for corresponding objects.The visual-prompt pipeline extracts multi-scale features, encodes boxes or masks, and matches the resulting embedding.
  • 3.3 Prompt-Free Instance Segmentation: Prompt-free inference uses a built-in vocabulary of 4,585 RAM++ categories and lazily retrieves category names only for relevant object regions.LRPC first identifies candidate regions, filters anchor points, matches the remaining regions to the vocabulary, and generates labels and masks.

4 TRAINING STRATEGY AND IMPLEMENTATION

YOLOE-26 integrates open-vocabulary prompt alignment with a deployment-first YOLOv26-style segmentation pipeline. Its training combines diverse supervision, multi-task objectives, staged prompt specialization, and Ultralytics-compatible workflows.

  • Datasets and supervision: YOLOE-26 uses detection and grounding datasets to provide diverse categories, language grounding, and visual variability for open-vocabulary training.Objects365, GQA, and Flickr30k Entities supply complementary supervision.
  • Datasets and supervision: Pseudo-masks generated from bounding boxes provide segmentation targets when large-scale detection and grounding datasets lack high-quality instance masks.Refinement can suppress leakage and fragmented or spurious regions while enforcing mask smoothness.
  • Datasets and supervision: Language phrases are mapped to prompt embeddings, keeping supervision consistent across datasets with differing vocabularies.Each training region includes a box, mask, and semantic label, which may originate from a grounding phrase.
  • Objective functions: A weighted multi-task objective combines semantic classification, box localization, mask segmentation, and reference-related losses.The weights balance semantic alignment, localization, and segmentation quality; dice loss may improve robustness to foreground-background imbalance.
  • Objective functions: The model predicts boxes, mask representations, and object embeddings, assigning categories through similarity between object and prompt embeddings rather than fixed logits.Similarity scores replace conventional closed-set class logits.
  • Optimization and implementation: Training is staged across text-prompt pretraining, visual-prompt adaptation, and prompt-free specialization to stabilize optimization and reduce compute.RepRTA supports compact inference re-parameterization, while SAVPE adaptation can freeze most of the network; models support Ultralytics training, validation, export, and deployment workflows.

5 Conclusion and Future Roadmap

YOLOE-26 integrates YOLOv26’s deployment-oriented, NMS-free design with YOLOE’s open-vocabulary learning to support real-time segmentation across prompted and prompt-free settings. The roadmap emphasizes autonomous perception, stronger supervision, and edge-focused optimization while acknowledging persistent performance and deployment limitations.

  • Conclusion: YOLOE-26 combines end-to-end YOLOv26 detection with YOLOE’s open-vocabulary paradigm for real-time instance segmentation.The framework supports text-prompted, visual-prompted, and prompt-free inference while retaining embedding-based similarity matching and re-parameterizable prompting components.
  • Limitations: Prompt-free performance remains below text- and visual-prompted settings, reflecting the difficulty of unconstrained object discovery in large-vocabulary environments.Other limitations include label noise from multi-source supervision, reduced boundary precision, prompt sensitivity, and incomplete separation of visually similar categories.
  • Future Roadmap: Edge-first optimization remains necessary for deployment under extreme compute and energy constraints, including quantization, distillation, prompt caching, and efficient vocabulary indexing.These directions target real-time operation on CPUs, embedded GPUs, and mobile NPUs.
  • Future Roadmap: Future work prioritizes agentic perception loops with prompt-free discovery, contextual prompt refinement, uncertainty-aware re-prompting, and temporal or multi-view self-verification.The roadmap also includes lightweight vision–language reasoning for more discriminative attribute-based or compositional prompts.
  • Future Roadmap: Continual, federated, self-supervised, and weakly supervised learning are proposed to expand categories, preserve privacy, reduce forgetting, and improve mask quality.Suggested techniques include iterative pseudo-label cleanup, boundary-aware losses, and temporal consistency.
  • Conclusion: YOLOE-26 demonstrates that open-vocabulary segmentation can achieve YOLO-level efficiency through tightly integrated end-to-end detection and lightweight vision–language embeddings.The paper identifies agentic learning as a pathway toward autonomous, self-improving, deployment-ready segmentation systems.
Loading 2602.00168v1…