PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLOE: Real-Time Seeing Anything

Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, Guiguang Ding

arXiv:2503.07465v2cs.CV

TL;DR

YOLOE addresses the challenge of achieving flexible open-prompt detection and segmentation without sacrificing the efficiency and deployability of YOLO models. It unifies text, visual, and prompt-free mechanisms through RepRTA, SAVPE, and LRPC, and reports strong performance with lower training cost and faster inference. On LVIS and COCO transfer, the model improves over the cited baselines and closed-set YOLOv8 variants under substantially reduced training cost or time.

  • Problem

    Efficient YOLO models are limited by predefined categories, while open-prompt methods often incur high computational or deployment costs.

  • Method

    YOLOE unifies detection and segmentation for text, visual, and prompt-free inputs using RepRTA, SAVPE, and LRPC.

  • Results

    YOLOE performs strongly across diverse open prompt mechanisms, including 3.5 AP higher LVIS performance with 3× less training cost and COCO transfer gains over YOLOv8 with nearly 4× less training time.

  • Takeaways & Limitations

    YOLOE provides a unified, efficient baseline for real-time open prompt-driven detection and segmentation.

Abstract

from arXiv · show

Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE's exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with 3$\times$ less training cost and 1.4$\times$ inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 AP$^b$ and 0.4 AP$^m$ gains over closed-set YOLOv8-L with nearly 4$\times$ less training time. Code and models are available at https://github.com/THU-MIG/yoloe.

1. Introduction

YOLOE addresses the gap between closed-set efficiency and open-prompt flexibility with one unified detection and segmentation model. It supports text, visual, and prompt-free mechanisms while targeting strong accuracy, low training cost, and efficient inference.

  • Closed-set YOLO models are efficient but restricted to predefined categories, limiting flexibility in open scenarios requiring arbitrary-object detection and segmentation.
  • Existing unified or prompt-specific open-set methods struggle to balance accuracy, efficiency, training cost, and deployment complexity.DINO-X has resource-intensive training and inference, while text and visual prompt methods introduce modality-fusion or encoder overhead.
  • YOLOE unifies detection and segmentation across text, visual, and prompt-free mechanisms using RepRTA, SAVPE, and LRPC.RepRTA improves pretrained textual embeddings, SAVPE encodes visual cues, and LRPC retrieves categories from a built-in vocabulary.
  • 3× less training cost and 1.4× faster inference accompany YOLOE-v8-S's 3.5 AP advantage over YOLO-Worldv2-S on LVIS.The reported speedups are measured on T4 and iPhone 12, with 1.4× and 1.3× gains respectively.
  • YOLOE-v8-L outperforms T-Rex2 by 3.3 APr and GenerateU by 0.4 AP with 2× less training data and 6.3× fewer parameters.

2. Related Work

Related work has expanded open-set detection and segmentation through text, visual, and prompt-free mechanisms, but unified solutions remain costly or difficult to deploy. YOLOE targets a more efficient single architecture for these prompt settings.

  • Traditional detection and segmentation: Traditional detection and segmentation methods primarily use closed-set paradigms, with two-stage frameworks emphasizing proposals and single-stage detectors emphasizing speed.
  • Text-prompted detection and segmentation: Text-prompted methods align visual features with textual embeddings to recognize novel categories, with systems such as GLIP, Grounding DINO, and YOLO-World advancing this direction.
  • Visual-prompted detection and segmentation: Visual prompts complement language by specifying objects through cues such as boxes or masks, especially when language alone is insufficient.
  • Prompt-free detection and segmentation: Prompt-free approaches generate object descriptions using language models, but this introduces substantial computational overhead.
  • Closing remarks: YOLOE aims to combine detection and segmentation across diverse open prompts in one architecture with real-time performance and easier edge deployment.The comparison specifically contrasts YOLOE with DINO-X's extensive training cost and inference overhead.

3. Methodology

YOLOE builds on a YOLO architecture and adds dedicated strategies for text, visual, and prompt-free inputs. These designs improve alignment or retrieval efficiency while preserving a low-cost deployment path.

  • Model architecture: YOLOE uses a YOLO-style backbone, PAN, regression head, segmentation head, and object embedding head to support detection and segmentation.The regression head predicts boxes, while the segmentation head produces prototypes and mask coefficients.
  • Re-parameterizable region-text alignment: The label computation contrasts anchor-point object embeddings with prompt embeddings, producing scores across N anchor points and C prompts.The embedding dimension is D.
  • Re-parameterizable region-text alignment: For text prompts, RepRTA improves pretrained textual embeddings with a lightweight auxiliary network and removes its deployment cost through re-parameterization.Text embeddings are cached before training, and the auxiliary network is integrated into the object embedding head after training.
  • Semantic-activated visual prompt encoder: SAVPE separates semantic and activation branches: one extracts prompt-agnostic semantic features, while the other produces grouped prompt-aware weights from visual masks and image features.Their aggregation yields a visual prompt embedding with low-dimensional processing and minimal cost.
  • Lazy region-prompt contrast: For prompt-free recognition, LRPC retrieves category names from a built-in vocabulary only for anchor points identified as objects, avoiding language-model dependency.A threshold δ filters anchor points before vocabulary matching, improving efficiency without performance drop.

4. Experiments

YOLOE is evaluated as a unified detection-and-segmentation model across text, visual, and prompt-free settings, emphasizing zero-shot performance, efficiency, and transferability. Results show strong performance across LVIS and COCO, supported by targeted ablations of RepRTA, SAVPE, and LRPC.

  • Implementation details: YOLOE uses YOLOv8 or YOLO11 backbones at small, medium, and large scales, with staged training for text, visual, and prompt-free capabilities.Text-prompt training lasts 30 epochs, SAVPE training 2 epochs, and specialized prompt-embedding training 1 epoch.
  • Segmentation evaluation: YOLOE-v8-M/L achieve 20.8/23.5 APm zero-shot on LVIS, exceeding fine-tuned YOLO-Worldv2-M/L by 3.0/3.7 APm.The comparison concerns segmentation performance on LVIS.
  • Prompt-free evaluation: YOLOE-v8-L exceeds GenerateU by 0.4 AP and 3.5 APr with 6.3× fewer parameters and 53× faster inference in the prompt-free setting.The prompt-free model uses a built-in large vocabulary and specialized prompt embedding to identify objects without explicit prompts.
  • Ablation study: RepRTA improves AP by 2.3% without inference overhead, SAVPE beats mask pooling by 1.5 AP, and LRPC provides up to 1.7× inference speedup at equal performance.Adding joint segmentation causes a 0.2% AP and 0.9 APf drop attributed to multi-task learning.

5. Conclusion

YOLOE integrates detection and segmentation across text, visual-cue, and prompt-free mechanisms in one efficient model. RepRTA, SAVPE, and LRPC support these prompt modes with favorable performance and low cost.

  • 5. Conclusion: YOLOE is a single highly efficient model for object detection and segmentation across diverse open prompt mechanisms.The supported mechanisms are textual prompts, visual cues, and a prompt-free paradigm.
  • 5. Conclusion: RepRTA, SAVPE, and LRPC enable YOLOE to process textual prompts, visual cues, and prompt-free inputs with favorable performance and low cost.The paper positions YOLOE as a baseline for further real-time open prompt-driven vision research.

A. More Implementation Details

The implementation uses Objects365 and GoldG data, pseudo masks generated with SAM-2.1, and standard optimization and augmentation procedures. Additional details cover training data, anchor filtering, and COCO-transfer optimization.

  • Data: YOLOE training uses Objects365 and GoldG, with SAM-2.1-Hiera-Large generating pseudo segmentation masks from ground-truth boxes.Masks are filtered, smoothed with Gaussian kernels, and refined.
  • Training: The implementation applies AdamW with initial learning rate 0.002, batch size 128, and weight decay 0.025.Augmentation includes color jittering, random affine transformations, horizontal flipping, and mosaic augmentation.
  • LRPC filtering: Figure 5 counts retained anchor points across LRPC filtering thresholds, with the dashed line marking the total number.The figure concerns anchor retention under different filtering thresholds.

B. More Analyses for LRPC

LRPC reduces the number of anchor points considered for category retrieval as its filtering threshold increases, lowering computational overhead relative to processing all anchors.

  • LRPC analysis: Increasing the LRPC filtering threshold substantially decreases retained anchor points across models.The baseline processes 8400 anchor points, so filtering reduces the retrieval workload.

C. More Visualization Results

YOLOE is qualitatively evaluated across zero-shot, customized-text, visual-prompt, and prompt-free scenarios. The visualizations show flexible prompting and identification of diverse objects across images.

  • Zero-shot inference on LVIS: YOLOE detects and segments diverse objects on LVIS using 1203 category names as text prompts.The results demonstrate zero-shot inference across various images.
  • Prompt with customized texts: Customized text prompts let YOLOE interpret generic and specific inputs for precise detection and fine-grained segmentation.Users can define prompts at varying levels of granularity to tailor model behavior.
  • Prompt with visual inputs: Visual prompts in the form of boxes, points, or handcrafted shapes guide YOLOE to find same-category instances across objects and images.The visual prompt can also be transferred across images.
  • Prompt-free inference: Without prompts, YOLOE effectively identifies diverse objects, supporting use when predefined inputs are unavailable or impractical.The prompt-free visualization omits segmentation masks for clearer display.
Loading 2503.07465v2…