PaperlayerView exact revision on arXiv

Source-linked AI summary

YOLO-World: Real-Time Open-Vocabulary Object Detection

Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, Ying Shan

arXiv:2401.17270v3cs.CV

TL;DR

YOLO detectors are efficient but limited to predefined categories, while existing open-vocabulary approaches face training-data, computation, and deployment challenges. YOLO-World adds vision-language modeling, RepVL-PAN, and region-text contrastive pre-training to enable efficient open-vocabulary detection, achieving strong zero-shot and downstream performance.

  • Problem

    Existing detectors handle fixed vocabularies, while prior open-vocabulary methods are limited by scarce diverse training data, heavy computation, and complicated edge deployment.

  • Method

    YOLO-World combines a YOLO architecture with CLIP text encoding, RepVL-PAN for vision-language interaction, region-text contrastive loss, and pseudo-labeled image-text pre-training.

  • Results

    YOLO-World demonstrates strong zero-shot and downstream open-vocabulary performance, achieving 35.4 AP on LVIS at 52.0 FPS on V100.

  • Takeaways & Limitations

    The pre-trained detector offers an efficient, deployable basis for real-world open-vocabulary detection and adaptation to tasks including instance segmentation and referring object detection.

  • Takeaways & Limitations

    Fine-tuning on O365 may degrade the generalization ability of pre-trained CLIP because O365 has only 365 categories and limited textual information.

Abstract

from arXiv · show

The You Only Look Once (YOLO) series of detectors have established themselves as efficient and practical tools. However, their reliance on predefined and trained object categories limits their applicability in open scenarios. Addressing this limitation, we introduce YOLO-World, an innovative approach that enhances YOLO with open-vocabulary detection capabilities through vision-language modeling and pre-training on large-scale datasets. Specifically, we propose a new Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate the interaction between visual and linguistic information. Our method excels in detecting a wide range of objects in a zero-shot manner with high efficiency. On the challenging LVIS dataset, YOLO-World achieves 35.4 AP with 52.0 FPS on V100, which outperforms many state-of-the-art methods in terms of both accuracy and speed. Furthermore, the fine-tuned YOLO-World achieves remarkable performance on several downstream tasks, including object detection and open-vocabulary instance segmentation.

1. Introduction

YOLO-World addresses the limited vocabulary and deployment costs of open-vocabulary detection by adapting YOLO with vision-language modeling, efficient inference, and large-scale pre-training.

  • Motivation: Fixed-vocabulary detectors cannot detect categories beyond those predefined and labeled in their training datasets.This limits their applicability to open scenes.
  • Motivation: Previous open-vocabulary detectors face heavy computation and complicated deployment, while small-detector pre-training remains underexplored.Distillation-based methods also suffer from limited training-data vocabulary diversity.
  • Approach: YOLO-World combines a standard YOLO architecture with a CLIP text encoder and RepVL-PAN to connect text and image features.The network supports cross-modal visual-semantic representation.
  • Approach: During inference, text embeddings can be re-parameterized into RepVL-PAN weights, allowing the text encoder to be removed for efficient deployment.The prompt-then-detect paradigm encodes user prompts into an offline vocabulary.
  • Pre-training: YOLO-World uses region-text contrastive pre-training with detection, grounding, and image-text data to support large-vocabulary recognition.The method unifies these sources into region-text pairs.

2. Related Works

Open-vocabulary detection extends beyond fixed categories but has been constrained by limited training vocabularies, heavy detector architectures, and costly online text encoding. YOLO-World uses a lightweight YOLO-based, prompt-then-detect design to target real-time deployment.

  • Detection paradigms: Traditional detectors operate within fixed category vocabularies defined by datasets such as COCO and Objects365.Open-vocabulary detection instead seeks to detect objects beyond predefined categories.
  • Open-vocabulary detection: Early open-vocabulary methods train on base classes and evaluate novel classes, but limited datasets and vocabularies restrict broader-domain generalization.This setting measures novel-object recognition while remaining limited for open scenarios.
  • Vision-language methods: Vision-language methods formulate detection as image-text matching and use large-scale image-text data to expand training vocabulary.Examples include OWL-ViT and GLIP-style detection and grounding approaches.
  • Open-vocabulary detection: Many recent methods use heavy detectors and Swin-L backbones, creating high computational demands and deployment challenges.These methods include approaches based on ATSS or DINO.
  • YOLO-World: YOLO-World targets efficient open-vocabulary detection with real-time inference and easier downstream deployment using lightweight YOLO detectors.Its prompt-then-detect paradigm uses an offline vocabulary rather than encoding prompts online for every prediction.
  • YOLO-World: YOLO-World is positioned as a lightweight alternative that combines strong open-vocabulary performance with practical efficiency.The paper contrasts this goal with prior heavy open-vocabulary detectors.

3. Method

YOLO-World reformulates detection around region-text pairs and combines a YOLO detector with CLIP text encoding, RepVL-PAN, contrastive training, and prompt-based vocabulary handling. Its pre-training uses detection, grounding, and image-text data, including pseudo-labeled region-text pairs.

  • 3.1. Pre-training Formulation: Region-Text Pairs: YOLO-World reformulates instance annotations as region-text pairs, where each region is paired with a category name, noun phrase, or object description.The model takes an image and text set as input and predicts bounding boxes with corresponding object embeddings.
  • 3.2. Model Architecture: The architecture combines a YOLO detector, a CLIP-pretrained Transformer text encoder, and RepVL-PAN to enhance image and text representations through cross-modal interaction.The YOLO detector uses a Darknet backbone, PAN feature pyramids, and a head for box regression and object embeddings.
  • 3.2. Model Architecture: The text contrastive head computes object-text similarity from L2-normalized object and text embeddings with learnable scaling and shifting factors.The L2 norms and affine transformations are described as important for stabilizing region-text training.
  • 3.2. Model Architecture: Training constructs online vocabularies from positive and negative nouns, while inference uses offline user-defined prompts to avoid repeated text encoding and adjust the vocabulary flexibly.The default online vocabulary contains at most 80 nouns.
  • 3.3. Re-parameterizable Vision-Language PAN: RepVL-PAN uses Text-guided CSPLayer to inject language into multi-scale image features and Image-Pooling Attention to enhance text embeddings with image-aware information.Image-Pooling Attention applies max pooling to multi-scale features to form 27 patch tokens, while offline vocabulary embeddings can be re-parameterized into layer weights during inference.
  • 3.4. Pre-training Schemes: The pre-training objective combines region-text contrastive loss with IoU and distributed focal losses, and pseudo-labeling produces 821k annotations across 246k CC3M images.Regression loss is applied only to samples with accurate bounding boxes because image-text datasets may contain noisy boxes.

4. Experiments

YOLO-World is evaluated through large-scale pre-training, zero-shot detection, ablations, fine-tuning, and open-vocabulary instance segmentation. Experiments show gains from richer pre-training data and RepVL-PAN, strong LVIS results, and a trade-off between segmentation fit and retained zero-shot capability.

  • Experimental Setup: YOLO-World uses small, medium, and large variants with a frozen pre-trained CLIP text encoder, and inference speed is measured on one NVIDIA V100 without FP16 or TensorRT.Pre-training uses AdamW for 100 epochs on 32 NVIDIA V100 GPUs with total batch size 512.
  • Zero-shot LVIS Detection: 35.4 AP with a 20× inference-speed increase over DetCLIP demonstrates comparable LVIS zero-shot performance at substantially higher speed.YOLO-World is compared on LVIS in a zero-shot setting, with fewer model parameters than some methods using more data.
  • Pre-training Data Ablation: An 8.4 AP gain on LVIS follows from adding GQA to Objects365 pre-training, while adding 8% of CC3M yields a further 0.5 AP gain and 1.3 AP on rare objects.Performance continues to improve as the amount of pre-training data increases.
  • Architecture Ablation: RepVL-PAN improves the YOLOv8-PAN baseline by 1.1 AP on LVIS, with larger improvements for rare categories and stronger gains when pre-training includes GQA.The ablation evaluates Text-guided CSPLayers and Image Pooling Attention under Objects365 and Objects365-plus-GQA settings.
  • Text Encoder Ablation: CLIP outperforms BERT by 10.1 AP on rare LVIS categories, whereas fine-tuning CLIP causes a severe performance drop during pre-training.Fine-tuning BERT improves performance by 3.7 AP, while the authors attribute CLIP’s drop to degraded generalization after fine-tuning on Objects365.
  • Downstream Fine-tuning: YOLO-World-L outperforms YOLOv8-L by 7.2 AP and 10.2 APr on fine-tuned LVIS detection, while fine-tuning all segmentation modules improves LVIS AP by 9.6 but drops box APr by 0.6.Fine-tuning only the segmentation head retains zero-shot capabilities, whereas fine-tuning all modules may degrade them.

5. Conclusion

YOLO-World reshapes YOLO into a vision-language architecture for efficient open-vocabulary detection. Its pre-training combines detection, grounding, and image-text data, supporting speed, open-vocabulary performance, and transfer to real-world tasks.

  • YOLO-World combines a YOLO detector with vision-language pre-training for open-vocabulary detection.The architecture connects vision and language information through RepVL-PAN.
  • RepVL-PAN connects vision and language information and can be re-parameterized for efficient deployment.
  • Pre-training with detection, grounding, and image-text data gives YOLO-World strong open-vocabulary detection capability.
  • Experiments demonstrate speed and open-vocabulary performance while indicating the effectiveness of vision-language pre-training on small models.

A.1. Re-parameterization for RepVL-PAN

RepVL-PAN is simplified for offline-vocabulary inference by converting text-guided operations into efficient convolutional or linear computations. The process pre-computes text embeddings and uses multi-scale image features to update them.

  • Offline-vocabulary inference pre-computes text embeddings through the text encoder before RepVL-PAN re-parameterization.
  • Text embeddings are reshaped into 1 × 1 convolution or linear-layer weights to simplify each T-CSPLayer.
  • The re-parameterized T-CSPLayer maps input and output image features with matrix multiplication involving reshaped text embeddings.
  • I-Pooling Attention aggregates multi-scale RepVL-PAN features using concatenation and 3 × 3 max pooling.
  • The flattened multi-scale features have shape B × D × 27 and are used to update the text embeddings.

A.2. Fine-tuning Details.

For COCO fine-tuning, YOLO-World removes its text-guided and image-pooling interaction modules because the dataset has 80 categories and lower visual-language dependence. The model starts from pre-trained weights and stores category embeddings in the classification layers.

  • YOLO-World removes T-CSPLayers and Image-Pooling Attention when transferring to 80-category COCO detection.
  • Fine-tuning initializes YOLO-World with pre-trained weights and uses a learning rate of 0.0002 with weight decay 0.05.
  • After fine-tuning, given COCO categories are encoded and stored in the classification-layer weights.

B. Automatic Labeling on Large-scale Image-Text Data

YOLO-World automatically labels large-scale image-text data by extracting object nouns, generating detector-based region proposals, and filtering them with CLIP-based scores. The resulting pseudo-annotations are used for pre-training without human verification.

  • The labeling pipeline extracts object nouns from captions with a simple n-gram algorithm.
  • A pre-trained open-vocabulary detector generates pseudo boxes and confidence scores for the extracted nouns.These form coarse region-text proposals.
  • CLIP computes image-text and region-text similarities for rescoring and filtering noisy proposals.
  • Optional re-labeling assigns each crop the noun with maximum CLIP similarity to correct detector labels.
  • Region-level NMS and confidence thresholds remove duplicate or low-confidence proposals before image-level filtering.The NMS threshold is 0.5 and the proposal confidence threshold is 0.3.
  • The automatic pipeline keeps images above an image-level score threshold of 0.3 and uses the labeled samples for YOLO-World pre-training.The procedure does not use human verification.

C. Pre-training YOLO-World at Scale

Pre-training data composition strongly affects small YOLO-World models: image-text data improves zero-shot performance, while excessive pseudo-labeled data can hurt and fine-grained annotations provide significant gains.

  • Adding image-text data improves the overall zero-shot performance of YOLO-World-S.
  • Fine-grained annotations provide significant improvements for small models, indicating the value of large-scale, high-quality annotated data.
  • Table 9 evaluates YOLO-World-S zero-shot performance on LVIS across different amounts of image-text data.
Loading 2401.17270v3…