Source-linked AI summary
Simple Open-Vocabulary Object Detection with Vision Transformers
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, Neil Houlsby
TL;DR
Object detection lacks well-established pretraining and scaling approaches, especially when localized data are scarce in long-tailed and open-vocabulary settings. The paper transfers contrastively pretrained image-text models using a minimally modified Vision Transformer and end-to-end detection fine-tuning. Scaling model size and pretraining improves detection, alongside strong zero-shot text-conditioned and one-shot image-conditioned results.
Problem
Pretraining and scaling are less established for long-tailed and open-vocabulary object detection, where localized training data are relatively scarce.
Method
The paper uses a standard Vision Transformer with minimal detection modifications, contrastive image-text pretraining, and end-to-end detection fine-tuning.
Results
The approach achieves 34.6% AP overall and 31.2% APrare on unseen LVIS classes, and improves one-shot unseen-COCO detection from 26.0 to 41.8 AP50.
Takeaways & Limitations
Simple, scalable image-text pretraining can transfer strongly to detection, including text-conditioned zero-shot and image-conditioned one-shot settings.
Abstract
from arXiv · showhide
Combining simple architectures with large-scale pre-training has led to massive improvements in image classification. For object detection, pre-training and scaling approaches are less well established, especially in the long-tailed and open-vocabulary setting, where training data is relatively scarce. In this paper, we propose a strong recipe for transferring image-text models to open-vocabulary object detection. We use a standard Vision Transformer architecture with minimal modifications, contrastive image-text pre-training, and end-to-end detection fine-tuning. Our analysis of the scaling properties of this setup shows that increasing image-level pre-training and model size yield consistent improvements on the downstream detection task. We provide the adaptation strategies and regularizations needed to attain very strong performance on zero-shot text-conditioned and one-shot image-conditioned object detection. Code and models are available on GitHub.
1 Introduction
The paper presents OWL-ViT, a simple recipe that transfers contrastively pretrained image-text models to open-vocabulary detection. Scaling pretraining and model size improves detection, while the same architecture supports strong text- and image-conditioned detection.
- Method: OWL-ViT transfers contrastively pretrained image-text encoders to open-vocabulary detection with lightweight detection heads and end-to-end fine-tuning.The architecture removes token pooling, attaches classification and localization heads to image tokens, and uses text embeddings for classification.
- Scaling: Increasing model size and pre-training duration continues improving detection performance beyond 20 billion image-text pairs.The paper emphasizes that image-text pairs are abundant compared with detection data, allowing further scaling.
- Image-conditioned detection: 41.8 AP50 improves over 26.0 for one-shot detection on unseen COCO categories, a reported improvement of 72%.The model uses image-derived query embeddings without architectural modification.
- Text-conditioned detection: 34.6% AP overall and 31.2% APrare are achieved on unseen classes on LVIS for text-conditioned detection.These results characterize open-vocabulary detection performance on a long-tailed benchmark.
- Contributions: The authors position OWL-ViT as a strong, simple baseline and flexible starting point for open-vocabulary localization research.The paper also lists a detailed scaling and ablation study as a contribution.
2 Related Work
Related work frames open-vocabulary detection as extending closed-vocabulary detectors with language representations, while image-conditioned detection uses a query image to match objects. OWL-ViT emphasizes simple, scalable models and extensive image-text pretraining rather than specialized cross-attention.
- Open-vocabulary detection: Traditional object detectors use fixed vocabularies, whereas open-vocabulary detectors replace fixed class representations with language embeddings.The central challenge is transferring image-level representations to detection with scarce localized annotations for rare classes.
- Terminology: Open-vocabulary detection permits category overlap between training and testing; zero-shot detection specifically targets categories without localized training instances.The paper distinguishes the broader open-vocabulary setting from its zero-shot subset.
- Image-conditioned detection: Image-conditioned detection matches objects using a single query image, enabling detection when object names are unknown or difficult to describe textually.This includes unique objects and specialized technical parts.
- Approach relative to prior work: Prior image-conditioned methods mainly emphasize architectural innovations such as sophisticated cross-attention, whereas this approach relies on a large model and extensive image-text pretraining.The comparison concerns the design emphasis rather than a claim that one ingredient alone determines performance.
3 Method
The method uses a two-stage recipe: contrastively pretrain image and text encoders on large-scale image-text data, then add detection heads and fine-tune on medium-sized detection data. Separate text or image queries support flexible open-vocabulary and few-shot detection.
- Training recipe: The recipe contrastively pre-trains image and text encoders, then adds detection heads and fine-tunes on detection data.The model can subsequently be queried in different ways for open-vocabulary or few-shot detection.
- Image encoder adaptation: OWL-ViT removes token pooling and projects each image-encoder output token into a per-object embedding for classification.The maximum number of predicted objects equals the image encoder’s token sequence length.
- Query-based classification: Text embeddings from category names or descriptions replace learned class embeddings, defining a separate discriminative label space for each image.Queries can vary across images, and using all category names recovers classical closed-vocabulary detection as a special case.
- Flexible querying: The absence of image-text fusion allows image-derived embeddings to replace text queries without modifying the model.This supports image-conditioned one-shot detection for objects that are difficult to describe in text.
- End-to-end fine-tuning: Both encoders are fine-tuned end-to-end after contrastive pretraining, while detection-specific heads contain at most 1.1% of model parameters.This leaves nearly all parameters eligible to benefit from image-level pretraining.
- Detection fine-tuning: Detection training uses per-image query labels and adapts bipartite matching with focal sigmoid cross-entropy for federated, non-disjoint label spaces.The paper notes that large detection datasets remain small relative to billions of image-level weak labels, motivating tuned regularization and augmentation.
4 Experiments
Experiments evaluate the model on open-vocabulary, zero-shot, and image-conditioned detection, while testing how pre-training, architecture, and detection data affect transfer. Scaling model size and pre-training generally improves detection, but transfer depends on architecture and sufficient detection data.
- Experimental setup: The study pre-trains image and text models on 3.6 billion image-text pairs, then fine-tunes them on object-level annotations.Detection uses publicly available datasets totaling around 2 million images, including OpenImages, Objects365, and Visual Genome.
- Experimental setup: LVIS v1.0 val is the main benchmark, with APLVIS rare measuring performance on categories whose localized annotations were removed from training.The evaluation queries all category names for each image and uses prompt ensembling.
- Open-vocabulary detection: 31.2% APLVIS rare is achieved by the best model, which uses a publicly available CLIP backbone.The method is competitive across architecture sizes in both open-vocabulary and zero-shot scenarios.
- Image-conditioned detection: 72% is the margin by which the model outperforms the best task-specific prior work across four COCO image-conditioned detection splits.The model also supports efficient simultaneous querying with thousands of image embeddings because query and target features are not entangled during inference.
- Image-conditioned detection: Averaging image embeddings from multiple query examples produces further significant improvements for few-shot detection.The approach extends directly from one-shot to multiple conditioning examples without changing the model.
- Scaling: High image-level performance is necessary but not sufficient for strong detection transfer, while longer pre-training and larger models extend detection gains.Detection performance can peak as pre-training increases, whereas increasing model size and improving fine-tuning can extend the positive trend.
- Scaling: Hybrid models are more efficient at small sizes, but pure ViTs scale better at large sizes and perform better on zero-shot detection than hybrids at comparable overall performance.Pure ResNets perform poorly in this setup, and the architecture difference diminishes as model size and performance increase.
- Transfer recipe: Detection training data is a limiting factor for performance, motivating careful use of available annotations and methodological regularization.The authors identify optimization stabilization, detection-data handling, and data augmentation as important components of the transfer recipe.
5 Conclusion
The paper presents a simple recipe for transferring contrastively trained image-text models to detection, achieving competitive zero-shot results and large-margin gains in image-conditioned detection.
- The method transfers contrastively trained image-text models to object detection using a simple recipe.
- The method achieves competitive zero-shot detection on LVIS despite using a simpler approach than competing methods.
- The method outperforms existing approaches on image-conditioned detection by a large margin.
- Pre-training on billions of image-text examples transfers strong generalization to detection with relatively limited object-level data.
- Scaling simple architectures and increasing pre-training data lead to strong zero-shot detection performance.
Appendix
The appendix provides additional qualitative examples, results, and methodological details, while directing remaining questions to the project code repository.
- The appendix provides additional examples, results, and methodological details.
- The project code is available in the Scenic OWL-ViT repository for remaining questions.
A1.1 Qualitative Examples
The qualitative examples illustrate text-conditioned and image-conditioned detection queries, including object categories and query patches with similarity scores.
- Text conditioning uses prompts of the form "an image of a {}" with {} replaced by object category names.
- The text-conditioned examples span household objects, food, vehicle parts, and other categories.
- Image conditioning displays query patches in the center column and detections with similarity scores in the outer columns.
A1.2 Detection Datasets
The work uses several object-detection datasets for fine-tuning and evaluation, combining large and densely annotated sources while filtering overlapping validation images from training data.
- Five object-detection datasets are used for fine-tuning and evaluation.
- COCO contains about 900k bounding-box annotations across 80 categories and uses the 2017 train, validation, and test splits.
- Visual Genome provides free-text annotations averaging 35 objects per COCO image, but only object annotations are used.
- Objects 365 provides over 10M bounding boxes across 365 categories, while LVIS contains over a thousand long-tailed categories.
- OpenImages V4 is described as the largest public object-detection dataset, with about 14.6 bounding-box annotations per image.
- Training combines OpenImages and Visual Genome, with additional Objects 365 training in some experiments, while COCO and LVIS are not used for training.
De-duplication
Table A2 reports train-dataset de-duplication statistics, distinguishing image examples from bounding-box instances.
- Table A2 summarizes train-dataset de-duplication statistics.
- The table uses ‘Examples’ to refer to images.
- The table uses ‘instances’ to refer to bounding boxes.
A1.3 Hyper-parameters
The appendix lists implementation settings and examines how CLIP’s class token is incorporated during detection fine-tuning.
- Table A3 provides an exhaustive overview of hyper-parameter settings for the main experiments.
- Training uses cosine learning-rate decay, focal loss, equal bounding-box, gIoU, and classification-loss weights, Adam, and global-norm gradient clipping.
- The text encoder input length is limited to 16 tokens for both LIT and CLIP-based models.
- Multiplying the CLIP class token with feature-map tokens followed by layer normalization performed best for most architectures.
CLIP-based models.
Table A3 lists the hyperparameters used across the paper’s models, including representation aggregation and encoder-specific regularization settings.
- Table A3 lists hyperparameters for all models shown in the paper.
- MAP and GAP denote multihead attention pooling and global average pooling for image-level representation aggregation.
- When two droplayer rates are reported, the first applies to the image encoder and the second to the text encoder.
A1.4 Pre-Training Image Resolution
The appendix evaluates pre-training resolution and describes augmentation, query construction, evaluation, and extended ablations for open-vocabulary detection.
- A1.4 Pre-Training Image Resolution: Larger pre-training images consistently improve zero-shot classification but make no significant difference to detection performance.
- A1.4 Pre-Training Image Resolution: Figure A3 shows image-size effects on zero-shot classification and detection for ViT-B/32, with trends extending to other architectures.
- Training uses random crops, square padding, and single-image or mosaic inputs to expose the detector to varied object scales while limiting padding.
- One-shot detection derives a query embedding from the class-head feature in the query patch’s source image and falls back to a text embedding in about 10% of cases.
- Evaluation reports AP50 on target images containing held-out categories, using paired query patches from the validation set.
- Table A4 reports open-vocabulary performance on COCO and O365, but the results are not strictly zero-shot because target categories were not removed from training data.
- A 7:3 OI:VG training ratio performs best, while VG region annotations hurt generalization and are omitted.
- Per-example loss normalization, lower text-encoder learning rates, location bias, and merging highly overlapping boxes are reported as beneficial design choices.