Simple Open-Vocabulary Object Detection with Vision Transformers

Method

• OWL-ViT transfers contrastively pretrained image-text encoders to open-vocabulary detection with lightweight detection heads and end-to-end fine-tuning.The architecture removes token pooling, attaches classification and localization heads to image tokens, and uses text embeddings for classification.Source: §1 · p. 2, Figure 1 · p. 3

Scaling

• Increasing model size and pre-training duration continues improving detection performance beyond 20 billion image-text pairs.The paper emphasizes that image-text pairs are abundant compared with detection data, allowing further scaling.Source: §1 · p. 2

Image-conditioned detection

• 41.8 AP50 improves over 26.0 for one-shot detection on unseen COCO categories, a reported improvement of 72%.The model uses image-derived query embeddings without architectural modification.Source: §1 · p. 2

Text-conditioned detection

• 34.6% AP overall and 31.2% APrare are achieved on unseen classes on LVIS for text-conditioned detection.These results characterize open-vocabulary detection performance on a long-tailed benchmark.Source: §1 · p. 2

Contributions

• The authors position OWL-ViT as a strong, simple baseline and flexible starting point for open-vocabulary localization research.The paper also lists a detailed scaling and ablation study as a contribution.Source: §1 · p. 2

Open-vocabulary detection

• Traditional object detectors use fixed vocabularies, whereas open-vocabulary detectors replace fixed class representations with language embeddings.The central challenge is transferring image-level representations to detection with scarce localized annotations for rare classes.Source: Contrastive Vision-Language Pre-Training · p. 3, Long-Tailed and Open-Vocabulary Object… · pp. 3–4

Terminology

• Open-vocabulary detection permits category overlap between training and testing; zero-shot detection specifically targets categories without localized training instances.The paper distinguishes the broader open-vocabulary setting from its zero-shot subset.Source: Long-Tailed and Open-Vocabulary Object… · p. 4

Image-conditioned detection

• Image-conditioned detection matches objects using a single query image, enabling detection when object names are unknown or difficult to describe textually.This includes unique objects and specialized technical parts.Source: Image-Conditioned Detection · p. 4

Approach relative to prior work

• Prior image-conditioned methods mainly emphasize architectural innovations such as sophisticated cross-attention, whereas this approach relies on a large model and extensive image-text pretraining.The comparison concerns the design emphasis rather than a claim that one ingredient alone determines performance.Source: Image-Conditioned Detection · p. 4

Training recipe

• The recipe contrastively pre-trains image and text encoders, then adds detection heads and fine-tunes on detection data.The model can subsequently be queried in different ways for open-vocabulary or few-shot detection.Source: §3 · p. 5

Image encoder adaptation

• OWL-ViT removes token pooling and projects each image-encoder output token into a per-object embedding for classification.The maximum number of predicted objects equals the image encoder’s token sequence length.Source: Architecture · p. 5

Query-based classification

• Text embeddings from category names or descriptions replace learned class embeddings, defining a separate discriminative label space for each image.Queries can vary across images, and using all category names recovers classical closed-vocabulary detection as a special case.Source: Open-vocabulary object detection · p. 5

Flexible querying

• The absence of image-text fusion allows image-derived embeddings to replace text queries without modifying the model.This supports image-conditioned one-shot detection for objects that are difficult to describe in text.Source: Open-vocabulary object detection · p. 5, One- or Few-Shot Transfer · p. 6

End-to-end fine-tuning

• Both encoders are fine-tuned end-to-end after contrastive pretraining, while detection-specific heads contain at most 1.1% of model parameters.This leaves nearly all parameters eligible to benefit from image-level pretraining.Source: Image-Level Contrastive Pre-Training · p. 6

Detection fine-tuning

• Detection training uses per-image query labels and adapts bipartite matching with focal sigmoid cross-entropy for federated, non-disjoint label spaces.The paper notes that large detection datasets remain small relative to billions of image-level weak labels, motivating tuned regularization and augmentation.Source: Training the Detector · pp. 6–7

Experimental setup

• The study pre-trains image and text models on 3.6 billion image-text pairs, then fine-tunes them on object-level annotations.Detection uses publicly available datasets totaling around 2 million images, including OpenImages, Objects365, and Visual Genome.Source: §4.1 · p. 7, §4.2 · p. 7
• LVIS v1.0 val is the main benchmark, with APLVIS rare measuring performance on categories whose localized annotations were removed from training.The evaluation queries all category names for each image and uses prompt ensembling.Source: Table 1 · p. 8, §4.3 · pp. 8–9

Open-vocabulary detection

• 31.2% APLVIS rare is achieved by the best model, which uses a publicly available CLIP backbone.The method is competitive across architecture sizes in both open-vocabulary and zero-shot scenarios.Source: §4.3 · p. 9

Image-conditioned detection

• 72% is the margin by which the model outperforms the best task-specific prior work across four COCO image-conditioned detection splits.The model also supports efficient simultaneous querying with thousands of image embeddings because query and target features are not entangled during inference.Source: §4.4 · p. 10
• Averaging image embeddings from multiple query examples produces further significant improvements for few-shot detection.The approach extends directly from one-shot to multiple conditioning examples without changing the model.Source: §4.4 · p. 10

Scaling

• High image-level performance is necessary but not sufficient for strong detection transfer, while longer pre-training and larger models extend detection gains.Detection performance can peak as pre-training increases, whereas increasing model size and improving fine-tuning can extend the positive trend.Source: §4.5 · p. 11
• Hybrid models are more efficient at small sizes, but pure ViTs scale better at large sizes and perform better on zero-shot detection than hybrids at comparable overall performance.Pure ResNets perform poorly in this setup, and the architecture difference diminishes as model size and performance increase.Source: §4.5 · p. 12

Transfer recipe

• Detection training data is a limiting factor for performance, motivating careful use of available annotations and methodological regularization.The authors identify optimization stabilization, detection-data handling, and data augmentation as important components of the transfer recipe.Source: Careful Use of Available Detection Data · p. 13

• The method transfers contrastively trained image-text models to object detection using a simple recipe.Source: §5 · p. 14
• The method achieves competitive zero-shot detection on LVIS despite using a simpler approach than competing methods.Source: §5 · p. 14
• The method outperforms existing approaches on image-conditioned detection by a large margin.Source: §5 · p. 14
• Pre-training on billions of image-text examples transfers strong generalization to detection with relatively limited object-level data.Source: §5 · p. 14
• Scaling simple architectures and increasing pre-training data lead to strong zero-shot detection performance.Source: §5 · p. 14

• The appendix provides additional examples, results, and methodological details.Source: Appendix · p. 18
• The project code is available in the Scenic OWL-ViT repository for remaining questions.Source: Appendix · p. 18

• Text conditioning uses prompts of the form "an image of a {}" with {} replaced by object category names.Source: Figure A1 · p. 18
• The text-conditioned examples span household objects, food, vehicle parts, and other categories.Source: Figure A1 · p. 18
• Image conditioning displays query patches in the center column and detections with similarity scores in the outer columns.Source: Figure A2 · p. 18

• Five object-detection datasets are used for fine-tuning and evaluation.Source: A1.2 Detection Datasets · p. 19
• COCO contains about 900k bounding-box annotations across 80 categories and uses the 2017 train, validation, and test splits.Source: A1.2 Detection Datasets · p. 19
• Visual Genome provides free-text annotations averaging 35 objects per COCO image, but only object annotations are used.Source: A1.2 Detection Datasets · p. 19
• Objects 365 provides over 10M bounding boxes across 365 categories, while LVIS contains over a thousand long-tailed categories.Source: A1.2 Detection Datasets · p. 19
• OpenImages V4 is described as the largest public object-detection dataset, with about 14.6 bounding-box annotations per image.Source: A1.2 Detection Datasets · p. 19
• Training combines OpenImages and Visual Genome, with additional Objects 365 training in some experiments, while COCO and LVIS are not used for training.Source: De-duplication · p. 20

• Table A2 summarizes train-dataset de-duplication statistics.Source: Table A2 · p. 20
• The table uses ‘Examples’ to refer to images.Source: Table A2 · p. 20
• The table uses ‘instances’ to refer to bounding boxes.Source: Table A2 · p. 20

• Table A3 provides an exhaustive overview of hyper-parameter settings for the main experiments.Source: A1.3 Hyper-parameters · p. 20
• Training uses cosine learning-rate decay, focal loss, equal bounding-box, gIoU, and classification-loss weights, Adam, and global-norm gradient clipping.Source: A1.3 Hyper-parameters · p. 20
• The text encoder input length is limited to 16 tokens for both LIT and CLIP-based models.Source: A1.3 Hyper-parameters · p. 20
• Multiplying the CLIP class token with feature-map tokens followed by layer normalization performed best for most architectures.Source: CLIP-based models · p. 20

• Table A3 lists hyperparameters for all models shown in the paper.Source: Table A3 · p. 21
• MAP and GAP denote multihead attention pooling and global average pooling for image-level representation aggregation.Source: Table A3 · p. 21
• When two droplayer rates are reported, the first applies to the image encoder and the second to the text encoder.Source: Table A3 · p. 21

A1.4 Pre-Training Image Resolution

• Larger pre-training images consistently improve zero-shot classification but make no significant difference to detection performance.Source: A1.4 Pre-Training Image Resolution · p. 21
• Figure A3 shows image-size effects on zero-shot classification and detection for ViT-B/32, with trends extending to other architectures.Source: A1.4 Pre-Training Image Resolution · p. 21, Figure A3 · p. 22

• Training uses random crops, square padding, and single-image or mosaic inputs to expose the detector to varied object scales while limiting padding.Source: A1.6 Image Scale Augmentation · p. 22, Figure A4 · p. 23
• One-shot detection derives a query embedding from the class-head feature in the query patch’s source image and falls back to a text embedding in about 10% of cases.Source: A1.7 One-shot (Image-Conditioned)… · p. 23, Extracting Image Embeddings to Use as… · p. 23
• Evaluation reports AP50 on target images containing held-out categories, using paired query patches from the validation set.Source: Image-Conditioned Evaluation Protocol · p. 23
• Table A4 reports open-vocabulary performance on COCO and O365, but the results are not strictly zero-shot because target categories were not removed from training data.Source: Table A4 · p. 24
• A 7:3 OI:VG training ratio performs best, while VG region annotations hurt generalization and are omitted.Source: Dataset ratios · p. 25
• Per-example loss normalization, lower text-encoder learning rates, location bias, and merging highly overlapping boxes are reported as beneficial design choices.Source: Loss normalization and gradient clipping · p. 25, Instance merging · p. 26, Learning rates · p. 26