Scaling Open-Vocabulary Object Detection

• Detection models typically have brief detection-training stages because detection data is scarce and pretrained representations are fragile.Source: §1 · p. 1
• Self-training uses an existing detector to predict bounding boxes on unlabeled Web images, creating open-vocabulary detection data from image-text pairs.Source: §1 · p. 1
• The approach improves as training scales and supports strong in- and out-of-distribution performance with weight ensembling.Source: Figure 1 · p. 2, §1 · p. 2
• The recipe uses caption N-grams as detection prompts and applies only weak confidence filtering instead of retaining a single pseudo-box per image.Source: §1 · p. 2
• OWL-ST combines self-training with OWLv2, an architecture optimized for training efficiency, and surpasses prior methods at moderate training scales.Source: §1 · p. 2
• 44.6% zero-shot LVIS mAPrare is achieved by a ViT-L/14 model trained on 2.3B image-text pairs and fine-tuned on LVISbase.Source: §1 · p. 2

• Vision-language models use abundant Web image-text pairs for scalable weak supervision and enable open-vocabulary applications.Source: §2.1 · p. 2
• Earlier VLM-based detectors transferred recognition to localization through region distillation or detection heads on frozen or fine-tuned encoders.Source: §2.2 · p. 3
• A central challenge is preventing VLMs from forgetting open-vocabulary knowledge when detection heads are trained on limited detection data.Source: §2.2 · p. 3
• Weakly supervised detection methods either self-train on pseudo-boxes or directly optimize grounding losses from image-level labels or captions.Source: §2.3 · p. 3
• Prior methods rarely went much beyond 10M examples, whereas this work keeps architecture and loss simple while scaling training data.Source: §2.3 · p. 3
• The paper connects large-scale self-training with a broader strategy for improving fine-grained vision tasks through abundant weak supervision.Source: §2.3 · p. 3

Web-scale annotation and training

• The pipeline predicts pseudo-boxes on WebLI images, self-trains a detector on them, and optionally fine-tunes it on human-annotated detection data.WebLI provides approximately 10B images and associated alt-text strings; OWL-ViT CLIP-L/14 generates the pseudo-annotations.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4

Label space

• Label-space design compares a human-curated vocabulary with machine-generated queries from each image’s associated text.The curated vocabulary combines LVIS, Objects365, OpenImagesV4, and Visual Genome; the machine-generated space extracts word N-grams up to length 10 without grammatical parsing.Source: §3.1 · pp. 3–4

Pseudo-annotation filtering

• OWL-ST uses all possible image-text N-grams as detection prompts and applies only weak confidence filtering to pseudo-labels.For each predicted box, the highest-scoring query is retained, and all boxes above a score threshold are kept as hard labels.Source: §3.1 · p. 4

Training efficiency

• OWLv2 improves training efficiency with token dropping, objectness-based instance selection, mosaics, and large-scale Transformer training practices.Instance selection computes boxes, class scores, and losses only for the top k tokens by objectness during training; approximately 10% of instances are selected.Source: §3.2 · pp. 4–5
• 50% lower training FLOPS and 2× higher throughput distinguish OWLv2 from the original OWL-ViT while leaving inference identical.For L/14 at 840 × 840 resolution on TPUv3, the reported values are 11’945.4 versus 5357.9 GFLOPs/example and 1.0 versus 2.2 examples/s/core.Source: §3.2 · p. 5

Fine-tuning

• Fine-tuning starts from the self-trained checkpoint and continues on target detection data while linearly cooling the learning rate to zero.The procedure retains the self-training learning rate and optimizer state before briefly training on the target dataset.Source: §3.3 · p. 5

Main results

• 29.6% LVIS mAPrare is achieved by OWL-ST B/16 without fine-tuning, 9 points above the equivalent OWL-ViT model.The G/14 model reaches 37.5% mAPrare, 4.7 points above the next-best literature model.Source: §4.2 · p. 6
• 47.2% LVIS mAPrare is reached after fine-tuning the best machine-query-trained model, improving 14.4 points over the next-best literature model.The improvement occurs even though self-training used only machine-generated queries.Source: §4.2 · p. 7

Pseudo-annotation label space

• Machine-generated queries improve performance on unseen classes and in-the-wild datasets, while curated label spaces help most when the target label space is known.A mixture performs well across settings, but does not significantly outperform purely machine-generated labels on in-the-wild datasets.Source: §4.3 · p. 7

Scaling

• Performance increases monotonically with training compute, and larger models are needed to benefit from more compute and data.The self-training dataset contains approximately 2B images, enabling scaling studies in a regime comparable to image-level training.Source: Figure 4 · p. 8, §4.5 · p. 8

Pseudo-annotation filtering

• Including all pseudo-annotations above a moderate 0.3 confidence threshold works well, whereas strict thresholds produce poor results.For the main results, annotations above 0.1 were retained, while images needed at least one annotation above 0.3.Source: Figure 3 · p. 7, §4.4 · p. 8

Fine-tuning and open-world performance

• Fine-tuning improves target-dataset performance but reduces ODinW13 out-of-distribution performance in proportion to fine-tuning duration.Weight-space ensembling improves this trade-off and the Pareto frontier without additional training cost.Source: §4.6 · pp. 8–9

Compute and data

• Self-training requires substantial compute and data, and further improvements come at increasingly large costs.The authors state that cost may increase faster than resources can realistically grow, eventually requiring new approaches.Source: §5 · p. 9

Performance trade-off

• Fine-tuned and open-vocabulary performance involve a trade-off similar to that in other open-vocabulary models.This trade-off is addressed in Section 4.6.Source: §5 · p. 9
• For out-of-distribution queries, fine-tuned models may produce poorly calibrated predictions that depend on the query wording.Weight ensembling can mitigate these issues, but the robustness of open-vocabulary models remains incompletely understood.Source: §5 · p. 9

6 Conclusion

• Self-training can be scaled up to overcome open-vocabulary detection's dependency on human annotations.Source: §6 · p. 9
• OWL-ST delivers large detection improvements using weak supervision from abundant Web data.Source: §6 · p. 9
• The resulting use of weak supervision is similar to what has been seen for image classification and language modelling.Source: §6 · p. 9

A Appendix

• The Appendix provides additional methodological details, model hyperparameters, and results.Source: A Appendix · p. 12
• The Appendix includes qualitative examples of the self-training data and model predictions.Source: A Appendix · p. 12
• The Appendix is structured as follows.Source: A Appendix · p. 12

A.1 Human-Curated Label Space

• The label space includes dataset class names available from sources such as TensorFlow Datasets.Source: §A.1 · p. 12
• For Visual Genome, the label space uses the 1600 most common label strings.Source: §A.1 · p. 12
• The label space adds OBJECTS365 and OPEN_IMAGES_V4_BOXABLE class names.Source: §A.1 · p. 12
• The processing removes duplicate queries and plural forms through explicit filtering steps.Source: §A.1 · p. 12

Label-space construction

• Image-associated text is processed separately for each image to obtain a machine-generated label space for pseudo-annotation.The resulting N-gram label space is used to generate pseudo-annotations.Source: §A.2 · p. 12

Text preprocessing

• Captions are lower-cased before query extraction.Source: §A.2 · p. 13
• Common generic words are removed from captions before generating candidate queries.The implementation filters words against a manually curated list of generic terms.Source: §A.2 · p. 13

Query filtering

• N-grams consisting entirely of stop words are excluded from the query list.The check tests whether all words in an n-gram belong to the English stopword set.Source: §A.2 · p. 13

Confidence balancing

• Human-curated queries tend to receive higher annotator scores than machine-generated queries because they are closer to the annotator’s training distribution.Source: §A.3 · p. 14
• Applying one confidence threshold to the combined label space would preferentially retain annotations from human-curated queries.The combined “N-grm+curated” label space therefore requires more even score balancing.Source: §A.3 · p. 14

Self-training augmentations

• Web images are augmented with mosaics of up to 6 × 6 tiles because they are typically smaller and contain fewer objects than LVIS images.Raw images are randomly resized to 0.5×–1.0× the width of a full mosaic tile and padded on the bottom.Source: §A.4 · p. 14
• Self-training omits left/right flipping, random cropping, and random prompt templates for pseudo-labels.Fine-tuning instead uses the augmentations from the prior OWL-ViT work.Source: §A.4 · p. 14

Token-dropping choice

• 50% token dropping is used in all experiments after remaining within one standard deviation of full performance in the token-drop ablation.The dropped patches are selected using image-patch pixel variance.Source: §A.5 · p. 14

Token selection

• Small noise is added before computing patch variance to inject stochasticity into token selection.The noise is uniformly sampled between 0.0 and 0.01 for images scaled to [0.0, 1.0].Source: §A.5 · p. 14

Large-scale optimization

• Large-scale training also uses an Adafactor variant and an open-ended inverse square-root schedule with a fixed 10’000-step timescale.These choices are adopted to reduce memory use and avoid fixing the total training duration in advance.Source: §A.6 · p. 14