PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Open-Vocabulary Object Detection

Matthias Minderer, Alexey Gritsenko, Neil Houlsby

arXiv:2306.09683v3cs.CV

TL;DR

Open-vocabulary detection remains limited by scarce detection training data, despite abundant Web image-text pairs. The paper introduces OWL-ST self-training and the efficient OWLv2 architecture, achieving stronger detection performance at comparable scales and further gains beyond 1B examples. Its results support Web-scale training for open-world localization, while compute and data costs and fine-tuning trade-offs remain important limitations.

  • Problem

    Open-vocabulary detection lacks abundant localization training data, limiting its ability to exploit Web-scale image-text supervision.

  • Method

    OWL-ST generates pseudo-boxes with an existing open-vocabulary detector, trains on them, and optionally fine-tunes on human annotations, while OWLv2 improves training efficiency.

  • Results

    OWL-ST with OWLv2 surpasses prior detectors at comparable scales and, with 2.3B image-text pairs, reaches 44.6% zero-shot LVIS mAPrare for ViT-L/14.

  • Takeaways & Limitations

    Self-training can overcome dependence on human box annotations and enable Web-scale training for open-vocabulary detection.

  • Takeaways & Limitations

    Self-training requires increasing amounts of compute and data, with costs likely rising faster than resources can realistically grow.

Abstract

from arXiv · show

Open-vocabulary object detection has benefited greatly from pretrained vision-language models, but is still limited by the amount of available detection training data. While detection training data can be expanded by using Web image-text pairs as weak supervision, this has not been done at scales comparable to image-level pretraining. Here, we scale up detection data with self-training, which uses an existing detector to generate pseudo-box annotations on image-text pairs. Major challenges in scaling self-training are the choice of label space, pseudo-annotation filtering, and training efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which address these challenges. OWLv2 surpasses the performance of previous state-of-the-art open-vocabulary detectors already at comparable training scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples, yielding further large improvement: With an L/14 architecture, OWL-ST improves AP on LVIS rare classes, for which the model has seen no human box annotations, from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale training for open-world localization, similar to what has been seen for image classification and language modelling.

1 Introduction

Open-vocabulary detection is constrained by scarce detection data despite scalable image-text supervision. The paper addresses this with scalable self-training, weak pseudo-label filtering, and an efficient architecture, achieving stronger results as training grows.

  • Detection models typically have brief detection-training stages because detection data is scarce and pretrained representations are fragile.
  • Self-training uses an existing detector to predict bounding boxes on unlabeled Web images, creating open-vocabulary detection data from image-text pairs.
  • The approach improves as training scales and supports strong in- and out-of-distribution performance with weight ensembling.
  • The recipe uses caption N-grams as detection prompts and applies only weak confidence filtering instead of retaining a single pseudo-box per image.
  • OWL-ST combines self-training with OWLv2, an architecture optimized for training efficiency, and surpasses prior methods at moderate training scales.
  • 44.6% zero-shot LVIS mAPrare is achieved by a ViT-L/14 model trained on 2.3B image-text pairs and fine-tuned on LVISbase.

2 Related Work

Vision-language pretraining made open-vocabulary recognition scalable, but localization remained limited by scarce detection supervision. Recent work introduced weakly supervised detection, while this paper emphasizes scaling self-training data beyond prior practice.

  • Vision-language models use abundant Web image-text pairs for scalable weak supervision and enable open-vocabulary applications.
  • Earlier VLM-based detectors transferred recognition to localization through region distillation or detection heads on frozen or fine-tuned encoders.
  • A central challenge is preventing VLMs from forgetting open-vocabulary knowledge when detection heads are trained on limited detection data.
  • Weakly supervised detection methods either self-train on pseudo-boxes or directly optimize grounding losses from image-level labels or captions.
  • Prior methods rarely went much beyond 10M examples, whereas this work keeps architecture and loss simple while scaling training data.
  • The paper connects large-scale self-training with a broader strategy for improving fine-grained vision tasks through abundant weak supervision.

3 Method

The method generates open-vocabulary pseudo-box annotations from WebLI image-text data, then self-trains and optionally fine-tunes detectors. OWL-ST broadens supervision through text-derived queries, while OWLv2 improves training efficiency for scaling.

  • Web-scale annotation and training: The pipeline predicts pseudo-boxes on WebLI images, self-trains a detector on them, and optionally fine-tunes it on human-annotated detection data.WebLI provides approximately 10B images and associated alt-text strings; OWL-ViT CLIP-L/14 generates the pseudo-annotations.
  • Label space: Label-space design compares a human-curated vocabulary with machine-generated queries from each image’s associated text.The curated vocabulary combines LVIS, Objects365, OpenImagesV4, and Visual Genome; the machine-generated space extracts word N-grams up to length 10 without grammatical parsing.
  • Pseudo-annotation filtering: OWL-ST uses all possible image-text N-grams as detection prompts and applies only weak confidence filtering to pseudo-labels.For each predicted box, the highest-scoring query is retained, and all boxes above a score threshold are kept as hard labels.
  • Training efficiency: OWLv2 improves training efficiency with token dropping, objectness-based instance selection, mosaics, and large-scale Transformer training practices.Instance selection computes boxes, class scores, and losses only for the top k tokens by objectness during training; approximately 10% of instances are selected.
  • Training efficiency: 50% lower training FLOPS and 2× higher throughput distinguish OWLv2 from the original OWL-ViT while leaving inference identical.For L/14 at 840 × 840 resolution on TPUv3, the reported values are 11’945.4 versus 5357.9 GFLOPs/example and 1.0 versus 2.2 examples/s/core.
  • Fine-tuning: Fine-tuning starts from the self-trained checkpoint and continues on target detection data while linearly cooling the learning rate to zero.The procedure retains the self-training learning rate and optimizer state before briefly training on the target dataset.

4 Experiments

Experiments show that machine-generated pseudo-annotations enable strong open-vocabulary detection, scale predictably with compute, and expose a trade-off between target-dataset performance and open-world generalization.

  • Main results: 29.6% LVIS mAPrare is achieved by OWL-ST B/16 without fine-tuning, 9 points above the equivalent OWL-ViT model.The G/14 model reaches 37.5% mAPrare, 4.7 points above the next-best literature model.
  • Main results: 47.2% LVIS mAPrare is reached after fine-tuning the best machine-query-trained model, improving 14.4 points over the next-best literature model.The improvement occurs even though self-training used only machine-generated queries.
  • Pseudo-annotation label space: Machine-generated queries improve performance on unseen classes and in-the-wild datasets, while curated label spaces help most when the target label space is known.A mixture performs well across settings, but does not significantly outperform purely machine-generated labels on in-the-wild datasets.
  • Scaling: Performance increases monotonically with training compute, and larger models are needed to benefit from more compute and data.The self-training dataset contains approximately 2B images, enabling scaling studies in a regime comparable to image-level training.
  • Pseudo-annotation filtering: Including all pseudo-annotations above a moderate 0.3 confidence threshold works well, whereas strict thresholds produce poor results.For the main results, annotations above 0.1 were retained, while images needed at least one annotation above 0.3.
  • Fine-tuning and open-world performance: Fine-tuning improves target-dataset performance but reduces ODinW13 out-of-distribution performance in proportion to fine-tuning duration.Weight-space ensembling improves this trade-off and the Pareto frontier without additional training cost.

5 Limitations

The method is limited by the compute and data required for self-training, with costs rising as performance improves. It also faces a trade-off between fine-tuned and open-vocabulary performance.

  • Compute and data: Self-training requires substantial compute and data, and further improvements come at increasingly large costs.The authors state that cost may increase faster than resources can realistically grow, eventually requiring new approaches.
  • Performance trade-off: Fine-tuned and open-vocabulary performance involve a trade-off similar to that in other open-vocabulary models.This trade-off is addressed in Section 4.6.
  • Performance trade-off: For out-of-distribution queries, fine-tuned models may produce poorly calibrated predictions that depend on the query wording.Weight ensembling can mitigate these issues, but the robustness of open-vocabulary models remains incompletely understood.

6 Conclusion

The paper argues that self-training can scale to reduce open-vocabulary detection's dependence on human annotations. OWL-ST uses abundant Web data to deliver large detection improvements, paralleling progress in image classification and language modelling.

  • 6 Conclusion: Self-training can be scaled up to overcome open-vocabulary detection's dependency on human annotations.
  • 6 Conclusion: OWL-ST delivers large detection improvements using weak supervision from abundant Web data.
  • 6 Conclusion: The resulting use of weak supervision is similar to what has been seen for image classification and language modelling.

A Appendix

The Appendix contains additional methodological details, model hyperparameters, results, and qualitative examples of self-training data and model predictions. It also describes its overall structure.

  • A Appendix: The Appendix provides additional methodological details, model hyperparameters, and results.
  • A Appendix: The Appendix includes qualitative examples of the self-training data and model predictions.
  • A Appendix: The Appendix is structured as follows.

A.1 Human-Curated Label Space

The human-curated label space is assembled from dataset class names and processed to remove duplicates and plural forms. The listed sources include common dataset label collections and Visual Genome labels.

  • A.1 Human-Curated Label Space: The label space includes dataset class names available from sources such as TensorFlow Datasets.
  • A.1 Human-Curated Label Space: For Visual Genome, the label space uses the 1600 most common label strings.
  • A.1 Human-Curated Label Space: The label space adds OBJECTS365 and OPEN_IMAGES_V4_BOXABLE class names.
  • A.1 Human-Curated Label Space: The processing removes duplicate queries and plural forms through explicit filtering steps.

A.2 Machine-Generated Label Space

The machine-generated label space derives query candidates from each image’s associated caption by preprocessing text and extracting n-grams.

  • Label-space construction: Image-associated text is processed separately for each image to obtain a machine-generated label space for pseudo-annotation.The resulting N-gram label space is used to generate pseudo-annotations.
  • Text preprocessing: Captions are lower-cased before query extraction.
  • Text preprocessing: Common generic words are removed from captions before generating candidate queries.The implementation filters words against a manually curated list of generic terms.
  • Query filtering: N-grams consisting entirely of stop words are excluded from the query list.The check tests whether all words in an n-gram belong to the English stopword set.

A.3 Combined Label Space

Combining human-curated and machine-generated queries requires separate confidence handling because the annotator scores the two label spaces differently.

  • Confidence balancing: Human-curated queries tend to receive higher annotator scores than machine-generated queries because they are closer to the annotator’s training distribution.
  • Confidence balancing: Applying one confidence threshold to the combined label space would preferentially retain annotations from human-curated queries.The combined “N-grm+curated” label space therefore requires more even score balancing.

A.4 Augmentations for Self-Training

Self-training uses stronger spatial augmentation for Web images, while omitting several augmentations that are unnecessary at the dataset’s scale.

  • Self-training augmentations: Web images are augmented with mosaics of up to 6 × 6 tiles because they are typically smaller and contain fewer objects than LVIS images.Raw images are randomly resized to 0.5×–1.0× the width of a full mosaic tile and padded on the bottom.
  • Self-training augmentations: Self-training omits left/right flipping, random cropping, and random prompt templates for pseudo-labels.Fine-tuning instead uses the augmentations from the prior OWL-ViT work.

A.5 Token Dropping

Token dropping improves training efficiency by removing low-variance image patches while preserving performance within the tested range.

  • Token-dropping choice: 50% token dropping is used in all experiments after remaining within one standard deviation of full performance in the token-drop ablation.The dropped patches are selected using image-patch pixel variance.
  • Token selection: Small noise is added before computing patch variance to inject stochasticity into token selection.The noise is uniformly sampled between 0.0 and 0.01 for images scaled to [0.0, 1.0].
  • Large-scale optimization: Large-scale training also uses an Adafactor variant and an open-ended inverse square-root schedule with a fixed 10’000-step timescale.These choices are adopted to reduce memory use and avoid fixing the total training duration in advance.
Loading 2306.09683v3…