PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Vocabulary Object Detection Using Captions

Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, Shih-Fu Chang

arXiv:2011.10678v2cs.CVcs.AIcs.LG

TL;DR

Object detection is difficult to scale because expanding categories requires extensive bounding-box supervision. The paper introduces open-vocabulary detection, combining caption-based vocabulary learning with limited box annotations, and reports stronger performance than zero-shot and weakly supervised baselines.

  • Problem

    Scaling object detection to many categories is costly because each category requires extensive bounding-box annotation, while existing zero-shot and weakly supervised approaches are limited.

  • Method

    OVR-CNN pretrains Faster R-CNN on image-caption pairs, then fine-tunes it on bounding-box annotations for base classes while preserving open-vocabulary knowledge.

  • Results

    27% mAP versus 10% for zero-shot learning and 40% mAP versus 26% for weakly supervised object detectors.

  • Takeaways & Limitations

    Open-vocabulary detection separates recognition learned from captions from localization learned from accurate bounding-box annotations.

  • Takeaways & Limitations

    Localization accuracy for target classes is limited, with overly loose or overly tight boxes because target-class ground-truth boxes are unavailable.

Abstract

from arXiv · show

Despite the remarkable accuracy of deep neural networks in object detection, they are costly to train and scale due to supervision requirements. Particularly, learning more object categories typically requires proportionally more bounding box annotations. Weakly supervised and zero-shot learning techniques have been explored to scale object detectors to more categories with less supervision, but they have not been as successful and widely adopted as supervised models. In this paper, we put forth a novel formulation of the object detection problem, namely open-vocabulary object detection, which is more general, more practical, and more effective than weakly supervised and zero-shot approaches. We propose a new method to train object detectors using bounding box annotations for a limited set of object categories, as well as image-caption pairs that cover a larger variety of objects at a significantly lower cost. We show that the proposed method can detect and localize objects for which no bounding box annotation is provided during training, at a significantly higher accuracy than zero-shot approaches. Meanwhile, objects with bounding box annotation can be detected almost as accurately as supervised methods, which is significantly better than weakly supervised baselines. Accordingly, we establish a new state of the art for scalable object detection.

1 Snap Inc., Seattle, WA 2 Columbia University, New York, NY

The paper lists affiliations for Snap Inc. and Columbia University and provides an open-source project repository.

  • The project repository is available at github.com/alirezazareian/ovr-cnn.

1. Introduction

The introduction motivates open-vocabulary object detection as a way to scale recognition beyond categories with costly box annotations. It combines caption-based vocabulary learning with limited bounding-box supervision and reports higher performance than zero-shot and weakly supervised approaches.

  • Extending detection from 600 to 60,000 categories would require 100 times more resources.Existing datasets cover limited categories despite extensive annotation resources.
  • Humans acquire rich visual-semantic vocabularies from natural supervision and generalize localization from few examples across object types.The motivation contrasts natural language and visual experience with category-by-category box annotation.
  • OVD first learns an unbounded vocabulary from image-caption pairs, then learns detection using annotations for only some object categories.The two stages separate broad vocabulary acquisition from object detection supervision.
  • Target classes can be any subset of the language vocabulary and need not be known during training.Training combines a broad caption vocabulary VC with localized annotations for base classes VB.
  • 27% mAP versus 10% for zero-shot learning, and 40% mAP versus 26% for weakly supervised detection.OVR-CNN is pretrained on image-caption data and fine-tuned on bounding-box data while preserving its learned vocabulary.

2. Related work

Related work includes zero-shot, weakly supervised, mixed-supervision, visual-grounding, and vision-language approaches. The paper positions OVD as combining open-vocabulary caption supervision with bounding-box supervision while avoiding predefined target classes.

  • Zero-shot detection generalizes from annotated seen classes to unseen categories but has practical limitations when unseen objects lack examples.Prior work addresses background modeling, proposal generalization, and hallucination of unseen classes.
  • Weakly supervised detection uses image-level labels, but localization is difficult and labels remain manually collected and taxonomy-limited.Multiple instance learning is commonly used to address the localization challenge.
  • Caption-based weakly supervised methods can be closed-vocabulary, require caption-parser labels, or discard captions’ richer semantic content.The cited approaches parse captions into predefined labels or use a predefined word set.
  • Mixed-supervision methods often require boxes for all classes, while OVD transfers caption-learned semantic knowledge to detection with base-class supervision.The paper describes this as the opposite direction from transferring supervised base-class detectors to weakly supervised target classes.
  • Visual grounding maps caption words to image regions but cannot detect objects without a caption at inference.The paper therefore adds another training phase to transfer grounding knowledge to object detection.
  • Vision-language transformers learn versatile image-caption features for downstream tasks, but the cited methods had not been applied to object detection.

3. Method

OVR-CNN pretrains a visual-semantic space from image-caption pairs, then transfers the learned visual backbone and V2L layer to Faster R-CNN for open-vocabulary detection. The design uses grounding and auxiliary language-image objectives to preserve a broad vocabulary while learning detection from base-class annotations.

  • Motivation and framework: OVD pretrains the visual backbone on a larger vocabulary VC rather than only base classes, reducing overfitting from projecting into a small embedding space.The paper notes that the practical vocabulary remains limited by pretrained word embeddings, although these are trained on very large text corpora.
  • Downstream detection: The trained ResNet and V2L layer initialize Faster R-CNN, whose detection classifier computes base-class scores by comparing region embeddings with pretrained word embeddings.A fixed all-zero background embedding contributes the denominator’s constant 1 and avoids pushing possible target-class regions toward an arbitrary embedding-space location.
  • Image-caption pretraining: The pretraining architecture processes image-caption pairs through a ResNet visual backbone, a pretrained BERT language backbone, and a multimodal transformer.The ResNet extracts image-region features, while BERT produces contextualized caption-token embeddings.
  • Image-caption pretraining: A linear V2L layer maps visual region representations into the language embedding space, allowing image regions and caption words to be compared semantically.The resulting image-region embeddings are used with caption embeddings in the grounding objectives.
  • Pretraining objectives: Visual grounding uses a weighted average of word-region similarity scores and batch negatives to align matching images and captions while separating non-matching pairs.The model defines two grounding objectives, using other images and captions in the batch as negative examples.
  • Pretraining objectives: Masked language modeling and image-text matching supplement grounding, and the total loss jointly trains the visual backbone, V2L layer, and multimodal transformer.The language backbone and its word embeddings remain fixed in the reported experiments.

4. Experiments

Experiments evaluate OVR-CNN on COCO against zero-shot and weakly supervised baselines, then analyze its data, pretraining, transfer, and embedding-space choices. The results show stronger target-class performance and effective open-vocabulary generalization, with trade-offs between target and base-class performance.

  • Experimental setup: OVR-CNN is evaluated on COCO using 48 base classes and 17 target classes, with mAP at IoU 0.5 reported separately for base, target, and generalized settings.COCO Captions and Conceptual Captions provide image-caption pretraining data, while bounding-box annotations are retained for base classes.
  • Baselines: Because OVD combines training data and supervision not used by prior tasks, comparisons use related but not identically matched zero-shot, weakly supervised, and mixed-supervision baselines.Some weakly supervised and mixed-supervision methods know target classes during training, unlike OVR-CNN and most zero-shot methods.
  • Results: OVR-CNN significantly improves target and generalized-target performance over all zero-shot baselines, and outperforms weakly supervised and mixed-supervision methods on target classes.It also substantially improves base-class and overall performance over weakly supervised and mixed-supervision baselines through bounding-box supervision for base classes.
  • Ablations: Multimodal pretraining is essential: removing image-caption pretraining causes poor performance, while grounding contributes more than auxiliary multimodal-transformer objectives.Transferring ResNet weights alone is insufficient; the V2L layer must also be transferred and frozen to preserve target-class generalization.
  • Ablations: Conceptual Captions reduce performance relative to COCO Captions but still outperform all zero-shot baselines, showing that automatically collected, lower-quality data remains useful.Random initialization also causes a performance drop, yet remains better than most ImageNet-initialized zero-shot baselines.
  • Visualization and discussion: With image-caption pretraining, target-class embeddings form well-defined clusters containing class prototypes in most cases, unlike the convoluted or collapsed baseline embedding space.The visualization supports the quantitative finding that multimodal data produces a more coherent visual-semantic space for generalization beyond base classes.

5. Conclusion

The conclusion presents OVD as a framework that separates recognition from localization using captions and bounding boxes, and introduces OVR-CNN as one implementation. Experiments report record performance against zero-shot and weakly supervised baselines, while the framework is positioned as extensible to other vision tasks.

  • Conclusion: OVD separates recognition and localization, learning recognition from open-vocabulary captions and localization from accurate bounding-box annotations.The framework uses different supervision sources for the corresponding tasks.
  • Conclusion: OVR-CNN pretrains Faster R-CNN on image-caption data, then transfers its open-vocabulary visual-semantic knowledge to downstream object detection.The method maintains vocabulary learned during pretraining while fine-tuning on localized annotations.
  • Conclusion: OVR-CNN achieves record performance compared with zero-shot and weakly supervised baselines, establishing a new state of the art for scalable object detection.The conclusion describes OVR-CNN as one possible implementation of the broader framework.
  • Conclusion: The general OVD idea can extend beyond object detection to other downstream computer-vision tasks.The conclusion connects this extensibility to open-vocabulary computer-vision technology.

6. Supplementary Material

The analysis examines how training-data frequency and object size relate to OVR-CNN performance, and identifies localization accuracy as a key limitation for target classes.

  • Data bias: Downstream frequency decreases exponentially, but performance remains stable except for target classes with zero downstream examples.Target classes are inevitably less accurate because they lack downstream bounding-box examples.
  • Data bias: Fixing the classification head, including the V2L layer and class embeddings, is proposed as the reason for robustness to downstream data bias.This differs from conventional classifiers that adapt classifier parameters, including an explicit bias term, to biased training data.
  • Pretraining bias: The least frequent pretraining words correlate with the least accurate classes, although the correlation is weak.This motivates future bias-mitigation mechanisms for naturally supervised image-caption settings.
  • Object size: Smaller objects such as knife and tie have lower performance because weakly supervised grounding is less likely to align them correctly to words.Their smaller feature-map footprint makes alignment more difficult.
  • Qualitative analysis: Qualitative results identify target-class localization as the main limitation, with bounding boxes that can be overly loose or overly tight.The limitation is attributed to having no ground-truth bounding boxes for target classes and motivates class-agnostic boundary refinement.
Loading 2011.10678v2…