PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Vocabulary Object Detection with Pseudo Bounding-Box Labels

Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, Caiming Xiong

arXiv:2111.09452v3cs.CV

TL;DR

Object detection remains limited by the cost of bounding-box annotation, and open-vocabulary methods are constrained by small base-category sets. This paper uses pretrained vision-language models to generate pseudo boxes from image-caption pairs and train detectors, reporting gains over existing approaches across four datasets.

  • Problem

    Bounding-box annotation is expensive, while open-vocabulary and zero-shot methods remain constrained by the small sets of annotated base categories used for training.

    Source: §1 · pp. 1–2

  • Method

    The method uses pretrained vision-language localization to generate pseudo bounding boxes for objects in image-caption pairs and trains an open-vocabulary detector with them.

    Source: §1 · p. 2

  • Results

    8% mAP on COCO novel categories, with reported gains of 6.3% on PASCAL VOC, 2.3% on Objects365, and 2.8% on LVIS over existing approaches.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The method can train detectors with pseudo boxes and, when trained only on those labels, achieves higher performance than state-of-the-art methods relying on manual boxes.

    Source: §1 · p. 3

    The evaluation uses LVIS v0.5 because the LVIS v1.0 validation set contains COCO train 2017 images used for fine-tuning in some experiments.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Despite great progress in object detection, most existing methods work only on a limited set of object categories, due to the tremendous human effort needed for bounding-box annotations of training data. To alleviate the problem, recent open vocabulary and zero-shot detection methods attempt to detect novel object categories beyond those seen during training. They achieve this goal by training on a pre-defined base categories to induce generalization to novel objects. However, their potential is still constrained by the small set of base categories available for training. To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs. Our method leverages the localization ability of pre-trained vision-language models to generate pseudo bounding-box labels and then directly uses them for training object detectors. Experimental results show that our method outperforms the state-of-the-art open vocabulary detector by 8% AP on COCO novel categories, by 6.3% AP on PASCAL VOC, by 2.3% AP on Objects365 and by 2.8% AP on LVIS. Code is available at https://github.com/salesforce/PB-OVD.

1 Introduction

Bounding-box annotation costs limit object detectors to small category sets, constraining open-vocabulary and zero-shot methods. This paper generates pseudo boxes from image-caption pairs using pretrained vision-language models and reports gains over existing detectors across four datasets.

1 Introduction

  • Existing open-vocabulary and zero-shot detectors rely on small base-category sets because diverse bounding-box annotations are expensive, limiting generalization to novel objects.Detecting a new category with conventional methods also requires collecting and manually annotating many boxes.Source: §1 · pp. 1–2
  • The method uses Grad-CAM maps from pretrained vision-language models to generate pseudo boxes for diverse captioned objects, then trains an open-vocabulary detector with those labels.The detector can also be fine-tuned with human-provided boxes when available.Source: §1 · pp. 2–3
  • When both methods are fine-tuned on COCO base categories, the method beats the best open-vocabulary detector by 8% mAP on COCO novel objects; on PASCAL VOC, Objects365, and LVIS, it beats existing approaches by 6.3%, 2.3%, and 2.8%, respectively.Without COCO base-category fine-tuning, it still beats a fine-tuned SOTA baseline by 3% mAP.Source: §1 · p. 3

2 Related Work

Related work addresses annotation burden through weak supervision, zero-shot detection, and open-vocabulary detection, but these approaches still rely on fixed or limited category sets. Vision-language pretraining offers a way to derive pseudo boxes from captioned images.

2 Related Work

  • Weakly supervised detectors use image-level categories, captions, or object counts, but support fixed category sets and require manual annotation and retraining for new categories.Traditional two-stage detectors first generate proposals and then classify them.Source: §2 · pp. 3–4
  • Zero-shot and open-vocabulary detectors train on annotated base classes and transfer to novel classes, but limited base-category coverage makes substantially different novel objects challenging.Prior approaches include visual-semantic alignment, synthesizing novel-class representations, and distilling vision-language models.Source: §2 · p. 4
  • Vision-language models trained on image-caption pairs can encode word-to-region localization information, which this work uses to generate diverse-object pseudo boxes.Such models have also succeeded in image-language tasks and zero-shot image classification.Source: §2 · p. 4

3 Our Approach

The approach generates pseudo bounding boxes from image-caption pairs using vision-language localization, then trains an open-vocabulary detector with the resulting labels. Its pipeline maps captioned object words to image regions and aligns region visual embeddings with text embeddings.

3 Our Approach

  • The framework combines an automated pseudo-box generator with an open-vocabulary detector trained on generated labels, with optional fine-tuning on human boxes when available.Pseudo-label generation is disentangled from detector training, allowing different detector architectures.Source: §3 · pp. 4–5, §3.2 · pp. 6–7

3.1 Generating Pseudo Box Labels

  • Grad-CAM uses gradients of image-caption similarity with respect to cross-attention scores to form object activation maps; proposal selection then chooses the box with the greatest activation overlap, normalized by proposal area.The approach maintains an object vocabulary and generates labels for its objects; the authors report that generated boxes are good quality and improve detector performance.Source: §3.1 · p. 6

3.2 Open Vocabulary Object Detection with Pseudo Labels

  • The detector extracts region visual embeddings and candidate text embeddings, training to pull matching pairs together and push mismatched pairs apart.During inference, proposals are matched to object classes by visual-text embedding similarity.Source: §3.2 · p. 7, Figure 3 · p. 7

4 Experiments

The experiments train detectors on pseudo labels from about one million image-caption pairs spanning 1,582 object categories, then evaluate COCO novel-category detection and transfer to other datasets. The method outperforms the strongest baseline across the reported fine-tuned benchmarks, while ablations examine proposal quality, vision-language models, vocabulary size, and training data.

4.1 Datasets and Object Vocabulary for Training

  • Training uses about one million images from COCO Caption, Visual Genome, and SBU Caption, with a union vocabulary of 1,582 object categories.The vocabulary combines object names from COCO, PASCAL VOC, Objects365, and LVIS.Source: §4.1 · p. 8

4.2 Evaluation Benchmarks

  • COCO evaluation uses 48 base and 17 novel classes, reports mean average precision over novel classes, and sets the IoU threshold to 0.5.Generalization is evaluated on PASCAL VOC, Objects365, and LVIS, which the models are not trained on.Source: §4.2 · pp. 8–9

4.3 Implementation Details

  • Pseudo labels come from ALBEF Grad-CAM activation maps and Mask-RCNN proposals; the proposal generator excludes COCO novel categories.The detector uses Mask-RCNN with ResNet-50 and a pre-trained CLIP text encoder.Source: §4.3 · pp. 9–10

4.4 Experimental Results

  • 8% improvement over Zareian et al. on COCO novel categories accompanies gains of 6.3% on PASCAL VOC, 2.3% on Objects365, and 2.8% on LVIS.The COCO comparison uses fine-tuning on COCO base categories; the cross-dataset gains are also reported for fine-tuned models.Source: Table 1 · p. 9, Table 2 · p. 9, §4.4 · pp. 10–11
  • 25.8% AP on COCO novel categories without COCO-base fine-tuning exceeds the fine-tuned Zareian et al. baseline by 3%.Without fine-tuning, the method also outperforms the baseline on Objects365 and LVIS, but underperforms its fine-tuned baseline on PASCAL VOC.Source: Table 1 · p. 9, Table 2 · p. 9, §4.4 · pp. 10–11

How does the quality of bounding-box proposals affect performance?

5 Closing Remarks

The paper trains open-vocabulary detectors with pseudo bounding-box labels generated from large-scale image-caption pairs using pre-trained vision-language models. It reports improved detection over state-of-the-art methods while noting that caption-language bias may affect the labels.

5 Closing Remarks

  • The framework trains open-vocabulary object detectors with pseudo bounding-box labels generated from large-scale image-caption pairs.The generator uses pre-trained vision-language models’ localization ability to annotate diverse objects embedded in captions.Source: §5 · p. 14
  • The pseudo labels are generated from objects embedded in image captions using pre-trained vision-language models’ localization ability.Source: §5 · p. 14
  • The method outperforms state-of-the-art zero-shot and open-vocabulary object detection methods by a large margin.Source: §5 · p. 14
  • Bias embedded in language descriptions may bias pseudo labels mined from captions without human intervention.The paper identifies manually filtering biased object names in the vocabulary as a possible solution.Source: §5 · p. 14

A Additional Analysis

The additional analysis finds that more pretraining data can reduce Zareian et al.’s performance, while this method retains an advantage with COCO Caption alone and benefits from more image-caption data.

A Additional Analysis

  • 4.2% and 10.8%: Zareian et al.’s performance on COCO novel categories drops with pretraining on 1M and 4M images, respectively, versus the COCO Caption baseline.Source: A Additional Analysis · p. 17
  • 5%: the method still outperforms its baseline when its vision-language model is trained with COCO Caption data only.The authors describe this setting as not leveraging their ability to employ more image-caption data without much extra cost.Source: A Additional Analysis · p. 17
  • 27.8 versus 30.8: the method performs better with more image-caption data, comparing COCO-caption-only pretraining with the main-paper result.The passage attributes 27.8 to Table 8 and 30.8 to Table 1 in the main paper.Source: A Additional Analysis · p. 17, Table 8 · p. 17

Why Zareian et al. cannot easily take advantage of more powerful VL

Zareian et al.’s approach ties the detector backbone to the vision-language model’s visual encoder, limiting which models it can use. This method separates those components, allowing it to use most recent vision-language models.

Why Zareian et al. cannot easily take advantage of more powerful VL

Loading 2111.09452v3…