PaperlayerView exact revision on arXiv

Source-linked AI summary

PromptDet: Towards Open-vocabulary Detection using Uncurated Images

Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, Lin Ma

arXiv:2203.16513v2cs.CV

TL;DR

Object detectors are difficult to scale beyond manually annotated base categories, motivating open-vocabulary detection with minimal human effort. PromptDet combines a two-stage detector, a frozen pretrained text encoder, regional prompt learning, and iterative self-training on uncurated web images. It achieves 21.4 AP on novel classes on LVIS and reports superior performance on LVIS and MS-COCO with fewer training costs.

  • Problem

    Existing detection training depends on extensive bounding-box annotations and therefore scales poorly to novel categories; image-text supervision lacks the spatial information needed for detection.

  • Method

    PromptDet uses class-agnostic proposals classified by a frozen pretrained text encoder, aligns text and regional visual features with regional prompt learning, and self-trains on sourced web images.

  • Results

    21.4 AP on novel classes on LVIS; PromptDet substantially outperforms previous state-of-the-art on LVIS and MS-COCO with uncurated web images and fewer training costs.

  • Takeaways & Limitations

    PromptDet demonstrates a zero-manual-annotation route to expanding an object detector toward novel categories using pretrained language representations and noisy web images.

  • Takeaways & Limitations

    Naive visual-text alignment yields limited performance, while class-name ambiguity and the scene-centric nature of web images create alignment challenges; learned prompts are assumed transferable across categories.

Abstract

from arXiv · show

The goal of this work is to establish a scalable pipeline for expanding an object detector towards novel/unseen categories, using zero manual annotations. To achieve that, we make the following four contributions: (i) in pursuit of generalisation, we propose a two-stage open-vocabulary object detector, where the class-agnostic object proposals are classified with a text encoder from pre-trained visual-language model; (ii) To pair the visual latent space (of RPN box proposals) with that of the pre-trained text encoder, we propose the idea of regional prompt learning to align the textual embedding space with regional visual object features; (iii) To scale up the learning procedure towards detecting a wider spectrum of objects, we exploit the available online resource via a novel self-training framework, which allows to train the proposed detector on a large corpus of noisy uncurated web images. Lastly, (iv) to evaluate our proposed detector, termed as PromptDet, we conduct extensive experiments on the challenging LVIS and MS-COCO dataset. PromptDet shows superior performance over existing approaches with fewer additional training images and zero manual annotations whatsoever. Project page with code: https://fcjian.github.io/promptdet.

1 Introduction

PromptDet targets the limited scalability of manually annotated, closed-vocabulary detection by combining a two-stage detector with a pretrained text encoder, regional prompt learning, and self-training on uncurated web images. It expands detection to novel categories without manual annotations and reports superior performance on LVIS and MS-COCO with fewer training costs.

  • Large-scale bounding-box annotation supports strong detectors but limits them to a small, closed set of categories.
  • Open-vocabulary detection seeks to localise and recognise novel categories using an existing detector trained on base categories with minimal human effort.
  • Regional prompt learning aligns textual embeddings with object-centric regional visual features, while iterative self-training sources candidate web images and pseudo-labels.
  • PromptDet uses a two-stage detector with class-agnostic proposals classified by a frozen text encoder from a pretrained visual-language model.
  • PromptDet substantially outperforms previous state-of-the-art on LVIS and MS-COCO using uncurated web images, smaller image resolution, and fewer epochs.

2 Related Work

Related work establishes the distinction between conventional fixed-vocabulary detection and open-vocabulary approaches that transfer semantic knowledge to novel categories. Prior methods use language embeddings, image-text training, or CLIP distillation, while Detic relies on annotated external data rather than uncurated web images.

  • Conventional two-stage and one-stage detectors generally require bounding-box annotations and recognise only categories present during training.
  • Open-vocabulary detection extends recognition beyond a closed set using language embeddings, external text, image-text contrastive learning, or visual-language distillation.
  • Unlike Detic, which self-trains on ImageNet21K images requiring substantial annotation effort, PromptDet trains directly on uncurated web images.
  • Zero-shot learning transfers knowledge from seen to novel classes through visual attributes or joint visual-semantic embedding spaces.
  • CLIP and ALIGN learn joint image-text representations from large-scale Internet image-caption pairs using noise contrastive learning.

3 Methodology

PromptDet expands a two-stage detector to novel categories by combining class-agnostic proposals with a frozen language-based classifier, regional prompt learning, and iterative self-training on web images. The method recalibrates textual embeddings for object-centric regions and uses sourced images with pseudo boxes to improve novel-category detection.

  • 3.1 Open Vocabulary Object Detector: The detector inherits a frozen CLIP text encoder as its classifier generator while training the visual backbone and class-agnostic RPN to align with textual category embeddings.This pairs RPN-derived visual features with language embeddings rather than learning a fixed closed-set classification layer.
  • 3.1 Open Vocabulary Object Detector: PromptDet formulates open-vocabulary detection as generating class-agnostic region proposals and classifying them beyond the fixed training vocabulary.The detector is based on a standard two-stage architecture with an encoder, region proposal network, and classification module.
  • 3.3 Alignment via Regional Prompt Learning: Regional prompt learning steers the textual latent space toward local, object-centric RPN features instead of scene-centric visual representations.The method addresses lexical ambiguity with category descriptions and inserts learnable continuous prompt vectors around tokenized category and description text.
  • 3.3 Alignment via Regional Prompt Learning: Prompt vectors are learned offline from base-category object crops using frozen visual and textual encoders and a cross-entropy objective.The resulting prompts recalibrate category embeddings before visual-backbone alignment, and experiments report noticeable improvements in open-vocabulary generalisation.
  • 3.4 PromptDet: Alignment via Self-training: The self-training framework iteratively learns prompts, sources high-precision web images for novel categories, generates pseudo ground-truth boxes, and trains with base LVIS images plus sourced images.Iterating prompt learning and image sourcing improves object-centric image mining, pseudo-box accuracy, and novel-category detection after self-training.

4 Experiment

Experiments evaluate PromptDet on LVIS and MS-COCO, including ablations of prompt learning, iterative sourcing, self-training, pseudo-label generation, and training choices. The method improves novel-category detection, with stronger results from learned prompts, iterative self-training, confident pseudo boxes, and more sourced images.

  • Ablation Study: Learned regional prompts improve novel-category performance to 29.0 AP, exceeding the manual prompt’s 27.4 AP by 1.6 AP.With 72 epochs, the model reaches 31.4 AP on the 21 novel MS-COCO categories.
  • Ablation Study: Self-training raises performance from 9.0 to 15.3 AP for manual prompts and from 11.1 to 15.9 AP for learned prompts.Two rounds of regional prompt learning and image sourcing reach 19.0 AP, while three rounds reach 19.3 AP on novel categories.
  • Ablation Study: Selecting the most confident proposal for pseudo ground truth substantially outperforms alternative box-generation strategies.The authors contrast this with max-sized-box heuristics and note that noisy uncurated images can produce erroneous supervision.
  • Ablation Study: Increasing sourced images from 50 to 300 improves novel-category performance monotonically from 14.6 to 19.0 AP.LAION-400M supports at most 300 images for most categories, so larger-scale sourcing is left for future work.
  • Ablation Study: Freezing the class-agnostic RPN and box regression reduces novel-category performance by 1.8 AP, from 19.0 to 17.2.The ablation therefore evaluates updating these components during self-training.
  • Ablation Study: The best pseudo-labeling performance comes from selecting the most confident prediction among the top-20 proposals; using all proposals gives 10.4 AP versus 19.0 AP.The experiments set K = 20 for pseudo-label generation.
  • Comparison with the State-of-the-Art: PromptDet reaches 21.4 AP on LVIS novel categories, surpassing ViLD-ens by 4.8 AP and Detic by 3.6 AP.The model uses a 72-epoch training schedule for this comparison.
  • Comparison with the State-of-the-Art: On MS-COCO, PromptDet achieves 26.6 AP novel-class mAP and 50.6 AP overall mAP, compared with Detic’s 24.1 and 44.7 AP.Both methods use 640×640 input resolution and PromptDet is trained for 24 epochs.

5 Conclusion

PromptDet detects novel categories without manual annotations by combining a frozen CLIP text encoder, regional prompt learning, and self-training on uncurated external images. On LVIS, it achieves 21.4 AP for novel classes with lower training costs.

  • PromptDet detects novel categories without any manual annotations.
  • A frozen CLIP text encoder generates classifier representations for a two-stage detector.
  • Regional prompt learning transforms the textual embedding space to align with object-centric visual representations.
  • Self-training selects high-quality candidate images from uncurated external images and iteratively trains the detector.
  • 21.4 AP is achieved for novel classes on LVIS, surpassing prior open-vocabulary detectors with lower training costs.

– Supplementary material – PromptDet: Towards Open-vocabulary

The supplementary material identifies the work as “Detection using Uncurated Images” and lists its authors and institutional affiliations.

  • The supplementary material is titled “Detection using Uncurated Images.”
  • The listed authors are Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma.
  • The authors are affiliated with Meituan Inc. and Shanghai Jiao Tong University.

1 Open-vocabulary COCO benchmark details

The open-vocabulary COCO benchmark uses selected base and novel categories from MS-COCO, supplements category descriptions, and combines COCO-base with LAION-novel data for self-training.

  • MS-COCO provides 48 base classes and 17 novel classes for the benchmark, using images containing at least one base class for training.
  • About 1500 LAION-400M images are downloaded for each novel category for self-training.
  • Category descriptions come from LVIS when available and otherwise are obtained through Google searches.
  • Mask-RCNN with a ResNet-50-FPN backbone is trained for 24 epochs using COCO-base, followed by self-training on COCO-base and LAION-novel.

2 Cross-dataset transfer

PromptDet is transferred directly from LVIS-base to MS-COCO without fine-tuning to assess generalisation to categories outside LVIS-base. Regional prompts improve novel-category performance, reaching 31.4 AP with longer training.

  • Cross-dataset transfer: The LVIS-trained detector is directly evaluated on MS-COCO without fine-tuning.
  • Cross-dataset transfer: MS-COCO contains 80 categories, of which 59 are covered by LVIS-base and 21 serve as novel categories for transfer evaluation.
  • Cross-dataset transfer: 29.0 AP is achieved on MS-COCO novel categories with regional prompts, exceeding manual prompts by 1.6 AP.
  • Cross-dataset transfer: 31.4 AP is achieved on the 21 novel MS-COCO categories after 72 epochs of training.

3 Qualitative Results

PromptDet qualitatively localises and recognises diverse novel-category objects on the open-vocabulary LVIS benchmark without manual annotations.

  • PromptDet accurately localises and recognises objects from diverse novel categories without manual annotations.
  • Figure 1 presents qualitative PromptDet results on images from the LVIS validation set.
  • Green boxes denote novel-category objects, while blue boxes denote base-category objects.
Loading 2203.16513v2…