PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, Guoqi Li

arXiv:2203.14940v1cs.CV

TL;DR

Open-vocabulary detection needs prompts that transfer from base classes to novel classes, but classification-oriented prompt learning and manual prompt engineering are poorly suited to detection. DetPro learns continuous detection prompts with background interpretation and foreground context grading, and its ViLD assembly outperforms the baseline across evaluated settings, including LVIS novel classes.

  • Problem

    Open-vocabulary detection requires detectors trained on base classes to detect novel classes, while effective prompts remain laborious to engineer and classification-oriented prompt learning is suboptimal for detection.

  • Method

    DetPro learns continuous prompt representations using background interpretation for negative proposals and context grading for tailored positive proposals, then replaces ViLD’s proposal classifier.

  • Results

    DetPro outperforms ViLD in all evaluated settings, including +3.4 APbox and +3.0 APmask on LVIS novel classes.

  • Takeaways & Limitations

    Experiments on LVIS and transfer learning to Pascal VOC, COCO, and Objects365 demonstrate DetPro’s effectiveness and generalization ability.

  • Takeaways & Limitations

    The setup trains on base-class annotations and evaluates recognition of novel classes during inference within the specified base/novel split.

Abstract

from arXiv · show

Recently, vision-language pre-training shows great potential in open-vocabulary object detection, where detectors trained on base classes are devised for detecting new classes. The class text embedding is firstly generated by feeding prompts to the text encoder of a pre-trained vision-language model. It is then used as the region classifier to supervise the training of a detector. The key element that leads to the success of this model is the proper prompt, which requires careful words tuning and ingenious design. To avoid laborious prompt engineering, there are some prompt representation learning methods being proposed for the image classification task, which however can only be sub-optimal solutions when applied to the detection task. In this paper, we introduce a novel method, detection prompt (DetPro), to learn continuous prompt representations for open-vocabulary object detection based on the pre-trained vision-language model. Different from the previous classification-oriented methods, DetPro has two highlights: 1) a background interpretation scheme to include the proposals in image background into the prompt training; 2) a context grading scheme to separate proposals in image foreground for tailored prompt training. We assemble DetPro with ViLD, a recent state-of-the-art open-world object detector, and conduct experiments on the LVIS as well as transfer learning on the Pascal VOC, COCO, Objects365 datasets. Experimental results show that our DetPro outperforms the baseline ViLD in all settings, e.g., +3.4 APbox and +3.0 APmask improvements on the novel classes of LVIS. Code and models are available at https://github.com/dyabel/detpro.

1. Introduction

Open-vocabulary object detection trains on base classes while detecting novel classes, but effective deployment depends on prompts whose manual design is laborious. DetPro learns detection-specific continuous prompts by incorporating background proposals and grading foreground context.

  • Motivation: Open-vocabulary object detection trains detectors on base classes while enabling detection of new classes, avoiding the need to collect more labeled data for every desired category.The alternative closed-set strategy incurs labeling costs and can produce long-tailed class distributions.
  • Motivation: ViLD generates class text embeddings from prompts through a pretrained vision-language text encoder and uses them to classify proposals during detector training.For open-set detection, embeddings for both base and novel classes replace base-only embeddings.
  • Motivation: Manual prompt engineering is crucial but laborious because slight word changes can positively or negatively affect detection performance.The process requires domain expertise and careful word tuning.
  • DetPro: Classification-oriented prompt learning is insufficient for detection because detectors must distinguish foreground from background and classify region proposals rather than only recognize image labels.DetPro is introduced to learn prompt representations specifically for open-vocabulary object detection.
  • DetPro: DetPro includes background interpretation for negative proposals and context grading for tailored prompt learning over foreground proposals.Negative embeddings are optimized away from all class embeddings, while foreground proposals are separated by context levels.

2. Related Work

Prompt learning transfers vision-language models to downstream tasks but remains difficult to engineer manually. DetPro extends classification-oriented prompt learning to open-vocabulary detection with strategies for foreground and background proposals, building on ViLD.

  • Prompt Learning: Pretrained vision-language models support downstream few-shot or zero-shot learning through fine-tuning or prompt engineering, but task-specific prompts can require laborious manual design.CLIP and ALIGN align image and text embeddings using large collections of web image-text pairs.
  • Prompt Learning: CoOp targets prompt representation learning for image classification, whereas DetPro is trained on base classes and designed to generalize to novel detection classes.DetPro extends the prompt-learning idea to the open-vocabulary object-detection setting.
  • Open-Vocabulary Object Detection: Zero-shot detection generalizes from annotated seen classes to unseen classes, but its overall performance remains far behind fully supervised methods.Open-vocabulary detection is presented as a broader paradigm that uses image-text pretraining and base-class bounding-box annotations.
  • Open-Vocabulary Object Detection: DetPro builds on ViLD, replacing its hand-crafted prompts with fine-grained automatic prompt learning and specialized background interpretation.ViLD distills knowledge from a pretrained vision-language model into a detector.

3. Problem Setting

DetPro replaces ViLD’s proposal classifier with learned prompt representations for open-vocabulary detection. Its design combines losses for positive and negative proposals, using context-tailored foreground sets and background interpretation.

  • Detection Prompt: DetPro learns continuous prompt representations for an open-vocabulary vision-language detector and replaces the proposal classifier in the ViLD pipeline.The method uses CLIP text embeddings generated from learned prompt representations.
  • Detection Prompt: Positive loss aligns visual embeddings of positive proposals with their corresponding class embeddings, while negative loss separates negative-proposal embeddings from all class embeddings.The two losses respectively supervise foreground classification and background interpretation.
  • Detection Prompt: Different tailored positive proposal sets defined by α < IoU(GT, Pos P) < β learn distinct prompt representations that are ensembled afterward.The grading scheme associates proposal groups with different context levels.
  • Problem Setting: The problem setting separates base classes used for training from novel classes available during inference, while the detector uses Faster R-CNN with ResNet-50 and FPN.The training dataset contains base-class annotations; the inference dataset contains both base and novel classes.

4. Method

DetPro adapts prompt representation learning to object detection by incorporating foreground and background proposals, then integrates the learned classifier into ViLD.

  • Detection Prompt: DetPro learns continuous prompt representations for open-vocabulary object detection using a pretrained vision-language model.The learned representations generate class embeddings that serve as a region-level classifier.
  • Motivation: Classification-oriented prompt learning is sub-optimal for detection because detectors must distinguish foreground from background and classify region proposals.Detection training therefore requires proposal-level supervision beyond recognizing an image-level label.
  • Context grading: Positive proposals are partitioned into disjoint IoU groups so each group learns prompt representations tailored to a similar foreground-context level.The resulting group-specific representations are averaged to obtain the final class prompt representation.
  • Background interpretation: Negative proposals are incorporated by encouraging their image embeddings to remain dissimilar to all base-class text embeddings.The method interprets background implicitly rather than forcing varied background content toward one explicit background embedding.
  • Assembly with ViLD: DetPro is assembled with ViLD by replacing its proposal classifier with fixed base-class embeddings generated from learned prompts and a learnable background embedding.ViLD retains image and text heads, while class-agnostic regression and mask modules remain part of the detector design described in the pipeline.

5. Experiment

Experiments train DetPro and its detector on LVIS base classes, evaluate rare classes, and transfer the LVIS-trained model to three datasets. DetPro improves ViLD* across the reported settings, while ablations examine background proposals, training data, ensemble strategies, context lengths, and class-token placement.

  • Dataset and Evaluation: DetPro and its open-vocabulary detector are trained on LVIS base classes and evaluated on LVIS novel classes, with transfer tests on Pascal VOC, COCO, and Objects365.LVIS uses frequent and common classes as 866 base classes and rare classes as 337 novel classes.
  • Main Results: DetPro improves ViLD* by +3.4 APr on LVIS object detection and +3.0 APr on instance segmentation.The re-implementation ViLD* uses a shorter training schedule than the original ViLD while achieving comparable AP.
  • Main Results: DetPro improves ViLD* on Pascal VOC, COCO, and Objects365, demonstrating effectiveness and generalization in the reported transfer experiments.The LVIS-trained model is transferred by directly replacing class tokens.
  • Ablation Study: Ablations compare negative-proposal inclusion strategies, training-data combinations, background-proposal counts, prompt ensembles, context lengths, and class-token positions.The default configuration uses 10% background proposals, and placing the class token at the end performs best in the reported experiment.
  • Ablation Study: The reported APr consistently declines as negative samples increase, so reducing negatives avoids background bias and speeds training.The paper’s default is 10 negative proposals according to the ablation discussion.
  • Visualization: DetPro embeddings are more discriminative than prompt-engineering embeddings in t-SNE visualizations of LVIS base and novel classes.The visualization is intended to assess the suitability of the embeddings as region classifiers.

6. Conclusion

The paper presents DetPro for learning continuous prompt representations tailored to open-vocabulary object detection. Its background interpretation and foreground context grading schemes support experiments showing effectiveness and generalization across LVIS and transferred datasets.

  • Conclusion: DetPro learns continuous prompt representations for open-vocabulary object detection using a pre-trained vision-language model.It is assembled with the ViLD open-vocabulary detection pipeline.
  • Conclusion: DetPro includes background interpretation for negative proposals and context grading for tailored positive-proposal prompt training.These schemes address both background and foreground proposals during detection-oriented prompt learning.
  • Conclusion: Experiments on LVIS and transfer learning to Pascal VOC, COCO, and Objects365 demonstrate the effectiveness and generalization ability of DetPro.The conclusion explicitly reports this cross-dataset evidence.

A. More Experiments and Analysis

Additional experiments evaluate DetPro variants, proposal-classification accuracy, assembly with ViLD, and transferred-dataset embeddings. These analyses further report improvements and more discriminative embeddings relative to prompt engineering.

  • Variants: DetPro-text removes the image head and uses only a text head for training and inference, following a corresponding ViLD-text variant.The comparison is conducted under the LVIS rare-class evaluation setting.
  • Proposal Classification: Tables 11 and 12 report top-1 and top-5 proposal-classification accuracy, respectively.These tables provide proposal-level classification analyses for the additional experiments.
  • Assembly: Assembling a trained ViLD with DetPro prompt representations improves novel-class performance under different ensemble settings.The class embeddings generated by DetPro replace the original image-head class embeddings at inference.
  • Visualization: Transferred-dataset t-SNE visualizations show DetPro embeddings are more discriminative than prompt-engineering embeddings on Pascal VOC, COCO, and Objects365.The paper connects this property with suitability as region classifiers for open-vocabulary detection.

B. More Implementation Details

The appendix specifies detector, proposal-generation, inference, and DetPro-training settings, alongside t-SNE visualizations for three transferred datasets.

  • DetPro Training: DetPro training uses batch size 512 with cross-entropy loss and temperature parameter 0.01.These are the additional DetPro training details reported in the appendix.
Loading 2203.14940v1…