PaperlayerView exact revision on arXiv

Source-linked AI summary

CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching

Xiaoshi Wu, Feng Zhu, Rui Zhao, Hongsheng Li

arXiv:2303.13076v1cs.CVcs.AI

TL;DR

Open-vocabulary detection must transfer recognition to novel categories while overcoming CLIP’s whole-image-to-region mismatch and the difficulty of unseen-class localization. CORA adapts CLIP in a DETR-style framework through Region Prompting and Anchor Pre-Matching, achieving state-of-the-art results across COCO and LVIS OVD benchmarks. On COCO, it improves novel-category AP50 by 2.4 over the previous best method without extra data, while extra data raises performance to 43.1 AP50.

  • Problem

    Open-vocabulary detection requires recognizing novel categories, but CLIP-based detectors face whole-image-to-region distribution mismatch and difficulty localizing unseen-class objects.

  • Method

    CORA adapts CLIP within a DETR-style detector using Region Prompting for region classification and Anchor Pre-Matching for class-aware object localization.

  • Results

    2.4 AP50 improvement over the previous best method is achieved on novel COCO categories without extra data, while extra data raises performance to 43.1 AP50 and LVIS performance to 28.1 APr.

  • Takeaways & Limitations

    CORA reports improved transfer from base to unseen novel classes through region adaptation and efficient class-aware localization on COCO and LVIS OVD benchmarks.

Abstract

from arXiv · show

Open-vocabulary detection (OVD) is an object detection task aiming at detecting objects from novel categories beyond the base categories on which the detector is trained. Recent OVD methods rely on large-scale visual-language pre-trained models, such as CLIP, for recognizing novel objects. We identify the two core obstacles that need to be tackled when incorporating these models into detector training: (1) the distribution mismatch that happens when applying a VL-model trained on whole images to region recognition tasks; (2) the difficulty of localizing objects of unseen classes. To overcome these obstacles, we propose CORA, a DETR-style framework that adapts CLIP for Open-vocabulary detection by Region prompting and Anchor pre-matching. Region prompting mitigates the whole-to-region distribution gap by prompting the region features of the CLIP-based region classifier. Anchor pre-matching helps learning generalizable object localization by a class-aware matching mechanism. We evaluate CORA on the COCO OVD benchmark, where we achieve 41.7 AP50 on novel classes, which outperforms the previous SOTA by 2.4 AP50 even without resorting to extra training data. When extra training data is available, we train CORA$^+$ on both ground-truth base-category annotations and additional pseudo bounding box labels computed by CORA. CORA$^+$ achieves 43.1 AP50 on the COCO OVD benchmark and 28.1 box APr on the LVIS OVD benchmark.

1. Introduction

CORA adapts CLIP within a DETR-style detector to address region-feature mismatch and unseen-class localization in open-vocabulary detection. It combines Region Prompting with Anchor Pre-Matching and reports state-of-the-art results on COCO and LVIS OVD benchmarks.

  • Motivation: Open-vocabulary detection extends beyond closed-category detection to objects from unseen categories, which otherwise require extra annotations and training.Large-scale vision-language models such as CLIP provide a basis for recognizing novel objects.
  • Approach: CORA incorporates CLIP into a DETR-based framework for open-vocabulary detection without additional image-text data.The framework uses a DETR-style object localizer and CLIP-based region classification.
  • Approach: Region Prompting adapts CLIP to region features by addressing the distribution gap between whole-image and pooled regional representations.The gap reduces classification accuracy when CLIP’s visual encoder is used for region recognition.
  • Approach: Anchor Pre-Matching enables class-aware localization by associating dynamic anchor boxes with input categories before box regression.This avoids repetitive per-class inference in the DAB-DETR localizer.
  • Results: 2.4 AP50 improvement over the previous best method is achieved on novel COCO categories without training on extra data.With extra training data, the method gains 3.8 AP50 over existing methods and achieves 22.2/28.1 APr with/without extra data on LVIS.
  • Results: Region Prompting raises novel-class classification performance from 63.9% to 74.1% on COCO base-class prompting.Other prompting or adaptation methods are reported to bias toward base classes.

2. Related Works

Related work applies vision-language pretraining, prompting, pseudo-label mining, and DETR-based matching to open-vocabulary detection. CORA is positioned among approaches addressing region adaptation and novel-class localization while fixing the pretrained vision-language model for generalization.

  • Open-Vocabulary Object Detection: OVR-CNN formulates open-vocabulary detection and aligns region features with nouns from image-paired captions.
  • Open-Vocabulary Object Detection: Prior methods mine pseudo labels, learn regional prompts, or adapt CLIP and other vision-language models for region detection and transfer.These approaches include pseudo-label mining, regional prompt learning, RegionCLIP, and joint localization-language alignment.
  • Open-Vocabulary Object Detection: OV-DETR uses conditional matching to address missing novel classes in assignment, but incurs inefficient inference.
  • Detection Transformers: DETR formulates object detection as set-to-set matching, while subsequent work improves its convergence through architectural changes and training strategies.Deformable attention is one cited architectural improvement for aggregating multi-scale features.
  • Prompt Tuning: Prompt tuning originated in NLP and later extended to continuous prompt vectors, pixel-space visual prompting, video recognition, and class-aware visual prompts.

3. Method

CORA adapts CLIP for open-vocabulary detection with a DETR-style localizer, region prompting, and anchor pre-matching. The framework addresses whole-image-to-region distribution mismatch and class-aware localization for unseen categories.

  • 3. Method: CORA uses a DETR-style framework that adapts CLIP for open-vocabulary detection with Region Prompting and Anchor Pre-Matching.CORA supports a setting without additional image-text data and a broader CORA+ setting with extra data.
  • 3. Method: The shared CLIP feature map supports both object localization and region classification, which are decoupled and conducted sequentially.Regional features are pooled from the feature map and classified using CLIP class-name embeddings.
  • 3.2. Region Prompting: Region Prompting adds learnable prompts to pooled region features before CLIP attention pooling and classification, addressing the whole-image-to-region distribution gap.The regional feature is augmented by element-wise addition with learnable prompts, then processed by the CLIP attention pooling module.
  • 3.2. Region Prompting: Compared with cropping each RoI as a separate image, pooled feature-map regions avoid repeated encoding for overlaps and preserve richer context.The paper presents regional prompting as more efficient than the conventional crop-based pipeline.
  • 3.3. Anchor Pre-Matching: Anchor Pre-Matching assigns object queries to class-name embeddings before decoding, enabling class-aware box regression and per-class bipartite matching.Each query is conditioned on its pre-matched class embedding, and training assignments are restricted to predictions with the same pre-matched label.
  • 3.3. Anchor Pre-Matching: CLIP-Aligned labeling relabels training boxes with the region classifier so more ground-truth boxes can be matched despite anchor or classifier misalignment.Without a same-label pre-matched query, a ground-truth box is ignored during training; relabeling addresses this alignment problem.

4. Experiments

CORA is evaluated on COCO and LVIS open-vocabulary detection benchmarks, including generalized detection, fairer comparisons with extra data, region classification, and ablations of its training components. The experiments show gains from region prompting and anchor pre-matching across the reported settings.

  • Datasets and evaluation: CORA is evaluated on COCO and LVIS OVD benchmarks, with COCO using 48 base and 17 novel classes.The COCO training split contains 107,761 images and 665,387 instances; evaluation uses 4,836 images and 33,152 instances.
  • Datasets and evaluation: COCO evaluation reports AP50 for generalized detection, while LVIS reports novel-class box APr on the full validation dataset.The generalized setting requires predictions for both base and novel classes before evaluation on novel objects.
  • Effectiveness of region prompting: Region prompting achieves 58.2 mAP on novel classes directly from CLIP and gains 6.9 mAP over CLIP after prompting.On the RN50x4 backbone, region prompting further boosts CLIP by 10.2% mAP; the comparison is against adapter and prompt-tuning baselines.
  • Effectiveness of anchor pre-matching: Full anchor pre-matching boosts novel-class performance by 10.8 AP50, while query conditioning without post-matching yields a 3.9 AP50 gain.The ablation separates query conditioning from per-class post-matching and also examines class dropout and CLIP-aligned labeling.

5. Conclusion

CORA targets knowledge transfer from base to unseen categories in OVD by adapting CLIP to regions and using class-aware localization. Experiments report a smaller novel-to-base performance gap than prior work.

  • 5. Conclusion: CORA adapts CLIP into a region classifier and uses region prompting to reduce the whole-image-to-region feature gap.The method also uses anchor pre-matching for class-aware object localization.
  • 5. Conclusion: Anchor pre-matching provides efficient class-aware localization instead of relying on a fixed RPN for novel classes.
  • 5. Conclusion: CORA transfers knowledge from base classes to unseen novel classes with a smaller performance gap than prior methods.

A. Extra Implementation Details

Training uses an exponential moving average, auxiliary decoder losses, one-time anchor pre-matching, matched loss coefficients, and standard geometric image augmentations.

  • A. Extra Implementation Details: The exponential moving average factor is set to 0.99996.
  • A. Extra Implementation Details: Auxiliary loss is applied to every intermediate decoder-layer output, while anchor pre-matching is conducted only once.
  • A. Extra Implementation Details: Matching-cost hyperparameters are identical to the corresponding loss coefficients.
  • A. Extra Implementation Details: Training augmentation uses random flipping, resizing, and cropping, with the smaller image edge resized between 480 and 800.

B. Localization Capability of CORA

CORA’s localization capability is evaluated independently of classification on the standard COCO OVD benchmark. Compared with RegionCLIP, CORA has a significantly smaller novel-to-base performance gap.

  • B. Localization Capability of CORA: Localization is evaluated using predicted box coordinates, with ground-truth matching assigning labels and highest IoU replacing confidence scores.These changes remove the classifier’s effect from the evaluation.
  • B. Localization Capability of CORA: The localization results are reported as AP50 in Table 6.
  • B. Localization Capability of CORA: CORA has a significantly lower novel-to-base performance gap than RegionCLIP, indicating better generalization toward novel classes.

C. Visualizations

The visualizations show CORA’s predictions on images containing novel objects, covering both base and novel classes.

  • C. Visualizations: Figure 5 visualizes CORA’s predictions on images with novel objects.
  • C. Visualizations: The figure presents predictions for both base and novel classes.
Loading 2303.13076v1…