PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection

Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, Fahad Shahbaz Khan

arXiv:2207.03482v3cs.CVcs.AI

TL;DR

Open-vocabulary detection must generalize beyond base classes, but CLIP and image-level supervision are not optimally aligned with precise object localization. The paper combines region-based CLIP alignment, pseudo-labeling from multimodal ViTs, and region-conditioned weight transfer, achieving gains on COCO and LVIS benchmarks.

  • Problem

    CLIP and image-level supervision provide image-centric cues that do not precisely specify local object regions for open-vocabulary detection.

  • Method

    The method distills object-centric information into CLIP region embeddings, uses multimodal ViT proposals for pseudo-labeling, and transfers region-level visual-language weights to image-level mapping.

  • Results

    The method improves performance across COCO and LVIS, including 36.6 AP on COCO novel classes and a 5.0 AP gain over ViLD on LVIS rare categories.

  • Takeaways & Limitations

    Object-centric visual and language alignment can be integrated within one OVD architecture to improve generalization to novel objects.

Abstract

from arXiv · show

Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local object regions. In this work, we propose to address this problem by performing object-centric alignment of the language embeddings from the CLIP model. Furthermore, we visually ground the objects with only image-level supervision using a pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training. We establish a bridge between the above two object-alignment strategies via a novel weight transfer function that aggregates their complimentary strengths. In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting. On the COCO benchmark, our proposed approach achieves 36.6 AP50 on novel classes, an absolute 8.2 gain over the previous best performance. For LVIS, we surpass the state-of-the-art ViLD model by 5.0 mask AP for rare categories and 3.4 overall. Code: https://github.com/hanoonaR/object-centric-ovd.

1 Introduction

Open-vocabulary detection seeks to detect novel classes beyond the base training vocabulary, but common image-centric supervision does not align precisely with object regions. The proposed framework combines region alignment, visually grounded pseudo-labeling, and weight transfer to bridge this gap.

  • OVD aims to detect novel classes defined by an open vocabulary beyond the limited base classes labeled during training.
  • Image-level supervision and pretrained CLIP embeddings are mismatched with object-centric detection because they lack precise local-region information.
  • High-quality class-agnostic and class-specific proposals from pretrained multimodal ViTs support region distillation and visually grounded pseudo-labeling.
  • A weight transfer function connects region-level and image-level visual-language mappings while combining their complementary strengths.
  • The framework uses region-based knowledge distillation to adapt image-centric CLIP embeddings for local regions.
  • 8.2 AP and 5.0 AP are the reported absolute gains on COCO novel classes and LVIS rare categories, respectively.

2 Related Work

Related work spans zero-shot detection, weakly supervised detection, and open-vocabulary detection. The paper positions its approach within weakly supervised detection while using pretrained multimodal ViT proposals to expand vocabulary and generalize to novel categories.

  • Zero-shot Object Detection (ZSD): Zero-shot object detection targets novel classes at inference without visual examples of those classes during training.
  • Weakly-supervised Object Detection (WSOD): Weakly supervised object detection uses image-level labels alone or alongside detection data to enlarge the detector vocabulary.
  • Weakly-supervised Object Detection (WSOD): Prior weakly supervised methods use region proposals, captions, clustering, attention, self-distillation, or max-score proposals to guide detection.
  • Weakly-supervised Object Detection (WSOD): This work uses high-quality object proposals from pretrained multimodal ViTs to enlarge the detector vocabulary and generalize toward novel categories.
  • Open-vocabulary Object Detection (OVD): Open-vocabulary detection commonly replaces classifier weights with text embeddings from models such as GloVe, BERT, or CLIP.

3 Object-centric Open-Vocabulary Detection

The framework bridges image- and object-centric representations in OVD by adapting CLIP embeddings to regions, grounding image-level labels with pseudo-boxes, and combining both through weight transfer.

  • 3.1 Detection Pipeline: Preliminaries: OVD replaces detector classifier weights with fixed CLIP text embeddings for target vocabulary categories, using base-category embeddings during training.
  • 3.1 Detection Pipeline: Preliminaries: Scene-centric CLIP training and weak image-level supervision provide cues that are poorly aligned with tightly bounded object regions.
  • 3.2 Region-based Knowledge Distillation: Region-based knowledge distillation uses high-quality class-agnostic MViT proposals to align detector region embeddings with CLIP region representations.
  • 3.2 Region-based Knowledge Distillation: Figure 2 reports improved separability of novel-class features for CLIP-aligned RKD and weight-transfer detector embeddings relative to supervised detector embeddings.
  • 3.3 Image-level Supervision with Pseudo Box Labels: Pseudo-box image-level supervision constructs class-specific queries and selects the highest-confidence proposal for each image-level category.The resulting pseudo-box labels supervise classification on the image classification dataset.
  • 3.4 Weight Transfer Function: Naively combining RKD and pseudo-box supervision causes competition, so a weight-transfer function maps frozen RKD projection weights toward the weak-supervision pathway.The transfer operation is represented as WT : WD → WP.

4 Experiments

Experiments evaluate the method on COCO and LVIS, including component ablations, strict open-vocabulary settings, and cross-dataset transfer. The results show gains from RKD, pseudo-labeled image supervision, and their weight-transfer combination.

  • Main results: 36.6 AP on COCO novel classes is achieved by weight transfer while maintaining base-class performance.The method combines the complementary benefits of RKD and pseudo-box-labeled image-level supervision.
  • Component analysis: 19.5 and 1.5 AP are gained on COCO novel and base classes, respectively, by region-based knowledge distillation.The result indicates that image-centric CLIP embeddings can be adapted to local regions.
  • Component analysis: 28.7 AP is added on COCO novel classes by pseudo-box-labeled image-level supervision, demonstrating vocabulary expansion to novel classes.Naively combining RKD and image-level supervision does not preserve the individual gains, whereas weight transfer does.
  • COCO results: 36.9 AP on COCO novel classes surpasses ViLD by 9.3.The reported comparison attributes complementary gains to the weight-transfer function.
  • LVIS results: 5.0 mask AP is gained over ViLD on LVIS rare classes with the 8x schedule, while frequent-class performance improves by 0.8.With the 3x schedule, the method already surpasses ViLD on rare classes by 1.0.
  • Strict OVD: 21.71 AP is achieved under the stricter LVIS setting using only LVIS base categories for image-level supervision, close to 22.75 AP using 997 categories.Expanding the vocabulary to five times the prior size slightly degrades performance, indicating sensitivity to image-level supervision.
  • Cross-dataset evaluation: Direct transfer of the LVIS-trained model to COCO, OpenImages, and Objects365 is evaluated without dataset-specific finetuning.The evaluation reports AP50 and is intended to assess adaptability across datasets.

5 Qualitative Results

Qualitative results show detections on COCO and LVIS, plus cross-dataset transfer to Objects365 and OpenImages. The COCO visualization distinguishes base and novel categories by color.

  • COCO qualitative results use purple for base categories and green for novel categories.
  • Cross-dataset qualitative results show high-quality detections on Objects365 and OpenImages without finetuning.

6 Conclusion

The paper combines region-based knowledge distillation with object-level pseudo-labeling to align visual-language representations for open-vocabulary detection. A weight-transfer module integrates both strategies, with encouraging generalization across four OVD benchmarks.

  • The framework targets the image-centric versus object-centric mismatch in existing open-vocabulary detection supervision.
  • Region-based knowledge distillation and object-level pseudo-labeling promote region-wise alignment between visual and language representations.
  • The weight-transfer module integrates the benefits of knowledge distillation and object-level pseudo-labeling.
  • The method demonstrates generalization ability on four popular open-vocabulary detection benchmarks.

Checklist

The checklist records empirical, implementation, ethical, licensing, and reproducibility disclosures, while noting that error bars were not reported because compute resources were limited.

  • Checklist: The paper states that its empirical findings are mainly based on experiments and empirical results rather than theoretical results requiring full assumptions and proofs.Relevant mathematical information is included in the theoretical formulations.
  • Checklist: The authors report providing code, reproduction instructions, training details, data splits, hyperparameter choices, and compute-resource details.The code and reproduction instructions are provided in the supplemental material, while training and resource details are referenced in the main paper.
  • Checklist: No error bars were reported because the authors cite limited compute resources as the reason for not running or reporting repeated statistics.This limits assessment of variation across random seeds in the reported experiments.
  • Checklist: The experiments use publicly available datasets, with dataset-use permissions checked and discussions of personally identifiable information and offensive content provided in the supplemental material.The paper states that explicit consent discussion was not included in the main paper because the datasets were publicly available and permitted for research.
  • Checklist: The paper reports discussing limitations, potential negative societal impacts, ethical implications, asset licenses, and dataset-related information in supplementary sections.It points readers to appendices covering limitations, societal impacts, ethics, and dataset license details.

A Implementation Details

The implementation combines standard Faster R-CNN and Mask R-CNN backbones with normalized embeddings, dataset-specific image-level supervision, cross-dataset evaluation, and a CLIP classification comparison.

  • A Implementation Details: COCO experiments use Faster R-CNN with a ResNet-50 C4 backbone, while LVIS experiments use Mask R-CNN with a ResNet-50 FPN backbone.Region and text embeddings are L2-normalized before RKD losses and final classification scores.
  • A Implementation Details: L2 normalization of region and text embeddings is used before RKD loss and final classification scores to stabilize training.The same implementation passage specifies the detector backbones used for COCO and LVIS.
  • A Implementation Details: COCO ILS uses COCO captions, whereas LVIS ILS uses ImageNet-21k, whose overlapping categories form a 997-category subset.Caption words are heuristically compared with COCO base and novel category names to generate positive image labels.
  • A Implementation Details: Cross-dataset evaluation uses OpenImages V5 and Objects365 V2 validation sets and reports AP50.The evaluation includes approximately 41K OpenImages images and 80K Objects365 images.
  • A Implementation Details: On COCO ground-truth object crops, pretrained CLIP shows stronger novel-class classification than the supervised-base model, while the proposed methods further improve performance.The comparison evaluates top-1 accuracy separately for novel and base classes.

C.1 Ablation Experiments on LVIS

LVIS ablations examine RKD, ILS, weight transfer, initialization, skip connections, proposal generation, and zero-shot classification, showing complementary roles for the proposed components.

  • C.1 Ablation Experiments on LVIS: ILS with class-specific pseudo-labels increases rare-category performance by 4.8 AP over the base model.The class-specific proposals are generated using pretrained multi-modal ViT outputs.
  • C.1 Ablation Experiments on LVIS: The weight-transfer scheme combines RKD and ILS, improving common and frequent categories while matching naive combination performance on rare classes.This is reported in the comparison between rows 4 and 5 of the LVIS component ablation.
  • C.1 Ablation Experiments on LVIS: RKD improves all reported LVIS metrics over the baseline, while ILS mainly improves rare-class performance.Simply combining RKD and ILS improves over baseline but does not retain the individual gains for common and frequent categories as well as weight transfer.
  • C.1 Ablation Experiments on LVIS: High-quality MViT proposals provide significant RKD gains over RPN proposals, and inter-embedding relations are additionally evaluated in the RKD ablation.The ablation compares alternative loss functions and proposal sources on LVIS.
  • C.1 Ablation Experiments on LVIS: Selecting the top-1 MViT proposal for each target category provides optimal rare-class performance among the evaluated ILS heuristics.The method uses target-specific queries such as ‘every {category}’.
  • C.1 Ablation Experiments on LVIS: RKD training from scratch lowers base-class AP, whereas initialization from the supervised base model recovers the loss and improves over the base model.The result indicates that region alignment requires mature feature distributions for effective CLIP knowledge distillation.
  • C.1 Ablation Experiments on LVIS: An MLP skip connection improves convergence and generally improves results by giving CLIP a direct region-classification path in ILS.The connection lets weight transfer focus on the residual signal in the ILS pathway.
  • C.1 Ablation Experiments on LVIS: Class-agnostic MViT proposals from the ‘all objects’ query are ranked by objectness, with the top 5 used for RKD; class-specific queries yield one top proposal per category for ILS.Proposal generation and corresponding CLIP region embeddings are computed offline for training efficiency.

E Limitations

The method retains a performance gap between base and novel classes and depends on the pretrained CLIP vocabulary; its flexible vocabulary and prompt tuning also create misuse and bias risks.

  • E Limitations: The reported COCO and LVIS results still show lower novel or rare-class performance than base or frequent-class performance.Examples include 56.7 versus 40.5 AP for COCO base and novel categories, and 29.1 versus 21.1 Mask AP for LVIS frequent and rare categories.
  • E Limitations: Open-vocabulary capability is limited by the vocabulary available in pretrained CLIP, which serves as the RKD teacher.This bounds vocabulary expansion despite the method’s object-centric alignment and weight transfer.
  • E Limitations: A detector with a highly flexible vocabulary can be used for inappropriate large-scale surveillance, while specialized prompt tuning can introduce biases and unfair predictions.The paper identifies these as potential negative societal impacts of flexible and targeted detection.

G Ethical Considerations

The detector’s category recognition depends on the image-text datasets used to pretrain its vision-language model, creating ethical concerns. Dataset bias and biased prompts can produce unfair or unethical human detections.

  • The detector’s category recognition strongly depends on the image-text datasets used to pretrain CLIP.
  • Internet-extracted pretraining datasets may contain racial and unethical bias that influences the detector’s ethical behavior.
  • The flexible detector vocabulary can be tuned with biased prompts to produce targeted human detections based on race.

H License Details

The experiments use COCO, LVIS, and ImageNet-21k under specified public, research, non-commercial, and Creative Commons licensing conditions summarized in Table 15.

  • COCO is available for non-commercial use under the Creative Commons Attribution 4.0 license.
  • LVIS is based on COCO and is licensed under both CC BY 4.0 and the COCO license.
  • ImageNet-21k is publicly available for research and non-commercial use under a CC BY-NC license.
  • Table 15 summarizes the dataset licenses used in the experiments.
Loading 2207.03482v3…