PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Vocabulary DETR with Conditional Matching

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, Chen Change Loy

arXiv:2203.11876v2cs.CVcs.AI

TL;DR

Open-vocabulary detectors must recognize novel classes despite lacking labeled training images, ideally from either text or exemplar-image queries. OV-DETR uses conditional binary matching in an end-to-end DETR framework with vision-language embeddings, achieving consistent improvements on LVIS and COCO. Its main limitation is slower inference when many classes must be detected because conditional decoding requires multiple forward passes.

  • Problem

    Object detectors are typically limited to fixed training vocabularies, while novel-class matching requires labels that are unavailable and collecting new labeled images is costly.

  • Method

    OV-DETR conditions the DETR Transformer decoder on CLIP-derived text or image queries and trains binary matching between each query and its corresponding object.

  • Results

    OV-DETR consistently improves performance on LVIS and COCO, achieving 17.4 mask mAP on novel LVIS classes and 29.4 box mAP on novel COCO classes.

  • Takeaways & Limitations

    Conditional matching turns an end-to-end Transformer detector into an open-vocabulary system that supports both natural-language and exemplar-image queries.

  • Takeaways & Limitations

    Inference becomes slow when detecting many classes, such as 1,203 on LVIS, because the conditional design requires multiple Transformer-decoder forward passes.

Abstract

from arXiv · show

Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR -- hence the name OV-DETR -- which, once trained, can detect any object given its class name or an exemplar image. The biggest challenge of turning DETR into an open-vocabulary detector is that it is impossible to calculate the classification cost matrix of novel classes without access to their labeled images. To overcome this challenge, we formulate the learning objective as a binary matching one between input queries (class name or exemplar image) and the corresponding objects, which learns useful correspondence to generalize to unseen queries during testing. For training, we choose to condition the Transformer decoder on the input embeddings obtained from a pre-trained vision-language model like CLIP, in order to enable matching for both text and image queries. With extensive experiments on LVIS and COCO datasets, we demonstrate that our OV-DETR -- the first end-to-end Transformer-based open-vocabulary detector -- achieves non-trivial improvements over current state of the arts.

1 Introduction

OV-DETR addresses the difficulty of detecting novel classes without labeled images by replacing conventional proposal-based detection with conditional matching in an end-to-end Transformer detector. It accepts either class-name text or exemplar-image queries and improves novel-class performance on LVIS and COCO.

  • Motivation: Fixed-vocabulary detectors cannot detect classes beyond their training categories, while adding labeled data for novel classes is costly and inefficient.COCO-trained detectors, for example, are limited to 80 classes.
  • Motivation: Existing open-vocabulary detectors align detector features with CLIP-like language or vision-language embeddings but often depend on unreliable region proposals for novel classes.Binary region proposal networks can overfit to seen classes and fail to cover novel objects.
  • OV-DETR: OV-DETR is an end-to-end DETR-based detector trained without an intermediate RPN to detect objects specified by class names or exemplar images.Conditional queries provide the Transformer decoder with text or image embeddings from a pre-trained vision-language model.
  • Conditional Matching: OV-DETR reformulates training as binary matching between conditional input queries and corresponding objects, enabling correspondence to generalize to unseen queries.This avoids calculating novel-class classification costs that require unavailable training labels.
  • Results: 17.4 mask mAP on novel LVIS classes and 29.4 box mAP on novel COCO classes surpass state-of-the-art methods by 1.3 and 1.8 mAP, respectively.The experiments report consistent improvements across the two open-vocabulary benchmarks.

2 Related Work

Prior open-vocabulary detectors use language models, feature alignment, distillation, or synthesized novel-class representations, while visual grounding addresses a narrower single-object localization problem. OV-DETR relates to MDETR through conditional DETR processing but targets generic open-vocabulary detection by addressing novel-class matching.

  • Open-Vocabulary Detection: OVR-CNN pre-trains Faster R-CNN with BERT on image-caption pairs, while ViLD aligns Mask R-CNN features with CLIP to synthesize novel-class classification weights.Other approaches use prompt tuning, GAN-generated features, or data augmentation to represent unseen classes.
  • Visual Grounding: Visual grounding localizes a particular object from natural language, whereas open-vocabulary detection aims to identify all target objects in an image.This difference prevents typical visual grounding methods from directly applying to generic object detection.
  • MDETR and OV-DETR: MDETR conditionally combines visual and textual features in DETR, but OV-DETR addresses open-vocabulary detection by bypassing classification-cost computation for novel classes.MDETR links DETR outputs with specific words but is not applicable to novel classes under the classification framework.

3 Open-Vocabulary DETR

OV-DETR reformulates DETR for open-vocabulary detection by conditioning its decoder on text or exemplar-image embeddings and matching conditional inputs to detected objects. It replaces closed-set classification matching with class-agnostic binary matchability, using parallel conditional queries to support flexible inference.

  • Overview: OV-DETR extends DETR to detect objects specified by arbitrary text inputs or exemplar images.The method uses conditional inputs from a pre-trained CLIP model.
  • Conditional Matching: OV-DETR reformulates fixed set matching as conditional binary matching between text or image queries and detection outputs.The binary objective measures whether a conditional input and a detection result are matched or not matched.
  • Challenge: Standard DETR cannot compute novel-class classification costs because their labeled categories are unavailable.Its bipartite matching requires class labels for the predictions and ground-truth annotations.
  • Conditional Inputs: CLIP supplies aligned text and image embeddings that condition the DETR decoder for training and enable arbitrary query inputs at test time.Training randomly selects text or image embeddings for balanced conditioning, while novel-class proposals use image embeddings when class names are unavailable.
  • Parallel Conditioning: The conditioned decoder copies object queries and processes multiple conditional inputs in parallel, producing N × R matching queries.An attention mask keeps query copies independent during conditioning.
  • Matching Objective: The binary matching cost combines a matchability loss with bounding-box localization loss.Matchability is represented as a one-dimensional sigmoid probability for matched versus not matched.

4 Experiments

Experiments evaluate OV-DETR on open-vocabulary LVIS and COCO benchmarks, including ablations, transfer, qualitative behavior, and inference speed. The method improves novel-class detection while supporting text and image queries, but inference slows as the number of classes grows.

  • Datasets: LVIS uses 1,203 classes, with 337 rare classes treated as novel and frequent/common classes used for training.
  • Ablation Studies: Conditional binary matching raises OV-LVIS novel APm from 9.5 to 17.4, showing its importance for open-vocabulary DETR training.
  • Generalization: OV-DETR improves transfer performance over ViLD when trained on LVIS and evaluated on PASCAL VOC and COCO by replacing category text embeddings.
  • Qualitative Results: Qualitative results show accurate localization and segmentation for novel objects using text queries, image queries, or mixtures, including visually different exemplar images.
  • Inference Time Analysis: With 1,203 LVIS classes, vanilla OV-DETR is 16× slower than Deformable DETR; parallel conditional queries reduce inference time by nearly 60%.

5 Conclusion

The paper presents OV-DETR as an end-to-end Transformer open-vocabulary detector using conditional matching and pre-trained vision-language models. It reports significant improvements over current state-of-the-art two-stage detectors.

  • OV-DETR turns an end-to-end Transformer detector into an open-vocabulary detector through conditional matching and pre-trained vision-language models.
  • The method addresses open-vocabulary detection, which is challenging because unseen classes lack training data.
  • OV-DETR significantly outperforms current state-of-the-art methods based on two-stage detectors despite using a simplified training pipeline.

A More Qualitative Results of OV-DETR

Additional qualitative experiments examine novel-class detection, generalization to anime characters, and failure cases. The results support novel-class activation and image-query generalization while exposing robustness limitations.

  • Open-Vocabulary COCO: OV-DETR shows higher activation values on novel-class objects than the RPN used by ViLD in Open-Vocabulary COCO visualizations.
  • Web Images: OV-DETR successfully detects regions matched with conditional image queries for anime characters absent from training.
  • Failure Cases: Detecting small or occluded objects with conditional image queries is hard for OV-DETR.
  • Failure Cases: The method is not robust to unrelated out-of-distribution text queries.

B Discussion of Object Proposals

The discussion compares OV-DETR's object proposals with ViLD's and reports similar novel-category recall, while ablations investigate object-query and copy counts.

  • Discussion of Object Proposals: ViLD trains an RPN on base classes to generate M pre-computed proposals that may contain novel-class objects.
  • Discussion of Object Proposals: OV-DETR uses a Def-DETR architecture instead of ViLD's RPN to generate object proposals.
  • Discussion of Object Proposals: 48.3 for ViLD and 47.6 for ours are the reported AR@300 values for novel categories, indicating similar high top-300 averaged recall.
  • Discussion of Object Proposals: The ablation study on N varies the number of object queries, while the study on R varies the number of copies.

C Importance of Lembed

OV-DETR includes an embedding reconstruction head for predicting the conditional input embedding, and Table 6 evaluates its loss.

  • Importance of Lembed: The embedding reconstruction head predicts the conditional input embedding z_text or z_image.
  • Importance of Lembed: The reconstruction head is optimized using the loss L_embed.
  • Importance of Lembed: Table 6 reports the efficacy of L_embed on LVIS.

D Importance of Multiple Queries for Training

The study evaluates cloning query features across different numbers of queries and repetitions, finding that three repetitions consistently help while larger settings face optimization or memory limits.

  • With N = 100, copying queries from R = 1 to 3 improves novel APm from 10.6 to 13.6.
  • With N = 100, increasing repetition to R = 9 slightly degrades novel APm, partly because of limited optimization capacity.
  • With N = 300, copying queries from R = 1 to 3 is beneficial, but R > 3 causes GPU out-of-memory errors.
  • The optimal configuration is N = 300 queries repeated R = 3 times.

E More Implementation Details

The implementation uses a ResNet50-C4 backbone, CLIP-based prompt ensembling, and qualitative evaluations spanning novel classes, anime characters, proposal use, and failure cases.

  • Implementation details: Models are trained on 8 Tesla V100 GPUs with ResNet50-C4 as the default backbone and specified loss weights for BCE, L1, GIoU, and embedding losses.The weights are LBCE = 3.0, LL1 = 5.0, LGIoU = 2.0, and Lembed = 1.0.
  • Text prompts: Each class name is wrapped in 63 prompt templates, whose CLIP text embeddings are averaged through prompt ensembling.An example template is “there is a {class name} in the photo”.
  • Qualitative evaluation: Qualitative COCO results visualize novel-class predictions and compare activation maps from OV-DETR with those from an RPN network.
  • Qualitative evaluation: Anime-character experiments use a model trained on LVIS to test matchability with conditional image queries.
  • Failure cases: Failure cases include missed small or occluded objects for image queries, partial improvement with text queries, and false positives for unrelated text queries.
  • Proposal comparison: Compared with ViLD, OV-DETR uses pre-computed object proposals to generate the conditional image query rather than applying proposal distillation.
Loading 2203.11876v2…