PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, Yin Cui

arXiv:2104.13921v3cs.CVcs.AIcs.LG

TL;DR

Open-vocabulary object detection seeks to recognize arbitrary text-described categories despite limited and costly detection annotations. ViLD distills text and image knowledge from a pretrained open-vocabulary classifier into a two-stage detector, achieving 16.1 AP on novel LVIS categories and transferring directly across datasets.

  • Problem

    Open-vocabulary detection aims to recognize text-described novel categories using only annotations for base categories, while expanding detection vocabularies requires costly long-tailed data collection.

  • Method

    ViLD distills text and image embeddings from a pretrained open-vocabulary image classifier into a two-stage detector with class-agnostic localization modules.

  • Results

    ViLD transfers directly to three detection datasets and outperforms the previous COCO open-vocabulary detector by 4.8 novel AP.

  • Takeaways & Limitations

    ViLD provides a scalable alternative to collecting expensive detection annotations for long-tailed categories.

  • Takeaways & Limitations

    The interactive attribute and fine-grained category analysis assumes category membership and attribute presence are conditionally independent given the region embedding.

Abstract

from arXiv · show

We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask AP$_r$ with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 AP$_r$. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP$_{50}$ on PASCAL VOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the-art by 4.8 on novel AP and 11.4 on overall AP. Code and demo are open-sourced at https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.

1 INTRODUCTION

The paper targets open-vocabulary object detection using only base-category detection annotations, addressing the limits of expanding detection vocabularies through increasingly large labeled datasets. ViLD uses pretrained vision-language representations to classify generalized object proposals and reports strong novel-category and cross-dataset performance.

  • Problem: Open-vocabulary detection aims to recognize novel categories described by text inputs using only detection annotations for base categories.The motivating example expands from a trained category such as “toy” to a novel category such as “toy elephant.”
  • Motivation: Existing detectors generally recognize dataset categories, while expanding vocabulary typically requires collecting images with more labeled categories.LVIS exemplifies this scaling effort with 1,203 categories.
  • Motivation: Abundant paired image-text data enable pretrained vision-language models to transfer to arbitrary text categories, but object-level open-vocabulary representations remain challenging.A joint vision-language model trained on 400 million image-text pairs transferred directly to over 30 datasets.
  • Approach: ViLD decomposes detection into generalized object proposal and open-vocabulary image classification, using a base-category region proposal model and a pretrained classifier on cropped proposals.The proposals may contain both base and novel categories.
  • Approach: ViLD distills text and image embeddings from an open-vocabulary image classification teacher into a two-stage detector for classifying detected regions.ViLD-text derives category embeddings from the pretrained text encoder, while ViLD-image uses teacher-inferred image embeddings.
  • Results: 16.1 AP for novel categories on LVIS exceeds the supervised counterpart by 3.8, while ALIGN raises novel AP to 26.3.The ALIGN result is only 3.7 worse than the fully-supervised 2020 LVIS Challenge winner.

2 RELATED WORK

Related work expands visual-recognition vocabularies through zero-shot transfer and large-scale image-text models, while object-detection research addresses costly data scaling through unified labels, incremental learning, and zero-shot detection. Unlike image-level open-vocabulary recognition, this work focuses on detecting objects from arbitrary text inputs.

  • Increasing vocabulary in visual recognition: Zero-shot recognition transfers knowledge to unseen categories using visual attributes, word embeddings, or knowledge graphs.Early methods represent categories with attribute-based binary codebooks, while later work incorporates word embeddings and knowledge graphs.
  • Increasing vocabulary in visual recognition: CLIP and ALIGN scale image-text pretraining to million-scale pairs, enabling strong transfer across classification datasets.These models use contrastive learning to jointly train image-text representations.
  • Increasing vocabulary in visual recognition: Where prior image-text models emphasize image-level open-vocabulary recognition, this work targets object detection with arbitrary text inputs.The distinction is explicitly drawn between image-level recognition and object-level detection.
  • Increasing vocabulary in object detection: Large-vocabulary object detection is addressed through multi-dataset label unification, incremental learning of unknown categories, and zero-shot detection.Zero-shot detection methods commonly align region features with pretrained text embeddings for base categories.

3 METHOD

ViLD adapts two-stage detection for open-vocabulary recognition by using class-agnostic localization and distilling pretrained vision-language embeddings into detector region embeddings. Its text and image distillation objectives share text-based inference, while ensembling strategies address performance and speed tradeoffs.

  • Localizing novel objects: Class-agnostic box regression and mask prediction replace class-specific localization modules, enabling proposals for both base and novel categories.Only annotations from base categories CB are used for training.
  • Text and image embeddings: Text embeddings are generated offline with prompt templates and synonym ensembling, while inference compares region and text embeddings using cosine similarity, softmax, and per-class NMS.During inference, the text encoder can represent CB ∪ CN or novel categories CN only.
  • ViLD-text: ViLD learns lightweight region embeddings from detector features so proposals can be classified directly against pretrained text embeddings.The region embedding is taken from the detector output before its classification layer.
  • ViLD-image: ViLD-image distills pretrained image-encoder embeddings into detector region embeddings using precomputed proposal features and an L1 loss.Offline proposals may contain objects from both CB and CN, unlike ViLD-text training, which uses only CB annotations.
  • Combined objectives: The total ViLD loss is a weighted sum of text cross-entropy and image-embedding distillation, but image distillation is used only during training.At inference, ViLD-image, ViLD-text, and ViLD use the same text embeddings and open-vocabulary detection architecture.
  • Model ensembling: Ensembling combines ViLD-text predictions with the teacher model or learns separate text- and image-oriented embedding heads to improve performance and mitigate slow inference.The teacher ensemble uses top k candidate regions, whereas separate heads address contention between the two training losses.

4 EXPERIMENTS

Experiments show that ViLD can detect novel categories without novel-category training, benefits from combining text and visual distillation and stronger teachers, and transfers across datasets. Qualitative studies further demonstrate interactive fine-grained recognition and systematic vocabulary expansion with attributes.

  • LVIS benchmark: LVIS uses 866 frequent/common base categories and 337 rare novel categories among 1,203 total categories, with rare-category AP as the main metric.The benchmark holds out all rare categories during training.
  • Novel-category localization: ∼2 AR separates base-only from base-plus-novel proposal training at 100, 300, and 1000 proposals, showing localization remains possible without novel-category training.The comparison uses Mask R-CNN region proposal networks with a ResNet-50 backbone.
  • Ablations: 10.1 APr for ViLD-text versus 3.0 APr for GloVe embeddings demonstrates the value of image-jointly-trained text representations.ViLD-text uses CLIP text embeddings, while the comparison uses GloVe embeddings pretrained on text-only data.
  • Ablations: 11.2 APr and 11.2 overall AP are achieved by ViLD-image without any object-category labels, demonstrating the effectiveness of visual distillation.ViLD-image distills CLIP image embeddings from cropped region proposals.
  • Ablations: 6.0 higher APr than ViLD-text and 4.9 higher APr than ViLD-image show that combining text classification and visual distillation improves novel-category performance.Separate-head ensembling further reduces the competition between the two components, mainly improving APc and APr.
  • COCO benchmark: 4.8 Novel AP improvement over Zareian et al. (2021) establishes ViLD’s stronger COCO benchmark performance under the generalized zero-shot setup.The comparison uses box AP at an IoU threshold of 0.5 with a ResNet-50 backbone.
  • Vocabulary expansion: p×q vocabularies can be formed from p categories and q attributes, enabling successful detection of fruits with color attributes.The expansion assumes category and attribute events are conditionally independent given the region embedding.

5 CONCLUSION

ViLD distills knowledge from open-vocabulary image classification models into open-vocabulary object detection and is evaluated on LVIS. It achieves 16.1 AP for novel categories with ResNet50, rising to 26.3 novel AP with ALIGN, while supporting systematic vocabulary expansion.

  • 5 CONCLUSION: ViLD distills knowledge from open-vocabulary image classification models for open-vocabulary object detection.The passage presents ViLD as an open-vocabulary detection method based on knowledge distillation.
  • 5 CONCLUSION: ViLD is the first open-vocabulary detection method evaluated on the challenging LVIS dataset.
  • 5 CONCLUSION: 16.1 AP for novel categories on LVIS is achieved with a ResNet50 backbone, surpassing the supervised counterpart at the same inference speed.
  • 5 CONCLUSION: 26.3 novel AP is achieved with the stronger ALIGN teacher model.
  • 5 CONCLUSION: 11× vocabulary expansion adds 11 color attributes to LVIS categories, enabling correct color assignment to fruit detections.The examples use class-agnostic NMS with threshold 0.9 and show the top 15 predictions.
  • 5 CONCLUSION: The detector trained on LVIS performs fine-grained detection of 200 bird species in CUB-200-2011 but fails on visually non-distinctive species.It incorrectly assigns “Western Gull” to “Horned Puffin” due to visual similarity.

ETHICS STATEMENT

The paper studies open-vocabulary object detection using knowledge distillation and relies on public datasets and publicly available or published pretrained models.

  • ETHICS STATEMENT: The experiments use public datasets and pretrained models that are publicly available or introduced in published papers.The method applies knowledge distillation, a technique used extensively in computer vision and natural language processing.

APPENDIX · A ADDITIONAL QUALITATIVE RESULTS

The appendix qualitatively shows that ViLD transfers effectively from LVIS to PASCAL VOC without finetuning, including detection of tiny objects in lower-resolution images. It also documents failures involving missed or visually confused novel objects and masks driven by low-level appearance rather than semantics.

  • A ADDITIONAL QUALITATIVE RESULTS: ViLD transfers effectively from LVIS to PASCAL VOC without finetuning.The qualitative results use the PASCAL VOC 2007 test set.
  • A ADDITIONAL QUALITATIVE RESULTS: On PASCAL VOC, ViLD correctly detects objects despite images usually having lower resolution than LVIS.
  • A ADDITIONAL QUALITATIVE RESULTS: ViLD detects tiny bottles in one PASCAL VOC image but fails to detect the person.
  • A ADDITIONAL QUALITATIVE RESULTS: A common LVIS failure on novel categories is missing the novel object entirely, such as an undetected elevator car.
  • A ADDITIONAL QUALITATIVE RESULTS: A less common failure misclassifies part of a novel object, such as detecting half of a waffle iron as a calculator.The example is attributed to visual similarity.
  • A ADDITIONAL QUALITATIVE RESULTS: PASCAL VOC mask prediction sometimes appears to rely on low-level appearance rather than semantics.This is presented as a failure case of mask prediction.

B ANALYSIS OF CLIP ON CROPPED REGIONS

The analysis identifies four common CLIP failure modes on cropped regions: visual similarity, extreme aspect ratios, multiple objects within a box, and confidence scores that do not reflect localization quality. It also reports that replacing ViT-B/32 CLIP with ALIGN improves detection performance and that class-agnostic masks can fail due to low-level appearance.

  • Visual similarity: Visual similarity causes confusion on large vocabularies, while replacing ViT-B/32 CLIP with EfficientNet-l2 ALIGN increases AP.The analysis attributes this limitation partly to the relatively small CLIP model and suggests higher-capacity pretrained models as a mitigation.
  • Mask quality: Class-agnostic mask prediction can produce poor-quality masks by following low-level appearance rather than semantics, failing to obtain a complete instance mask.Figure 10 illustrates this failure on PASCAL VOC.
  • Stronger classification models: ALIGN on cropped regions achieves superior APr and demonstrates that a stronger open-vocabulary classification model can improve detection performance by a large margin.Table 6 reports box APs for this comparison.
  • Aspect ratio: Fixed 224×224 preprocessing can distort proposals with extreme aspect ratios or tiny sizes because proposals are directly resized to that resolution.CLIP normally resizes the shorter image edge and center-crops, whereas region proposals may differ substantially from its training images.
  • Multiple objects in a bounding box: Multiple objects within one bounding box interfere with classification because CLIP image-caption pretraining emphasizes salient objects in entire images.The analysis notes that this issue is difficult to mitigate at the open-vocabulary classification model’s end.
  • Confidence scores and localization: CLIP confidence scores do not reflect localization quality, so correctly classified objects may receive their highest scores on partial detection boxes.The analysis re-scores proposals using the geometric mean of CLIP confidence and objectness from the proposal model.

C ADDITIONAL QUANTITATIVE RESULTS

Additional experiments show that visual distillation benefits novel-category performance, deeper backbones improve all metrics, and prompt ensembling provides a small APr gain. The visual-distillation trade-off favors APr at the expense of APc and APf, while ViLD-ensemble performs best across box and mask APr.

  • Hyperparameter sweep for visual distillation: The visual-distillation trade-off improves APr at the expense of APc and APf, suggesting competition between ViLD-text and ViLD-image.Additional learning from image embeddings generally improves novel-category performance compared with no distillation.
  • Hyperparameter sweep for visual distillation: L1 loss is better than L2 loss, and increasing visual-distillation weight raises APr while lowering APf,c.Across all parameter combinations, ViLD outperforms ViLD-text on APr under the shorter 84,375-iteration training setup with a ResNet-50 backbone.
  • Box APs and ResNet-152 backbone: ResNet-152 improves all metrics, with relative performance trends consistent across box and mask AP and different backbones.ViLD-ensemble achieves the best box and mask APr, while box AP is generally slightly higher than mask AP.
  • Ablation study on prompt engineering: Ensembling synonyms and 63 prompt templates slightly improves APr by 0.4 compared with a single prompt template.The single-template comparison uses “a photo of {article} {category}”.

D MORE IMPLEMENTATION DETAILS

The implementation details specify ViLD-ensemble’s dual-head design, proposal and suppression settings for R-CNN-style experiments, and shared class-agnostic proposal machinery across ViLD variants. They also describe stronger-teacher configurations and prompt ensembling for open-vocabulary classification.

  • ViLD-ensemble architecture: ViLD-ensemble uses two separate heads with identical architecture to reduce competition between ViLD-text and ViLD-image objectives, then ensembles their inference results.Its learning objectives are otherwise similar to ViLD.
  • R-CNN style experiments: R-CNN-style experiments generate up to 1000 proposals using class-agnostic NMS at 0.9, then retain the top 300 detections after class-specific NMS at 0.6.The objectness score is one minus the background score, and refined second-stage boxes are treated as proposals for mask AP reporting.
  • ViLD variants: All ViLD variants use standard two-stage Mask R-CNN with class-agnostic localization modules, obtaining both offline M proposals and online N proposals from the first-stage RPN.R-CNN-style methods and ViLD variants share the concept of class-agnostic object proposals.
  • Stronger teacher models: Experiments with CLIP ViT-L/14 and ALIGN use EfficientNet-b7 backbones, ViLD-ensemble, P3-only RoI features, and reduced large-scale jittering of [0.5, 2.0].For CLIP ViT-L/14, the Faster R-CNN head and FPN dimensions are increased to 1,024 and 512; ALIGN uses EfficientNet-l2 and BERT-large as teacher encoders.
  • Text prompts: Category names are inserted into an ensemble of 63 prompt templates, including detection-oriented prompts containing the phrase “in the scene”.The templates adapt models trained on full image-text sentences to object detection.
Loading 2104.13921v3…