PaperlayerView exact revision on arXiv

Source-linked AI summary

DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection

Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, Hang Xu

arXiv:2209.09407v2cs.CV

TL;DR

Open-world detection must recognize arbitrary object categories despite limited, heterogeneous annotations and inefficient category interaction in prior grounding formulations. DetCLIP parallelizes concept processing across detection, grounding, and image-text data, while a description-rich dictionary enriches concepts, supplies negatives, and completes labels. On LVIS, DetCLIP-T improves mAP by 9.9% over GLIP-T and shows gains on total and rare categories against an equally backed fully supervised baseline.

  • Problem

    Open-world detection is limited by scarce rare-category examples, heterogeneous label spaces, partial annotations, and inefficient sequential category interaction.

  • Method

    DetCLIP parallelizes separate concept encoding across detection, grounding, and image-text data and uses a description-rich concept dictionary for enrichment, negative sampling, and label completion.

  • Results

    9.9% mAP improvement over GLIP-T on LVIS without using LVIS images during pretraining, with 2.3% total and 13.5% rare-class gains over an equally backed ATSS model.

  • Takeaways & Limitations

    DetCLIP supports open-domain detection by unifying heterogeneous supervision and enriching concepts with external knowledge, improving both common and rare LVIS categories.

  • Takeaways & Limitations

    Prior open-domain approaches can depend on costly cropped-region extraction, limited caption phrases, or pretrained vision-language model performance.

Abstract

from arXiv · show

Open-world object detection, as a more general and challenging goal, aims to recognize and localize objects described by arbitrary category names. The recent work GLIP formulates this problem as a grounding problem by concatenating all category names of detection datasets into sentences, which leads to inefficient interaction between category names. This paper presents DetCLIP, a paralleled visual-concept pre-training method for open-world detection by resorting to knowledge enrichment from a designed concept dictionary. To achieve better learning efficiency, we propose a novel paralleled concept formulation that extracts concepts separately to better utilize heterogeneous datasets (i.e., detection, grounding, and image-text pairs) for training. We further design a concept dictionary~(with descriptions) from various online sources and detection datasets to provide prior knowledge for each concept. By enriching the concepts with their descriptions, we explicitly build the relationships among various concepts to facilitate the open-domain learning. The proposed concept dictionary is further used to provide sufficient negative concepts for the construction of the word-region alignment loss\, and to complete labels for objects with missing descriptions in captions of image-text pair data. The proposed framework demonstrates strong zero-shot detection performances, e.g., on the LVIS dataset, our DetCLIP-T outperforms GLIP-T by 9.9% mAP and obtains a 13.5% improvement on rare categories compared to the fully-supervised model with the same backbone as ours.

1 Introduction

Open-world detection seeks recognition and localization beyond fixed category sets, but limited labels, rare-category data, and heterogeneous supervision hinder progress. DetCLIP addresses these challenges with parallel concept processing and a dictionary that enriches concepts and supervision, achieving strong LVIS improvements.

  • Motivation: Open-world detection aims to recognize and localize objects described by arbitrary category names, beyond predefined detection categories.Existing datasets contain limited category coverage and require costly annotations, especially for rare classes.
  • Motivation: Image-text data offer abundant supervision, but prior approaches rely on costly cropped-region extraction or limited caption phrases.These approaches also inherit discrepancies between image-level and instance-level features or depend on pretrained vision-language performance.
  • DetCLIP: GLIP concatenates all detection category names into one prompt, creating inefficient category interactions and input-length constraints.DetCLIP instead extracts concepts separately and feeds them to the text encoder in parallel.
  • DetCLIP: DetCLIP’s concept dictionary enriches concepts with descriptions, models relationships across heterogeneous label spaces, supplies negative concepts, and completes missing labels.The dictionary is built from online resources and large-scale detection datasets.
  • Results: 9.9% mAP improvement over GLIP-T on LVIS is achieved without using LVIS images during pretraining.DetCLIP-T also improves total LVIS performance by 2.3% and rare-class performance by 13.5% over an equally backed ATSS model trained on LVIS.

2 Related Work

Related work spans vision-language pretraining, conventional predefined-category detection, and zero-shot or open-vocabulary detection. DetCLIP differs by targeting open-domain recognition while integrating heterogeneous supervision for detection.

  • Vision-Language Pre-training: Vision-language pretraining learns cross-modal representations from large-scale image-text pairs, but commonly targets whole-image retrieval or generation rather than object detection.Such pretrained models therefore cannot be directly applied to object detection.
  • General Object Detection: General object detectors use predefined categories, whereas DetCLIP seeks recognition of a wide range of concepts and arbitrary categories.The related methods include one-stage, two-stage, and transformer-based detectors.
  • Zero-shot and Open-Vocabulary Detection: Zero-shot object detection traditionally generalizes from known training categories to unknown inference categories using semantic features or knowledge graphs.This setting differs from open-world detection’s broader goal of detecting arbitrary concepts.

3 The Proposed Approach

DetCLIP unifies heterogeneous detection, grounding, and image-text data through parallel concept inputs and a dictionary that enriches concepts with descriptions and negative samples. This design supports more efficient open-world pre-training and addresses inconsistent labels and partial annotations.

  • 3 The Proposed Approach: DetCLIP uses hybrid supervision from detection, grounding, and image-text pair data for open-world detection pre-training.The framework is designed to support heterogeneous supervision through a unified training pipeline.
  • 3.1 Paralleled Concept Formulation: Paralleled formulation feeds each concept separately to the text encoder instead of concatenating all category names into one sentence.Grounding phrases are extracted for bounding boxes, while detection categories are handled as individual inputs.
  • 3.1 Paralleled Concept Formulation: 27.8% v.s. 23.7%: paralleled formulation learns faster and achieves better performance than the alternative formulation.The comparison is reported in Figure 4 as an efficiency and performance result.
  • 3.2 Concept Dictionary: The concept dictionary forms a unified concept space by linking concepts across datasets through definitions and prior knowledge.It aggregates noun phrases and category names from image-text, detection, and manually collected concept sources.
  • 3.2.1 Constructing the Concept Dictionary: The dictionary contains about 14k concepts with definitions and can be updated by adding new concepts and corresponding definitions.These definitions explicitly provide relationships among concepts from different data sources.
  • 3.2.2 Knowledge Enrichment with Concept Dictionary: DetCLIP samples dictionary concepts as negatives for alignment and uses dictionary concepts to complete missing labels in image-text pair data.This targets both missing negative annotations and partial positive annotations in grounding and image-text pair datasets.

4 Experimental Results

Experiments evaluate DetCLIP on LVIS and downstream datasets, showing strong detection performance, efficiency, and gains from its formulation and concept dictionary.

  • Open-world Detection Results: DetCLIP-T achieves 35.9% AP on LVIS when trained with detection, grounding, and image-text pair data.Adding heterogeneous training data improves DetCLIP-T from 28.8% AP to 35.9% AP.
  • Open-world Detection Results: 9.9% mAP improvement over GLIP-T is achieved by DetCLIP-T on LVIS, despite using fewer epochs and less training data.DetCLIP-T also exceeds GLIPv2-T by 6.9%.
  • Efficiency Comparison: 2.3 FPS versus 0.12 FPS is achieved by DetCLIP-T and GLIP-T, respectively, during LVIS inference on a single V100.GLIP-T requires about 10.7K GPU hours for training, approximately five times DetCLIP-T’s training time.
  • Qualitative Visualizations: DetCLIP-T produces more complete and precise LVIS predictions than GLIP-T, especially for rare classes.Concept-dictionary label completion adds categories absent from captions, supports finer-grained labels, and alleviates caption label noise.
  • Impact of Concept Dictionary’s Size: Larger concept dictionaries provide richer negative concepts and improve discriminative feature learning.The dictionaries combine category names from Objects365, Things, and noun phrases extracted from YFCC100m.

5 Conclusion

The conclusion presents DetCLIP as an open-world detection pre-training framework that combines parallel concept processing with dictionary-based knowledge enrichment. Experiments report improved performance across common and rare LVIS categories and multiple downstream datasets.

  • 5 Conclusion: DetCLIP unifies detection, grounding, and image-text supervision through a paralleled concept formulation.The framework is designed to improve open-domain ability and learning efficiency.
  • 5 Conclusion: The concept dictionary imports external knowledge to improve discovery and coverage of novel concepts.Its uses improve open-world detection results on both common and rare LVIS categories.
  • 5 Conclusion: Experiments on multiple downstream detection datasets suggest that DetCLIP is more powerful than current state-of-the-art open-world detectors such as GLIP.

A Negative Impacts and Limitations

The paper reports no ethical risk from dataset usage or privacy violation because the benchmarks are publicly available and transparent, while localization remains constrained by bounding-box annotations.

  • Potential Negative Social Impact: The authors report no ethical risk from dataset usage or privacy violation because all benchmarks are publicly available and transparent.
  • Limitations and Future Works: Localization ability remains limited by the quality of bounding-box annotations.The authors suggest adding more weak supervision and extending evaluation to larger Internet image-text datasets.

B Dataset Details

The appendix describes pseudo-label generation for YFCC image-text pairs, compares DetCLIP and GLIP training data, and documents downstream dataset prompts.

  • Pseudo Labeling on Image-Text Pair Data: YFCC pseudo labels are generated by applying an Objects365-pretrained RPN to extract and filter object proposals.Boxes with objectness below 0.3 or area below 6000 are removed before further category prediction and sampling.
  • Pseudo Labeling on Image-Text Pair Data: CLIP image and text encoders score normalized cropped proposals against category embeddings during pseudo-label generation.A second-stage filter removes proposals with classification scores below 0.24, and 1M images are sampled for training.
  • Training Data Comparison: DetCLIP-T uses less than half the training data of GLIP-T, while DetCLIP-L uses less than 10% of GLIP-L’s training data.
  • Training Data Comparison: The training-data comparison distinguishes O365 versions, four detection datasets, grounding data, and web-crawled image-text datasets.Dataset volumes are reported in the corresponding table.
  • Downstream Dataset Prompts: Manually designed prompts are provided for six downstream detection datasets.

C More Results on LVIS and 13 Detection Datasets

DetCLIP shows strong zero-shot transfer across LVIS evaluations and compares favorably with GLIP and VILD under their respective protocols.

  • LVIS and 13 Detection Datasets: DetCLIP models dominate their GLIP counterparts in most LVIS full-validation cases despite using much less training data.The exception is APf on LVIS for DetCLIP-L.
  • LVIS and 13 Detection Datasets: Zero-shot transfer performance is also detailed across 13 detection datasets.Table 8 reports the detailed AP results for DetCLIP on these datasets.
  • LVIS and 13 Detection Datasets: 4.8% mAP separates DetCLIP from VILD under the VILD protocol, with DetCLIP reaching 27.3 mAP versus VILD’s 22.5 mAP.The comparison uses LVIS-base training and includes GoldG and YFCC1M.

D Ablation Studies

The ablations support parallel concept inputs and concept definitions as important design choices, while also identifying training techniques that stabilize DetCLIP.

  • Ablation Studies: +1.4% LVIS minival zero-shot transfer AP follows randomly shuffled grounding word order, supporting parallel noun-phrase inputs over sequential context.DetCLIP therefore drops context information and feeds each noun phrase separately.
  • Ablation Studies: Adding class definitions during inference significantly improves zero-shot transfer performance.This experiment tests concept enrichment by comparing text inputs with and without definitions.
  • Ablation Studies: 28.8 AP with FILIP exceeds 28.3 AP without a language model for dictionary retrieval, whereas RoBERTa performs similarly to no language model.The comparison evaluates pretrained language models used to retrieve definitions for concepts absent from WordNet.
  • Ablation Studies: A small learning rate for the pretrained language backbone preserves learned language knowledge, while removing regression loss for non-detection data reduces localization error impact.These are reported as important training techniques for vision-language pretraining.

E Qualitative Results

Qualitative analyses show that the concept dictionary enriches pseudo labels and expands the semantic coverage of concepts beyond original captions and dataset names.

  • Qualitative Results: Concept-dictionary pseudo labels are finer-grained and higher quality than labels generated from original captions.The dictionary alleviates the partial-label problem in image-text pair data.
  • Qualitative Results: Dictionary retrieval finds proper synonyms for class names absent from the dictionary using semantic similarity from a pretrained language model.Exact matches use stored definitions; otherwise, the closest concept is retrieved semantically.
  • Qualitative Results: Concepts from image-text pairs cover both fine-grained categories such as cotswold and cuniculus paca and broader classes such as giant and cathedral.These examples illustrate the dictionary’s wider semantic range.
Loading 2209.09407v2…