PaperlayerView exact revision on arXiv

Source-linked AI summary

DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment

Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Hang Xu

arXiv:2304.04514v1cs.CV

TL;DR

Open-vocabulary detection aims to recognize arbitrary categories, but prior approaches rely on pre-trained vision-language models or pseudo-labels and face limits in domain coverage or training scale. DetCLIPv2 jointly trains on detection, grounding, and image-text data, aligning words with regions. It reports stronger LVIS zero-shot results than prior detectors and uses more image-text pairs at similar training time.

  • Problem

    Open-vocabulary detection seeks to recognize arbitrary categories, while prior approaches rely on pre-trained vision-language models or pseudo-labeling and face domain-gap, label-quality, and scaling constraints.

    Source: §1 · pp. 1–2

  • Method

    DetCLIPv2 jointly trains on detection, grounding, and image-text data, using optimal matching-based region-word similarity for contrastive learning and low-resolution image-text inputs.

    Source: §1 · p. 2

  • Results

    40.4% zero-shot AP on LVIS with Swin-T surpasses GLIP/GLIPv2/DetCLIP by 14.4/11.4/4.5% AP, respectively; DetCLIPv2 also uses 13× more image-text pairs than DetCLIP with similar training time.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents scaling image-text pairs as a possible route toward open-world detection and reports broad domain coverage.

    Source: §5 · p. 8

    Internet-crawled captions describe only a fraction of image objects, and including image-to-text matching noticeably degrades performance.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

This paper presents DetCLIPv2, an efficient and scalable training framework that incorporates large-scale image-text pairs to achieve open-vocabulary object detection (OVD). Unlike previous OVD frameworks that typically rely on a pre-trained vision-language model (e.g., CLIP) or exploit image-text pairs via a pseudo labeling process, DetCLIPv2 directly learns the fine-grained word-region alignment from massive image-text pairs in an end-to-end manner. To accomplish this, we employ a maximum word-region similarity between region proposals and textual words to guide the contrastive objective. To enable the model to gain localization capability while learning broad concepts, DetCLIPv2 is trained with a hybrid supervision from detection, grounding and image-text pair data under a unified data formulation. By jointly training with an alternating scheme and adopting low-resolution input for image-text pairs, DetCLIPv2 exploits image-text pair data efficiently and effectively: DetCLIPv2 utilizes 13X more image-text pairs than DetCLIP with a similar training time and improves performance. With 13M image-text pairs for pre-training, DetCLIPv2 demonstrates superior open-vocabulary detection performance, e.g., DetCLIPv2 with Swin-T backbone achieves 40.4% zero-shot AP on the LVIS benchmark, which outperforms previous works GLIP/GLIPv2/DetCLIP by 14.4/11.4/4.5% AP, respectively, and even beats its fully-supervised counterpart by a large margin.

1. Introduction

DetCLIPv2 addresses the challenge of detecting arbitrary categories without costly instance-level annotations by jointly training on detection, grounding, and image-text data. It learns word-region alignment from image-text pairs and reports stronger open-vocabulary results while using substantially more such pairs at similar training time.

1. Introduction

  • Open-vocabulary detection targets arbitrary categories, which traditional detectors miss without costly additional instance-level bounding-box annotations, especially for rare classes.The introduction motivates this need with downstream scenarios such as autonomous driving, where accurate predictions across road-object classes are required.Source: §1 · p. 1
  • Prior image-text approaches rely on pseudo-boxes whose quality is limited by detectors trained on annotated concepts or by vision-language models with a domain gap.Using detection-resolution inputs for massive image-text datasets also imposes a large computational burden.Source: §1 · p. 2
  • DetCLIPv2 jointly trains on detection, grounding, and image-text data, using optimal matching-based region-word set similarity to guide contrastive learning without teacher-provided pseudo-labels.Alternating data types is described as a flywheel: localization supports region extraction, while image-text contrastive learning broadens concepts and can improve localization of unseen categories.Source: §1 · p. 2
  • Low-resolution inputs for image-text pairs reduce training computation because captions typically describe only the image’s main objects.Figure 2 contrasts VL-model distillation, pseudo-labeling, and the paper’s end-to-end joint-training paradigm.Source: §1 · p. 2, Figure 2 · p. 2
  • 40.4% zero-shot AP on LVIS with Swin-T surpasses GLIP/GLIPv2/DetCLIP by 14.4/11.4/4.5% AP, respectively.DetCLIPv2 also reports state-of-the-art fine-tuning performance on LVIS and ODinW13.Source: §1 · p. 2
  • Compared with DetCLIP, DetCLIPv2 exploits 13× more image-text pairs with similar training time; using vanilla ATSS and Swin-T, it reaches 40.4% zero-shot AP on LVIS, surpassing GLIP/GLIPv2/DetCLIP by 14.4/11.4/4.5 AP, respectively.DetCLIPv2 also achieves state-of-the-art fine-tuning performance on LVIS and ODinW13.Source: §1 · p. 2

2. Related Work

Related work spans general vision-language pre-training, open-vocabulary detection, and methods using unlabeled or weakly labeled data. DetCLIPv2 differs by targeting end-to-end word-region alignment from massive image-text pairs in an open-domain setting.

2. Related Work

  • CLIP [38] and ALIGN [23] use large-scale image-text pairs for cross-modal contrastive learning, while FILIP [54] introduces word-patch late interaction.The paper identifies adapting FILIP’s alignment idea to open-vocabulary detection as non-trivial.Source: §2 · p. 2
  • Open-vocabulary detectors transfer pre-trained vision-language knowledge [19] [49] [56] [59] or use image-text pairs through pseudo-labeling [16] [17] [22] [30] [53].XDETR [5] uses standard contrastive learning for image-to-text alignment, while Detic [60] assigns labels from max-size region proposals.Source: §2 · pp. 2–3
  • DetCLIPv2 instead targets end-to-end learning of word-region alignment from massive image-text pairs without a teacher model.The related work distinguishes this from approaches that transfer pre-trained VL knowledge or expand coverage through pseudo-labeling.Source: §2 · p. 3
  • SSOD [45] [46] [51] and WSOD [3] [10] use labeled data or image-level labels but assume predefined categories; DetCLIPv2 targets an open-domain setting.The paper frames its goal as learning unlimited concepts from massive image-text pairs.Source: §2 · p. 3

3. The Proposed Approach

DetCLIPv2 jointly trains on detection, grounding, and image-text data, using annotation-based alignment where instance labels exist and contrastive matching for image-text pairs.

3.1. A Unified Data Formulation

  • A shared sample format represents images, annotated boxes, and concepts; image-text pairs omit boxes and derive concepts from captions and extracted noun phrases.Detection and grounding boxes can be labeled with concepts, while image-text pairs have no instance-level annotations.Source: §3.1 · p. 3

3.2. Learning from Image-text Pairs

  • For each noun phrase, DetCLIPv2 selects its best-matching region and aggregates matches in text-to-image contrastive learning to learn fine-grained correspondences.The method is designed to learn word-region alignment without instance-level annotations for image-text pairs.Source: §3.2 · p. 4
  • Internet captions describe only a small fraction of image objects, leaving most proposals unmatched, so the method excludes image-to-text matching.The paper reports that including this matching can noticeably degrade performance.Source: §3.2 · p. 4
  • For image-text contrastive learning, region objectness is the maximum similarity to local-batch text concepts; top-k proposals are retained after NMS.This selects regions relevant to text and hard negative concepts described in other texts.Source: §3.2 · p. 5

3.3. Model Architecture and Training Objective

  • A lightweight deformable convolution routes classification-head gradients into the regression head, which receives no direct regression-branch supervision on image-text pairs.The design aggregates backbone features and is intended to help the regression head acquire localization ability.Source: §3.3 · p. 5
  • Losses are task-specific: detection combines alignment, regression, and centerness, grounding uses alignment, and image-text pairs use contrastive loss.Grounding omits localization loss because its bounding-box annotations are inaccurate.Source: §3.3 · p. 5

3.4. Joint Training

  • Training alternates among data types, using high-resolution, small-batch inputs for detection and grounding and low-resolution, large-batch inputs for image-text pairs.The image-text configuration increases contrastive negative samples and reduces training cost.Source: §3.4 · p. 5

4. Experimental Results

DetCLIPv2’s experiments test its training setup, image-text alignment choices, and scaling behavior. The results show that adding data improves detection performance while the framework uses substantially more image-text pairs at similar training time.

4.1. Implementation Details

  • Training uses O365 detection data, GoldG grounding data, and CC3M/CC12M image-text pairs, with 320 × 320 inputs for image-text pairs.The reported zero-shot evaluation uses LVIS with 1203 categories, and generalization is also evaluated on ODinW13’s 13 downstream tasks.Source: §4.1 · pp. 5–6

4.2.1 Ablations for Image-text Contrastive Learning

  • 31.3 AP is reached with batch-wise similarity and closest-region word matching; 100 proposals, τ = 0.5, and λ = 0.1 are the selected settings.Using image-to-text loss alone gives 29.8 AP, while adding centerness to batch-wise similarity lowers performance to 30.9 AP. The default λ = 1 and τ = 0.07 perform poorly.Source: §4.2.1 · p. 6

4.2.2 Effectiveness of Deformable Module

  • The deformable module improves weakly supervised grounding/image-text training but hurts strongly supervised detection, with negligible iteration-time cost.Source: §4.2.2 · p. 6, Table 3 · p. 7

4.2.3 Incorporating More Data Helps Learning

  • 40.4 overall AP is achieved with CC12M, surpassing the fully supervised method as training data is added across sources.Including CC3M raises overall AP from 28.6 to 31.3, with rare-category AP rising from 24.2 to 29.4 (+5.2 AP); GoldG raises overall AP to 38.4.Source: §4.2.3 · pp. 6–7, Table 4 · p. 7

4.2.4 Training Efficiency

  • 13× more image-text pairs are used than in DetCLIP at similar training time, with FPS increasing from 4.1 to 25.7.The authors attribute the efficiency design partly to low-resolution image-text inputs and a maximum text-token length of 16.Source: §4.2.4 · p. 7, Table 5 · p. 7

4.3. Main Results

DetCLIPv2 outperforms prior methods on zero-shot LVIS detection and transfers strongly to fine-tuning benchmarks. Its zero-shot results also exceed fully supervised counterparts in the reported comparisons.

4.3.1 Zero-shot Performance on LVIS

  • 40.4 AP with Swin-T exceeds GLIP-T/GLIPv2-T by 14.4/11.4 AP and DetCLIP by 4.5 AP on zero-shot LVIS; the Swin-L gain over DetCLIP is 6.1 AP.The zero-shot models also beat their fully supervised counterparts by 6.8/0.8 AP for Swin-T/Swin-L; rare-category improvements exceed 10 AP on both models.Source: §4.3.1 · pp. 7–8, Table 6 · p. 8

4.3.2 Transfer Results with Fine-tuning

  • On ODinW13, DetCLIPv2-T outperforms GLIP-T/GLIPv2-T by 3.1/1.5 average AP, while DetCLIPv2-L matches GLIPv2-H at 70.4 average AP.On LVIS fine-tuning without mask annotations, DetCLIPv2-T reaches 50.7 AP versus 50.6 AP for GLIPv2-T.Source: §4.3.2 · p. 8, Table 7 · p. 8

4.4. Visualizations and Analyses

The visualizations illustrate word-to-region alignment on image-text pairs, including cases beyond standard detection concepts. The analysis also reports localization-recall gains across object sizes when image-text pairs are included.

Visualizations of word-region alignment

  • The alignments cover unseen concepts, shifted image styles, and co-reference expressions, such as “juvenile” referring to “young bird.”Examples include parsley, sketch and cartoon images, and “curator” referring to a person.Source: §4.4 · p. 8

Learning from image-text pairs benefits localization

  • Adding image-text pairs improves LVIS average recall across small, medium, and large objects over IoU thresholds 0.5–0.95.Source: §4.4 · p. 8, Table 8 · p. 8

5. Conclusion

DetCLIPv2 uses best-matching region-word similarity to guide contrastive learning from image-text pairs. The authors report superior open-vocabulary performance and broad domain coverage, and propose scaling image-text pairs as a possible route to open-world detection.

5. Conclusion

  • DetCLIPv2 uses best-matching similarity between regions and words to guide its contrastive objective on massive image-text pairs.The conclusion presents this as a way to leverage image-text pairs for object detection.Source: §5 · p. 8
  • Experiments demonstrate superior open-vocabulary performance and broad domain coverage for DetCLIPv2.Source: §5 · p. 8
  • The authors propose scaling image-text pairs as a possible way to achieve open-world detection and leave further scaling to future work.Source: §5 · p. 8

A. Limitations

The authors identify three boundaries: localization still relies strongly on detection-data boxes, Internet image-text pairs are noisy and incomplete, and scaling those pairs beyond detection data may create imbalance.

A. Limitations

  • Localization capability still relies strongly on bounding-box annotations from detection data.Source: A. Limitations · p. 9
  • Noisy Internet image-text pairs with incomplete descriptions undermine the efficiency of word-region alignment.The authors point to improving data quality as a direction for further design.Source: A. Limitations · p. 9
  • Scaling image-text pairs until they overwhelm detection data may cause imbalanced training that hurts performance.The authors identify this as a potential issue for future exploration.Source: A. Limitations · p. 9

B. More Implementation Details

Pre-training uses AdamW with warm-up and cosine decay, while fine-tuning uses dataset-specific schedules. The implementation also reduces text-encoder learning rates and uses memory-saving techniques.

B. More Implementation Details

  • Pre-training uses AdamW, linear warm-up, and cosine decay, with peak learning rates of 2.8e-4 for Swin-B and 4e-4 for Swin-L models.The detailed pre-training settings distinguish Swin-T/L models and detection versus grounding data.Source: B. More Implementation Details · p. 9, Table 9 · p. 9
  • The text encoder learning rate is reduced by 0.1, and mixed precision plus gradient checkpointing reduce GPU memory cost and allow larger contrastive-learning batches.The text encoder is initialized from pre-trained FILIP.Source: B. More Implementation Details · p. 9
  • LVIS fine-tuning uses 24 epochs and a smaller learning rate, while ODinW13 uses a shared configuration with plateau-based decay and early stopping.The ODinW13 schedule avoids dataset-specific hyperparameter searches; detailed fine-tuning settings are tabulated for LVIS and ODinW13.Source: B. More Implementation Details · p. 9, Table 10 · p. 9, Table 11 · p. 10

C. More Experimental Results

The added experiments report strong LVIS results, broad concept coverage, and better average ODinW13 fine-tuning AP than GLIP/GLIPv2. They also examine image-text input resolution and classification-data incorporation, finding that classification data helps most when only detection data is used.

C. More Experimental Results

  • 320 × 320 is selected for image-text inputs: increasing resolution from 256 to 320 raises AP from 30.5 to 31.5, while 384 adds limited gains with considerable overhead.The reported trade-off is between computational cost and model performance.Source: C. More Experimental Results · p. 10
  • Adding ImageNet1k classification data raises AP from 28.6 to 30.4 with detection-only training, including rare-category AP from 24.4 to 32.2 (+8 AP).Against CC3M, its overall AP is slightly worse in that setting; with all data, it adds only 0.2 overall AP for Swin-T and none for Swin-L, so the authors exclude it.Source: C. More Experimental Results · p. 10, Table 13 · p. 10
  • 15.6/4.4 AP gains are reported for DetCLIPv2-T over GLIP-T/DetCLIP-T on the complete LVIS validation set.The evaluation includes 20k images and 1,203 categories; Table 14 reports fixed AP and labels DetCLIPv2 state of the art.Source: C. More Experimental Results · p. 10, Table 14 · p. 11
  • About 15 overall AP improvement over the fully-supervised method is observed for both Swin-T and Swin-L models.Source: C. More Experimental Results · p. 11, Table 14 · p. 11
  • Image-text pre-training covers concepts absent from detection and grounding data, including abstract concepts and examples such as tensioners, tattoos, and lifebuoys.Source: C. More Experimental Results · p. 11
  • DetCLIPv2-L/T surpass their GLIP/GLIPv2 counterparts in average AP across the 13 ODinW datasets.Table 15 provides detailed fine-tuning AP results for the datasets.Source: C. More Experimental Results · p. 11, Table 15 · p. 11

D. More Visualization Results

Figures 1-a and 1-b show DetCLIPv2 aligning textual concepts with image regions across a broad range of domains.

D. More Visualization Results

Related papers

Loading 2304.04514v1…