Source-linked AI summary
Learning Object-Language Alignments for Open-Vocabulary Object Detection
Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, Jianfei Cai
TL;DR
Open-vocabulary detection needs to recognize novel classes, but image-text pairs lack fine-grained object-language alignments and grounding annotations are costly. VLDet learns region-word alignments by bipartite matching directly from image-text pairs. It reports state-of-the-art results on open-vocabulary COCO and LVIS, with novel-class gains over competing methods.
Problem
Image-text pairs lack fine-grained object-language alignments for detection, while grounding annotations used by prior methods are costly and classification-model distillation is limited by global rather than region-word matching.
Source: §1 · pp. 1–2Method
VLDet treats image regions and caption-noun embeddings as sets, aligns them by bipartite matching, and trains the detector directly from image-text pairs.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
Visualized CC3M matches include novel objects such as jar, typewriter, and camera, indicating vocabulary expansion and generalization to novel classes.
Source: §4.4 · p. 7The study does not account for vision-language data bias and does not investigate larger data volumes such as Conceptual Captions 12M.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations. Natural language supervision is an attractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine-grained object-language alignments. Previous solutions rely on either expensive grounding annotations or distilling classification-oriented vision models. In this paper, we propose a novel open-vocabulary object detection framework directly learning from image-text pair data. We formulate object-language alignment as a set matching problem between a set of image region features and a set of word embeddings. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way. Extensive experiments on two benchmark datasets, COCO and LVIS, demonstrate our superior performance over the competing approaches on novel categories, e.g. achieving 32.0% mAP on COCO and 21.7% mask mAP on LVIS. Code is available at: https://github.com/clin1223/VLDet.
1 INTRODUCTION
VLDet learns region–word alignments directly from image-text pairs for open-vocabulary detection, without expensive grounding annotations or distilling classification-oriented vision models. It reports gains over prior methods on novel categories in COCO and LVIS.
1 INTRODUCTION
- Open-vocabulary recognition targets arbitrary novel categories, while language supervision offers rich semantics, broad data sources, and nearly free annotation.The paper positions open-vocabulary object detection as a natural extension of open-vocabulary image classification.Source: §1 · p. 1
- Learning detection from language requires object-language annotations, and grounding annotations used by prior open-vocabulary methods are even costlier than detection data.Distillation from classification-oriented models is also limited by pretraining on global image-text rather than region-word matching.Source: §1 · pp. 1–2
- VLDet replaces detection classification loss with optimal region-word alignment loss, using bipartite matching between image regions and words.The framework trains directly from image-text pairs without expensive grounding annotations or distilling classification-oriented models.Source: §1 · p. 2
- VLDet surpasses PB-OVD by 1.2% mAP on COCO and DetPro by 1.9% mAPmask on LVIS for novel categories.The experiments reserve localized annotations for base categories and use image-text pairs for novel categories.Source: §1 · p. 2
2 RELATED WORK
Related work addresses open-vocabulary detection through weak supervision, grounding-based multimodal detection, and transfer or distillation from vision-language models. VLDet instead matches caption words to image regions from image-text pairs without bounding-box annotations for novel classes.
Weakly-Supervised Object Detection (WSOD)
- Weakly supervised object detection trains from image-level labels without boxes, whereas open-vocabulary detection benefits from fully annotated base classes and targets classes not known in advance.Directly applying WSOD can favor base classes because they receive full supervision.Source: §2 · pp. 2–3
Multi-Modal Object Detection
- Grounding-based multimodal detectors align language and objects, but GLIP, MDETR, and XDETR rely fully or partly on annotated object-language pairs.VLDet instead learns novel-class region-word alignments by bipartite matching from image-text pairs without bounding-box annotations.Source: §2 · p. 3
Open-Vocabulary Object Detection (OVOD)
- Prior open-vocabulary approaches transfer or distill vision-language knowledge, generate pseudo-region annotations, or use classification data; VLDet matches image regions to caption words.The paper also contrasts its image-text-pair vocabulary expansion with methods that use image classification data or predicted masks.Source: §2 · p. 3
3 METHOD
VLDet extends detection to novel classes by learning alignments between image regions and caption words. Its method combines set matching, a caption-derived vocabulary, and a two-stage detector adapted for open-vocabulary classification.
3.1 OVERVIEW
- OVOD extends base-class detection to novel classes using image-caption data with a large vocabulary, which may overlap with the base and novel classes.VLDet formulates this as matching object candidates with word candidates from image-text pairs.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 LEARNING OBJECT-LANGUAGE ALIGNMENTS BY BIPARTITE MATCHING
- For each image-caption pair, VLDet matches region features to embeddings of caption nouns using dot-product alignment scores and the Hungarian algorithm.The matched pairs train the classifier with cross-entropy, while an additional image-text loss treats each image and its caption as a positive pair and other batch captions as negatives.Source: §3.2 · pp. 4–5
- VLDet uses all nouns in training-batch captions as its object vocabulary, rather than restricting vocabulary to dataset category labels.The authors report that this vocabulary is larger than the dataset label space and achieves better performance than previous works.Source: §3.2 · p. 5
3.3 NETWORK ARCHITECTURE
- The network uses Faster R-CNN with class-agnostic localization and language embeddings in place of trainable classifier weights.Base-class images follow classical detection training; image-text pairs supply regions and noun embeddings for matching, and inference incorporates language embeddings into the classifier.Source: §3.3 · p. 5
4 EXPERIMENTS
Experiments evaluate VLDet on OV-COCO and OV-LVIS, test vocabulary, matching, and loss choices, and assess transfer without retraining. VLDet outperforms competing methods on novel categories in the reported benchmarks.
4.1 DATASETS
- OV-COCO uses 48 base and 17 novel classes; OV-LVIS uses 866 base and 337 rare novel classes, with COCO Caption and CC3M image-text data respectively.OV-LVIS removes novel-class annotations during training and predicts all categories at test time.Source: §4.1 · p. 5
4.3 OPEN-VOCABULARY COCO
- VLDet is best on novel COCO classes and beats PB-OVD on all metrics using 10K COCO Caption images versus PB-OVD’s about 1M image-text pairs.The comparison uses the same COCO Caption training data as OV-RCNN.Source: Table 1 · p. 6, §4.3 · pp. 6–7
4.4 OPEN-VOCABULARY LVIS
- VLDet exceeds DetPro by 1.9% AP on novel classes; Swin-B adds 4.5% mAP over ResNet on novel classes and gains 2.4% over Detic.The LVIS comparison includes ResNet50 and Swin-B backbones.Source: Table 2 · p. 6, §4.4 · p. 7
4.2 IMPLEMENTATION DETAILS
- Training mixes base detection and image-text data at 1:4 with fixed CLIP text embeddings; COCO uses Faster R-CNN/ResNet50-C4, while LVIS uses CenterNet2/ResNet50.Training initializes from a detector trained on fully supervised base-class detection data.Source: §4.2 · p. 6
4.5 ABLATION STUDIES
- The full noun vocabulary improves novel-class performance by 1.8% on OV-COCO and 1.5% on OV-LVIS, while image-text alignment loss further improves performance.The vocabulary ablation replaces the full vocabulary with benchmark category names; region-word alignment loss alone also achieves promising results.Source: §4.5 · pp. 8–9, Table 3 · p. 8, Table 5 · p. 9
- One-to-one Hungarian assignment reaches 32.0 AP and 21.7 AP, exceeding Sinkhorn’s one-to-many 29.1 AP and 18.5 AP.Global bipartite matching also performs best for novel-class mAP against max-score and max-size alignment on both OV-COCO and OV-LVIS.Source: §4.5 · p. 8, Table 4 · p. 8, Figure 4 · p. 9
4.6 TRANSFER TO OTHER DATASETS.
- Without retraining, COCO-trained VLDet improves box AP50 by 2.5% on PASCAL VOC test and 2.0% on LVIS validation.The transfer setup replaces the classifier head’s class embeddings and evaluates the model on both datasets without training images.Source: Table 6 · p. 9, §4.6 · p. 9
5 CONCLUSIONS AND LIMITATIONS
VLDet learns annotation-free object-language alignments from image-caption pairs using bipartite matching. The conclusion reports state-of-the-art performance on open-vocabulary COCO and LVIS, while identifying data bias and untested larger data volumes as limitations.
5 CONCLUSIONS AND LIMITATIONS
- VLDet learns annotation-free object-language alignments from image-caption pairs by matching regions with words from a global perspective.The conclusion describes a single object detection model supervised by image-text pairs, without object-language annotations for novel classes.Source: §5 · p. 9
- The conclusion reports new state-of-the-art performance on open-vocabulary COCO and LVIS.The supplied conclusion text names COCO and LVIS across the passage boundary.Source: §5 · pp. 9–10
- The authors identify vision-language data bias and untested larger data volumes, such as Conceptual Captions 12M, as limitations.They leave both issues for future work.Source: §5 · p. 10
1. Detailed Framework
VLDet combines a base-class detector with a separate image-text training component for learning fine-grained region-word alignments.
1. Detailed Framework
- The framework has two training parts: a Faster R-CNN path and an image-text alignment component.The figure marks the Faster R-CNN path with red arrows and the alignment component with black arrows.Source: Figure 5 · p. 14
- The bottom Faster R-CNN path is also the inference path and is trained with well-annotated base-class data.Source: Figure 5 · p. 14
- The top component learns fine-grained object-word alignments by bipartite matching between image regions and word candidates.The figure notes that Faster R-CNN regression heads are omitted for illustration.Source: Figure 5 · p. 14
2. The Standard LVIS Benchmark
On the standard LVIS benchmark, the comparison uses all-class annotations and additional CC3M image-text pairs. VLDet improves rare-class mAP over Detic and is comparable on common and frequency classes.
2. The Standard LVIS Benchmark
- VLDet improves rare-class mAP by 0.8% over Detic and achieves comparable results on common and frequency classes.Source: §2 · p. 14, Table 7 · p. 14
- The Box-Supervised method uses all training-set annotations, including rare and base classes.Source: §2 · p. 14
- The benchmark comparison evaluates Detic and VLDet using all LVIS training-class annotations and additional CC3M image-text pairs.Source: Table 7 · p. 14, §2 · p. 14
3. Visualization of OV-COCO
The OV-COCO visualization presents region-word matches learned from image-text data, including examples that extend detection vocabulary to novel classes.
3. Visualization of OV-COCO
- Examples of matched words include “ostrich,” “duffle bag,” and “adult bear,” extending the object vocabulary.Source: §3 · p. 14
- The authors interpret the matching examples as evidence of generalization to novel classes.Source: §3 · p. 14
- Figure 6 visualizes bipartite matching results in the open-vocabulary COCO setting.The authors describe the displayed matches as promising region-word pairs extracted from image-text data.Source: §3 · p. 14, Figure 6 · p. 15
4. Failure Cases
VLDet’s failure cases expose two distinct boundaries: missing region proposals and captions that mention objects absent from the image.
4. Failure Cases
- VLDet cannot handle a caption mentioning “port” when that object is absent from the image, as shown in Figure 7(2).The authors leave this problem for future study.Source: §4 · p. 15, Figure 7 · p. 15
- The proposed region-proposal improvement addresses missing candidate regions, while the absent-object caption case remains for future study.Source: §4 · p. 15
5. Vision and Language Representation Learning
Prior vision-language work uses object-detection regions to support multimodal representation learning, while this paper focuses on object-detection generalization through weak language supervision.
5. Vision and Language Representation Learning
- Oscar uses object tags and region features with a universal Transformer for cross-domain semantics, improving visual-language understanding and generation tasks (Li et al., 2020).Source: §5 · p. 16
- ImageBERT uses masked object classification and region-feature regression as pre-training tasks (Radford et al., 2021).Source: §5 · p. 16
- UNITER learns contextualized embeddings through word-region alignments that encourage similar output distributions across modalities (Chen et al., 2020).Source: §5 · p. 16
- Unlike work targeting a universal semantic space for cross-modal understanding, this paper targets object-detection generalization through weak language supervision.Source: §5 · p. 16