Source-linked AI summary
Aligning Bag of Regions for Open-Vocabulary Object Detection
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, Chen Change Loy
TL;DR
Open-vocabulary detectors traditionally align individual regions, leaving potentially learned compositional scene structure in VLMs underused. BARON groups related regions, encodes them as pseudo-word sentences through a frozen text encoder, and aligns the resulting representation with VLM image features. It surpasses previous results on novel categories of OV-COCO and OV-LVIS.
Problem
Existing OVD methods align individual region embeddings, while VLMs may encode compositional relationships among multiple semantic concepts.
Method
BARON samples contextually interrelated regions, projects them into word-embedding space, and uses a frozen VLM text encoder to obtain bag-of-regions embeddings for alignment.
Results
BARON achieves 34.0 box AP50 on novel OV-COCO categories and 22.6 mask mAP on novel OV-LVIS categories, increases of 4.6 and 2.8, respectively.
Takeaways & Limitations
Aligning bags of regions enables BARON to model co-existing visual concepts and achieve new state-of-the-art performance across multiple OVD benchmarks.
Takeaways & Limitations
The explored compositional structure mainly concerns object co-occurrence, while more complex language compositionality remains under-explored.
Abstract
from arXiv · showhide
Pre-trained vision-language models (VLMs) learn to align vision and language representations on large-scale datasets, where each image-text pair usually contains a bag of semantic concepts. However, existing open-vocabulary object detectors only align region embeddings individually with the corresponding features extracted from the VLMs. Such a design leaves the compositional structure of semantic concepts in a scene under-exploited, although the structure may be implicitly learned by the VLMs. In this work, we propose to align the embedding of bag of regions beyond individual regions. The proposed method groups contextually interrelated regions as a bag. The embeddings of regions in a bag are treated as embeddings of words in a sentence, and they are sent to the text encoder of a VLM to obtain the bag-of-regions embedding, which is learned to be aligned to the corresponding features extracted by a frozen VLM. Applied to the commonly used Faster R-CNN, our approach surpasses the previous best results by 4.6 box AP50 and 2.8 mask AP on novel categories of open-vocabulary COCO and LVIS benchmarks, respectively. Code and models are available at https://github.com/wusize/ovdet.
1. Introduction
Open-vocabulary detection addresses unseen categories, but existing distillation methods align regions individually and underuse VLMs’ compositional scene structure. BARON groups related regions, encodes them as pseudo-word sentences through a frozen VLM text encoder, and improves novel-category detection and segmentation results.
- Open-vocabulary object detection targets categories absent from training, extending beyond traditional detectors’ fixed category sets.
- Distillation-based OVD methods transfer VLM knowledge by aligning each individual region embedding with corresponding frozen image-encoder features.
- BARON addresses compositional structure by aligning embeddings of contextually interrelated region bags rather than isolated regions.The method explicitly models co-existing visual concepts in a scene.
- BARON projects regional features into word-embedding space, forms pseudo-word sentences, and processes them with a frozen VLM text encoder.Contrastive learning aligns detector bag embeddings with teacher embeddings from image crops enclosing each region bag.
- 34.0 box AP50 on novel OV-COCO categories and 22.6 mask mAP on novel OV-LVIS categories surpass prior results by 4.6 and 2.8, respectively.With caption supervision, BARON achieves 32.7 box AP50 on novel OV-COCO categories.
2. Related Work
Vision-language pre-training aligns image and text representations using large-scale image-text data and supports zero-shot transfer. Related dense-prediction work suggests that VLMs implicitly capture compositional structure, motivating bag-of-regions alignment for OVD.
- Vision-language pre-training learns aligned image and text representations from large-scale, including billion-scale, image-text pairs.
- Pre-trained VLMs show zero-shot performance when transferred to image classification tasks.
- Dense-prediction studies apply vision-language alignment to segmentation and detection, including pixel- or region-level alignment.
- MaskCLIP indicates that VLM image encoders capture multiple concepts in complex scenes and align their pixel embeddings with text representations.This motivates exploring compositional alignment between bags of regions and bags of words.
3. Method
BARON extends open-vocabulary detection from aligning individual regions to aligning bags of contextually related regions, using VLM encoders to capture co-occurring concepts. It forms spatially neighboring bags, represents regions as positional pseudo words, and aligns student and teacher bag embeddings contrastively.
- 3.1. Preliminaries: BARON modifies Faster R-CNN by replacing its classifier with a linear projection that maps region features into pseudo words.The approach is designed for arbitrary vocabularies and can also apply to other OVD architectures.
- 3.2. Forming Bag of Regions: Each bag combines a region proposal with independently sampled, spatially neighboring boxes generated around it.Eight same-shaped candidate boxes are sampled around each proposal; candidates extending beyond the image boundary are discarded.
- 3.2. Forming Bag of Regions: The sampled bags can include co-occurring objects and occasional novel categories, providing scene context for learning region concepts.Examples include a carriage with a horse and a cup appearing alongside other objects.
- 3.3. Representing Bag of Regions: BARON encodes each region as pseudo words, adds positional embeddings, and feeds the sequence to the frozen VLM text encoder for a student bag embedding.Positional embeddings preserve relative spatial information that concatenating pseudo words alone would lose.
- 3.3. Representing Bag of Regions: The teacher bag embedding comes from the VLM image encoder applied to a crop enclosing the grouped regions, with outside content masked in attention layers.Student and teacher embeddings are defined for the same sampled region groups.
- 3.4. Aligning Bag of Regions: BARON uses InfoNCE to pull corresponding student–teacher bag embeddings together and push non-corresponding pairs apart, using queues for more negatives.The framework can also align bag-of-regions text embeddings with caption embeddings instead of image embeddings.
4. Experiments
BARON is evaluated on OV-COCO and OV-LVIS through benchmark comparisons, ablations of its bag construction, and qualitative analyses of its predictions and VLM representations.
- Ablation Study: Neighborhood sampling achieves 32.2 mAP50 on novel categories, exceeding grid sampling at 25.4 and random sampling at 27.3.It samples neighboring boxes around region proposals to capture nearby objects and preserve the bag representation for the teacher encoder.
- Further Analysis: Adding object concepts to text descriptions increases image-text similarity, indicating that the evaluated VLM captures co-occurrence among multiple concepts.Qualitative visualizations also show stronger responses to novel-category objects for BARON than for the individual-level baseline.
5. Discussion and Conclusion
The paper extends open-vocabulary object detection from individual-region alignment to bag-of-regions alignment, using compositional structure in vision-language models. Its explored structure mainly concerns object co-occurrence, while richer language composition remains open.
- BARON aligns bag-of-regions representations rather than only individual regions, exploring compositional structure learned by large-scale vision-language models.
- The method groups contextually related regions and uses contrastive learning to align detector and pre-trained VLM representations.
- The explored compositional structure mainly captures object co-occurrence and behaves like a bag of words.
- More complex compositional structure in language and whether modern VLMs capture it remain open problems.
A1. Implementation Details
The implementation specifies proposal filtering, contrastive alignment losses, temperature settings, and auxiliary engineering choices for BARON.
- Sampling: Region proposals are filtered by objectness, aspect ratio, and area ratio before non-maximum suppression.The passage gives thresholds of 0.85 objectness, aspect ratios between 0.25 and 4.0, area ratio at least 0.01, and IoU threshold 0.1.
- Alignment Loss: The bag alignment loss compares teacher image and student text embeddings for sampled bags using temperature-scaled cosine similarity.
- Alignment Loss: The individual-region alignment loss similarly compares teacher and student embeddings for individual regions with a separate temperature.
- Alignment Loss: BARON uses different bag and individual temperatures on OV-COCO and OV-LVIS to make contrastive learning harder on LVIS.The settings are τbag = 30.0 and τindividual = 50.0 on OV-COCO, versus τbag = 20.0 and τindividual = 30.0 on OV-LVIS.
- Implementation Choices: The implementation uses one linear layer to map detector region features into pseudo-words, while the appendix compares sampling strategies and engineering choices.
A2. Sampling Strategy
BARON compares global and local sampling strategies, progressively shifting from whole-image bags toward neighboring regions and addressing imbalance and redundant content.
- Sampling Strategies: Grid and random sampling form bags representing the whole image, while additional strategies shift attention toward local neighboring regions.
- Sampling Strategies: Random-Tight uses the crop tightly enclosing randomly sampled regions, whereas Random-Neighbor samples nearby proposals around center regions.
- Sampling Strategies: The neighborhood strategy samples two nearby proposals with GIOU larger than 0.5 for each center proposal and uses 12 centers for 36 total regions.
- Figure A1: Figure A1 compares region proposals, sampled region boxes, and the image crop enclosing a region group.
- Sampling Strategies: The sampling strategies progressively move from whole-image coverage toward neighboring local regions, but can retain box-size imbalance and redundant image content.
A3. Pseudo Word Encoding
BARON projects region features into pseudo-words and sends them through the VLM text encoder, producing more diverse and distinct category clusters in the reported visualization.
- Pseudo Word Encoding: BARON projects region features into word-embedding space to obtain pseudo-words before text-encoder processing.
- Detection Visualization: Figure A3 identifies red boxes as novel categories and blue boxes as base categories in OV-COCO detection visualizations.
- Pseudo Word Encoding: After pseudo-words are encoded by the text encoder, categories form clusters with more diverse distributions and distinct boundaries.
A4. Image-Guided Inference
Image-guided inference allows the LVIS-trained detector to detect objects shown in a reference image, including cartoon characters.
- The detector uses a reference image to guide inference and detect the depicted object.
- It recognizes cartoon characters such as ‘pikachu’ and ‘winnie pooh’ in reference images.
A5. Detection Results
BARON detects novel and rare categories across COCO, LVIS, and transferred Objects365 settings, demonstrating broad object-concept recognition.
- BARON detects novel COCO categories including bus, keyboard, and couch, and rare LVIS categories such as salad plate, fedora hat, and gas mask.
- Figure A4 shows image-guided recognition of cartoon characters including ‘pikachu’ and ‘winnie pooh’.
- The LVIS-trained detector correctly recognizes a wide range of object concepts when transferred to Objects365.
A6. Potential Negative Societal Impacts
The models inherit knowledge from large-scale web image-text pretraining and may consequently inherit or reinforce harmful biases and stereotypes.
- The models learn from VLMs pretrained on large-scale web image-text pairs.
- The pretrained VLMs may transmit and reinforce harmful biases and stereotypes in the resulting models.
- The paper recommends scrupulous probing before applying the models for any purpose.