Source-linked AI summary
GLIPv2: Unifying Localization and Vision-Language Understanding
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, Jianfeng Gao
TL;DR
GLIPv2 targets the difficult unification of localization and vision-language understanding, which traditionally differ in modality and output requirements. It converts both data types into grounded VL learning with phrase grounding, region-word contrastive learning, and masked language modeling. A shared-weight model achieves near SoTA performance across tasks while supporting open-vocabulary transfer and grounded VL outputs.
Problem
Unifying localization and vision-language understanding remains difficult because localization requires fine-grained outputs while VL tasks require multimodal semantic understanding.
Method
GLIPv2 turns localization and vision-language data into grounded VL data and pre-trains with phrase grounding, inter-image region-word contrastive learning, and masked language modeling.
Results
A single shared-weight GLIPv2 model achieves near SoTA performance across localization and understanding tasks, with strong zero-shot and few-shot open-vocabulary transfer and grounded VL capability.
Takeaways & Limitations
Grounded VL pre-training provides one model for localization and understanding while enabling language-guided localization and grounded answers or captions.
Takeaways & Limitations
Additional analysis is needed before deployment because large-scale web data may contain private information, unsuitable content, or bias leakage.
Abstract
from arXiv · showhide
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks. Code will be released at https://github.com/microsoft/GLIP.
1 Introduction
GLIPv2 addresses the challenge of unifying localization and vision-language understanding by reformulating both as grounded VL understanding. Its shared model and region-word contrastive learning support mutual benefits, broad task coverage, and grounded outputs.
- Motivation: Localization and VL understanding differ in output and modality requirements, making their unification a long-standing challenge.Localization requires fine-grained boxes or masks, whereas VL understanding emphasizes multimodal fusion and semantic outputs.
- Motivation: Earlier unified models shared a visual encoder but retained separate branches, producing marginal mutual benefit or performance degradation.These localization components remained vision-only and could not exploit semantics from vision-language data.
- Approach: GLIPv2 converts localization and vision-language data into grounded VL data so one model can perform unified grounded VL understanding.Localization categories become synthesized language input, while image-text data can be converted through self-training.
- Approach: Inter-image region-word contrastive learning creates additional negatives from phrases in other sentences within the same batch.This strengthens grounding beyond the easy 1-in-3 phrase-matching setup by using more information from each annotation.
- Results: A single GLIPv2 model with shared weights achieves near SoTA performance across localization and understanding tasks.The model also supports zero-shot and few-shot open-world detection and segmentation, language-guided localization, and grounded VL outputs.
2 Related Work
Prior work addressed localization and vision-language understanding separately or through incomplete multi-task sharing. GLIPv2 instead uses grounded pre-training to simplify transfer and strengthen mutual task benefits.
- Localization models: Localization models traditionally use single-modality inputs and output boxes or masks, but struggle to generalize to rare and novel concepts.A central difficulty is collecting localization data that covers many rare categories.
- Vision-language understanding models: Vision-language models often rely on pre-trained localization encoders, creating a prolonged localization-to-VLP pre-training pipeline.This dependency motivates unified alternatives that combine localization and VL representation learning.
- Unifying localization and understanding: Earlier unification methods shared only a low-level visual encoder while using separate task branches, without evident mutual benefit.Their localization tasks remained single-modality despite the multimodal nature of VL tasks.
- GLIPv2 versus GLIP: GLIPv2 extends grounded pre-training from localization to VL understanding and introduces a stronger inter-image region-word contrastive loss than GLIP.The loss is presented as a region-word-level generalization of image-level contrastive learning.
3 GLIPv2: Unifying Localization and VL Understanding
GLIPv2 unifies localization and vision-language understanding as grounded VL tasks through task-agnostic vision-language matching and shared architecture. Its pre-training combines grounding, intra- and inter-image region-word contrastive learning, and masked language modeling, while supporting shared-weight transfer and grounding across downstream tasks.
- Unified architecture: Architecture Π uses visual and language dual encoders followed by a fusion encoder, with lightweight task-specific heads operating on fused representations for VL tasks.The model extracts pre-fusion image and text features, then produces fused features used by downstream heads such as the MLM head.
- Unified formulation: GLIPv2 reformulates fixed-vocabulary classification as task-agnostic region-word matching, unifying detection, grounding, and segmentation within grounded VL tasks.Detection replaces class logits with region-word similarity logits, while segmentation uses the same classification-to-matching principle.
- Pre-training: GLIPv2 pre-trains with phrase grounding, intra-image region-word alignment, inter-image region-word contrastive learning, and masked language modeling.The grounding loss combines localization supervision with intra-image alignment; the inter-image loss contrasts regions and phrases across image-text pairs.
- Pre-training: Inter-image contrastive learning increases negatives from tens to thousands by using phrases from other image-text pairs, with nearly negligible additional computational cost.It computes similarities before VL fusion and uses label propagation rather than treating every unpaired region-text pair as negative.
- Transfer and grounded understanding: A single shared-weight GLIPv2 model transfers to multiple tasks with few or no task-specific parameters, including zero-shot or prompt-tuning localization.Prompt tuning adapts a small prompt embedding matrix, whose conditioning affects text, visual, and fused representations in the deep-fused model.
- Transfer and grounded understanding: Grounded VL fine-tuning retains localization alongside downstream VL prediction, adding grounding information that increases model interpretability.For VQA, training jointly predicts answers and grounds entities in the image.
4 Experiments
GLIPv2 is evaluated as a unified vision system across localization and vision-language tasks, using one architecture and shared weights. It achieves near state-of-the-art performance, strong zero- and few-shot transfer, data-efficient ODinW adaptation, and grounded VL outputs.
- One Model Architecture for All: GLIPv2 achieves near state-of-the-art performance across localization and vision-language tasks with one shared model architecture.The evaluation covers eight downstream tasks and compares GLIPv2 with specialized and unified systems.
- Localization Results: 5.8 AP improves COCO-Det performance over DyHead-T, with GLIPv2-T reaching 55.5 versus 49.7.
- Localization Results: 59.8 versus 24.2 bbox AP on LVIS shows GLIPv2-T outperforming supervised MDETR on rare and unseen categories.
- Unified-Model Comparisons: 18.3 AP separates GLIPv2-H from UniT on COCO object detection, while GLIPv2 also surpasses UniT on VQA by 6.9 points.
- Few-Shot and Zero-Shot Transfer: 48.5 zero-shot average AP for GLIPv2-T on ODinW surpasses GLIP-T at 46.5, while one-shot GLIPv2-H reaches 61.3 versus 60.8 for all-shot DyHead-T.
- Pre-training Analysis: Adding inter-image region-word contrastive loss improves object detection across COCO, ODinW, and LVIS, while MLM benefits Flickr30k, VQA, and captioning.
- Grounded VL Understanding: GLIPv2 can produce semantic outputs and supporting localization results simultaneously for captioning and other VL tasks.This grounded VL understanding capability is quantitatively evaluated on COCO Caption and Flickr30K Grounding.
5 Conclusion and Social Impacts
GLIPv2 is a unified framework serving localization and VL understanding tasks with shared model weights. The paper reports competitive performance while identifying risks requiring further analysis before practical deployment.
- GLIPv2 serves both localization and VL understanding tasks within one unified framework.
- A single model architecture achieves near SoTA performance across localization and understanding tasks.
- Social Impacts: Additional analysis is needed before deployment because large-scale web data may contain private information, unsuitable content, or bias leakage.
- GLIPv2 enables grounded outputs that combine visual-language predictions with localization results.
B.1 (Language-guided) object detection and phrase grounding
The appendix evaluates GLIPv2 across object detection, grounding, segmentation, VQA, and captioning datasets. These benchmarks cover conventional localization as well as language-guided and grounded tasks.
- Object detection and instance segmentation: COCO provides object detection annotations for 80 categories, while LVIS covers over a thousand long-tailed instance-segmentation categories.
- Phrase grounding: Flickr30k-Entities evaluates phrase grounding by requiring bounding boxes for each provided phrase and reporting Recall.
- Object detection and instance segmentation: ODinW uses 13 Roboflow datasets to assess object detection across varied domains.
- Referring image segmentation: PhraseCut evaluates referring-expression segmentation with masks for expressions spanning objects, attributes, and relations.
- VL understanding: VQA2.0 tests answer prediction from images and questions, while COCO Captioning tests natural-language image description using BLEU-4, CIDEr, and SPICE.
C Difference between inter-image region-word contrastive loss with other "region-word" losses.
GLIPv2’s inter-image region-word contrastive loss differs from prior full-batch approaches in its use of true region-word matching and positive-label propagation. The design explicitly incorporates multi-entity grounding data.
- GLIPv2 uses true region-word loss, whereas three compared methods use region-sentence similarity with a sentence-level [CLS] representation.
- Unlike the compared methods, GLIPv2 uses phrase grounding and pseudo-grounding data containing multiple entities in one sentence.
- Positive label propagation avoids assuming that every unpaired region-word pair is negative.
- The authors characterize the inter-image region-word contrastive loss as novel relative to the discussed prior works.
D.1 Pre-training
GLIPv2 pre-training combines detection, grounding, and web image-text data, using multiple losses and shared architecture across downstream tasks. Training choices vary by model size and target task.
- Pre-training data: Pre-training uses detection data, human-annotated grounding data, and web image-text pairs with pseudo boxes.
- Downstream tasks: Grounded VQA jointly predicts answers and boxes for entities mentioned in the question and answer text.
E Analysis on the effect of different language encoders and pre-trained weights
GLIPv2-T and GLIPv2-B use different image- and language-encoder initialization strategies. An ablation found nearly identical results for UniCL and BERT language encoders.
- GLIPv2-T initializes its image encoder from ImageNet-pre-trained Swin-Transformer and its language encoder from BERT-base-uncased.
- GLIPv2-B initializes with UniCL’s paired pre-trained image-language encoder.UniCL uses CLIP-like pre-training.
- The UniCL and BERT language-encoder ablation produces nearly identical results.The comparison is shown in Figure 9.
F More analysis on pre-training data
Scaling pre-training data benefits both in-domain and large-domain-gap tasks, while inter-image region-word contrastive learning provides additional gains at fixed data scale. GLIPv2 improves considerably as caption data grows from 0M to 12M, suggesting further scaling potential.
- Adding more weak image-text pair data benefits both standard tasks, including COCO and Flickr30K, and large-domain-gap tasks, including ODinW and LVIS.These findings are reported from the last-checkpoint results in Table 5.
- Inter-image region-word contrastive learning further helps when pre-training data is fixed at the same scale.
- Inter-image region-word contrastive loss provides additional improvement on large-domain-gap tasks.The passage specifically identifies this effect for large-domain-gap evaluation.
- GLIPv2 improves considerably as caption data increases from 0M to 12M.The authors hypothesize that even larger-scale web image-text pairs could yield further growth.
G Experiments on grounded image captioning
Grounded image captioning requires joint caption generation and phrase-to-region grounding, with caption quality evaluated on COCO Captions and grounding evaluated on Flickr30K. The appendix also reports task-specific inference-speed comparisons with MDETR.
- Grounded captioning outputs text captions, predicted object regions, and grounding correspondences between phrases and regions.
- Caption metrics are evaluated on COCO Captions, while grounding metrics are evaluated on Flickr30K using established benchmarks.
- Grounded captioning results report BLEU@4, CIDEr, and SPICE for captions, plus R@1, R@5, and R@10 for grounding.
- Table 10 compares GLIPv2 inference speed with MDETR on a V100 GPU using batch size 1.
- Inference speed is reported in FPS, defined as images processed per second per GPU, with higher values preferred.
J All results for ODinW
The ODinW analysis reports zero-shot performance across 13 datasets and per-dataset results for multiple detector variants under prompt tuning, full-model tuning, and full-data settings.
- The ODinW results include 0-, 1-, 3-, 5-, and 10-shot settings, full-data evaluation, prompt tuning, and full-model tuning.
- Table 11 reports zero-shot performance on 13 ODinW datasets.
- Table 12 compares per-dataset performance for DyHead, GLIP-T, GLIP-L, GLIPv2-T, GLIPv2-B, and GLIPv2-H.
- For PascalVOC, Table 12 reports mAP at IoU=0.50:0.95 using the COCO evaluation script for consistency across all 13 datasets.