GLIPv2: Unifying Localization and Vision-Language Understanding

Motivation

• Localization and VL understanding differ in output and modality requirements, making their unification a long-standing challenge.Localization requires fine-grained boxes or masks, whereas VL understanding emphasizes multimodal fusion and semantic outputs.Source: §1 · p. 1
• Earlier unified models shared a visual encoder but retained separate branches, producing marginal mutual benefit or performance degradation.These localization components remained vision-only and could not exploit semantics from vision-language data.Source: §1 · p. 1

Approach

• GLIPv2 converts localization and vision-language data into grounded VL data so one model can perform unified grounded VL understanding.Localization categories become synthesized language input, while image-text data can be converted through self-training.Source: §1 · p. 2
• Inter-image region-word contrastive learning creates additional negatives from phrases in other sentences within the same batch.This strengthens grounding beyond the easy 1-in-3 phrase-matching setup by using more information from each annotation.Source: §1 · p. 2

Results

• A single GLIPv2 model with shared weights achieves near SoTA performance across localization and understanding tasks.The model also supports zero-shot and few-shot open-world detection and segmentation, language-guided localization, and grounded VL outputs.Source: §1 · pp. 2–3

Localization models

• Localization models traditionally use single-modality inputs and output boxes or masks, but struggle to generalize to rare and novel concepts.A central difficulty is collecting localization data that covers many rare categories.Source: §2 · p. 3

Vision-language understanding models

• Vision-language models often rely on pre-trained localization encoders, creating a prolonged localization-to-VLP pre-training pipeline.This dependency motivates unified alternatives that combine localization and VL representation learning.Source: §2 · p. 3

Unifying localization and understanding

• Earlier unification methods shared only a low-level visual encoder while using separate task branches, without evident mutual benefit.Their localization tasks remained single-modality despite the multimodal nature of VL tasks.Source: §2 · p. 3

GLIPv2 versus GLIP

• GLIPv2 extends grounded pre-training from localization to VL understanding and introduces a stronger inter-image region-word contrastive loss than GLIP.The loss is presented as a region-word-level generalization of image-level contrastive learning.Source: §2 · p. 3

Unified architecture

• Architecture Π uses visual and language dual encoders followed by a fusion encoder, with lightweight task-specific heads operating on fused representations for VL tasks.The model extracts pre-fusion image and text features, then produces fused features used by downstream heads such as the MLM head.Source: §3.1 · p. 4

Unified formulation

• GLIPv2 reformulates fixed-vocabulary classification as task-agnostic region-word matching, unifying detection, grounding, and segmentation within grounded VL tasks.Detection replaces class logits with region-word similarity logits, while segmentation uses the same classification-to-matching principle.Source: §3.1 · pp. 3–4, §3 · p. 3

Pre-training

• GLIPv2 pre-trains with phrase grounding, intra-image region-word alignment, inter-image region-word contrastive learning, and masked language modeling.The grounding loss combines localization supervision with intra-image alignment; the inter-image loss contrasts regions and phrases across image-text pairs.Source: §3.2 · pp. 4–5
• Inter-image contrastive learning increases negatives from tens to thousands by using phrases from other image-text pairs, with nearly negligible additional computational cost.It computes similarities before VL fusion and uses label propagation rather than treating every unpaired region-text pair as negative.Source: §3.2 · p. 5

Transfer and grounded understanding

• A single shared-weight GLIPv2 model transfers to multiple tasks with few or no task-specific parameters, including zero-shot or prompt-tuning localization.Prompt tuning adapts a small prompt embedding matrix, whose conditioning affects text, visual, and fused representations in the deep-fused model.Source: §3.3 · pp. 5–6
• Grounded VL fine-tuning retains localization alongside downstream VL prediction, adding grounding information that increases model interpretability.For VQA, training jointly predicts answers and grounds entities in the image.Source: §3.3 · pp. 5–6

One Model Architecture for All

• GLIPv2 achieves near state-of-the-art performance across localization and vision-language tasks with one shared model architecture.The evaluation covers eight downstream tasks and compares GLIPv2 with specialized and unified systems.Source: §4 · p. 6, §4.1 · pp. 7–8

Localization Results

• 5.8 AP improves COCO-Det performance over DyHead-T, with GLIPv2-T reaching 55.5 versus 49.7.Source: §4.1 · p. 7
• 59.8 versus 24.2 bbox AP on LVIS shows GLIPv2-T outperforming supervised MDETR on rare and unseen categories.Source: §4.1 · p. 7

Unified-Model Comparisons

• 18.3 AP separates GLIPv2-H from UniT on COCO object detection, while GLIPv2 also surpasses UniT on VQA by 6.9 points.Source: §4.1 · p. 8

Few-Shot and Zero-Shot Transfer

• 48.5 zero-shot average AP for GLIPv2-T on ODinW surpasses GLIP-T at 46.5, while one-shot GLIPv2-H reaches 61.3 versus 60.8 for all-shot DyHead-T.Source: §4.3 · p. 9, §4.2 · p. 9

Pre-training Analysis

• Adding inter-image region-word contrastive loss improves object detection across COCO, ODinW, and LVIS, while MLM benefits Flickr30k, VQA, and captioning.Source: §4.4 · p. 9

Grounded VL Understanding

• GLIPv2 can produce semantic outputs and supporting localization results simultaneously for captioning and other VL tasks.This grounded VL understanding capability is quantitatively evaluated on COCO Caption and Flickr30K Grounding.Source: §4.4 · p. 10

• GLIPv2 serves both localization and VL understanding tasks within one unified framework.Source: §5 · p. 10, Figure 4 · p. 15, A Visualization · p. 15
• A single model architecture achieves near SoTA performance across localization and understanding tasks.Source: §5 · p. 10

Social Impacts

• Additional analysis is needed before deployment because large-scale web data may contain private information, unsuitable content, or bias leakage.Source: §5 · p. 10

• GLIPv2 enables grounded outputs that combine visual-language predictions with localization results.Source: Figure 4 · p. 15, A Visualization · p. 15, Figure 5 · p. 16

Object detection and instance segmentation

• COCO provides object detection annotations for 80 categories, while LVIS covers over a thousand long-tailed instance-segmentation categories.Source: §B.1 · p. 16, §B.2 · p. 17

Phrase grounding

• Flickr30k-Entities evaluates phrase grounding by requiring bounding boxes for each provided phrase and reporting Recall.Source: §B.1 · p. 16

Object detection and instance segmentation

• ODinW uses 13 Roboflow datasets to assess object detection across varied domains.Source: §B.1 · p. 16, Table 7 · p. 17

Referring image segmentation

• PhraseCut evaluates referring-expression segmentation with masks for expressions spanning objects, attributes, and relations.Source: §B.2 · p. 18

VL understanding

• VQA2.0 tests answer prediction from images and questions, while COCO Captioning tests natural-language image description using BLEU-4, CIDEr, and SPICE.Source: §B.3 · p. 18

• GLIPv2 uses true region-word loss, whereas three compared methods use region-sentence similarity with a sentence-level [CLS] representation.Source: C Difference between inter-image… · p. 19
• Unlike the compared methods, GLIPv2 uses phrase grounding and pseudo-grounding data containing multiple entities in one sentence.Source: C Difference between inter-image… · p. 19
• Positive label propagation avoids assuming that every unpaired region-word pair is negative.Source: C Difference between inter-image… · p. 19
• The authors characterize the inter-image region-word contrastive loss as novel relative to the discussed prior works.Source: C Difference between inter-image… · p. 19

Pre-training data

• Pre-training uses detection data, human-annotated grounding data, and web image-text pairs with pseudo boxes.Source: §D.1 · p. 19

Downstream tasks

• Grounded VQA jointly predicts answers and boxes for entities mentioned in the question and answer text.Source: §D.2 · p. 21

• GLIPv2-T initializes its image encoder from ImageNet-pre-trained Swin-Transformer and its language encoder from BERT-base-uncased.Source: E Analysis on the effect of different… · p. 21
• GLIPv2-B initializes with UniCL’s paired pre-trained image-language encoder.UniCL uses CLIP-like pre-training.Source: E Analysis on the effect of different… · p. 21
• The UniCL and BERT language-encoder ablation produces nearly identical results.The comparison is shown in Figure 9.Source: E Analysis on the effect of different… · p. 21

• Adding more weak image-text pair data benefits both standard tasks, including COCO and Flickr30K, and large-domain-gap tasks, including ODinW and LVIS.These findings are reported from the last-checkpoint results in Table 5.Source: F More analysis on pre-training data · p. 21
• Inter-image region-word contrastive learning further helps when pre-training data is fixed at the same scale.Source: F More analysis on pre-training data · p. 21
• Inter-image region-word contrastive loss provides additional improvement on large-domain-gap tasks.The passage specifically identifies this effect for large-domain-gap evaluation.Source: F More analysis on pre-training data · p. 21
• GLIPv2 improves considerably as caption data increases from 0M to 12M.The authors hypothesize that even larger-scale web image-text pairs could yield further growth.Source: F More analysis on pre-training data · p. 22

• Grounded captioning outputs text captions, predicted object regions, and grounding correspondences between phrases and regions.Source: G Experiments on grounded image… · p. 22
• Caption metrics are evaluated on COCO Captions, while grounding metrics are evaluated on Flickr30K using established benchmarks.Source: G Experiments on grounded image… · p. 22
• Grounded captioning results report BLEU@4, CIDEr, and SPICE for captions, plus R@1, R@5, and R@10 for grounding.Source: Table 9 · p. 22
• Table 10 compares GLIPv2 inference speed with MDETR on a V100 GPU using batch size 1.Source: H Inference speed · p. 22, Table 10 · p. 24
• Inference speed is reported in FPS, defined as images processed per second per GPU, with higher values preferred.Source: Table 10 · p. 24

• The ODinW results include 0-, 1-, 3-, 5-, and 10-shot settings, full-data evaluation, prompt tuning, and full-model tuning.Source: J All results for ODinW · p. 24
• Table 11 reports zero-shot performance on 13 ODinW datasets.Source: Table 11 · p. 24
• Table 12 compares per-dataset performance for DyHead, GLIP-T, GLIP-L, GLIPv2-T, GLIPv2-B, and GLIPv2-H.Source: Table 12 · p. 25
• For PascalVOC, Table 12 reports mAP at IoU=0.50:0.95 using the COCO evaluation script for consistency across all 13 datasets.Source: Table 12 · p. 25