Grounded Language-Image Pre-training

Liunian Harold Li; Pengchuan Zhang; Haotian Zhang; Jianwei Yang; Chunyuan Li; Yiwu Zhong; Lijuan Wang; Lu Yuan; Lei Zhang; Jenq-Neng Hwang; Kai-Wei Chang; Jianfeng Gao

Source-linked AI summary

Grounded Language-Image Pre-training

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, Jianfeng Gao

arXiv:2112.03857v2Dec 7, 2021cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Fixed-category visual recognition limits generalization to new visual concepts, while fine-grained tasks require object-level representations. GLIP addresses this by unifying phrase grounding and object detection with language-aware fusion, enabling strong transfer across benchmarks and downstream tasks, including 1-shot performance rivaling fully supervised Dynamic Head.

Problem
Fixed pre-determined object categories limit recognition models' generalization to new visual concepts, while many fine-grained tasks require object-level visual representations.
Method
GLIP unifies phrase grounding and object detection to learn object-level, language-aware, semantic-rich representations, using deep fusion between image and language encoders.
Results
GLIP transfers strongly across zero-shot, few-shot, and fine-tuning settings, including 49.8 AP on COCO val2017, 26.9 AP on LVIS val, and 1-shot performance rivaling a fully supervised Dynamic Head.
Takeaways & Limitations
GLIP supports one-model transfer across object detection tasks and prompt tuning that can match full fine-tuning while tuning only a fraction of model parameters.
Takeaways & Limitations
Prompt input is limited to 256 tokens in the implementation, so datasets with large vocabularies require multiple prompts and incur a minor performance drop.

Abstract

from arXiv · show

This paper presents a grounded language-image pre-training (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representation semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines. 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head. Code is released at https://github.com/microsoft/GLIP.

1. Introduction

GLIP addresses the limits of fixed-category visual recognition by unifying object detection and phrase grounding into an object-level, language-aware pre-training framework. It scales grounding with image-text data and transfers strongly across zero-shot, few-shot, and downstream detection settings.

Motivation: Fixed-category visual recognition requires additional labeled data to generalize to new visual concepts and domains.Object-level representations are needed for fine-grained tasks such as detection, segmentation, pose estimation, and scene understanding.
Approach: GLIP uses phrase grounding as a scalable pre-training task for object-level, language-aware, semantic-rich visual representations.Phrase grounding identifies fine-grained correspondences between sentence phrases and image regions.
Approach: GLIP reformulates object detection as phrase grounding by aligning region features with language features from text prompts.The model jointly trains image and language encoders and adds deep cross-modality fusion to learn language-aware visual representations.
Scalable pre-training: 27M grounding examples combine human-annotated data with web-crawled image-text pairs whose grounding boxes are generated automatically.The generated annotations include semantic-rich concepts such as rare entities, attributes, and abstract words.
Transfer results: 49.8 AP on COCO val2017 and 26.9 AP on LVIS val were achieved by GLIP-L without seeing COCO images during pre-training.These results surpassed many supervised baselines.
Transfer results: A zero-shot GLIP-L outperformed a 10-shot supervised Dynamic Head baseline, while a 1-shot GLIP-L rivaled a fully supervised Dynamic Head across 13 detection datasets.Prompt tuning can adapt task-specific embeddings while keeping model parameters unchanged, reducing fine-tuning and deployment cost.

2. Related Work

GLIP builds on open-vocabulary vision-language recognition while targeting domain transfer for object detection. Its distinctive contribution is combining semantically rich grounding with efficient prompt-based adaptation across tasks and domains.

Detection and grounding: GLIP addresses the cost and limited vocabulary of human-annotated detection data by reformulating detection as phrase grounding.This enables use of grounding data and massive image-text pairs.
Vision-language recognition: Unlike CLIP and ALIGN classification-focused approaches or MDETR's multimodal alignment training, GLIP targets object detection transfer with language-aware representations.The paper reports improved transferability to downstream detection tasks alongside strong object detection performance.
Open-vocabulary detection: GLIP conditions object proposals on the text prompt, so it need not propose every possible novel object from an open set.This provides an alternative perspective on open-world object proposal challenges.
Efficient transfer: Prompt tuning seeks strong transfer performance with only a few task-specific parameters or none, reducing training and storage cost.GLIP's deep vision-language fusion makes prompt tuning effective, whereas it is less effective for shallow-fused models.

3. Grounded Language Image Pre-training

GLIP unifies object detection and phrase grounding by replacing fixed-class classification with language-conditioned region–phrase alignment, then deep-fuses visual and linguistic features. This formulation supports scalable grounding pre-training from semantically rich image-text data.

3.1. Unified Formulation: Object detection is reformulated as grounding each image region against candidate class phrases in a text prompt.The prompt contains the detection task’s object categories, treating each class name as a phrase to be grounded.
3.1. Unified Formulation: Region–phrase alignment scores replace fixed classifier logits, using contextual token features from a language encoder alongside visual region features.The image and text encoders produce features whose dot products provide alignment scores for grounding.
3.1. Unified Formulation: Detection and phrase grounding are theoretically equivalent for training and inference, while the reformulation enables direct zero-shot transfer to arbitrary detection tasks.The authors also report identical COCO validation performance for a DyHead detector before and after reformulation.
3.2. Language-Aware Deep Fusion: Deep fusion inserts cross-modality communication between image and language encoders rather than combining them only at the final alignment step.The cross-modality multi-head attention module lets each modality attend to the other, followed by single-modality fusion updates.
3.2. Language-Aware Deep Fusion: Deep-fused representations improve phrase grounding and make visual predictions conditioned on the text prompt, allowing one model to serve multiple detection tasks.Language conditioning is identified as crucial for using a single model across downstream detection tasks.
3.3. Pre-training with Scalable Semantic-Rich Data: Grounding data expand the visual vocabulary beyond conventional detection datasets, with Flickr30K and VG Caption containing 44,518 and 110,689 unique phrases, respectively.The authors contrast these phrase vocabularies with detection datasets whose largest scaling efforts cover no more than 2,000 categories.

4. Transfer to Established Benchmarks

GLIP transfers strongly across COCO, LVIS, and Flickr30K, including zero-shot and fine-tuned settings. Ablations show that grounding data and image-text data improve transfer, especially for rare categories.

60.8 AP on COCO val2017 and 61.5 AP on test-dev were achieved by fine-tuned GLIP-L, surpassing prior state of the art.
Gold grounding data improved MiniVal APr by 4.2 points, while image-text data added a further 3.1 points on LVIS.
87.1 Recall@1 on Flickr30K was achieved by GLIP-L, outperforming the previous state of the art by 2.8 points.
Adding grounding data consistently improved performance across different detection datasets, while grounding data plus image-text data outperformed scaling detection data alone on COCO and LVIS APr.The comparison used 0.66M detection data and 0.8M gold grounding data versus 2.66M detection data; adding image-text data widened the LVIS APr gap to 20.8 versus 15.0.

5. Object Detection in the Wild

GLIP transfers to diverse object-detection tasks with strong data efficiency and language-based adaptation. Grounding data and prompt tuning support recognition of novel concepts while reducing task-specific training or deployment requirements.

5. Object Detection in the Wild: GLIP is evaluated on 13 public datasets spanning distinct localization skills and real-world application scenarios.Examples include hand localization, pothole detection, and infrared dog-and-person detection.
5.1. Data Efficiency: GLIP reaches comparable performance with significantly less task-specific data than baselines, including zero-shot GLIP-T outperforming 5-shot DyHead-T.A one-shot GLIP-L is competitive with a fully supervised DyHead-T.
5.1. Data Efficiency: Grounding data substantially improves zero-shot performance on tasks involving novel concepts such as potholes and hands.Models with grounding data outperform corresponding models without grounding data on Pothole and EgoHands.
5.2. One Model for All Tasks: Expressive prompt descriptions can inject domain knowledge and improve zero-shot localization of novel entities such as stingrays.Adding the attributes “flat and round” enables successful localization of stingrays in the Aquarium dataset.
5.2. One Model for All Tasks: Prompt tuning and linear probing can approach full-model tuning for GLIP-T and GLIP-L, enabling more efficient deployment.The method adapts task-specific language inputs while retaining the grounding model.

6. Conclusion

GLIP unifies detection and phrase grounding to learn object-level, language-aware, semantic-rich representations. The paper reports promising zero-shot and fine-tuning transfer results while leaving scaling with text-image data size for future work.

6. Conclusion: GLIP unifies object detection and phrase grounding to learn object-level, language-aware, semantic-rich visual representations.The appendix includes empirical discussion of detection-grounding equivalence and visualizations of grounding predictions.
6. Conclusion: GLIP shows promising results in zero-shot and fine-tuning settings on established benchmarks and 13 downstream tasks.The appendix provides per-dataset results for the Section 5 experiments and additional teacher-model visualizations.
6. Conclusion: The paper leaves detailed analysis of how GLIP scales with text-image data size to future work.
Appendix: The appendix analyzes language-aware deep-fusion computational cost and reports that deep fusion adds less than 1x additional computational cost.Its efficiency table reports FPS and GPU memory consumption.

B. Equivalence Discussion between Detection and Grounding

The paper empirically examines when grounding and detection are equivalent and how prompt-length limits affect the reformulation. Equivalence holds when all categories fit in one prompt, while multiple prompts cause a minor performance drop.

When all object categories fit into a single prompt: 49.4 AP is achieved by both the classical detection model and its grounding reformulation on COCO 2017val.The matching performance supports equivalence when all object categories fit into a single prompt.
When all object categories fit into a single prompt: When all categories fit into one prompt, the paper concludes that grounding and detection tasks are equivalent.
When not all object categories can fit into a single prompt: Prompt length limits prevent large-vocabulary datasets such as Objects365 from fitting all category names into one prompt.The implementation limits input length to 256 tokens, below BERT’s 512-token capacity.
When not all object categories can fit into a single prompt: Splitting category names across multiple prompts is a practical solution during training and inference, but it incurs a minor performance drop.On COCO zero-shot, DyHead-T reaches 43.6 while grounding-reformulated GLIP-T (A) reaches 42.9 after Objects365 pre-training.

C.1. Pre-training Details

GLIP pre-training uses large-scale image-text and detection data with prompt construction and caption augmentations designed for language-grounded learning. The implementation also manages input-length and checkpoint-selection constraints during evaluation.

Pre-training configuration: Pre-training uses Swin-Tiny models with 32 GPUs or Swin-Large models with 64 GPUs, each with batch size 64.The base learning rates are 1 × 10^-5 for the language backbone and 1 × 10^-4 for other parameters.
Prompt design for detection data: Detection prompts randomly down-sample and shuffle category names while always retaining positive classes.Negative classes are sampled to construct prompts of up to 85 categories.
Augmentation for image-text data with generated boxes: Image-text examples with generated boxes are augmented by mixing negative captions with the matched positive caption.This trains the model to localize and align boxes with the correct phrases in a more difficult text input.
Computational cost: Deep fusion adds less than 1x additional computational cost according to the appendix’s computational-cost table.The table measures speed in FPS and memory consumption in GPU GB.
Evaluation details: LVIS categories are split into chunks of 40 for zero-shot evaluation, and validation performance selects checkpoints for LVIS and Flickr30K.The procedure addresses prompt-length limits and observed overfitting during pre-training.

C.3. Difference Between Public Data and Web-Crawled Data

GLIP-T models pretrained on public versus web-crawled image-text data show only slight differences, with each data source favoring a different benchmark.

The Cap4M model performs better on LVIS, while the CC3M+SBU model performs better on COCO.The authors conjecture that public data contains more common categories and fewer rare concepts because it is more extensively screened.

D. Computation Cost and Performance Analysis of Deep Fusion

The section analyzes language-aware deep fusion’s computational cost and task-dependent performance effects. Deep fusion adds limited overall computation but its benefits vary across datasets, especially on LVIS.

Computation cost: Less than 1x additional computational cost is associated with language-aware deep fusion overall.Inference comparisons use a P100 GPU with batch size 1, while training uses 16 V100 GPUs with gradient checkpointing for the fusion module.
Performance effects: Deep fusion benefits COCO, Flickr30K grounding, and low-resource ODinW transfer tasks.The analysis evaluates four GLIP-T variants pretrained with different kinds of data.
Performance effects: On LVIS, deep fusion’s effect is unclear: it can reduce common-category performance while improving rare-category performance when grounding data are present.With detection-only training, the authors report degradation; they hypothesize that language-model overfitting to training categories may be amplified by fusion.
Performance effects: For models trained only on O365, Flickr30K performance is significantly worse because the models are not trained to ground natural-language captions.This comparison is reported in the language-aware fusion ablation table.

E.4. One Model for All Tasks

GLIP supports deployment with prompt tuning and full-model tuning across tasks, while linear probing is generally less effective. Prompt design can provide language guidance without annotation or retraining.

Deployment efficiency: Prompt tuning performance for GLIP is competitive with full tuning, demonstrating deployment efficiency.The comparison is made on ODinW transfer datasets using results from prompt tuning and full-model tuning.
Deployment efficiency: Linear probing does not work well compared with full tuning, unlike prior reports for classical detection models.The authors suggest this may reflect the novel-domain transfer setting, where performance gaps are large for some tasks such as AerialDrone.
Evaluation setting: The evaluation covers 13 ODinW datasets and summarizes each dataset’s objects of interest and split image counts.The datasets are selected from Roboflow, excluding cases considered too challenging or requiring specific domain knowledge.
Prompt guidance: Manually designed prompts can improve CLIP-T AP without annotation or model retraining by adding attributes, common category names, or language context.Prompts are designed for six datasets, with a Pothole prompt retained because it benefits CLIP-T (C).

E.5. All Results

The paper reports comprehensive zero-shot, few-shot, and tuning results across 13 ODinW object-detection datasets. These results compare linear probing, prompt tuning, and full-model tuning for GLIP and relevant baselines.

All results: Performance is reported under 0-, 1-, 3-, 5-, and 10-shot settings as well as full-data training.The results are organized across Tables 14–16 and include multiple tuning strategies.
Zero-shot evaluation: Table 14 reports zero-shot performance on 13 ODinW datasets.
Model and tuning comparisons: Table 15 compares DyHead, GLIP-T, and GLIP-L using linear probing, prompt tuning, and full-model tuning.PascalVOC uses mAP at IoU=0.50:0.95 under the COCO evaluation script for consistency across datasets.
Model variants: Table 16 reports per-dataset performance for GLIP-T variants A, B, and C under linear probing, prompt tuning, and full-model tuning.PascalVOC is evaluated with the COCO mAP protocol for consistency with the other 12 datasets.