PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Yan Zeng, Xinsong Zhang, Hang Li

arXiv:2111.08276v3cs.CLcs.CV

TL;DR

Existing vision-language pre-training methods struggle to learn multi-grained alignments and relations among multiple objects. X-VLM addresses this by locating text-associated visual concepts and aligning texts with them across granularities, consistently outperforming existing methods across downstream tasks.

  • Problem

    Existing fine-grained object-centric methods cannot easily represent relations among multiple objects, while coarse-grained methods cannot effectively learn object-level vision-language alignments.

  • Method

    X-VLM performs multi-grained pre-training by locating visual concepts using associated texts and aligning those texts with relevant concepts at multiple granularities.

  • Results

    X-VLM consistently outperforms existing methods across image-text retrieval, visual reasoning, visual grounding, and image caption generation tasks.

  • Takeaways & Limitations

    X-VLM provides an efficient approach for learning fine-grained vision-language alignments with moderate pre-training data and model size.

  • Takeaways & Limitations

    Evaluation with different X-VLM backbones is left for future work.

Abstract

from arXiv · show

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

1. Introduction

Existing vision-language pre-training methods struggle to learn alignments spanning object, region, and image levels, especially for relations among multiple objects. X-VLM addresses this by locating text-associated visual concepts and aligning them at multiple granularities, yielding strong downstream results.

  • Motivation: Object-centric methods use detected regions but cannot easily represent relations among multiple objects, while coarse-grained methods miss effective object-level alignments.These limitations affect tasks including visual reasoning, visual grounding, and image captioning.
  • Approach: X-VLM reformulates training data so captions, region descriptions, and object labels directly associate text with image-level, region-level, or object-level visual concepts.The reformulation supports diverse text descriptions and visual concepts without restricting alignment to one granularity.
  • Approach: X-VLM jointly locates visual concepts from associated texts and aligns texts with those concepts using box regression, intersection over union, contrastive, matching, and masked language modeling losses.The model contains image, text, and cross-modal encoders for learning these alignments.
  • Results: 4.65% absolute R@1 gain on MSCOCO image-text retrieval over VinVL demonstrates the benefit of X-VLM’s learned multi-grained alignments.X-VLM also outperforms ALIGN, ALBEF, and METER despite their greater data or parameter resources.
  • Results: 0.79% on VQA, 1.06% on NLVR2, and 4.5% on RefCOCO+ are absolute improvements over specified prior systems.The reported comparisons include VinVL for VQA and NLVR2, and UNITER for RefCOCO+ grounding.
  • Results: X-VLM consistently outperforms existing state-of-the-art methods across many downstream vision-language tasks during fine-tuning.The paper reports effectiveness on retrieval, reasoning, grounding, and caption generation.

2. Related Work

Related vision-language pre-training methods primarily learn either object-level alignments through object detection or image-level alignments from overall image features. X-VLM instead learns visual concepts at multiple granularities in a unified way without restricting alignment to object or image levels.

  • Fine-grained approaches: Fine-grained methods represent images with object-centric features extracted by object detectors and align text with those detected regions.Detectors identify candidate object regions and classify each region before representation learning.
  • Coarse-grained approaches: Coarse-grained methods encode overall image features using convolutional networks or vision transformers for vision-language alignment.The related-work discussion states that their performance is usually lower than fine-grained approaches.
  • Hybrid approaches: Some methods learn object-level and image-level alignments but still rely on object detectors, retaining the associated limitations.The paper contrasts this detector dependence with X-VLM’s approach.
  • X-VLM: X-VLM learns visual concepts across granularities in a unified way rather than separately aligning objects through classification and images through caption generation.Its alignments are not restricted to object-level or image-level representations.

3. Method

X-VLM builds multi-grained visual concept representations and aligns them with texts at image, region, and object levels. It combines text-conditioned box localization with contrastive, matching, and masked-language objectives.

  • X-VLM uses image, text, and cross-modal Transformer encoders, with cross-attention fusing vision and language features.
  • Images are reformulated with captions and multiple box-associated object or region descriptions, while some examples may lack captions or boxes.
  • Vision Encoding: The image encoder aggregates position-preserving patch features into representations for each object, region, or the entire image.
  • Bounding Box Prediction: X-VLM predicts each visual concept’s bounding box from image and text representations using a linear combination of L1 and generalized IoU losses.
  • Cross-Modal Alignment: Text–concept alignment uses contrastive learning, hard-negative matching prediction, and masked language modeling extended across objects, regions, and images.
  • Pre-training Objective: The overall X-VLM pre-training objective combines bounding-box prediction with contrastive, matching, and masked-language losses.

4. Experiment

X-VLM is evaluated in 4M and 16M pre-training settings across image-text retrieval, visual reasoning, visual grounding, and image captioning. The experiments use cleaned datasets to avoid overlap with downstream evaluation images and adapt the model to each task.

  • Pre-training datasets: The 4M and 16M settings combine in-domain COCO and Visual Genome data with additional out-of-domain caption datasets in the larger setting.The 4M setting uses COCO and Visual Genome annotations, while the 16M setting adds Conceptual Captions, Objects365, and OpenImages.
  • Pre-training datasets: The 4M setting uses 2.5M object annotations and 3.7M region annotations, reformulated so each box is paired with a text describing its visual concept.Descriptions may refer to an object, region, or the image itself.
  • Experimental controls: Pre-training and downstream evaluation data are cleaned to remove overlapping images and reduce information leakage.The filtering excludes downstream validation and test images and co-occurring Flickr30K images matched by URL.
  • Downstream tasks: X-VLM is adapted to image-text retrieval, visual question answering, NLVR2, visual grounding, and image captioning using task-specific fine-tuning procedures.Retrieval uses Recall@K, VQA constrains generation to 3,129 candidate answers, NLVR2 extends reasoning to two images, and captioning adapts the model to a multimodal decoder.
  • Evaluation: The evaluation reports image-text retrieval results on MSCOCO and Flickr30K and broader downstream results in Table 3.Table 3 covers VQA, NLVR2, RefCOCO+, and COCO Captioning, including indicators for weak supervision and CIDEr optimization.
  • Downstream tasks: For visual grounding, X-VLM directly predicts target bounding boxes and is also evaluated in a weakly supervised setting using only image-text pairs.This contrasts with prior ranking formulations that rely on object-detector region proposals.

4.4. Results on Image-Text Retrieval

X-VLM is evaluated against retrieval and broader vision-language baselines, including models using object-centric or overall-image features. It reports strong results across retrieval, reasoning, grounding, captioning, and ablations supporting multi-grained alignment.

  • Image-text retrieval: X-VLM(4M) outperforms previous methods by a large margin on image-text retrieval despite fewer parameters or less training data.Increasing training instances to 16M further improves R@1 and substantially improves zero-shot image-text retrieval.
  • Visual grounding: X-VLM(4M) achieves absolute improvements of 4.5% over UNITER and 1.1% over MDETR on RefCOCO+ grounding.Unlike ranking-based approaches using detector proposals, X-VLM directly predicts target boxes.
  • Image captioning: X-VLM performs comparably to SimVLM on image caption generation, including with 16M pre-training instances versus SimVLM’s 1.8B image-text pairs and large text corpus.X-VLM is described as an encoder-only model in this comparison.
  • Ablation study: Removing object or region concepts hurts performance, while removing bounding-box loss produces the lowest Meta-Sum.With only 210K densely annotated images in the 4M setting, Meta-Sum increases from 580.6 to 605.2 when the multi-grained components are used.

5. Conclusion and Discussion

The paper proposes X-VLM as an efficient multi-grained vision-language pre-training approach based on locating visual concepts and aligning text with them. Across downstream tasks, it outperforms existing methods that may use larger models or more data, while supporting fine-grained understanding and moderate-size deployment.

  • X-VLM learns multi-granular alignments by locating visual concepts from associated texts and aligning the texts with relevant concepts.
  • Across image-text retrieval, visual reasoning, visual grounding, and image caption generation, X-VLM outperforms existing methods that may use larger models or more pre-training data.
  • X-VLM shows better fine-grained vision-language understanding and is described as easier to deploy in a moderate model size.The paper gives image-caption detail and assistance for people with vision disabilities as an application example.

A.1. Statistics of Object and Region Annotations

The pre-training annotation statistics cover object and region annotations across datasets, with region annotations available only in Visual Genome. OpenImages additionally supplies relationship, property, and action annotations that can serve as region annotations.

  • Table 5 reports statistics of object and region annotations used in pre-training.
  • Only Visual Genome contains region annotations among the listed datasets.
  • OpenImages provides relationship, object-property, and human-action annotations that can also be viewed as region annotations.Examples include woman playing guitar, beer on table, table is wooden, and woman is jumping.
  • Pre-processing removes invalid, very small, or highly overlapping region samples before retaining the reported annotation counts.Examples of retained counts include 446,873 COCO objects, 2,043,927 Visual Genome objects, and 3,699,598 Visual Genome regions.

A.2. Implementation Details of Downstream Tasks

The downstream evaluations follow established fine-tuning settings while adapting training objectives and model components to each vision-language task. These adaptations include multi-image reasoning, answer generation, visual grounding, and image captioning.

  • General settings: Fine-tuning generally follows existing methods, with cleaned pre-training datasets used to avoid overlap-related data leaks.The cleaned datasets address overlaps between downstream tasks and COCO or Visual Genome images.
  • Image-Text Retrieval: Image-text retrieval uses MSCOCO and Flickr30K with Karpathy splits, multiple-positive contrastive targets, and top-k cross-modal reranking.The model is fine-tuned for 10 epochs; k is 256 for MSCOCO and 128 for Flickr30K.
  • Visual Question Answering: VQA trains a six-layer Transformer decoder to generate answers from cross-modal encoder outputs, restricting inference to 3,129 candidate answers.The decoder is fine-tuned for 10 epochs, and the candidate set includes a NULL answer.
  • Natural Language for Visual Reasoning: NLVR2 extends the cross-modal encoder for two-image reasoning and adds one epoch of pre-training that assigns a text to the first image, second image, or neither.The model is subsequently fine-tuned for 10 epochs.
  • Visual Grounding: Visual grounding is evaluated in supervised and weakly supervised RefCOCO+ settings, with an additional one-epoch Lbbox pre-training step in the supervised case.Both settings are followed by 10 epochs of fine-tuning; the supplied passage truncates the weakly supervised procedure.
  • Image Captioning: Image captioning adapts X-VLM to a multimodal decoder without adding a decoder, using one epoch of language-modeling pre-training on 4M data before COCO fine-tuning.Evaluation uses BLEU-4 and CIDEr on the Karparthy test split.

A.3. Zero-Shot Image-Text Retrieval Results

X-VLM is evaluated for zero-shot image-text retrieval on MSCOCO and Flickr30K against dual-encoder methods trained only with retrieval objectives. Despite combining objectives, it achieves competitive zero-shot retrieval results.

  • Zero-Shot Image-Text Retrieval Results: Table 6 reports zero-shot image-text retrieval results on MSCOCO and Flickr30K for image retrieval and text retrieval.The table compares X-VLM with dual-encoder state-of-the-art methods including CLIP and ALIGN.
  • Zero-Shot Image-Text Retrieval Results: X-VLM achieves very competitive zero-shot retrieval results despite being pre-trained with a combination of objectives rather than only the retrieval objective.The comparison is against dual-encoder SoTAs CLIP and ALIGN.

A.4. Case Study

The case study visualizes X-VLM’s region predictions and word-level alignments on unseen RefCOCO+ test images. The examples show accurate localization when descriptions differ by a single word and alignment of relational concepts to image regions.

  • Case Study: X-VLM predicts correct regions in supervised and weakly supervised settings even when textual descriptions differ by a single word.Supervised examples use predicted bounding boxes, while weakly supervised examples use Grad-CAM visualizations from fourth-layer cross-attention maps.
  • Case Study: Grad-CAM visualizations correspond to individual words, with the subject word shown for descriptions such as “brown dog.”The figure presents the visualization of the subject word, such as “dog.”
  • Case Study: X-VLM aligns relational concepts such as “pulling” and “holding” with their corresponding image regions.Figure 5 illustrates both bounding-box prediction and per-word visualization on unseen images.
Loading 2111.08276v3…