PaperlayerView exact revision on arXiv

Source-linked AI summary

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, Lijuan Wang

arXiv:2206.07643v2cs.CVcs.CLcs.LG

TL;DR

Existing VLP systems typically specialize in either image-level tasks or region-level understanding, leaving a unified end-to-end approach with broad task coverage underdeveloped. FIBER addresses this gap with deep backbone fusion and a coarse-to-fine pre-training pipeline, achieving consistent improvements across diverse VL benchmarks while reducing dependence on box-level annotations.

  • Problem

    Existing end-to-end VLP approaches generally focus on image-level tasks or region-level tasks, while fine-grained pre-training requires costly high-resolution inputs and box annotations.

  • Method

    FIBER inserts cross-attention into image and text backbones and uses coarse image-text pre-training followed by fine-grained image-text-box pre-training.

  • Results

    FIBER delivers consistent performance improvements over strong baselines across image-level and region-level VL tasks, including VQA, captioning, retrieval, grounding, and detection.

  • Takeaways & Limitations

    The unified architecture and two-stage strategy support broad VL task coverage while reducing the requirement for expensive box-level annotations.

  • Takeaways & Limitations

    The approach may inherit societal biases from pre-training data and can incur environmental costs during pre-training.

Abstract

from arXiv · show

Vision-language (VL) pre-training has recently received considerable attention. However, most existing end-to-end pre-training approaches either only aim to tackle VL tasks such as image-text retrieval, visual question answering (VQA) and image captioning that test high-level understanding of images, or only target region-level understanding for tasks such as phrase grounding and object detection. We present FIBER (Fusion-In-the-Backbone-based transformER), a new VL model architecture that can seamlessly handle both these types of tasks. Instead of having dedicated transformer layers for fusion after the uni-modal backbones, FIBER pushes multimodal fusion deep into the model by inserting cross-attention into the image and text backbones, bringing gains in terms of memory and performance. In addition, unlike previous work that is either only pre-trained on image-text data or on fine-grained data with box-level annotations, we present a two-stage pre-training strategy that uses both these kinds of data efficiently: (i) coarse-grained pre-training based on image-text data; followed by (ii) fine-grained pre-training based on image-text-box data. We conduct comprehensive experiments on a wide range of VL tasks, ranging from VQA, image captioning, and retrieval, to phrase grounding, referring expression comprehension, and object detection. Using deep multimodal fusion coupled with the two-stage pre-training, FIBER provides consistent performance improvements over strong baselines across all tasks, often outperforming methods using magnitudes more data. Code is available at https://github.com/microsoft/FIBER.

1 Introduction

FIBER addresses the gap between end-to-end models for image-level VL tasks and region-level systems by combining deep multimodal fusion with coarse-to-fine pre-training. Its single architecture supports tasks from VQA and retrieval to grounding and detection, while reducing reliance on box-level data.

  • Motivation: Existing end-to-end VLP models mainly target image-level understanding, whereas region-level systems target bounding-box prediction and generally do not support captioning or retrieval.Fine-grained pre-training also requires costly high-resolution inputs and cumbersome box annotations.
  • Architecture: FIBER inserts cross-attention directly into the image and text backbones instead of using only dedicated fusion layers above them.The architecture can switch cross-attention off for dual-encoder retrieval and on for fusion-based VQA and captioning.
  • Pre-training: FIBER uses one architecture across coarse-grained image-text pre-training and fine-grained image-text-box pre-training.Coarse training uses image-text matching, masked language modeling, and contrastive losses; fine-grained training adds localization and word-region alignment losses.
  • Results: FIBER provides consistent improvements over strong baselines across image-level and region-level VL tasks, including VQA, captioning, retrieval, grounding, and detection.The evaluated tasks include VQAv2, NLVR2, COCO captioning, NoCaps, retrieval, phrase grounding, referring expression comprehension, and multiple detection benchmarks.

2 Related Work

Related work spans classical and end-to-end VLP, vision-oriented VL modeling, and unified systems. FIBER is positioned as a broader end-to-end model supporting both image-level and region-level tasks, including dual-encoder retrieval.

  • Classical VL pre-training: Early VLP methods commonly used pre-trained object detectors to provide offline image-region features, while newer systems use images directly with convolutional or vision-transformer backbones.The related work contrasts detector-based approaches such as UNITER, OSCAR, VILLA, and VinVL with end-to-end models.
  • Positioning FIBER: FIBER is presented as the only compared VLP model supporting all considered tasks, including VQA, captioning, O(n + m) retrieval, visual grounding, and object detection.Here O(n + m) retrieval means the model backbones process n images and m text sentences O(n + m) times.
  • VL for vision tasks: Vision-oriented VL methods use image-text data for image encoders, while MDETR extends end-to-end detection with contrastive and alignment objectives for region-text correspondence.This line of work connects image-level pre-training with phrase grounding and detection.
  • Unified VL modeling: Unified VL systems differ in task coverage: some support generation or selected task combinations, but are not demonstrated across standard grounding, detection, captioning, and retrieval benchmarks.The cited comparison identifies missing combinations across VL-T5, UniT, GPV, FLAVA, UniTAB, and OFA.

3 Method

FIBER inserts gated cross-modal fusion inside the vision and language backbones, then uses a two-stage pre-training paradigm to support both image-level and region-level VL tasks. Cross-attention can be enabled for fusion-based tasks or disabled for dual-encoder retrieval, while fine-grained training passes text-aware image representations to detection components.

  • Architecture: FIBER inserts multimodal fusion inside the vision and language backbones rather than stacking fusion layers on top.A gating mechanism is included for the cross-modal layers.
  • Architecture: α is a learnable parameter initialized to 0, allowing cross-modal interactions to begin without altering the backbones' original computational flow.Setting α to 0 switches off interactions and enables the dual-encoder setting.
  • Two-stage pre-training: The two-stage paradigm first pre-trains shared models for coarse-grained VL tasks, then uses fine-grained pre-training for region-level outputs.The method is designed to share parameters across both task categories.
  • Coarse-grained pre-training: Coarse-grained pre-training uses masked language modeling, image-text matching, and image-text contrastive objectives.Cross-attention is switched off for image-text contrastive learning and switched on for masked language modeling and image-text matching.
  • Fine-grained pre-training: Fine-grained pre-training switches on cross-attention so high-resolution image representations passed to the FPN are already text-aware.Swin Transformer hierarchical representations and an FPN support object detection training across image scales.
  • Downstream tasks: FIBER adapts to downstream tasks by using fusion encoding for VQA and related classification tasks, dual encoding for retrieval, and grounding-based detection outputs.Its downstream adaptations are described across classification, retrieval, and region-level tasks.

4 Experiments

FIBER is evaluated across image-level and region-level vision-language tasks using coarse- and fine-grained data, with strong performance and efficiency relative to baselines.

  • Experimental Setup: FIBER uses about 4M images for coarse-grained pre-training and about 0.8M images for fine-grained pre-training, excluding downstream validation and test data.Coarse data comes from COCO, Conceptual Captions, SBU Captions, and Visual Genome; fine-grained data comes from MDETR-curated data and Objects365.
  • Experimental Setup: FIBER inserts cross-attention into the top six vision and text backbone blocks, using 384 × 384 inputs for coarse pre-training and 800×1,333 inputs for fine-grained pre-training.The hierarchical vision transformer supports the higher-resolution fine-grained setting more efficiently.
  • Vision-Language Classification: FIBER achieves the best reported performance on VQAv2 and NLVR2 in its setting, surpassing BLIP and SimVLM despite using 4M rather than 129M or 1.8B images.These results support fusion in the backbone as an alternative to appending fusion layers above unimodal backbones.
  • Image Captioning: FIBER improves captioning over models trained on the same data, and integrating GOLD yields results that outperform models trained with hundreds of millions of images.The model also establishes absolute state-of-the-art COCO CIDEr scores for base-size models.
  • Phrase Grounding: FIBER achieves 87.4 Recall@1 on Flickr30k Entities without subsequent fine-tuning, surpassing a larger model while using 25× less fine-grained data.It also outperforms similarly sized GLIP-B without coarse-grained VL training, supporting the value of backbone fusion for fine-grained understanding.

5 Conclusion

The paper proposes FIBER and a coarse-to-fine pre-training pipeline, reporting consistent improvements across diverse tasks while reducing reliance on expensive box-level annotations. It also identifies potential bias and environmental costs as deployment concerns.

  • The approach delivers consistent improvements over strong baselines across diverse vision-language tasks.
  • The results set new state-of-the-art scores while reducing the requirement for expensive box-level annotations.
  • The approach may inherit undesirable societal biases from its pre-training data and incur environmental costs during pre-training.

Checklist

The checklist reports that the paper describes its contributions, limitations, societal impacts, assets, and experimental reproducibility materials, but does not report error bars from repeated experiments.

  • The paper states that it describes its contributions and limitations.
  • It reports addressing potential negative societal impacts and conforming to ethics review guidelines.
  • The paper states that it provides code, data, instructions, and training details needed to reproduce the experiments.
  • The experiments do not report error bars from multiple runs with respect to random seeds.
  • The paper states that it cites existing assets, discusses their licenses, and includes no new assets.

A.1 Implementation Details

Implementation details specify task-specific fine-tuning procedures for classification, retrieval, captioning, grounding, referring expression comprehension, and object detection.

  • Vision-Language Classification: Classification uses task-specific resolutions, learning rates, batch size 512, and ten training epochs.
  • Image-Text Retrieval: Retrieval removes cross-attention layers to use the dual encoder and evaluates Recall@1,5,10 for text and image retrieval.
  • Image Captioning: Captioning keeps image-to-text attention, applies causal decoding, and trains autoregressively.
  • Phrase Grounding: Phrase grounding on Flickr30k is evaluated directly after fine-grained pre-training without further fine-tuning.
  • REC and object detection use task-specific fine-tuning schedules, learning rates, epochs, augmentation choices, and early stopping.
  • Object Detection: Object detection represents category names as text separated by full stops and learns to ground image regions to those names.

A.2 Ablation Study

Ablations examine fusion strategies, pre-training objectives, the two-stage pipeline, and backbone choices. They support controllable cross-attention fusion, the value of all coarse-grained objectives, and the effectiveness of coarse-to-fine pre-training.

  • Ablation Study on the Fusion Strategies: Introducing α permits more fusion layers with robust performance, leading the authors to fuse the top six backbone layers.
  • Ablation Study on the Fusion Strategies: Merged attention and co-attention perform comparably without α, while increasing fusion layers can reduce performance.
  • Ablation Study on Pre-training Objectives: Removing any coarse-grained pre-training objective reduces performance, while hard negative mining improves both VQA and retrieval.
  • Ablation Study on Pre-training Objectives: Masked language modeling is most effective for VQA, whereas removing it does not hurt retrieval performance.
  • Ablation Study on the Two-Stage Pre-training: Coarse-grained pre-training improves both evaluated tasks and can outperform a larger model trained with more data on RefCOCO+.
  • Ablation Study on Different Backbones: RoBERTa with Swin Transformer is selected because it performs slightly better before VLP and is more suitable for region-level tasks than CLIP-ViT.

A.3 Additional Results

The additional experiments examine captioning, open-ended VQA, unimodal performance, and the effects of using different pre-training checkpoints. Results show strong transfer across modalities, while VQA benefits from retaining the coarse-grained checkpoint.

  • Image Captioning: GOLD and CIDEr optimization improve image-captioning performance across metrics on COCO and NoCaps.NoCaps evaluation covers in-domain, near-domain, out-domain, and entire-domain settings.
  • Image Captioning: The seq2seq and ladder architectures achieve comparable image-captioning performance when adapting FIBER.The seq2seq design feeds the final encoding layer into image-to-text cross-attention modules, while the ladder structure more closely matches pre-training.
  • Open-ended VQA: FIBER’s generative model outperforms VL-T5 and VL-BART on open-ended VQA but trails SimVLM, especially for out-of-domain questions.The authors attribute SimVLM’s advantage possibly to its training on over a billion image-caption pairs.
  • Unimodal Performance: After coarse-grained pre-training, the text encoder remains stronger than SimVLM on GLUE despite marginal drops on some unimodal tasks.The authors suggest the drops may reflect exposure to simple text captions during vision-language pre-training.
  • Unimodal Performance: After coarse-grained pre-training, the image encoder achieves comparable or better image classification and semantic segmentation performance even without fusion modules.This suggests the image encoder remains powerful after vision-language pre-training when used for vision-only tasks.
  • Checkpoint Selection: VQAv2 test-dev performance drops from 78.55 to 74.3 when the fine-grained checkpoint is fine-tuned for VQA.The result supports using different pre-training stages for tasks with different characteristics.

A.4 Visualization after coarse-grained pre-training

Grad-CAM visualizations of the coarse-grained checkpoint show cross-attention aligning caption tokens with corresponding image regions. These examples suggest that visual grounding can emerge implicitly during coarse-grained pre-training.

  • Cross-attention Visualization: Grad-CAM visualizations show the model attending from caption tokens to their corresponding image regions.The visualizations use examples from the NoCaps validation set.
  • Cross-attention Visualization: The observed token-region alignments suggest that the model learns visual grounding implicitly during coarse-grained pre-training.The conclusion is based on qualitative visualizations rather than a quantitative grounding evaluation.

A.5 Visualization after fine-grained pre-training

The paper provides qualitative visualizations after fine-grained pre-training across grounding, referring-expression, and scene-prompt examples. It also compares alternative image-captioning adaptation architectures and reports backbone information.

  • Fine-grained Visualizations: Fine-grained pre-training visualizations include phrase-grounding examples from Flickr30k Entities.The examples are drawn from the validation set.
  • Fine-grained Visualizations: The visualizations include referring expression comprehension examples from the RefCOCO+ validation set.They are presented as qualitative examples after fine-grained pre-training.
  • Fine-grained Visualizations: Additional examples show images paired with prompts for various items in the scene.These examples are part of the post-fine-grained-pre-training visualization set.
  • Backbone Comparison: The appendix identifies Transformer backbones for different models in its backbone comparison.The table notes that XFM stands for Transformer.
Loading 2206.07643v2…