PaperlayerView exact revision on arXiv

Source-linked AI summary

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi

arXiv:2201.12086v2cs.CV

TL;DR

Existing vision-language pre-training models typically specialize in understanding or generation and rely on noisy web supervision. BLIP unifies both capabilities while bootstrapping cleaner image-text data, achieving state-of-the-art results across vision-language tasks and zero-shot video-language transfer.

  • Problem

    Existing vision-language models often specialize in either understanding or generation, while scaled web image-text pairs provide suboptimal supervision.

  • Method

    BLIP combines a multimodal encoder-decoder with caption generation and filtering to create cleaner training pairs from noisy web data.

  • Results

    BLIP achieves state-of-the-art performance across understanding and generation tasks, including zero-shot transfer to text-to-video retrieval and video question answering.

  • Takeaways & Limitations

    BLIP supports a unified approach to vision-language understanding and generation while bootstrapping datasets for further vision-language research.

  • Takeaways & Limitations

    The benchmark omits SNLI-VE because its test data has been reported to be noisy.

Abstract

from arXiv · show

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner. Code, models, and datasets are released at https://github.com/salesforce/BLIP.

1. Introduction

BLIP addresses limitations in existing vision-language pre-training by unifying understanding and generation within a flexible model while improving supervision from noisy web image-text pairs. Its MED architecture and CapFilt bootstrapping method support broad downstream transfer and strong vision- and video-language performance.

  • Motivation: Existing methods typically specialize in either encoder-based understanding or encoder-decoder generation, limiting direct transfer across retrieval and text-generation tasks.Encoder-based models are less straightforward to transfer to image captioning, while encoder-decoder models had not been successfully adopted for image-text retrieval.
  • Motivation: Web-collected image-text pairs provide scalable but noisy supervision that is suboptimal for vision-language learning.The paper identifies dataset scaling as a source of performance gains despite the shortcomings of noisy web text.
  • Method: CapFilt bootstraps noisy image-text data by generating synthetic captions and filtering noisy original and synthetic captions.A pre-trained MED is fine-tuned into a captioner and a filter for this process.
  • Method: BLIP introduces a multimodal mixture of Encoder-Decoder that operates as a unimodal encoder, image-grounded text encoder, or image-grounded text decoder.It is jointly pre-trained with image-text contrastive learning, image-text matching, and image-conditioned language modeling.
  • Results: Captioner-filter cooperation substantially improves downstream performance, with larger gains from more diverse captions.This observation is reported across various downstream tasks.
  • Results: BLIP achieves state-of-the-art results across image-text retrieval, image captioning, visual question answering, visual reasoning, and visual dialog, plus zero-shot video-language transfer.The zero-shot transfer covers text-to-video retrieval and videoQA.

2. Related Work

Related work identifies noisy web alt-text supervision and the difficulty of unifying vision-language understanding with generation. BLIP addresses these gaps with a flexible multimodal mixture of encoder-decoder and synthetic-caption-based pre-training.

  • Web-crawled image-alt-text pairs are widely used in VLP because human-annotated texts are prohibitively expensive, despite residual noise after rule-based filtering.
  • Unifying image-text retrieval and image captioning remains challenging because existing encoder, encoder-decoder, and unified encoder-decoder architectures cannot excel flexibly at both task types.
  • BLIP uses a multimodal mixture of encoder-decoder model to provide greater flexibility and downstream performance while keeping pre-training simple and efficient.
  • Unlike prior synthetic-example methods focused on low-resource language-only tasks, BLIP demonstrates the advantage of synthetic captions for large-scale vision-language pre-training.

3. Method

BLIP combines the multimodal mixture of encoder-decoder (MED) with three pre-training objectives to support both vision-language understanding and generation. Its CapFilt procedure bootstraps noisy web image-text data by generating synthetic captions and filtering mismatched texts.

  • MED architecture: MED is a multi-task architecture that operates as a unimodal encoder, image-grounded text encoder, or image-grounded text decoder.These functionalities enable a unified model for understanding and generation.
  • Pre-training objectives: BLIP jointly optimizes ITC, ITM, and LM objectives, requiring one visual-transformer pass and three text-transformer passes per image-text pair.ITC aligns visual and textual representations, while ITM learns fine-grained matching through binary classification with hard negatives.
  • Pre-training objectives: LM trains the image-grounded decoder autoregressively to generate image-conditioned text, using cross entropy with label smoothing of 0.1.Unlike MLM, LM provides caption-generation capability from visual information.
  • CapFilt: The filter uses ITC and ITM to determine whether each text matches its image, addressing web alt-texts that often inaccurately describe visual content.The captioner is an image-grounded text decoder fine-tuned with LM, whereas the filter is an image-grounded text encoder.
  • CapFilt: CapFilt improves noisy web corpora with a captioner that generates one synthetic caption per image and a filter that removes noisy original and synthetic texts.Both modules are initialized from the same pre-trained MED model and individually fine-tuned on COCO.

4. Experiments and Discussions

Experiments show that CapFilt improves downstream vision-language performance by combining synthetic caption generation with noise filtering, and that its gains scale with dataset and backbone size. Ablations further favor nucleus sampling and selective parameter sharing, while sharing captioner and filter parameters introduces confirmation bias.

  • Pre-training details: BLIP pre-trains for 20 epochs on 14M images, using ViT-B by default and initializing its vision and text transformers from ImageNet-pretrained ViT and BERTbase.The models use AdamW and 224 × 224 crops during pre-training, increasing resolution to 384 × 384 during finetuning.
  • CapFilt analysis: Applying the captioner or filter alone improves performance on 14M images, while combining them produces substantial gains over the original noisy web texts.The complementary effect is evaluated on image-text retrieval and image captioning under finetuned and zero-shot settings.
  • CapFilt analysis: CapFilt scales with both data and model size, and a ViT-L captioner and filter improve the base model even when the base model uses ViT-B.This supports scalability across larger datasets and larger vision backbones.
  • Synthetic caption generation: Nucleus sampling with p = 0.9 outperforms beam search for synthetic caption generation despite producing a higher filter noise ratio.The paper attributes this to more diverse and surprising captions, whereas beam search tends to produce safer, common captions.
  • Parameter sharing: Sharing text-encoder and decoder parameters except for self-attention layers improves performance and training efficiency, whereas sharing self-attention causes task conflict and degradation.During pre-training, the encoder and decoder share all parameters except the self-attention layers.
  • Parameter sharing: Sharing parameters between the captioner and filter decreases downstream performance because confirmation bias makes noisy captions less likely to be removed, with noise ratio falling from 25% to 8%.The captioner and filter are individually end-to-end finetuned on COCO when parameters are not shared.

5. Comparison with State-of-the-arts

BLIP improves over prior vision-language methods across retrieval, captioning, and VQA while using comparable or substantially less pre-training data than several strong baselines. It also transfers zero-shot to video-language tasks, achieving state-of-the-art results despite lacking temporal modeling.

  • Image-text retrieval: +2.7% average recall@1: BLIP outperforms ALBEF on COCO retrieval using the same 14M pre-training images.BLIP also achieves a large-margin zero-shot retrieval result on Flickr30K after transfer from COCO.
  • Image captioning: BLIP with 14M pretraining images substantially outperforms captioning methods using a similar amount of pre-training data.With 129M images, BLIP achieves competitive performance with LEMON, which uses 200M images; BLIP is detector-free and uses 384×384 inputs versus LEMON’s computationally heavy detector and 800×1333 inputs.
  • Visual question answering: +1.64%: BLIP using 14M images outperforms ALBEF on the VQA test set.Using 129M images, BLIP also achieves better performance than SimVLM, which uses 13× more pre-training data and a larger vision backbone.
  • Video-language transfer: +12.4% recall@1: zero-shot BLIP outperforms models finetuned on the target video dataset for text-to-video retrieval.BLIP achieves state-of-the-art performance on both text-to-video retrieval and video question answering despite domain differences and no temporal modeling.

6. Additional Ablation Study

Additional ablations show that CapFilt’s gains are not explained by longer training on noisy web texts. They also indicate that training a new model on the bootstrapped dataset is more effective than continuing from a previous pre-trained model.

  • Additional Ablation Study: The additional experiments evaluate whether CapFilt’s improvement comes from longer training or from how the bootstrapped dataset is used.The study includes ablations on CapFilt, including matched-sample training and continued training from a previous pre-trained model.
  • Training Duration: Replicating original web texts to match the bootstrapped dataset’s samples per epoch verifies that CapFilt’s improvement is not due to longer training time.Using the same number of training samples per epoch isolates dataset bootstrapping from the extra training caused by the larger bootstrapped dataset.
  • Model Initialization: Training a new model on the bootstrapped dataset provides more gain than continuing training from the previous pre-trained model.Table 13 reports that continued training does not help, consistent with the stated knowledge-distillation practice that a student model is not initialized from its teacher.

7. Conclusion

BLIP is a unified vision-language pre-training framework that achieves state-of-the-art performance across understanding and generation tasks by bootstrapping noisy image-text data. The released bootstrapped dataset supports future research, while further gains may come from repeated bootstrapping, larger synthetic-caption corpora, and CapFilt ensembles.

  • Conclusion: BLIP combines a multimodal encoder-decoder model with bootstrapped noisy image-text data for vision-language understanding and generation tasks.The data process injects diverse synthetic captions and removes noisy captions.
  • Conclusion: The bootstrapped dataset is released to facilitate future vision-language research.
  • Future directions: Potential improvements include multiple bootstrapping rounds, multiple synthetic captions per image, and ensembles of captioners and filters in CapFilt.These directions target both model and data improvements.

A. Downstream Task Details

Downstream vision-language tasks are fine-tuned with shared optimization settings and task-specific image-resolution and dataset protocols. The section specifies evaluation and inference procedures for retrieval, captioning, VQA, NLVR2, and VisDial.

  • Optimization and resolution: All downstream tasks use AdamW with 0.05 weight decay, cosine learning-rate scheduling, and 384 × 384 images except VQA at 480 × 480.Table 14 summarizes the downstream fine-tuning hyperparameters.
  • Image-Text Retrieval: Image-text retrieval uses the Karpathy split for COCO and Flickr30K, with 113/5k/5k and 29k/1k/1k train/validation/test images, respectively.The split is applied to both datasets.
  • Image Captioning: Image captioning fine-tunes on COCO’s Karpathy train split and evaluates on its Karpathy test and NoCaps validation splits.Inference uses beam search with beam size 3 and maximum generation length 20.
  • VQA: VQA uses VQA2.0, whose train/validation/test splits contain 83k/41k/81k images, with training and validation data plus Visual Genome samples.At inference, the decoder ranks 3,128 candidate answers.
  • NLVR2 and VisDial: NLVR2 uses its official split, while VisDial v1.0 uses its training split for fine-tuning and validation set for evaluation.These protocols are specified for the respective downstream tasks.

B. Additional Examples of Synthetic Captions

Figure 6 presents additional image-text examples in which noisy web captions are filtered out and synthetic captions are retained as clean training samples.

  • Additional Examples of Synthetic Captions: Figure 6 shows images and texts where web captions are filtered out while synthetic captions are kept as clean training samples.These examples illustrate the caption-filtering process used to obtain cleaner training data.

C. Pre-training Dataset Details

This section presents the statistics of the pre-training datasets and illustrates how web and synthetic captions are filtered. Accepted synthetic texts are distinguished from rejected ones in the examples.

  • Dataset statistics: Table 15 reports the statistics of the pre-training datasets.The supplied passage identifies Table 15 as the source of these dataset statistics.
  • Text examples: Figure 6 shows examples of web text Tw and synthetic text Ts.The figure compares examples from the web-text and synthetic-text sources.
  • Text filtering: The filter accepts green texts and rejects red texts in the illustrated examples.Color coding distinguishes texts retained by the filter from those removed.
Loading 2201.12086v2…