Source-linked AI summary
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi
TL;DR
Vision-language pre-training is computationally expensive because large models are trained end to end. BLIP-2 uses a lightweight, two-stage Querying Transformer with frozen image encoders and language models, achieving state-of-the-art results across tasks, including an 8.7% VQAv2 gain over Flamingo with 54× fewer trainable parameters.
Problem
End-to-end vision-language pre-training with increasingly large models and datasets incurs high computation costs and limits use of readily available unimodal models.
Method
BLIP-2 bridges frozen image encoders and large language models with a lightweight Querying Transformer trained in representation-learning and generative-learning stages.
Results
8.7% higher zero-shot VQAv2 performance than Flamingo with 54× fewer trainable parameters, alongside state-of-the-art results across vision-language tasks.
Takeaways & Limitations
BLIP-2 demonstrates compute-efficient vision-language pre-training and zero-shot instructed image-to-text generation with emerging visual reasoning and conversation capabilities.
Takeaways & Limitations
BLIP-2 does not improve in-context VQA with examples and can generate unsatisfactory or harmful outputs inherited from its frozen language models.
Abstract
from arXiv · showhide
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.
1. Introduction
BLIP-2 addresses the high computation cost of vision-language pre-training by keeping pre-trained image encoders and language models frozen and bridging them with a lightweight Q-Former. Its two-stage training supports effective cross-modal alignment, state-of-the-art vision-language performance, and instructed zero-shot image-to-text generation.
- Method: BLIP-2 pre-trains the Q-Former in two stages: vision-language representation learning followed by vision-to-language generative learning.The first stage learns text-relevant visual representations; the second trains representations interpretable by the frozen language model.
- Motivation: BLIP-2 bootstraps vision-language pre-training from frozen, off-the-shelf vision and language models to reduce computation cost and counteract catastrophic forgetting.Pre-trained vision models provide visual representations, while large language models provide language generation and zero-shot transfer abilities.
- Method: The lightweight Q-Former uses learnable query vectors to extract visual features and acts as an information bottleneck between the frozen image encoder and frozen language model.It addresses the challenge that frozen language models have not seen images during unimodal pre-training.
- Results: BLIP-2 achieves state-of-the-art performance across visual question answering, image captioning, and image-text retrieval.The framework effectively leverages both frozen pre-trained image models and language models.
- Capabilities: BLIP-2 supports zero-shot image-to-text generation that follows natural language instructions, enabling visual knowledge reasoning and visual conversation.The capability is powered by large language models such as OPT and FlanT5.
- Results: 8.7%: BLIP-2 outperforms Flamingo on zero-shot VQAv2 while using 54× fewer trainable parameters.The efficiency comes from frozen unimodal models and the lightweight Q-Former.
2. Related Work
Vision-language pre-training spans several model architectures, but most methods rely on costly end-to-end training and cannot flexibly reuse unimodal pre-trained models. BLIP-2 instead leverages both frozen image encoders and frozen LLMs for vision-language tasks, targeting stronger performance at lower computation cost.
- Vision-language pre-training uses dual-encoder, fusion-encoder, and encoder-decoder architectures for different downstream tasks.
- Most VLP methods perform end-to-end pre-training on large-scale image-text datasets, creating high computation costs as model size increases.
- End-to-end pre-training is inflexible for leveraging readily available unimodal pre-trained models, including large language models.
- BLIP-2 differs from prior methods by effectively and efficiently using both frozen image encoders and frozen LLMs across vision-language tasks.The approach is described as achieving stronger performance at a lower computation cost.
3. Method
BLIP-2 bridges a frozen image encoder and frozen LLM with a trainable Querying Transformer (Q-Former) pre-trained in two stages. Q-Former uses a compact set of query embeddings and complementary representation-learning objectives to extract text-relevant visual information before conditioning language generation.
- Q-Former architecture: Q-Former bridges a frozen image encoder and frozen LLM through two stages: vision-language representation learning followed by vision-to-language generative learning.The first stage uses a frozen image encoder, while the second uses a frozen LLM.
- Q-Former architecture: Q-Former contains image and text transformer submodules sharing self-attention layers, with cross-attention connecting learnable queries to frozen image features.The image transformer extracts visual features, while the text transformer can act as an encoder or decoder.
- Q-Former architecture: 32 queries of dimension 768 produce a 32 × 768 representation, forming a bottleneck much smaller than frozen image features such as 257 × 1024 for ViT-L/14.This bottleneck works with the pre-training objectives to force extraction of visual information most relevant to text.
- First-stage pre-training: The first stage jointly optimizes image-text contrastive learning, image-grounded text generation, and image-text matching with distinct attention masks.These objectives respectively align image and text representations, train text generation conditioned on images, and learn fine-grained matched-versus-unmatched alignment.
- Second-stage pre-training: The second stage linearly projects Q-Former outputs into the frozen LLM’s text-embedding dimension and prepends them as soft visual prompts.Decoder-based LLMs use language modeling loss, while encoder-decoder LLMs use prefix language modeling loss.
4. Experiment
BLIP-2 delivers strong zero-shot and fine-tuned performance across vision-language tasks with substantially fewer trainable parameters, while its instruction-following image-to-text generation and modular design support broad capabilities. Experiments further show that representation learning, stronger encoders or LLMs, and image-grounded generation objectives improve alignment and downstream results.
- BLIP-2 achieves improved performance over previous state-of-the-art models across zero-shot vision-language tasks with substantially fewer trainable parameters.
- Zero-shot VQA: 8.7%: BLIP-2 outperforms Flamingo80B on VQAv2 while using 54x fewer trainable parameters, achieves state-of-the-art results on VQAv2 and GQA, and ranks second on OK-VQA.
- Zero-shot VQA: A stronger image encoder or LLM improves performance: ViT-g beats ViT-L, larger models beat smaller models within an LLM family, and FlanT5 beats OPT on VQA.
- Effect of Vision-Language Representation Learning: Without first-stage representation learning, the Q-Former fails to bridge the modality gap, causing substantially lower zero-shot VQA performance and catastrophic forgetting in OPT during training.
- Image Captioning: BLIP-2 achieves state-of-the-art image-captioning performance with significant improvement on NoCaps, demonstrating strong generalization to out-of-domain images.
- Fine-tuned VQA and Image-Text Retrieval: BLIP-2 achieves state-of-the-art results in fine-tuned open-ended VQA and zero-shot image-text retrieval, while ITC, ITM, and ITG objectives support retrieval and ITG improves vision-language alignment.
5. Limitation
BLIP-2 does not exhibit improved VQA performance from in-context examples, likely because its pretraining samples contain only one image-text pair. Its image-to-text generation can also fail because of LLM knowledge, reasoning, and information limitations, while frozen models inherit broader LLM risks.
- In-context learning: BLIP-2 shows no improved VQA performance when given few-shot in-context VQA examples.The paper attributes this limitation to pretraining data containing only a single image-text pair per sample, preventing learning correlations among multiple pairs in one sequence.
- In-context learning: Single-pair image-text pretraining prevents BLIP-2 from learning correlations among multiple image-text pairs in one sequence.The paper notes that the same lack of in-context learning is also reported in the Flamingo paper.
- Generation and safety: BLIP-2 image-to-text generation may produce unsatisfactory results because of inaccurate LLM knowledge, incorrect reasoning paths, or outdated information about new image content.These failure modes are illustrated in Figure 7.
- Generation and safety: Frozen models cause BLIP-2 to inherit LLM risks, including offensive language, social-bias propagation, and private-information leakage.The passage identifies these as risks inherited from the frozen models used by BLIP-2.
6. Conclusion
BLIP-2 is a generic, compute-efficient vision-language pre-training method that uses frozen pretrained image encoders and LLMs. It achieves state-of-the-art performance with few trainable parameters and demonstrates zero-shot instructed image-to-text generation.
- Conclusion: BLIP-2 leverages frozen pretrained image encoders and large language models for compute-efficient vision-language pre-training.The method uses only a small amount of trainable parameters during pre-training.
- Conclusion: BLIP-2 achieves state-of-the-art performance across various vision-language tasks with few trainable parameters.
- Conclusion: BLIP-2 demonstrates emerging zero-shot capabilities for instructed image-to-text generation.The authors view BLIP-2 as an important step toward a multimodal conversational AI agent.