PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi

arXiv:2305.06500v2cs.CVcs.LG

TL;DR

General-purpose vision-language models struggle to generalize across diverse tasks, while instruction tuning remains under-explored. InstructBLIP addresses this with diverse instruction data and instruction-aware visual feature extraction, achieving state-of-the-art zero-shot performance across unseen vision-language tasks and downstream fine-tuning.

  • Problem

    Vision-language task diversity makes unified models difficult to generalize, while instruction tuning remains under-explored and prior approaches generalize poorly beyond visual descriptions.

  • Method

    InstructBLIP instruction-tunes pretrained BLIP-2 with 26 datasets across 11 task categories and extracts visual features conditioned on textual instructions.

  • Results

    InstructBLIP achieves state-of-the-art zero-shot performance across a wide range of unseen vision-language tasks and state-of-the-art downstream fine-tuning performance.

  • Takeaways & Limitations

    InstructBLIP provides a generalized vision-language model with instruction-following capabilities across complex reasoning, knowledge-grounded description, and multi-turn conversation tasks.

  • Takeaways & Limitations

    The Visual Dialog evaluation reports MRR instead of NDCG because MRR is considered better aligned with zero-shot evaluation.

Abstract

from arXiv · show

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.

1 Introduction

InstructBLIP addresses the diversity and generalization challenges of vision-language tasks with instruction tuning over diverse datasets and instruction-aware visual feature extraction. It achieves state-of-the-art zero-shot and downstream finetuning performance across broad vision-language tasks.

  • Introduction: Vision-language tasks are more diverse than NLP tasks because they combine visual inputs from various domains, making unified generalization especially challenging.Prior approaches include multitask learning and extending pretrained language models with visual components.
  • Introduction: Image-caption data alone is too limited to support broad generalization beyond visual-description tasks.This limitation motivates training with diverse instruction data.
  • Introduction: InstructBLIP is a vision-language instruction tuning framework that uses diverse instruction data and initializes training from pretrained BLIP-2 components.BLIP-2 combines an image encoder, an LLM, and a Q-Former that bridges them.
  • Introduction: 26 datasets are transformed into instruction-tuning format, with 13 held-in datasets for tuning and 13 held-out datasets for zero-shot evaluation.The study groups the datasets into 11 task categories and withholds four entire categories for task-level zero-shot evaluation.
  • Introduction: Instruction-aware visual feature extraction conditions the Q-Former on textual instructions to extract informative features from a frozen image encoder.The framework also uses balanced sampling to synchronize learning across datasets and evaluates model families based on FlanT5 and Vicuna.

2 Vision-Language Instruction Tuning

This section presents InstructBLIP’s vision-language instruction-tuning framework, covering diverse datasets, held-in/held-out evaluation, instruction-aware visual feature extraction, dataset balancing, inference procedures, and implementation choices.

  • 2.1 Tasks and Datasets: The instruction-tuning collection spans 11 task categories and 26 publicly available vision-language datasets transformed into instruction format.The datasets cover diverse capabilities, including captioning, visual reasoning, and image question answering.
  • 2.2 Training and Evaluation Protocols: The 26 datasets are divided into 13 held-in datasets for training and evaluation and 13 held-out datasets for zero-shot evaluation.Held-out evaluation includes unseen datasets for held-in tasks and datasets with entirely unseen tasks, introducing distribution shift.
  • 2.2 Training and Evaluation Protocols: Instruction tuning mixes held-in training sets, samples instruction templates uniformly per dataset, and trains with standard language-modeling loss to generate responses.OCR tokens are added to instructions for datasets involving scene text, while contamination is controlled by excluding evaluation data from training clusters.
  • 2.3 Instruction-aware Visual Feature Extraction: The instruction-aware Q-Former incorporates instruction tokens through self-attention to extract task-relevant image features for the frozen language model.It extends BLIP-2’s Q-Former, whose query embeddings interact with frozen image-encoder outputs before projection into the LLM.
  • 2.4 Balancing Training Datasets: To balance datasets of different sizes, InstructBLIP samples datasets with probabilities proportional to the square root of their training-set sizes.Manual weight adjustments lower A-OKVQA’s weight and increase OKVQA’s weight to reflect differences in task and training intensity.
  • 2.5 Inference Methods: Inference uses direct response generation for most datasets and vocabulary ranking for classification and multiple-choice VQA tasks.Evaluation metrics are computed by comparing generated responses with ground truth, while classification uses expanded positive and negative verbalizers.
  • 2.6 Implementation Details: The implementation uses four BLIP-2 variants sharing a ViT-g/14 image encoder and pairing it with frozen FlanT5XL, FlanT5-XXL, Vicuna-7B, or Vicuna-13B LLMs.The modular BLIP-2 architecture enables adaptation to multiple language models.

3 Experimental Results and Analysis

InstructBLIP achieves strong zero-shot and task-specific performance, with instruction-aware features and diverse instruction tuning supporting generalization, visual reasoning, and coherent responses. Ablations and comparisons further examine the contributions of its architecture, training strategy, and frozen language models.

  • Zero-shot evaluation: InstructBLIP achieves new zero-shot state-of-the-art results on all 13 held-out datasets and consistently surpasses BLIP-2 across language models.InstructBLIP FlanT5XL yields a 15.0% average relative improvement over BLIP-2 FlanT5XL.
  • Ablation studies: Removing instruction-aware visual features significantly degrades performance, especially on spatial and temporal visual reasoning datasets.The ablation study also removes balanced data sampling and evaluates held-in and held-out datasets across multiple tasks.
  • Qualitative analysis: InstructBLIP’s qualitative outputs show complex visual reasoning, more appropriate visual details, and more logically coherent responses than concurrent multimodal models.Compared with LLaVA and MiniGPT-4, it can adapt response length to user intent; these advantages are attributed to diverse instruction tuning data and architectural design.
  • Instruction tuning versus multitask learning: Instruction tuning and multitask learning exhibit similar performance on held-in datasets under identical BLIP-2 FlanT5XL training configurations.The comparison tests whether zero-shot gains arise from instruction formatting or merely from multitask training.
  • Downstream finetuning: 224×224 image resolution and a frozen visual encoder reduce trainable parameters from 1.2B to 188M during downstream finetuning.InstructBLIP outperforms BLIP-2 on all reported finetuning datasets and achieves state-of-the-art results on ScienceQA (IMG), OCR-VQA, and A-OKVQA.

4 Related Work

Prior work adapts instruction-tuned language models to vision-to-language generation by injecting visual information, while instruction data is built from converted datasets or LLM-generated examples. InstructBLIP broadens the data scope, introduces instruction-aware visual feature extraction, and analyzes generalization to unseen tasks.

  • Instruction tuning: Instruction tuning improves language-model generalization by teaching models to follow natural-language instructions.Instruction data can come from templates that convert existing NLP datasets or from LLM-generated examples with greater diversity.
  • Vision-language instruction tuning: Vision-to-language methods inject visual information into instruction-tuned language models using architectures such as BLIP-2’s Q-Former.BLIP-2 uses frozen FlanT5 models and trains a Q-Former to extract visual features; MiniGPT-4 reuses BLIP-2’s visual encoder and Q-Former with Vicuna.
  • InstructBLIP: InstructBLIP covers a wider range of vision-language instruction data and introduces instruction-aware visual feature extraction.Its data includes both template-based converted data and LLM-generated data, alongside comprehensive analysis of generalization to unseen tasks.

5 Conclusion

InstructBLIP is presented as an instruction-tuning framework for generalized vision-language models that generalizes to unseen tasks with state-of-the-art performance. It also supports complex instruction-following capabilities and provides an enhanced initialization for downstream fine-tuning.

  • InstructBLIP is a simple, novel instruction-tuning framework for generalized vision-language models that generalizes across unseen tasks with state-of-the-art performance.
  • Qualitative examples demonstrate instruction-following capabilities including complex visual reasoning, knowledge-grounded image description, and multi-turn conversations.
  • InstructBLIP provides an enhanced initialization for downstream task fine-tuning, achieving state-of-the-art results.

A Broader Impact

InstructBLIP inherits hallucination and bias risks from its frozen off-the-shelf LLMs, while grounding improvements and diverse, high-quality instruction tuning mitigate—but do not eliminate—these shortcomings.

  • Limitations: InstructBLIP inherits shortcomings from its frozen off-the-shelf LLMs, including hallucinated ungrounded text and biased outputs.These limitations arise from the original language models used by InstructBLIP.
  • Mitigation: The authors mitigate these shortcomings by improving grounding on visual and instruction inputs and tuning on diverse, high-quality datasets.The mitigation combines stronger grounding with broader vision-language instruction tuning data.
  • Deployment caution: The authors do not recommend deploying InstructBLIP models in downstream applications without prior assessment.The passage cautions that mitigation does not justify unassessed downstream use.

B More Case Studies

The case studies show InstructBLIP producing more comprehensive, visually grounded, and logical responses than comparison models. It also adapts response length and connects textual knowledge with visual input.

  • B More Case Studies: InstructBLIP’s response is more comprehensive than GPT-4, more visually grounded than LLaVA, and more logical than MiniGPT-4.The comparisons use responses from the respective papers or official demos.
  • B More Case Studies: InstructBLIP adaptively adjusts response length to address user intent, whereas other models tend to generate lengthy, less-relevant paragraphs.The case study illustrates that long-form responses are not always preferable.
  • B More Case Studies: InstructBLIP connects textual knowledge with visual input, while other models depict the picture plainly.MiniGPT-4 performs more poorly, possibly because it was trained only with long captions.

C Instruction Tuning Datasets

Table 4 describes the datasets used for held-in instruction tuning and held-out zero-shot evaluation.

  • Table 4 provides descriptions of datasets used in held-in instruction tuning and held-out zero-shot evaluations.

D Instruction Templates

The section presents instruction templates for converting held-in datasets into instruction-tuning data, with a specific extension for datasets containing OCR tokens.

  • D Instruction Templates: Instruction templates transform held-in datasets into instruction-tuning data.These templates are documented in Table 5.
  • D Instruction Templates: For datasets with OCR tokens, the format adds “OCR tokens:” after the image query embeddings.

E Instructions for Zero-shot Inference

The section specifies task-specific instruction templates for zero-shot inference, covering image question answering, captioning, OCR-based answering, multiple-choice reasoning, classification, truth evaluation, and visual dialogue. Options are ordered alphabetically when instructions include choices.

  • Formatting: For instructions with options, the options are separated in alphabetical order.The examples use the ordering “(a) blue (b) yellow (c) pink (d) black”.
  • Question Answering: GQA, VizWiz, iVQA, MSVD, and MSRVTT use image-question prompts requesting short answers.The template is “<Image> Question: {} Short answer:”.
  • Image Captioning: NoCaps and Flickr30k use image prompts requesting a short image description.The template is “<Image> A short image description:”.
  • Structured Question Answering: TextVQA includes OCR tokens and a question, while IconQA and ScienceQA include answer options in their prompts.TextVQA requests a short answer; IconQA requests a short answer after options; ScienceQA provides context, question, options, and an answer field.
  • Classification and Dialogue: HatefulMemes asks whether overlaid text is hateful, VSR asks whether an image-based statement is true or false, and Visual Dialog provides dialog history with a question.The HatefulMemes and VSR templates request answers, while Visual Dialog requests a short answer.
Loading 2305.06500v2…