PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

arXiv:2304.08485v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Multimodal instruction-following data is limited, so the paper uses GPT-4 to generate language-image instructions for training LLaVA. LLaVA shows strong visual-chat and instruction-following performance, including 67.3% overall on LLaVA-Bench In-the-Wild and new state-of-the-art accuracy on ScienceQA.

  • Problem

    Multimodal instruction-following data remains limited and difficult to create, hindering development of general-purpose visual assistants.

  • Method

    The paper uses GPT-4 to convert image-text pairs into instruction-following data and trains LLaVA by connecting a visual encoder with a language decoder.

  • Results

    LLaVA achieves strong visual-chat and instruction-following performance, including 67.3% overall on LLaVA-Bench In-the-Wild and new state-of-the-art accuracy on ScienceQA.

  • Takeaways & Limitations

    Visual instruction tuning provides an initial approach for building multimodal assistants that follow human intent on real-life visual tasks.

  • Takeaways & Limitations

    LLaVA can struggle with high-resolution perception, multilingual understanding, and broad knowledge coverage in challenging real-world images.

Abstract

from arXiv · show

Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

1 Introduction

The introduction motivates a general-purpose assistant that follows multimodal vision-and-language instructions, noting that visual foundation models and aligned LLMs provide complementary capabilities. It presents visual instruction-tuning as the first extension of instruction tuning to language-image data, alongside a generated-data pipeline and an end-to-end multimodal model.

  • Motivation: A general-purpose assistant should follow multimodal vision-and-language instructions aligned with human intent across real-world tasks.Vision and language each represent and communicate distinct concepts, supporting broader world understanding.
  • Background: Language-augmented vision models support open-world visual understanding, generation, and editing across diverse tasks.Reported capabilities include classification, detection, segmentation, captioning, visual generation, and editing.
  • Background: Aligned LLMs establish language as a universal interface for representing task instructions and guiding assistants to solve different tasks.ChatGPT and GPT-4 exemplify instruction-following success, while machine-generated instruction data improves alignment in text-only systems.
  • Contributions: Visual instruction-tuning is introduced as the first extension of instruction tuning to the language-image multimodal space.The paper addresses scarce vision-language instruction data by reforming image-text pairs with ChatGPT/GPT-4 and connects CLIP’s visual encoder to Vicuna’s language decoder for end-to-end fine-tuning.

2 Related Work

Related work includes multimodal instruction-following agents, spanning end-to-end models for specific visual tasks, and instruction tuning methods developed for language models. Prior NLP research applied instruction tuning to models including GPT-3, T5, PaLM, and OPT, producing instruction-following variants.

  • Multimodal Instruction-following Agents: Multimodal instruction-following agents include end-to-end models separately developed for specific research topics.Examples include vision-language navigation and Habitat, where embodied agents follow natural-language instructions and act in visual environments.
  • Instruction Tuning: Instruction tuning has been explored to help LLMs follow natural-language instructions and complete real-world tasks.The passage names GPT-3, T5, PaLM, and OPT as instruction-tuned language-model families.
  • Instruction Tuning: Instruction tuning produced counterparts including InstructGPT/ChatGPT, FLAN-T5, FLAN-PaLM, and OPT-IML.The passage states that this simple approach can effectively improve zero-shot capabilities, though the supplied text is truncated.

3 GPT-assisted Visual Instruction Data Generation

The section presents a GPT-4-assisted method for generating multimodal instruction-following data from image-text pairs, using symbolic visual representations and three instruction types. It produces 158K samples, with GPT-4 yielding higher-quality data than ChatGPT in early ablations.

  • Motivation: Available multimodal instruction-following data is limited because human creation is time-consuming and less well-defined, motivating GPT-4-assisted collection from existing image-pair data.The method is inspired by GPT models’ success in text-annotation tasks.
  • Symbolic Visual Representation: Captions and bounding boxes encode image content and spatial locations as symbolic, LLM-recognizable sequences for text-only GPT-4 or ChatGPT.Captions describe visual scenes from multiple perspectives, while boxes localize objects and encode their concepts and spatial locations.
  • Instruction Types: The generated data covers conversations, detailed descriptions, and complex reasoning, with questions targeting visual content and reasoning requiring step-by-step logic.Conversation questions include object types, counts, actions, locations, and relative positions; complex reasoning follows rigorous logic.
  • Dataset Scale: 158K unique language-image instruction-following samples comprise 58K conversations, 23K detailed descriptions, and 77K complex-reasoning samples.The samples are generated using COCO images, with manually designed examples serving as the only human annotations during collection.
  • Model Comparison: GPT-4 consistently provides higher-quality instruction-following data than ChatGPT in early ablations, including for spatial reasoning.The comparison concerns the use of ChatGPT and GPT-4 during data generation.

4 Visual Instruction Tuning

LLaVA connects a pretrained CLIP visual encoder to a language model through a lightweight trainable projection, producing visual tokens in the word-embedding space. It trains on multimodal instruction-following sequences through autoregressive prediction, using two stages for feature alignment and end-to-end fine-tuning.

  • 4.1 Architecture: LLaVA uses CLIP ViT-L/14 to encode an input image into visual features, including grid features before and after the final Transformer layer.The visual encoder computes Zv = g(Xv).
  • 4.1 Architecture: A trainable linear projection W converts visual features Zv into visual tokens Hv matching the language embedding dimensionality.The mapping is Hv = W · Zv.
  • 4.2 Training: Multimodal training organizes each image’s generated multi-turn conversations as a unified instruction-following sequence, treating answers as assistant responses.The number of turns varies with the instruction-following data.
  • 4.2 Training: The model is instruction-tuned with the original autoregressive objective to predict assistant answers and stopping tokens, using only prediction tokens for loss computation.The training format follows Vicuna-v0 for the system message and uses ### as the stop marker.
  • 4.2 Training: Stage 1 aligns features by converting 595K filtered CC3M image-text pairs into single-turn instruction-following examples.For each image, a question is randomly sampled to construct the instruction input.
  • 4.2 Training: Stage 2 fine-tunes end-to-end while freezing the visual encoder and updating the projection layer and LLM parameters.The trainable parameters are θ = {W, ϕ}.
  • 4.2 Training: The method is evaluated in two use cases: a multimodal chatbot trained on 158K language-image instruction-following examples and the ScienceQA benchmark.Chatbot training uniformly samples multi-turn conversation and two single-turn response types.

5 Experiments

LLaVA is evaluated on multimodal chat, instruction-following benchmarks, and ScienceQA. It shows GPT-4-like visual reasoning on examples and achieves strong benchmark performance, while challenging in-the-wild cases expose limitations in knowledge, multilingual understanding, and high-resolution detail extraction.

  • Multimodal chatbot: LLaVA shows similar reasoning results to multimodal GPT-4 on out-of-domain examples while following user instructions rather than merely describing images.The comparison includes BLIP-2 and OpenFlamingo, which focus on image description.
  • Quantitative Evaluation: 90 questions from 30 COCO-Val-2014 images form LLaVA-Bench (COCO), which tests alignment and capabilities under consistent visual inputs.Questions cover conversation, detailed description, and complex reasoning, while training datasets are varied to study instruction-following data.
  • Quantitative Evaluation: LLaVA-Bench (In-the-Wild) uses 24 diverse images and 60 questions to evaluate challenging-task performance and generalizability to novel domains.The benchmark includes indoor and outdoor scenes, memes, paintings, and sketches, with highly detailed manually curated descriptions.
  • Limitations: LLaVA-Bench (In-the-Wild) reveals limitations involving broad knowledge, multilingual understanding, Internet-retrieved multimodal information, and high-resolution image details.Some questions require identifying a restaurant, describing side dishes, or extracting fine-grained visual details.
  • ScienceQA: 90.92% accuracy places LLaVA close to the ScienceQA state of the art at 91.68%.LLaVA predicts reasons before answers, uses visual features before the last layer, and is trained for 12 epochs.
  • ScienceQA: 82.69% accuracy for two-shot GPT-4 is a 7.52% absolute gain over GPT-3.5’s 75.17% on ScienceQA.The paper also reports that ensembling the novel model with text-only GPT-4 consistently improves performance across categories and sets a new state of the art.

6 Conclusion

The paper demonstrates that visual instruction tuning can produce LLaVA, a multimodal model trained on automatically generated language-image instruction-following data. LLaVA achieves state-of-the-art ScienceQA accuracy after fine-tuning, strong visual chat capabilities, and supports a benchmark for multimodal instruction following.

  • Conclusion: The paper introduces an automatic pipeline for creating language-image instruction-following data and trains LLaVA to follow human intent on visual tasks.LLaVA is described as a multimodal model trained using the generated data.
  • Conclusion: LLaVA achieves new state-of-the-art accuracy when fine-tuned on ScienceQA.The conclusion attributes this result to LLaVA after ScienceQA fine-tuning.
  • Conclusion: LLaVA exhibits excellent visual chat capabilities when fine-tuned on multimodal chat data.The reported capability is specifically associated with fine-tuning on multimodal chat data.
  • Conclusion: The paper presents the first benchmark for studying multimodal instruction-following capability.The benchmark is intended to study multimodal instruction following.

A Broader Impact

LLaVA’s release brings potential benefits and risks inherited from its language and vision components, including misuse, hallucination, bias, energy concerns, and evaluation challenges. The authors argue that releasing it enables research, mitigation improvements, and responsible vision-language applications despite these risks.

  • Risks and benefits: LLaVA inherits risks from its LLM and vision-encoder foundations, while its visual nature introduces additional deployment and release considerations.The model is built upon LLaMA, Vicuna, and CLIP, and the paper outlines both risks and mitigation strategies.
  • Mitigations: Two filters mitigate malicious inputs: OpenAI Filter API blocks harmful text instructions, while an NSFW Filter detects and blocks harmful uploaded images.These precautions target both user-provided text and visual content.
  • Limitations: LLaVA may produce factually ungrounded outputs, particularly concerning in critical applications such as medicine, and may inherit biases from CLIP, LLaMA, and Vicuna.Inherited bias can lead to biased outcomes or unfair representations of diverse content.
  • Limitations: Energy consumption is currently less concerning because LLaVA uses a smaller pretraining dataset, but scaling the dataset or model could increase the concern.The paper specifically cites scaling to a larger LLaMA version such as the 65B model.
  • Limitations: Evaluating LLaVA is challenging because it combines language and visual tasks, requiring assessment of accuracy, concept coverage, reasoning, creativity, hallucination, and fine-grained visual understanding.Text-only GPT-4-based multimodal evaluation was consistent and accurate in the study, but the passage indicates robustness concerns.
  • Risks and benefits: The authors judge that release benefits outweigh potential harm because community access can improve mitigation, stimulate applications and research, and support responsible foundation-model deployment.They emphasize ongoing investigation and improvement by the research community.

B More Results

Additional qualitative results show that LLaVA exhibits emergent visual reasoning, code-generation, OCR, and generalization abilities beyond its training coverage. These results also reveal output limitations and motivate further study of the mechanisms, robustness, biases, and alignment of multimodal models.

  • Qualitative capabilities and limitations: LLaVA generates HTML/JS/CSS for an interactive joke website from a simplified user sketch, though the output requires minor correction and better intent alignment.The interface works after fixing a minor error, but the joke and punchline presentation could better match the user’s intent.
  • Emergent visual generalization: LLaVA recognizes Elon Musk in both a headshot and doge meme despite no Elon Musk examples in either visual alignment or instruction-tuning data.This suggests generalization to visual content not covered during training, although the pretrained CLIP encoder may have seen such images.
  • Emergent visual generalization: LLaVA demonstrates impressive OCR ability despite OCR rarely being covered in its training data.The observed OCR behavior appears in Table 9 and Figure 2.
  • Future work: Further work should investigate LLaVA’s emergent behaviors and mechanisms to improve multimodal robustness, reduce biases, and expand aligned vision-language representations.The paper frames these investigations as important for building better large multimodal models.
  • Qualitative capabilities and limitations: LLaVA matches GPT-4 in explaining a humorous meme, while BLIP-2 and OpenFlamingo fail to follow the user’s instructions.GPT-4 provides a more concise answer than LLaVA in the comparison.

C Training Details

LLaVA is pretrained on filtered CC-595K data and fine-tuned on LLaVA-Instruct-158K using staged optimization, memory-saving methods, and 8× A100 GPUs.

  • Optimization schedule: Pretraining runs for 1 epoch on filtered CC-595K with a 2e-3 learning rate and batch size 128, followed by 3 fine-tuning epochs on LLaVA-Instruct-158K.Fine-tuning uses a 2e-5 learning rate and batch size 32.
  • Optimization schedule: Training follows Vicuna with Adam, no weight decay, cosine learning-rate decay, and a 3% warmup ratio.
  • Memory optimization: FSDP and gradient checkpointing reduce GPU memory use during fine-tuning, without offloading.
  • Compute: All models are trained with 8× A100s; pretraining, Instruct-158K fine-tuning, and ScienceQA fine-tuning take within 4, 10, and 4 hours, respectively.

D Assets

The authors publicly provide the LLaVA source code, generated instruction-tuning data, and proposed benchmark through an anonymized GitHub repository. They also list supporting materials, including prompts, examples, benchmark data, and model checkpoints, with checkpoint release handled separately because of file-size limits.

  • The source code, generated instruction-tuning data, and proposed benchmark are uploaded to an anonymized GitHub repository.
  • Released materials include a README, demo-launch instructions, GPT-4 querying prompts and few-shot examples, LLaVA-Instruct-158K, and LLaVA-Bench.
  • 25GB compressed model checkpoints exceed GitHub LFS’s 5GB limit, so they will be released publicly or provided to reviewers upon request.

E Data · F Prompts

The E Data section defines varied brief and detailed image-description instructions and filters CC3M by noun-phrase frequency, yielding about 595K image-text pairs while retaining concepts occurring more than three times. The F Prompts section documents few-shot prompt construction for generating image-based conversations and instruction-following responses from textual context rather than directly using the visual image.

  • E Data: Brief image-description instructions express the same meaning through natural-language variation.The instruction list is provided in Table 11.
  • E Data: Detailed image-description instructions likewise provide varied natural-language formulations for describing image content in detail.The instruction list is provided in Table 12.
  • E Data: CC3M filtering extracts noun-phrases with Spacy and skips phrases occurring fewer than 3 times as usually rare concept-attribute combinations.The filtering targets combinations considered already covered by other captions.
  • E Data: 595K image-text pairs result from sampling captions containing the lowest-frequency remaining noun-phrases, with phrases above 100 frequency capped at 100 randomly selected captions.This procedure builds a candidate pool while limiting overrepresented noun-phrases.
  • E Data: The filtered CC3M dataset retains good coverage of concepts occurring more than 3 times while using fewer image-text pairs.Figure 7 compares noun-phrase statistics before and after filtering and reports total unique noun-phrases in its legend.
  • F Prompts: Few-shot in-context learning constructs ChatGPT/GPT-4 prompts that collect query[‘response’] from query[‘context’] using examples from fewshot_samples.Table 13 identifies messages as the final prompt and refers to Tables 15 and 16 for the in-context examples.
  • F Prompts: The instruction-following example prompts GPT with textual contexts such as captions and boxes, while the visual image is shown only as a reference.Table 14 presents three response types beneath the context block.
Loading 2304.08485v2…