PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Baselines with Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee

arXiv:2310.03744v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Training recipes for general-purpose multimodal assistants remain unclear. This paper systematically studies LLaVA design choices, yielding LLaVA-1.5, which achieves state-of-the-art across 11 tasks using public data and improved data efficiency.

  • Problem

    The best training recipe for general-purpose multimodal assistants remains unclear, and the causes of capability differences across LMMs are elusive.

  • Method

    A controlled study improves LLaVA with an MLP cross-modal connector, academic-task VQA data, and response-format prompting for short answers.

  • Results

    LLaVA-1.5 achieves state-of-the-art across 11 tasks using only public data and is significantly more data-efficient than previous approaches.

  • Takeaways & Limitations

    LLaVA-1.5 provides a simple, effective, data-efficient, and easily reproducible baseline for open-source LMM research.

  • Takeaways & Limitations

    Limitations include prolonged high-resolution training, no multiple-image understanding, limited problem-solving in some fields, and hallucinations requiring caution in critical applications.

Abstract

from arXiv · show

Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.

1. Introduction

The paper systematically studies how to train large multimodal models toward general-purpose assistants, using LLaVA to evaluate design choices across input, model, and data. It introduces LLaVA-1.5, a simple and reproducible baseline that achieves state-of-the-art across 11 tasks while improving data efficiency.

  • Motivation: The best recipe for training LMMs toward general-purpose assistants remains unclear because models excel at different benchmark styles.LLaVA performs strongly on conversational visual reasoning, whereas InstructBLIP excels at traditional VQA requiring short answers.
  • Core improvements: An MLP cross-modal connector and academic-task-related data such as VQA are simple, orthogonal improvements that strengthen LLaVA’s multimodal understanding.The study finds the fully connected vision-language connector surprisingly powerful and data-efficient.
  • Open problems: Higher-resolution inputs improve detailed perception and reduce hallucination while preserving data efficiency through simple image-grid division.The LLaVA architecture scales to higher resolutions by dividing images into grids.
  • Results: LLaVA-1.5 uses only public data, achieves state-of-the-art across 11 tasks, and is significantly more data-efficient than previous approaches.The improved baseline is presented as a simple, effective approach for balancing multitask learning and scaling.

2. Related Work

The related work covers common LMM architectures and visual instruction-following data, emphasizing LLaVA’s simple design and the importance of instruction-data quality. It motivates investigating how multimodal models can balance natural conversations with academic tasks.

  • Instruction-following large multimodal models (LMMs): LMMs commonly combine a pretrained visual backbone, a pretrained LLM, and a vision-language cross-modal connector.Visual resamplers such as Qformer can optionally reduce the number of visual features.
  • Instruction-following large multimodal models (LMMs): LLaVA is presented as perhaps the simplest architecture for instruction-following LMMs.Its architecture uses the visual backbone, language model, and cross-modal connector framework described above.
  • Multimodal instruction-following data: Instruction-following data quality strongly affects resulting model capability, while FLAN shows that academic language tasks can improve generalization.LLaVA pioneered using text-only GPT-4 to expand COCO bounding-box and caption data for visual instruction tuning.
  • Multimodal instruction-following data: The work investigates why multimodal models struggle to balance natural conversations and academic tasks.This question follows from the contrast between conversational instruction data and academic-task instruction tuning.

3. Approach

The approach improves LLaVA through explicit response-format prompts, a two-layer MLP connector, academic-task-oriented VQA data, and scaled resolution and model capacity. These modifications produce LLaVA-1.5, while patch-based encoding extends the model to arbitrary image resolutions as LLaVA-1.5-HD.

  • Response formatting: A single response-formatting prompt directs short-answer VQA outputs as one word or phrase, enabling LLaVA to adjust output format to user instructions.The prompt is appended to VQA questions when promoting short answers and avoids postprocessing answers with ChatGPT.
  • MLP vision-language connector: A two-layer MLP vision-language connector increases representation power and improves LLaVA’s multimodal capabilities over the original linear projection.The connector change is motivated by improved performance from MLPs in self-supervised learning.
  • Academic-task-oriented data: Four additional datasets—OKVQA, A-OKVQA, OCRVQA, and TextCaps—expand open-knowledge VQA and OCR capabilities, while A-OKVQA uses letter-only answer formatting.With only a subset of InstructBLIP’s datasets, LLaVA surpasses it on all three tasks in Table 2.
  • Additional scaling: LLaVA-1.5 combines 3362 input resolution, CLIP-ViT-L-336px, GQA, ShareGPT data, and a 13B LLM; scaling to 13B yields the largest MM-Vet improvement.The final model with all modifications significantly outperforms original LLaVA.
  • Higher-resolution scaling: LLaVA-1.5-HD divides images into encoder-sized patches, merges their feature maps, and concatenates a downsampled-image feature to provide global context at arbitrary resolutions.This split-encode-merge design maintains LLaVA-1.5’s data efficiency and reduces artifacts from patch processing.

4. Empirical Evaluation

LLaVA-1.5 achieves the best overall performance across 12 benchmarks while using substantially less pretraining and instruction-tuning data than other methods. Higher-resolution features, format prompts, and stronger base language models further improve or extend its capabilities.

  • Benchmark suite: 12 benchmarks span academic VQA, scientific question answering, text-rich images, hallucination, perception, robustness, and diverse instruction-following domains.The evaluation includes VQA-v2, GQA, VizWiz, ScienceQA, TextVQA, POPE, MME-Perception, MMBench, and other instruction-following benchmarks.
  • Overall results: LLaVA-1.5 achieves the best overall performance on 12 benchmarks despite using magnitudes smaller pretraining and instruction-tuning datasets than other methods.It also significantly outperforms LLaVA on all benchmarks for instruction-following LMMs.
  • High-resolution evaluation: 448^2 resolution with LLaVA-1.5-HD further improves performance on all benchmarks, especially detail-dependent tasks such as OCR and detailed image description.Adding global context helps recover from split-and-merge artifacts and locate relevant regions in high-resolution features.
  • Format instruction generalization: 11.1% →67.8% on unanswerable VizWiz questions shows that response-format prompts generalize beyond the limited format instructions used during training.The model also qualitatively follows verification and constrained JSON-format instructions.
  • LLM choices: Vicuna-v1.5 achieves the best overall performance, while LLaMA-2-based models generally outperform LLaMA-1-based models, highlighting the importance of the base language model.Despite shared ShareGPT instruction-tuning data, Vicuna-v1.3 generalizes significantly worse to Chinese on MMBench-CN than Vicuna-v1.5.

5. Open Problems in LMMs

The section examines open problems in LMM data efficiency, hallucination, and compositional capabilities using LLaVA-1.5. It finds strong performance with less data and robustness to some annotation errors, while identifying resolution-dependent hallucination and unresolved capability combinations.

  • Data Efficiency: 50% of the samples retains more than 98% of full-dataset performance, while the full mixture achieves the best overall performance.The full mixture provides the best knowledge coverage; the result suggests room for further data-efficiency improvements.
  • Data Efficiency: At 50% sampling, performance on MMBench, ScienceQA, and POPE does not decrease, and MMBench slightly improves; performance remains steady down to 30%.These findings suggest a possible less-is-more benefit for multimodal models.
  • Hallucination: Hallucination is significantly reduced when training excludes the cited hallucinated-content data, indicating robustness to a few training-data errors.The passage attributes hallucination concerns to errors in training data, including detailed descriptions in LLaVA-Instruct.
  • Compositional Capabilities: LLaVA-1.5 generalizes independently trained capabilities to tasks requiring their combination without explicit joint training.The section presents this as an interesting compositional capability.
  • Compositional Capabilities: ShareGPT improves language, multilingual, and response-detail capabilities, while academic-task datasets improve visual grounding, but some capability combinations and Korean conversations remain difficult.Correctly answering an object attribute in VQA does not guarantee accurately depicting it in a detailed whole-image description.
  • Compositional Capabilities: Compositional capabilities may improve performance without exhaustively including all task combinations, but their mechanisms remain an open avenue for improving capability and data efficiency.The passage calls for deeper investigation of these mechanisms in LMMs and LLaVA-1.5.

6. Conclusion · Appendix

LLaVA-1.5 is presented as a simple, effective, data-efficient baseline while also exposing open problems and capabilities in large multimodal models. The conclusion notes limitations, and the appendix provides implementation details, qualitative analyses, and expanded limitations discussion.

  • 6. Conclusion: LLaVA-1.5 is proposed as a simple, effective, and data-efficient baseline for large multimodal models.The paper frames this proposal as a step toward demystifying large multimodal model design.
  • 6. Conclusion: The paper explores visual instruction tuning, higher-resolution LMMs, hallucination, and compositional capabilities.These are presented as open problems and findings arising from the study.
  • 6. Conclusion: LLaVA-1.5 remains limited by prolonged high-resolution training, lack of multiple-image understanding, and constrained problem solving in some fields.These limitations are explicitly identified despite the model’s promising results.
  • 6. Conclusion: LLaVA-1.5 can still produce hallucinations and should be used cautiously in critical applications such as medicine.The appendix is referenced for a more detailed discussion of these limitations.
  • 6. Conclusion: The work acknowledges support from NSF CAREER IIS2150012 and Korea government-funded IITP grants.The grants concern AI agent collaboration and efficient pre-training of large Korean language models.
  • Appendix: The appendix covers LLaVA-1.5-HD implementation details, data and prompts, and hyperparameters in Section A.Section A is divided into implementation details (A.1), data and prompts (A.2), and hyperparameters (A.3).
  • Appendix: Section B presents qualitative results for response format prompts and compositional capabilities.These topics are addressed in Sections B.1 and B.2, respectively.
  • Appendix: Section C discusses the model’s limitations in greater detail.The appendix identifies this discussion as its own section.

A. Implementation Details · A.1. LLaVA-1.5-HD · A.1.1 Preprocessing

LLaVA-1.5-HD preprocesses images by selecting an efficient target resolution, encoding grid patches independently with CLIP, and assembling them into language-model-ready visual features. Postprocessing removes padding, marks row boundaries, and preserves fixed-resolution image features alongside the high-resolution representation.

  • A.1.1 Preprocessing: CLIP-ViT-L-14 (224^2) encodes image patches split across grids after each input image is selected and padded to a target resolution.The patches are encoded separately before their features are merged into one large feature map.
  • A.1.1 Preprocessing: The merged feature map is postprocessed into a flattened list of features for language-model processing.This follows independent encoding of all image patches and recombination into a single feature map.
  • A.1.1 Preprocessing: Features from a fixed-resolution image are additionally concatenated with the high-resolution image features.This fixed-resolution representation supplements the features produced from the selected target resolution.
  • A.1.1 Preprocessing: The preprocessing supports up to six grids, including 1x1 through 1x6, 2x2, 2x3, and their transposes.The predefined resolution set determines how images are partitioned into grid layouts.
  • A.1.1 Preprocessing: 672x448 is the maximum supported resolution, with 448x672 also available for transposed layouts.Target-resolution selection balances preserving image detail against avoiding unnecessarily large resource consumption.
  • A.1.1 Preprocessing: Padding-only features are discarded to reduce the number of visual tokens processed by the language model.Padding removal improves processing efficiency by eliminating features corresponding exclusively to padded regions.
  • A.1.1 Preprocessing: A special Rowend Token is appended to each feature row to explicitly indicate the image’s shape.The token is part of a three-step postprocessing procedure designed for effective and efficient language-model processing.

A.1.2 Training

Training uses visual features computed at the encoder’s original 2242 resolution without additional pretraining, including high-resolution projector pretraining, and applies visual instruction tuning directly to higher-resolution images.

  • Training: The model skips additional pretraining because visual features are computed at the vision encoder’s original 2242 resolution.This includes no additional pretraining for the visual projectors.
  • Training: Visual instruction tuning is performed directly on higher-resolution images without high-resolution pretraining for the visual projectors.

A.2. Data … B.1. Response Format Prompts

LLaVA-1.5 combines diverse multimodal and conversational data with efficiency-oriented sampling and training choices. Qualitative results show that it follows requested response formats, adapts response granularity and format mid-conversation, and supports constrained prompt generation, while also generalizing multilingual visual conversation with some language-specific errors.

  • A.2. Data: The training mixture includes VQA, OCR, region-level VQA, visual conversation, and language conversation datasets.QA pairs from the same training image are merged into a single conversation, while invalid ShareGPT conversations are filtered.
  • A.2. Data: 25% faster training is achieved by sampling each batch from a single modality without affecting the final outcome.All data splits are concatenated and sampled with the same probability.
  • A.3. Hyperparameters: The latest Vicuna v1.5 is used as the base LLM, with greedy decoding applied during evaluation for reproducibility.LLaVA-1.5 otherwise uses the original LLaVA hyperparameters, except for a halved pretraining learning rate with the MLP projection layer.
  • A.3. Hyperparameters: The pretraining learning rate is halved relative to original LLaVA because LLaVA-1.5 uses an MLP rather than a linear projection layer.Table 9 states that the remaining hyperparameters are the same as the original LLaVA.
  • B.1. Response Format Prompts: LLaVA-1.5 follows requested response formats, generalizes to unseen format instructions, and switches formats within conversations.It can provide details at different granularities and alter formats when requested by the user.
  • B.1. Response Format Prompts: LLaVA-1.5 can generate constrained prompts for Stable Diffusion models, including anime prompts.Additional examples demonstrate generalization to different unseen response format prompts.
  • B. Qualitative Results: LLaVA-1.5 provides more detailed, visually grounded responses for writing tasks with visual inputs than LLaVA.The comparison is summarized in Table 11.
  • B.1. Response Format Prompts: LLaVA-1.5 generalizes to multilingual visual conversations despite English-only visual instruction data, although some languages such as Korean still contain errors.The multilingual capability arises from combining visual instruction-following data with multilingual text-only ShareGPT data.

B.2. Compositional Capabilities

LLaVA-1.5 demonstrates compositional capabilities in multilingual visual conversations and visually grounded writing tasks. These capabilities emerge despite English-only visual instruction training and after adding multilingual ShareGPT text data.

  • Multilingual visual conversations: LLaVA-1.5 adapts its output language to the user’s input in multilingual visual conversations, despite no multilingual visual instruction training.The paper presents this as a qualitative compositional capability.
  • Multilingual visual conversations: The authors hypothesize that multilingual visual behavior combines English-only visual conversations with multilingual text-only ShareGPT data.This explanation is explicitly described as a hypothesis.
  • Visually grounded writing: After including ShareGPT data, LLaVA-1.5 produces more detailed and visually grounded responses in writing tasks with visual inputs than LLaVA.This is reported as another observed compositional capability.

C. Limitations

LLaVA-1.5 has efficiency, multimodal-input, and domain-specific problem-solving limitations. Future improvements include sample-efficient visual resamplers, multiple-image instruction-following data, longer context, and more capable language models.

  • Efficiency: Full image patches may prolong each LLaVA-1.5 training iteration.Visual resamplers reduce visual patches but do not yet converge as efficiently with comparable training data, probably because they have more trainable parameters.
  • Efficiency: Sample-efficient visual resamplers could enable future scaling of instruction-following multimodal models.The passage identifies sample efficiency as the development needed to address current resampler limitations.
  • Multimodal input: LLaVA-1.5 cannot yet process multiple images because of insufficient instruction-following data and limited context length.
  • Problem solving: LLaVA-1.5’s problem-solving capabilities remain limited in certain domains despite proficiency with complex instructions.The passage suggests improvement through a more capable language model and additional unspecified development.
Loading 2310.03744v2…