Source-linked AI summary
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, Yelong Shen
TL;DR
Most open-source LMM studies had used models of 13B parameters or smaller, leaving the effects of larger-scale visual instruction tuning less explored. This study scales LLaVA and tests tuning and data choices; scaling improves performance, while LoRA/QLoRA perform comparably to full-model fine-tuning.
Problem
Most existing open-source LMM studies use models of 13B parameters or smaller, leaving the impact of scaling to larger sizes unexplored.
Source: §1 · p. 1Method
The study scales LLaVA to larger language models and evaluates model size, tuning methods, image resolution, and multimodal-language data mixing.
Source: §1 · p. 1, §2 · p. 2, §3.2 · p. 4Results
Scaling LMM consistently improves performance, and LoRA/QLoRA tuning performs comparably to full-model fine-tuning.
Source: §4 · p. 6Takeaways & Limitations
LoRA/QLoRA offer a performance-cost trade-off for some real-world settings with limited GPU memory, while 33B tuning can balance cost and performance.
Source: §3.2 · p. 5, §4 · p. 6
Abstract
from arXiv · showhide
Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open-source LMM are performed using models with 13B parameters or smaller. In this paper we present an empirical study of scaling LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language capabilities when completing real-world tasks in the wild. We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning. Additionally, the study highlights the importance of higher image resolutions and mixing multimodal-language data to improve LMM performance, and visual instruction tuning can sometimes improve LMM's pure language capability. We hope that this study makes state-of-the-art LMM research at a larger scale more accessible, thus helping establish stronger baselines for future research. Code and checkpoints will be made public.
1 Introduction
Open-source LMMs such as LLaVA and MiniGPT-4 have shown promise, but high compute costs have kept most studies to 7B and 13B models. This study scales LLaVA to larger sizes and reports new state-of-the-art performance on LLaVA-Bench and MM-VET.
Research gap
- High compute costs have kept most existing open-source LMM studies at 7B or 13B, leaving the impact of scaling to 33B and 65B unexplored.Prior work on visual instruction tuning includes LLaVA and MiniGPT-4.Source: §1 · p. 1
Study aims
- The study scales LLaVA to investigate larger language models for LMMs and establish stronger baselines for future research.It explores model size, tuning methods, and data mixing.Source: §1 · p. 1
Contribution
- The scaling recipe achieves new state-of-the-art performance on LLaVA-Bench and MM-VET.The authors hope the findings and larger LLaVA checkpoints will provide a reference for future visual instruction-tuning research.Source: §1 · p. 1
2 Experiment Setup
The experiments scale LLaVA across language-model sizes and vary tuning methods and training data. They use two-stage visual instruction tuning and compare full-model training with parameter-efficient approaches and multimodal-language data mixing.
Model checkpoints
- The study increases LLM size to 33B and 65B alongside the 7B and 13B sizes used in existing LMMs.For 65B, the authors train their own Vicuna model because a public Vicuna-65B checkpoint was unavailable.Source: §2 · p. 2
Training stages
- The two-stage LLaVA training first aligns visual features to the LLM word-embedding space, then tunes on LLaVA-80K multimodal instructions.Stage 1 trains the linear projection layer using 558K concept-balanced LAION-CC-SBU samples.Source: §2 · p. 2
Training schedules
- The experiments explore training schedules intended to support following diverse instructions and completing tasks in the wild.The setup also searches LoRA hyperparameters; larger alpha or learning rate was found crucial for the best performance.Source: §2 · p. 2
Tuning methods and data
- The tuning comparisons include full-model fine-tuning, LoRA, and QLoRA, with the latter methods targeting lower-compute training.The authors compare multimodal-only instruction tuning with mixing ShareGPT language-only instructions into LLaVA-80K.Source: §2 · p. 2
3 Results
Larger LLaVA models improve results on LLaVA-Bench and MM-VET, while scaling experiments examine model size, image resolution, data mixing, and parameter-efficient tuning. The findings include gains from higher resolution and mixed instruction data, as well as language-capability results after visual instruction tuning.
3.1 Comparisons on Benchmarks
- On LLaVA-Bench, 33B and 65B LLaVA outperform 13B LLaVA and Bing Chat; on MM-VET, performance improves from 13B through 65B, with the largest model setting open-source SoTA.LLaVA-Bench is small, so the authors caution that its comparisons might not be statistically significant.Source: Table 1 · p. 3, Table 2 · p. 3, §3.1 · p. 4
3.2 Scaling up LLaVA
- Combining model size, image resolution, and data mixing improves the baseline LLaVA 7B model on MM-Bench.MM-Bench contains 2,974 questions evaluating reasoning skills across six categories.Source: §3.2 · p. 4
- Higher image resolution yields 2–3 points of improvement across all four LLM sizes, and ShareGPT mixing adds 2 points versus multimodal-only instruction data.Increasing model size also consistently improves overall performance; the authors conjecture that larger models need more training data.Source: §3.2 · p. 4, Table 3 · p. 5
- At 13B, LoRA rank 64 can deliver performance comparable to full-model tuning; increasing rank from 8 to 64 nearly matches rank-64 LoRA on 65B at 50% of its training cost.For 33B and larger models, higher LoRA ranks increase both performance and cost, while tuning 33B is reported as a good cost/performance trade-off.Source: §3.2 · pp. 4–5, Table 4 · p. 5
- 65B QLoRA fits on GPU with ZeRO2 where LoRA has an OOM issue; reducing LoRA learning rate to 2 × 10^-5 and alpha to 16 drops LLaVA-Bench from 71.8 to 65.5 at rank 64.The authors report that larger learning rate and alpha improve LoRA results, while raising rank from 64 to 128 and 512 yields smaller gains.Source: §3.2 · p. 5
- After training solely on multimodal instruction data, LLaVA has language capability comparable to its Vicuna initialization; adding language-only instruction data boosts multimodal but not language ability.The authors partially attribute retained language capability to complex reasoning questions and long-form answers in LLaVA-Instruct-158K.Source: §3.2 · p. 5
- LLaVA-70B improves LLaMA-2-70B-Chat by 2.4 MMLU points to 65.1, reported as the best 70B-model score according to [17] and the Chatbot Arena Leaderboard.Source: §3.2 · p. 6
4 Conclusions and Limitations
Scaling LMM consistently enhances performance and language capabilities, while parameter-efficient tuning and data curation offer additional gains. The findings remain preliminary because the study uses small training datasets.
4 Conclusions and Limitations
- Scaling LMM consistently enhances performance and language capabilities, primarily through increased LLM size.The authors leave scaling the vision encoder for future work on visual capabilities and vision tasks.Source: §4 · p. 6
- LoRA/QLoRA achieve performance comparable to full-model fine-tuning, with a good performance-cost trade-off in some settings with limited GPU memory.The authors report significant cost reductions in both training and serving.Source: §4 · p. 6
- Properly selecting image resolutions and mixing multimodal-language data can significantly improve resultant LMM performance.Source: §4 · p. 6
- Visual instruction tuning can improve LMM language capability, but the study's findings are preliminary because its training datasets are small.The authors plan to investigate data selection and mixing with much larger datasets.Source: §4 · p. 6