Source-linked AI summary
Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, Rongrong Ji
TL;DR
Existing vision-language adaptations of large language models require costly pre-training and substantial parameter updates. The paper introduces MMA, which uses lightweight adapters and modality-based routing to adapt LLaMA into LaVIN efficiently. LaVIN achieves competitive multimodal performance while requiring 1.4 training hours and 3.8M trainable parameters, though it remains limited on fabricated responses and extremely fine-grained visual content.
Problem
Existing multimodal LLM solutions require excessive parameter optimization and additional large-scale pre-training, making vision-language adaptation expensive.
Method
MMA connects image encoders and LLMs through lightweight adapters and dynamically routes single- and multimodal instructions during end-to-end optimization.
Results
LaVIN delivers competitive performance and superior training efficiency, requiring 1.4 hours with 8 A100 GPUs and 3.8M updated parameters on ScienceQA.
Takeaways & Limitations
MMA provides an affordable route to vision-language instruction tuning while supporting multimodal chatbot capabilities across tasks such as coding, mathematics, and image captioning.
Takeaways & Limitations
LaVIN may fabricate responses and cannot reliably identify extremely fine-grained visual content such as text characters.
Abstract
from arXiv · showhide
Recently, growing interest has been aroused in extending the multimodal capability of large language models (LLMs), e.g., vision-language (VL) learning, which is regarded as the next milestone of artificial general intelligence. However, existing solutions are prohibitively expensive, which not only need to optimize excessive parameters, but also require another large-scale pre-training before VL instruction tuning. In this paper, we propose a novel and affordable solution for the effective VL adaption of LLMs, called Mixture-of-Modality Adaptation (MMA). Instead of using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters, to bridge the gap between LLMs and VL tasks, which also enables the joint optimization of the image and language models. Meanwhile, MMA is also equipped with a routing algorithm to help LLMs achieve an automatic shift between single- and multi-modal instructions without compromising their ability of natural language understanding. To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision-language instructed model as LaVIN. To validate MMA and LaVIN, we conduct extensive experiments under two setups, namely multimodal science question answering and multimodal dialogue. The experimental results not only demonstrate the competitive performance and the superior training efficiency of LaVIN than existing multimodal LLMs, but also confirm its great potential as a general-purpose chatbot. More importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4 training hours with 3.8M trainable parameters, greatly confirming the effectiveness of MMA. Our project is released at https://luogen1996.github.io/lavin.
1 Introduction
Existing multimodal LLM adaptations are costly because they rely on large auxiliary systems, extensive pre-training, or broad parameter updates. MMA instead uses lightweight adapters and routing to efficiently adapt LLaMA into LaVIN, which achieves competitive performance with substantially lower training and storage costs.
- 1 Introduction: Existing expert-system approaches incur computation and storage overhead, while modular training requires an additional neck branch and large-scale image-text pre-training.Modular approaches may also update most LLM parameters, further limiting instruction-tuning efficiency.
- 1 Introduction: MMA connects image encoders and LLMs with lightweight adapters, jointly optimizing the multimodal model without another large-scale pre-training stage.Its routing scheme dynamically selects adaptation paths for single- and multimodal instructions while preserving NLP capability.
- 1 Introduction: LaVIN achieves competitive ScienceQA performance while reducing training time by up to 71.4% and storage costs by 99.9%.The comparison is reported against existing multimodal LLMs such as LLaVA.
- 1 Introduction: LaVIN on ScienceQA requires only 1.4 hours with 8 A100 GPUs and updates 3.8M parameters.The model was also extended to multimodal dialogue using text-only instructions and text-image pairs.
- 1 Introduction: The paper proposes LaVIN as a multimodal LLaMA-based model and reports potential for general-purpose chatbot use.Qualitative results cover coding, mathematics, image captioning, and vision-language understanding.
2 Related Work
Related work connects parameter-efficient adaptation and instruction tuning to the development of multimodal language models. The paper positions MMA and LaVIN within these approaches using modality-specific adapters and end-to-end mixture-of-modality training.
- Parameter-Efficient Transfer Learning: Parameter-efficient transfer learning inserts or fine-tunes a small number of parameters to adapt large language models while reducing training and storage overhead.Adapters are described as small MLP networks inserted into language models to project hidden features into task-relevant semantic spaces.
- Instruction Tuning: Instruction tuning fine-tunes language models on diverse natural-language task descriptions and has improved the performance and generalization of models such as InstructGPT and FLAN-T5.
- Multimodal Instruction-Following Models: MMA uses Mixture-of-Modality Adapters for instructions from different modalities, while LaVIN is optimized end-to-end through Mixture of Modality Training.
3 Method
MMA adapts LLMs to vision-language instructions with lightweight modality-aware adapters and joint end-to-end optimization. LaVIN implements this design with a compact visual interface and achieves comparable performance to prior models with substantially lower training and storage costs.
- MMA overview: MMA combines the Mixture-of-Modality Adapter with Mixture-of-Modality Training to support multimodal adaptation through lightweight modules and cheap joint optimization.The large image encoder and LLM are frozen while inserted adapters are fine-tuned end-to-end.
- Mixture-of-Modality Adapter: The MM-Adapter uses modality tokens and a routing function to select adaptation paths for single-modal and multimodal inputs.The routing weights are computed from modality information, while shared downsampling projections reduce parameter costs.
- Mixture-of-Modality Training: MMT trains on randomly mixed text-only and text-image instructions, enabling the multimodal model to execute instructions from different modalities.The training objective jointly optimizes the multimodal model through adapter parameters.
- Large Vision-language Instructed Model: LaVIN connects LLaMA with CLIP-ViT using visual adapters that transform visual features into the LLM’s representation space.Text-only inputs use the modality token and text embeddings, while text-image inputs concatenate the modality token, visual features, and text embeddings.
- Large Vision-language Instructed Model: LaVIN’s visual neck is 6 times smaller than LLaVA’s while achieving close performance, and its ScienceQA evaluation uses a comparison table spanning question classes and trainable parameters.The cited table distinguishes natural, social, and language science questions, context types, grade ranges, and trainable-parameter counts.
4 Experiments
Experiments evaluate LaVIN across multimodal science question answering, image captioning, zero-shot benchmarks, training efficiency, instruction following, and multimodal dialogue. Across these settings, LaVIN combines competitive performance with substantially lower training and storage costs.
- Results on ScienceQA: LaVIN achieves competitive ScienceQA performance while offering a better performance–efficiency trade-off than existing multimodal methods.LaVIN-7B outperforms LLaMA-Adapter by +4.22, LaVIN-13B gains +5.64, and LaVIN-13B remains comparable to LLaVA while performing better in some question classes.
- Results on ScienceQA: LaVIN without VL pre-training significantly outperforms comparable approaches, gaining +5.02 over LLaVA on ScienceQA.The compared LLaVA and LLaMA-Adapter models achieve 85.81 and 85.19 average accuracy, respectively.
- Results on COCO Captioning: LaVIN remains competitive on COCO captioning while reducing data, parameter, and training requirements relative to larger pre-training-based models.With 0.6M pre-training examples and 5.4M updated parameters, LaVIN reaches 131.7 CIDEr in 4 GPU hours, while BLIP-2 requires more than 300 GPU hours.
- Zero-shot evaluation on NLP and multimodal benchmarks: LaVIN improves zero-shot TruthfulQA performance over LLaMA by +9.2% and remains competitive on MME, including 963.6 versus MiniGPT-4’s 866.5 on MME-C.These results support LaVIN’s reported generalization across NLP and multimodal benchmarks.
- Ablation study: Ablations show that mixture-of-modality training provides the largest reported gain, +4.69, while joint vision-modality training contributes up to +3.67.The study reports that each MMA and LaVIN design contributes to final performance.
- Comparison of training efficiency: LaVIN reduces training time by about 80% for 7B and 71.4% for 13B, while saving more than 40% GPU memory and 99.9% disk storage versus existing approaches.The training-efficiency comparison evaluates all methods on 8 A100 GPUs.
- Qualitative evaluation: LaVIN produces stronger qualitative instruction-following and multimodal-dialogue responses than compared models across coding, mathematics, image captioning, and multi-turn conversations.In multimodal dialogue, GPT-4 assigns LaVIN higher scores among the compared models; the instruction-following examples also report a correct mathematical answer of 28.8.
5 Limitations and Broader Impact
LaVIN shares limitations with existing multimodal LLMs: it may produce incorrect or fabricated responses and struggles with extremely fine-grained visual content. The authors identify visual recognition improvement as future work.
- LaVIN may generate incorrect or fabricated responses, similar to existing multimodal LLMs.
- LaVIN cannot identify extremely fine-grained visual content such as text characters, leaving recognition ability for future improvement.
6 Conclusions
MMA offers an affordable vision-language instruction-tuning approach using lightweight adapters, joint optimization, and routing across instruction modalities. LaVIN achieves superior reasoning ability across various instruction-following tasks.
- MMA connects the image encoder and LLM through lightweight adapters for end-to-end joint optimization with few parameters.Its routing algorithm shifts reasoning paths automatically between single- and multimodal instructions.