LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

Peng Gao; Jiaming Han; Renrui Zhang; Ziyi Lin; Shijie Geng; Aojun Zhou; Wei Zhang; Pan Lu; Conghui He; Xiangyu Yue; Hongsheng Li; Yu Qiao

Source-linked AI summary

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, Yu Qiao

arXiv:2304.15010v1Apr 28, 2023cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

LLaMA-Adapter V2 addresses the limited open-ended visual instruction following of earlier parameter-efficient LLaMA adaptations. It combines broader parameter tuning, early visual fusion, disjoint-parameter joint training, and inference-time experts, yielding stronger visual and language instruction following while remaining parameter efficient.

Problem
Earlier LLaMA-Adapter multimodal models could produce prompted captions but could not generalize to open-ended visual reasoning and visual question answering.
Method
The method combines bias tuning, early fusion of visual prompts, joint training on image-text and instruction-following data with disjoint parameters, and plug-in expert models.
Results
LLaMA-Adapter V2 becomes a zero-shot visual instruction model with stronger multimodal and language instruction-following ability, including multi-turn dialogue.
Takeaways & Limitations
The framework supports parameter-efficient multimodal reasoning using small-scale image-text and instruction-following data, while experts can further enhance image understanding without training costs.
Takeaways & Limitations
The model can produce inaccurate or unrelated image descriptions, especially for out-of-distribution images, motivating the use of additional expert systems.

Abstract

from arXiv · show

How to efficiently transform large language models (LLMs) into instruction followers is recently a popular research direction, while training LLM for multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter demonstrates the potential to handle visual inputs with LLMs, it still cannot generalize well to open-ended visual instructions and lags behind GPT-4. In this paper, we present LLaMA-Adapter V2, a parameter-efficient visual instruction model. Specifically, we first augment LLaMA-Adapter by unlocking more learnable parameters (e.g., norm, bias and scale), which distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, we propose an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset. During inference, we incorporate additional expert models (e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal instructions by merely introducing 14M parameters over LLaMA. The newly designed framework also exhibits stronger language-only instruction-following capabilities and even excels in chat interactions. Our code and models are available at https://github.com/ZrrSkywalker/LLaMA-Adapter.

1. Introduction

LLaMA-Adapter V2 targets open-ended visual instruction following with a parameter-efficient design that combines broader tuning, early visual fusion, joint training, and modular experts. It aims to improve multimodal reasoning and language instruction following without relying on massive multimodal instruction datasets.

Motivation: LLaMA-Adapter’s multimodal version was limited to prompted captioning and could not handle open-ended visual reasoning or visual question answering.The limitation was linked to a lack of multimodal instruction-tuning data.
Motivation: Visual features can dominate adaptation prompts during image-text alignment, causing instruction-following ability to deteriorate rapidly.
Method: LLaMA-Adapter V2 distributes visual prompts only to early LLM layers, reducing interference between image-text alignment and language instruction tuning.The visual prompts are placed in the first K layers, where K < N − L and N is the total number of Transformer layers.
Method: The method unlocks normalization, bias, and scale parameters so instruction-following knowledge is distributed across the LLM while adding only approximately 0.04% tunable parameters.
Method: Joint training uses disjoint parameters for image-text pairs and instruction-following data, while expert models such as captioning, detection, and OCR systems enhance image understanding.The framework is designed to avoid end-to-end pre-training on massive image-text pairs.
Results: LLaMA-Adapter V2 surpasses LLaMA-Adapter in language instruction following and supports multi-turn dialogue, while providing balanced visual instruction tuning without multimodal instruction data.

2. Related Work

Related work develops instruction-following language models, visual instruction systems, parameter-efficient fine-tuning, and expert-model integration. These lines of research motivate combining efficient adaptation with broader multimodal reasoning capabilities.

Instruction-following Language Models: Instruction-following language models adapt pretrained LLMs through instructional datasets to improve their ability to respond to user directions.
Visual Instruction Models: Traditional captioning and VQA systems provide limited forms of visual instruction because captioning produces short descriptions and VQA generally lacks explicit explanations.
Visual Instruction Models: GPT-4 demonstrates broader visual instruction following across tasks including meme explanation, document-level reasoning, and exam problem-solving.
Parameter-efficient Fine-tuning: Parameter-efficient fine-tuning freezes most foundation-model parameters and optimizes only a small portion as model scale makes full fine-tuning increasingly impractical.
Integration of Expert Systems: Integrating multiple expert models is an established strategy for improving performance and robustness, particularly in computer vision.

3. A Revisit of LLaMA-Adapter

LLaMA-Adapter provides parameter-efficient instruction following by injecting learned prompts into a frozen LLaMA and can incorporate visual features through a projection layer. Its open-ended multimodal capability is limited because learned visual cues can override instruction-following behavior.

Zero-initialized Attention: LLaMA-Adapter freezes LLaMA and adds a lightweight 1.2M-parameter adapter with learnable soft prompts at higher Transformer layers.
Zero-initialized Attention: A progressively increasing gating magnitude gradually injects instruction-following ability while preserving language generation early in training.
Simple Multi-modal Variant: For multimodal inputs, a visual encoder extracts features that are aggregated and projected into the linguistic embedding space.
Open-ended Multi-modal Reasoning: Fine-tuning adapter and visual projection layers on COCO Caption exposed a failure mode in which visual cues dominate prompts and override inherent instruction-following characteristics.

4. LLaMA-Adapter V2

LLaMA-Adapter V2 expands parameter-efficient tuning with bias and scale updates, jointly trains visual and language data through disjoint parameters, and separates visual from adaptation prompts. It also uses expert systems at inference to supplement visual understanding while demonstrating stronger instruction-following behavior.

Bias Tuning of Linear Layers: LLaMA-Adapter V2 unfreezes normalization layers and adds learnable bias and scale factors to each Transformer linear layer.The modified layer is y = s · (W · x + b).
Bias Tuning of Linear Layers: 0.04% (∼5M) of the entire LLaMA is added through bias tuning, preserving a highly parameter-efficient approach.Bias and scale factors are initialized with zeros and ones to stabilize early training.
Joint Training with Disjoint Parameters: Joint training uses image-text captioning data and language-only instruction examples while optimizing disjoint parameter groups.The strategy addresses the data-volume imbalance between 500K image-text pairs and 50K instruction data.
Early Fusion of Visual Knowledge: Visual tokens and adaptation prompts are injected into different Transformer layers, with visual prompts placed early rather than fused into adaptation prompts.This early-fusion design prevents direct interactions between the two prompt types and, together with joint training, resolves their fine-tuning conflict.
Integration with Experts: The model uses smaller-scale common image-captioning data than recent visual instruction models, but its image understanding remains relatively weak and can produce inaccurate or unrelated responses.The paper addresses this limitation by integrating expert systems instead of collecting more image-text data or adopting stronger multi-modal modules.
Integration with Experts: LLaMA-Adapter V2 supplements image understanding during inference with captioning, detection, OCR, or other expert systems.An expert generates textual context for the input image, and the approach can switch among image-to-text models or search engines.

5. Experiments

Experiments evaluate LLaMA-Adapter V2 as a language instruction model, image captioner, and visual instruction model. Results show stronger language responses, detailed captioning, multi-modal reasoning, and gains from expert systems, while image understanding remains imperfect.

Experimental Setup: 52K single-turn instructions, 567K COCO captions, and 80K conversations support the instruction, captioning, and chatbot experiments.The model is trained without visual instruction data.
Language Instruction: LLaMA-Adapter V2 provides comprehensive answers and detailed explanations, whereas LLaMA-Adapter produces relatively short answers.The comparison attributes this improvement to bias tuning and higher-quality instruction data.
Language Instruction: GPT4 evaluation on 80 questions finds LLaMA-Adapter V2 beats ChatGPT on total scores and 50/80 questions.The model uses LLaMA-65B while Vicuna uses LLaMA-13B; only 14M parameters are fine-tuned for LLaMA-Adapter V2.
Image Captioning: Early fusion and joint training let LLaMA-Adapter V2 perform language instruction-following and image captioning with natural, detailed descriptions.LLaMA-Adapter instead produces short, close-form descriptions and is sensitive to prompts such as “Generate caption for this image”.
Limitations: LLaMA-Adapter V2 fails on an out-of-distribution cartoon image, generating incorrect descriptions, possibly because it lacks an image-text alignment stage.This limitation motivates adding captioning and other expert systems.
Visual Understanding: Visual examples show the model integrates image information with language context for “why” and “what should” questions and can provide plausible explanations beyond direct image evidence.The authors connect these results to addressing interference between image-text pairs and instruction data.
Expert Systems: OCR expert outputs provide specific cues, such as a glasses price, enabling accurate answers to image questions.The authors state that expert integration extends the model’s multi-modal reasoning capability.

6. Conclusion

The paper presents LLaMA-Adapter V2 as a parameter-efficient visual instruction tuning system. Joint training, bias tuning, early fusion, and expert visual systems improve zero-shot visual and language instruction following, although visual understanding still trails LLaVA and expert errors can affect responses.

Conclusion: LLaMA-Adapter V2 is a parameter-efficient visual instruction tuning system.The method converts pre-trained LLMs into a zero-shot visual instruction model through joint training on image-text pairs and instruction-following data.
Conclusion: Joint training, bias tuning, and early fusion alleviate interference between image-text pairs and instruction-following data.The paper reports stronger language instruction following and multi-turn chatbot ability alongside zero-shot visual instruction following.
Conclusion: OCR and image-captioning experts provide more accurate image-related information without changing the training procedure.The experts address inaccurate image descriptions during visual instruction following.
Conclusion: The model still lags behind LLaVA in visual understanding and is susceptible to inaccurate information from expert systems.These are stated scope boundaries of the reported approach.