PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Yu Qiao

arXiv:2303.16199v3cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Open instruction-following models remain costly to develop because full fine-tuning of large language models is resource-intensive. LLaMA-Adapter freezes LLaMA and adds prompted, zero-gated attention modules for efficient adaptation. It uses 1.2M parameters and trains in less than one hour while achieving comparable instruction-following proficiency, and extends to multimodal and traditional vision-language settings.

  • Problem

    Full fine-tuning of large-scale LLaMA remains time-consuming, computation-intensive, and cumbersome to transfer across downstream scenarios.

  • Method

    LLaMA-Adapter freezes LLaMA, prepends learnable adaptation prompts in higher transformer layers, and uses zero-initialized gated attention to inject instructional signals.

  • Results

    1.2M parameters and less than one hour of training achieve instruction-following proficiency comparable to Alpaca’s full 7B-parameter fine-tuning.

  • Takeaways & Limitations

    The approach extends to image-conditioned LLaMA and parameter-efficient fine-tuning across traditional vision, language, and vision-language tasks.

Abstract

from arXiv · show

We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the word tokens at higher transformer layers. Then, a zero-initialized attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With our efficient training, LLaMA-Adapter can generate high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Besides language commands, our approach can be simply extended to multi-modal instructions for learning image-conditioned LLaMA model, which achieves superior reasoning performance on ScienceQA and COCO Caption benchmarks. Furthermore, we also evaluate the zero-initialized attention mechanism for fine-tuning other pre-trained models (ViT, RoBERTa) on traditional vision and language tasks, demonstrating the superior generalization capacity of our approach. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.

1 INTRODUCTION

LLaMA-Adapter targets the high cost of open instruction-model development by adapting frozen LLaMA with a lightweight, zero-initialized attention mechanism. It reports efficient instruction tuning, adapter-based transfer, multimodal reasoning, and generalization to traditional vision and language tasks.

  • Closed-source restrictions and high development costs impede the broader adoption of instruction-following models.
  • LLaMA-Adapter freezes LLaMA and appends learnable adaptation prompts in higher transformer layers, using zero-initialized gating to progressively inject instructional signals.The gating mechanism is intended to preserve pretrained knowledge and stabilize training.
  • 1.2M learnable parameters achieve instruction-following proficiency comparable to Alpaca’s full 7B-parameter fine-tuning.
  • Less than one hour on 8 A100 GPUs makes LLaMA-Adapter three times faster to train than Alpaca.
  • A 1.8M adapter can provide scenario-specific expertise or modality inputs without storing a complete 13G LLaMA copy.
  • The method extends to image-conditioned LLaMA, showing competitive reasoning capacity on MME, MMBench, and LVLM-eHub benchmarks.
  • Zero-initialized attention generalizes to parameter-efficient fine-tuning of ViT, ReBERTa, and CLIP across downstream vision, language, and vision-language tasks.

2 RELATED WORK

Prior work established instruction tuning and parameter-efficient fine-tuning as routes to stronger downstream models. Alpaca demonstrated open instruction tuning with 52K self-instruct examples but still fine-tuned all 7B LLaMA parameters.

  • Instruction tuning trains language models on instruction-output pairs to generate responses to natural-language commands.
  • InstructGPT improved instruction-following ability, but its closed-source status limited community access.
  • Stanford Alpaca fine-tuned all 7B LLaMA parameters using 52K self-instruct data to promote open instruction models.
  • Parameter-efficient fine-tuning freezes most pretrained parameters while seeking comparable downstream capabilities to full fine-tuning.

3 LLAMA-ADAPTER

LLaMA-Adapter inserts learnable prompts into higher LLaMA layers and uses zero-initialized attention to progressively inject instructional or visual information. The same framework supports language instruction following and image-conditioned multi-modal reasoning.

  • 3.1 LEARNABLE ADAPTION PROMPTS: Learnable adaption prompts are inserted into LLaMA’s topmost L transformer layers and prepended to word tokens as prefixes.At an inserted layer, the prompt and M word tokens form [P_l; T_l] with K+M tokens.
  • 3.2 ZERO-INITIALIZED ATTENTION: Zero-initialized attention uses a learnable gating factor to suppress under-fitted prompt influence initially and increase instructional information during training.Separate softmax components control prompt and word-token attention, while the gate is initialized at zero and regulated by tanh.
  • 3.2 ZERO-INITIALIZED ATTENTION: Zero-initialized attention progressively injects instructional signals while retaining LLaMA’s pre-trained knowledge for response generation.The mechanism is designed to avoid early disturbance from randomly initialized prompts and preserve the original model behavior during early training.
  • 3.3 MULTI-MODAL REASONING: A pre-trained image encoder extends LLaMA-Adapter to image-conditioned instruction following by combining projected image tokens with adaption prompts.The projected image token is repeated K times and added element-wise to prompts at all L inserted layers.
  • 3.3 MULTI-MODAL REASONING: The multi-modal training strategy freezes the image encoder and LLaMA for ScienceQA while training only a projection network and zero-initialized attention mechanisms.For zero-shot multi-modal evaluation, the method uses a two-stage training procedure and evaluates MME, MMBench, and LVLM-eHub.

4 EXPERIMENT

Experiments evaluate LLaMA-Adapter for language instruction following, multi-modal reasoning, ablations, and parameter-efficient fine-tuning of other large models. The method achieves competitive language and multi-modal performance with substantially lower tuning cost, while zero-initialized attention improves training and task accuracy.

  • Instruction-following evaluation: LLaMA-Adapter produces instruction-following responses comparable to Alpaca, and receives more GPT-4-evaluation wins than Alpaca and Alpaca-LoRA.The evaluation uses GPT-4 to assess response quality on 80 questions.
  • Instruction-following evaluation: 1.2M parameters, 4.9M storage, and one-hour training provide a lightweight alternative to updating Alpaca’s 7B parameters.Training time is measured on 8 A100 GPUs, and only the lightweight module’s gradients need transfer among nodes.
  • Multi-modal evaluation: 78.31% accuracy is achieved by text-only LLaMA-Adapter on ScienceQA, while visual conditioning adds 6.88 percentage points and reaches leading results.The multi-modal model adds a 0.6M projection network and is evaluated alongside other visual question-answering methods.
  • Ablation study: Inserting attention into more transformer layers improves ScienceQA accuracy until an optimal insertion depth, after which excessive insertion can disturb early word encoding.When resources are limited, inserting into all transformer layers is described as a generally good solution.
  • Ablation study: Zero-initialized attention trains faster initially and converges to zero loss, whereas random initialization approaches 0.15 without fully converging.The loss-curve comparison links the slower, incomplete convergence of random initialization to a large performance drop.
  • Zero-initialized attention for other large models: The zero-initialized attention method generalizes to ViT, RoBERTa, and CLIP, outperforming or leading prior fine-tuning methods across visual, language, and vision-language tasks.Evaluations use VTAB-1k, SQuAD, and base-to-novel classification benchmarks.

5 CONCLUSION

LLaMA-Adapter combines learnable gating with zero-initialized attention to inject instructional signals while preserving LLaMA’s pre-trained knowledge. With 1.2M parameters and one-hour training, it supports efficient instruction following, multi-modal reasoning, and fine-tuning across traditional vision and language tasks.

  • LLaMA-Adapter uses zero-initialized attention with a learnable gating factor to increasingly incorporate instructional signals while preserving LLaMA’s pre-trained knowledge.
  • With 1.2M parameters and one-hour training, LLaMA-Adapter efficiently fine-tunes instruction-following models compared with the 7B-parameter Alpaca.
  • The method extends to image-conditioned generation and achieves competitive results on visual question-answering benchmarks.
  • Zero-initialized attention attains favorable fine-tuning performance on traditional vision and language tasks, indicating strong generalization capacity.

B MORE DETAILS OF MULTI-MODAL EVALUATION

The multi-modal evaluation reports competitive performance across visual question-answering benchmarks and illustrates detailed zero-shot responses. ScienceQA samples combine visual and textual context with questions, options, and answers.

  • ScienceQA samples contain visual context, textual context, a question, multiple options, and a correct answer.
  • Across MME and LVLM-eHub evaluations, LLaMA-Adapter consistently achieves competitive results on visual question-answering problems.The evaluation follows the official procedures for three multi-modal benchmarks.
  • The section also presents open-domain zero-shot visual-question examples in which the model generates detailed responses in natural language.
  • The reported MME tables separate perception and cognition results.
  • The LVLM-eHub results cover object counting, multi-class identification, key information extraction, visual entailment, knowledge-grounded image description, and visual commonsense reasoning.

D.1 DETAILED RESULTS ON VISION MODELS

On VTAB-1k, zero-initialized attention is evaluated across 19 visual tasks spanning natural, specialized, and structured image domains. It outperforms VPT on most datasets and exceeds full fine-tuning and other methods by large margins.

  • VTAB-1k evaluates 19 downstream visual tasks: 7 natural, 4 specialized, and 8 structured tasks.The base model is supervised ImageNet-21k-pretrained ViT-B/16, and results use top-1 accuracy.
  • 16 out of 19 datasets show zero-initialized attention outperforming VPT.
  • Zero-initialized attention surpasses full fine-tuning and other fine-tuning methods by large margins.

D.2 MORE EXPERIMENTS ON LANGUAGE TASKS

The paper extends zero-initialized attention to RoBERTa-large language-task experiments beyond the main-paper extractive question-answering evaluation. These experiments cover named entity recognition and semantic role labeling across four CoNLL datasets.

  • RoBERTa-large is fine-tuned on named entity recognition and semantic role labeling in addition to extractive question answering.
  • The evaluation uses CoNLL03 and CoNLL04 for named entity recognition and CoNLL05 and CoNLL12 for semantic role labeling.
  • Table 13 reports micro-f1 scores, with asterisks marking reproduced results.

D.3 DETAILED RESULTS ON VISION-LANGUAGE MODELS

The vision-language evaluation applies zero-initialized attention to CLIP on base-to-novel generalization. With the entire CLIP frozen, the method achieves the best average classification accuracy on both base and novel categories.

  • Table 14 reports classification accuracy and harmonic mean for the vision-language comparison.
  • The entire CLIP model is frozen while adaption prompts with zero-initialized attention are inserted into its encoders.
  • The approach achieves the best average classification accuracy on both base and novel categories.

E.1 EVALUATION ON COUNTERFACTUAL REASONING

The paper evaluates multimodal counterfactual reasoning using C-VQA, a benchmark designed to assess reasoning about alternatives to observed states or past events.

  • C-VQA evaluates counterfactual reasoning, which involves processing alternatives to observed states or past events.
  • C-VQA contains 2K counterfactual question-and-answer pairs collected from VQAv2 and supplemented by ChatGPT.
  • The evaluation results are presented in Table 15.

E.2 EVALUATION ON OBJECT HALLUCINATION

The paper evaluates multimodal LLMs for object hallucination, using POPE to test whether generated descriptions remain consistent with target images.

  • POPE evaluates object hallucination by converting it into a binary classification task.
  • POPE includes 500 MSCOCO images with six questions per sample.
  • The object-hallucination evaluation results are reported in Table 16.

G COMPARISON WITH LLAMA-I

LLaMA-Adapter produces instruction-following responses comparable to the larger LLaMA-I model while tuning only 1.2M parameters on LLaMA 7B. Qualitative examples cover factual answers, lists, translations, creative writing, and code-related instructions.

  • 1.2M parameters are fine-tuned for LLaMA-Adapter while its responses remain comparable to instruction-fine-tuned LLaMA-I 65B.
  • LLaMA-Adapter lists three popular chess openings: the Sicilian Defense, French Defense, and Caro-Kann Defense.
  • LLaMA-Adapter generates JavaScript XMLHttpRequest code for sending an HTTP GET request.
  • LLaMA-Adapter generates conversations and fictional theories in response to creative-writing instructions.
Loading 2303.16199v3…