Source-linked AI summary
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Ka-Wei Lee
TL;DR
Open-source LLM adaptation remains costly with full-model fine-tuning, and the best PEFT module, placement, and configuration for different tasks are unclear. The paper introduces LLM-Adapter, training datasets, and a broad empirical study, finding that smaller adapter-tuned models can match or surpass larger models on specific reasoning tasks.
Problem
Full-model fine-tuning is computationally and storage-intensive, while the best PEFT module, placement, and hyperparameter configuration for LLM tasks remains unclear.
Method
The paper develops LLM-Adapter, constructs math and commonsense fine-tuning datasets, and evaluates diverse PEFT methods across open-source LLMs and reasoning tasks.
Results
LLaMA-13B with LoRA outperforms GPT-3.5 (>175B) on MultiArith, AddSub, and SingleEq, while ID-fine-tuned LLaMA-13B adapters outperform ChatGPT on commonsense reasoning tasks.
Takeaways & Limitations
Adapter-based PEFT enables smaller open-source LLMs to achieve competitive or superior performance to much larger models on specific reasoning tasks.
Takeaways & Limitations
The study could not evaluate larger models such as LLaMA-33B and LLaMA-65B and does not explore combining different adapters.
Abstract
from arXiv · showhide
The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance. To enable further research on PEFT methods of LLMs, this paper presents LLM-Adapters, an easy-to-use framework that integrates various adapters into LLMs and can execute these adapter-based PEFT methods of LLMs for different tasks. The framework includes state-of-the-art open-access LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as Series adapters, Parallel adapter, Prompt-based learning and Reparametrization-based methods. Moreover, we conduct extensive empirical studies on the impact of adapter types, placement locations, and hyper-parameters to the best design for each adapter-based methods. We evaluate the effectiveness of the adapters on fourteen datasets from two different reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few extra trainable parameters yields comparable, and in some cases superior, performance to powerful LLMs (175B) in zero-shot inference on both reasoning tasks.
1 Introduction
LLM-Adapters addresses the computational burden and unresolved design choices of adapting open-source LLMs by providing a unified PEFT framework and studying adapter configurations across reasoning tasks. The study finds that carefully configured adapters can make smaller models competitive with, and sometimes superior to, much larger models.
- Motivation: Full-model fine-tuning of open-source LLMs is computationally and storage-intensive, motivating parameter-efficient alternatives.PEFT fine-tunes a small set of external parameters while retaining comparable or superior performance to full-model fine-tuning.
- Research scope: The paper studies PEFT placement, adapter performance, and ID-versus-OOD behavior across BLOOM, GPT-J, and LLaMA.These questions target which module, layer placement, and hyperparameter configuration best suit different tasks and datasets.
- Findings: The optimal placements are after MLP layers for series adapters, parallel with MLP layers for parallel adapters, and after both attention and MLP layers for LoRA.These placement findings are reported as the study’s adapter-design results.
- Findings: LLaMA-13B with LoRA can outperform GPT-3.5 (>175B) on MultiArith, AddSub, and SingleEq, while ID-fine-tuned LLaMA-13B adapters outperform ChatGPT on commonsense reasoning.The results indicate that smaller PEFT-adapted models can exceed larger models on specific tasks and data settings.
- Study design: The study constructs two training datasets for math and commonsense reasoning and conducts a comprehensive empirical study of PEFT methods.The contributions include dataset construction, framework development, and extensive experiments addressing three research questions.
- Framework: LLM-Adapter is a user-friendly framework that integrates diverse adapters into LLMs for implementation across a wide range of tasks.The framework supports adapter-based PEFT research on open-source language models.
2 PEFT Overview
The paper reviews four PEFT families for LLMs: prompt-based learning, reparametrization, series adapters, and parallel adapters. These methods add or modify a small set of parameters through distinct architectural mechanisms.
- Prompt-based learning: Prompt-based learning replaces hard-prompt optimization with trainable continuous prompts added to input embeddings or hidden states.Prompt tuning adds a trainable prefix to input embeddings, while Prefix Tuning adds soft prompts to hidden states at all layers.
- Architectural overview: Figure 1 compares the architectures of Prefix-Tuning, LoRA, Series Adapter, and Parallel Adapter.The figure provides a visual overview of the four adapter designs discussed in the section.
- Reparametrization-based method: Reparametrization methods reduce trainable parameters by representing weight updates with low-rank matrices.LoRA decomposes the update to a pretrained weight matrix into lower-rank matrices A and B, with r ≪ d as an important hyper-parameter.
- Series Adapter: Series adapters insert learnable modules sequentially within a sublayer, commonly after attention or feed-forward layers.The adapter down-projects a layer output, applies a nonlinear function, and up-projects it back to the original dimension.
- Parallel Adapter: Parallel adapters add learnable modules alongside distinct backbone sublayers rather than sequentially within them.Variants include Multi-head Parallel Adapter, Scaled Parallel Adapter, and Ladder Side-Tuning with a lightweight side network receiving intermediate activations.
3 Experiment Setup
The experiments evaluate adapter-based PEFT across fourteen arithmetic and commonsense reasoning benchmarks. The study constructs dedicated math and commonsense fine-tuning datasets and uses LLaMA, BLOOMz, and GPT-J as base models.
- 3.1 Benchmarks: 14 benchmark datasets cover arithmetic reasoning and commonsense reasoning tasks.The arithmetic group includes GSM8K, SVAMP, MultiArith, AddSub, AQuA, and SingleEq; the commonsense group includes BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-c, ARC-e, and OBQA.
- 3.2 Fine-tuning Data Collection: Math10K contains 10K math reasoning samples assembled from GSM8K, MAWPS, MAWPS-single, and selected AQuA examples.The source datasets primarily provide equations and answers, forming the basis for the math fine-tuning collection.
- 3.2 Fine-tuning Data Collection: Commonsense170K contains 170K samples formatted from eight commonsense reasoning training sets using predefined task templates.Each template describes the task goal, presents the content, and supplies the answer format.
- 3 Experiment Setup: The experiments use LLaMA 7B and 13B, BLOOMz 7B, and GPT-J 6B as base models.These open-access models provide the backbone models for evaluating the adapter methods.
4 Experiment Results
The experiments identify adapter placements and configurations for math reasoning, then compare PEFT methods with large-model baselines across arithmetic and commonsense tasks. Results show strong task- and data-distribution dependence, including cases where smaller adapted models outperform larger baselines.
- Placement: Series Adapter performs best after MLP layers, Parallel Adapter within MLP layers, and LoRA across both Multi-head Attention and MLP layers.The corresponding average accuracies are 59.5%, 61.7%, and 60%, respectively, on math reasoning datasets.
- Configuration: Prefix-Tuning reaches 42.0% average accuracy with 10 virtual tokens, while Series and Parallel Adapters perform best with bottleneck size 256.Increasing the bottleneck size to 512 decreases the accuracy of both Series and Parallel Adapters.
- Commonsense Reasoning: LLaMA-13B with Parallel Adapter achieves 81.5% average accuracy, exceeding ChatGPT by 4.5% on commonsense reasoning.The commonsense adapters are fine-tuned on Commonsense170K, which includes the training sets of the evaluated commonsense datasets.
- ID and OOD Analysis: PEFT methods show more remarkable results on commonsense reasoning, where ID fine-tuning enables smaller LLaMA models to surpass larger models such as ChatGPT and PaLM.For math reasoning, PEFT methods outperform GPT-3.5 on selected OOD datasets but retain a gap on the ID datasets GSM8K and AQuA.
5 Qualitative Study
The qualitative study compares ChatGPT with LLaMA-13B outputs using several PEFT methods on a GSM8K example. Series Adapter produces a correct solution, while other adapted outputs exhibit directional or arithmetic errors.
- GSM8K Example: ChatGPT correctly solves the GSM8K example through two explicit calculation steps.It computes 24 square feet and then 576 mosaic tiles.
- Adapter Outputs: LLaMA-13B with Series Adapter gives a high-quality answer with the crucial two steps and correct calculations.LLaMA-13B-Parallel and LLaMA-13B-LoRA generate nearly identical rationales, but Parallel Adapter produces an incorrect final calculation.
6 Conclusion
The paper develops LLM-Adapter, a user-friendly framework for implementing diverse adapter-based PEFT methods in LLMs. It also constructs two fine-tuning datasets for math and commonsense reasoning to support empirical evaluation and further research.
- Framework: LLM-Adapter integrates diverse adapters into LLMs for implementing adapter-based PEFT methods across tasks.The framework is intended to support researchers evaluating different PEFT methods on downstream tasks.
- Datasets: The paper constructs two high-quality fine-tuning datasets to enhance PEFT performance on math reasoning and commonsense reasoning tasks.These datasets are used with the toolkit to conduct the paper’s evaluations.
- Future Research: The authors hope the work will encourage further research on PEFT methods for LLMs.
7 Limitations
The paper identifies two limitations: it does not evaluate larger LLaMA models because of computing constraints, and it does not explore combining different adapters.
- Model Scale: Computing constraints prevent evaluation of larger models such as LLaMA-33B and LLaMA-65B.The authors anticipate that these larger models may yield superior performance because of enhanced language understanding capabilities.
- Adapter Combinations: The study does not explore combinations of different adapters because their search space is extensive.The authors identify adapter combinations as a direction for future research.
A.1 Math Reasoning Prompt Templates
The math-reasoning data-collection prompts ask ChatGPT to provide step-by-step rationales and final answers as Arabic numerals, supporting adapter fine-tuning and answer extraction.
- The prompts request reasoning steps so rationale information can be used to fine-tune adapters.
- The prompts request Arabic numerals for final answers, making answers easier to extract from generated outputs.
A.2 Commonsense Data Templates
Commonsense datasets use structured templates that describe each task’s goal before presenting its content and answer.
- Each commonsense data template presents the task goal, corresponding content, and answer in sequence.
A.3 Placement Analysis
The analyses examine adapter placement and PEFT hyperparameters on Math10K with LLaMA-7B, while accompanying tables document templates and example outputs across reasoning datasets.
- Adapter placement: 59.5% average accuracy is achieved by placing the Series Adapter after the MLP layers on math reasoning datasets.The analysis uses Math10K for fine-tuning and LLaMA-7B as the base model.
- Adapter placement: The Parallel Adapter performs best when placed within the MLP layers on the evaluated math reasoning datasets.