Source-linked AI summary
HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, Chengzhong Xu
TL;DR
PEFT methods such as LoRA reduce the resources needed for adaptation but can underperform full fine-tuning, especially on heterogeneous corpora and complex domains. HydraLoRA addresses this with an asymmetric shared-A, multiple-B architecture that improves domain adaptation while increasing efficiency and limiting parameter growth.
Problem
LoRA-like methods can fall short of full fine-tuning because restricting tuned parameters limits learning from diverse, heterogeneous datasets.
Method
HydraLoRA uses a shared A matrix and multiple B matrices, with lightweight LoRA experts and a router trained to handle distinct intrinsic components.
Results
HydraLoRA improves domain adaptation and fine-tuning and inference efficiency while maintaining minimal parameter growth.
Takeaways & Limitations
The asymmetric design offers a pathway to balance learning from diverse datasets with a lean model.
Takeaways & Limitations
LoRA-Split introduces parameter redundancy because its segmented architecture can increase total parameters manifoldly compared with a single LoRA model.
Abstract
from arXiv · showhide
Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases.
1 Introduction
PEFT methods reduce the cost of adapting LLMs but can lose performance on complex, heterogeneous domains. HydraLoRA addresses this trade-off with an asymmetric architecture motivated by LoRA analyses of shared and task-specific structure.
- PEFT freezes the backbone and fine-tunes a small set of task-specific parameters, reducing the memory and computation required for adaptation and deployment.
- LoRA can underperform full fine-tuning because its reduced parameter count limits learning, especially for domains containing complex sub-domains and diverse tasks.
- The paper asks which architecture can preserve reduced-parameter efficiency while delivering superior model performance.
- Exploratory experiments on LLaMA2 across multiple downstream tasks motivate replacing one domain-wide LoRA with smaller heads dedicated to specific components.
- Parameter visualization suggests that some parameters capture cross-data commonalities while others capture intrinsic-component-specific aspects, motivating an explicit asymmetric structure.
- HydraLoRA uses one shared A matrix for parameter efficiency and multiple B matrices, while autonomously separating samples into intrinsic components during fine-tuning.
2 Background and Motivation
PEFT methods reduce trainable parameters but can lose performance on heterogeneous datasets, where task interference becomes more pronounced. Experiments with LoRA indicate that multiple task-specific heads and asymmetric sharing of A and B matrices can improve efficiency and adaptation.
- Practical dilemma: LoRA and related PEFT methods tune limited parameters, improving training efficiency but often underperforming full fine-tuning on diverse datasets.The performance gap widens as corpus heterogeneity increases.
- Practical dilemma: Figure 2 associates greater performance losses for parameter-efficient approaches with heterogeneous corpora containing varied content and styles.Such heterogeneity can produce interference during adaptation.
- Observation I: With the same parameter count, multiple smaller LoRA heads dedicated to individual downstream tasks outperform a single LoRA trained across the entire domain dataset.This observation suggests that separating task-specific adaptation can reduce interference.
- Observation II: Across independently trained heads, A matrices tend to converge whereas B matrices remain distinguishable across tasks.The analysis uses t-SNE visualizations of LoRA submodules for summarization, closed QA, and information extraction.
- Asymmetric architecture: The proposed architecture shares A parameters across sub-domains or tasks while using multiple B matrices for different intrinsic components.Mixture-of-Experts methods can automatically route samples when intrinsic components are not manually identified.
3 HydraLoRA
HydraLoRA introduces an asymmetric LoRA architecture that shares one A matrix across samples while using multiple B matrices for specialized functionality. Its MoE workflow identifies intrinsic components, routes samples to expert adapters during fine-tuning, and dynamically merges B matrices during inference.
- 3 HydraLoRA: HydraLoRA uses a shared A matrix and multiple distinct B matrices to combine shared knowledge with specialized functionalities.The asymmetric structure is designed to avoid redundancy across a heterogeneous corpus.
- 3 HydraLoRA: During fine-tuning, HydraLoRA adaptively identifies and initializes intrinsic components, then uses an MoE router to assign samples to B-matrix experts.The router and experts are trained while the remaining LLM parameters stay frozen.
- 3 HydraLoRA: The router combines expert contributions using gating scores produced from token representations by a trainable transformation matrix followed by softmax.The gate function regulates the contributions of the N experts.
- 3 HydraLoRA: During inference, HydraLoRA flexibly and dynamically merges multiple B matrices through the trained router.Because the B matrices operate as linear functions, their weighted average is computed before applying the PEFT transformation.
4 Experiments
The experiments evaluate HydraLoRA across single-domain, mixed-task, efficiency, ablation, and cluster-count settings. Results indicate that its asymmetric shared-A, multiple-B design improves performance and efficiency while addressing task interference, though LoRA-Split increases parameter costs.
- Overall performance: HydraLoRA consistently outperforms competing schemes across diverse single-domain and mixed-task evaluations.The comparisons include conventional PEFT methods, LoRA variants, and merge methods.
- Overall performance: LoRA-Split outperforms a single LoRA with the same parameter count across evaluated scenarios by separating tasks into four components.The comparison uses LoRA rank 32 versus four LoRA-Split components with rank 8 each.
- Overall performance: LoRA-Split reduces task interference but introduces parameter redundancy and additional computational-resource and optimization challenges.Its multiple vanilla LoRAs can substantially increase total parameters relative to a singular LoRA.
- Overall performance: HydraLoRA improves parameter efficiency by sharing matrix A across samples while using separately trained B matrices and a trainable router.The design separates commonalities from diversities and composes multiple B matrices.
- Energy and throughput analysis: 1.96× faster training and 49.6% lower energy cost are reported for HydraLoRA versus LoRA with rank 32.The comparison uses LLaMA2-7B fine-tuned on GSM-8K; LoRA-Split exceeds LoRA rank 32 in energy consumption and latency.
- Ablation and hyperparameters: The ablation study tests the contributions of the MoE architecture, gate function, and Hydra structure, while separate experiments vary the number of intrinsic components.The cluster-count experiment evaluates fine-tuned Dolly-15K on MMLU.
5 Related work
Prior work develops parameter-efficient adaptation through adapters and multi-LoRA architectures. These approaches respectively add trainable layers or combine multiple adapters to reduce adaptation costs or improve task handling.
- Parameter-efficient fine-tuning: Adapters add trainable dense layers to a frozen pretrained model to reduce memory and storage costs during adaptation.The related work describes adapters as a prominent PEFT approach used across domains.
- Multi-LoRA architecture: LoraHub selects combinations of trained adapters during inference, while MultiLoRA horizontally splits LoRA modules to reduce parameter dependence.Both methods represent multi-LoRA approaches with different mechanisms for combining or scaling adapters.
6 Conclusion
The paper identifies domain interference from using a single LoRA across varied tasks and proposes HydraLoRA as an asymmetric alternative. Its shared and component-specific matrices aim to improve adaptation and inference efficiency with minimal parameter growth.
- Conclusion: A single LoRA can suffer from domain interference when adapting LLaMA2 across varied tasks, motivating HydraLoRA.The conclusion frames HydraLoRA as a response to limitations observed in single-LoRA adaptation.
- Conclusion: HydraLoRA uses one shared matrix for all samples and distinct matrices for each intrinsic component.The asymmetric design selectively focuses on distinct components during adaptation.
- Conclusion: The paper positions HydraLoRA as a pathway to improve LLMs while minimizing parameter growth.The stated conclusion emphasizes balancing diverse-dataset learning capability with a lean model.
A.1 Datasets
The evaluation covers general, medical, law, and mixed multi-task/domain settings, using task-specific fine-tuning datasets and corresponding benchmarks.
- General: The general setting fine-tunes on databricks-dolly-15k and evaluates generic language capability with MMLU.
- Medical: The medical setting uses GenMedGPT and clinic-10k, evaluating clinical knowledge, professional medicine, and college medicine tasks in MMLU.
- Law: The law setting fine-tunes on Lawyer-Instruct and US-Terms, then evaluates law tasks in MMLU.
- Multi-task Domain: The mixed multi-task/domain setting uses Flanv2 subsets spanning NLU and NLG, grouped into 10 task clusters and evaluated with BBH.
- Dataset Details: The dataset inventory includes structured-to-text conversion, translation, commonsense reasoning, sentiment analysis, and paraphrase detection tasks.
PEFT methods
The compared PEFT methods range from full fine-tuning and prompt-based adaptation to LoRA variants that route, combine, or cluster specialized modules.
- Baseline and Prompt Methods: Full fine-tuning updates all model parameters from pretrained weights and biases, whereas prompt tuning updates task-specific prompts while freezing pretrained parameters.
- Baseline and Prompt Methods: P-Tuning optimizes trainable prompt embeddings with a prompt encoder and permits prompt tokens throughout the input sequence.
- Baseline and Prompt Methods: Prefix Tuning learns task-specific vectors inserted across model layers while keeping the pretrained model frozen.
- Adaptive PEFT Methods: IA3 inserts learned vectors into transformer architectures to reduce trainable parameters and inference latency, while AdaLoRA allocates more parameters to important matrices and layers.
- LoRA Variants: LoRA MoE uses a router and gating scores to weight outputs from multiple parameterized experts, enabling specialized processing.
- LoRA Variants: LoraHub randomly aggregates 20 LoRAs and uses black-box optimization to learn their parameter-level weights without large-model gradient calculations.
- HydraLoRA Initialization: HydraLoRA initializes the number of matrix B modules by extracting TF-IDF features, clustering them, and selecting K through the elbow method based on SSE.
- Breakdown Analysis: The GSM8K breakdown analyzes 32 decoder-layer modules, each containing q_proj_A, q_proj_B, v_proj_A, and v_proj_B submodules across IID subsets.
D More Results
Table 4 compares Base, LoRA, LoraHub, LoRA MoE, and HydraLoRA on the BBH benchmark for a mixed-task domain using LLaMA2-7B.
- Comparative Performance: Table 4 evaluates five tuning schemes—Base, LoRA, LoraHub, LoRA MoE, and HydraLoRA—on mixed-task BBH with LLaMA2-7B in a 3-shot setting.
- Breakdown Analysis: Figure 10 presents a T-SNE breakdown analysis of LoRA modules for Dolly-15K subsets across different layers.
- Comparative Performance: The reported comparison concerns mixed-task-domain performance on the BBH benchmark rather than a single-domain evaluation.
E Limitation
The paper identifies higher training cost as a limitation and leaves several PEFT configurations and pre-training evaluation for future work, while also discussing broader societal risks.
- Limitations: HydraLoRA requires 1 to 2 times more training iterations than typical PEFT methods because it employs multiple adapter copies.
- Limitations: The study evaluates established PEFT techniques such as LoRA but does not test prompt-tuning or adapter configurations.
- Limitations: The evaluation is limited to fine-tuning, with efficacy during pre-training left for future research.
- Broader Impacts: The broader-impacts discussion identifies misuse, job displacement, privacy, and security as potential negative societal consequences of easier LLM deployment.