Source-linked AI summary
Mixture of LoRA Experts
Xun Wu, Shaohan Huang, Furu Wei
TL;DR
Existing LoRA composition methods face challenges including potential effects on the pretrained model’s original weights as more LoRAs are combined. MOLE addresses this with layer-wise learnable gating and achieves improved composition performance across NLP and V&L evaluations, while remaining challenged by very large-scale composition.
Problem
Existing LoRA composition can affect the pretrained model’s original weights as the number of combined LoRAs increases, diminishing generative capabilities.
Method
MOLE composes multiple trained LoRAs through hierarchical layer-wise weight control using learnable gating functions and domain-specific training objectives.
Results
MOLE outperforms existing LoRA composition methods, with extensive evaluations demonstrating enhanced composition performance in NLP and V&L domains.
Takeaways & Limitations
MOLE provides a versatile and dynamic approach for composing multiple trained LoRAs across NLP and V&L domains.
Takeaways & Limitations
When the number of LoRAs becomes very large, such as 128, performance decreases for MOLE and other LoRA composition methods.
Abstract
from arXiv · showhide
LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. Direct arithmetic merging may result in the loss of the original pre-trained model's generative capabilities or the distinct identity of LoRAs, thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based fusion exhibits limitations concerning the requisite flexibility for the effective combination of multiple LoRAs. In response to these challenges, this paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach not only achieves superior LoRA fusion performance in comparison to direct arithmetic merging but also retains the crucial flexibility for combining LoRAs effectively. Extensive experimental evaluations conducted in both the Natural Language Processing (NLP) and Vision & Language (V&L) domains substantiate the efficacy of MoLE.
1 INTRODUCTION
MOLE addresses the challenge of composing multiple trained LoRAs dynamically and efficiently while preserving their individual characteristics. It uses hierarchical, layer-wise gating to improve composition flexibility and performance across NLP and V&L tasks.
- Limitations of Existing Methods: Linear arithmetic composition can impair pretrained-model generation when combining multiple LoRAs, while normalization may weaken their unique characteristics.Reference tuning can perform better but requires manually designed masks and full-model retraining, limiting flexibility and increasing costs.
- MOLE: MOLE treats each LoRA layer as an expert and learns layer-specific composition weights through gating functions.The method modulates different trained LoRAs within each layer according to a domain objective.
- MOLE: During training, MOLE learns gating functions while keeping other parameters frozen, reducing computational costs.At inference, it can use all trained LoRAs or mask undesired LoRAs and proportionally redistribute weights without changing the gating weights.
- Motivation: MOLE targets dynamic, efficient composition of multiple trained LoRAs while preserving their individual characteristics.
- Evaluation: Extensive experiments in NLP and V&L show that MOLE improves LoRA composition performance and mitigates issues associated with existing methods.
2 BACKGROUND
LoRA composition combines specialized, parameter-efficient adaptations, but existing arithmetic and reference-tuning approaches trade off generative quality, individual characteristics, flexibility, and computational cost.
- LoRA Composition: LoRA is a parameter-efficient method for adapting large models, and multiple specialized LoRAs can be composed for more comprehensive representations.Examples include combining LoRAs specialized in clothing or facial features.
- Linear Arithmetic Composition: Direct arithmetic composition can diminish the pretrained model’s generative capabilities as the number of composed LoRAs increases.The composition affects the original parameter W when N increases.
- Linear Arithmetic Composition: Weight normalization protects the original model embedding but can reduce the individual characteristics of each LoRA as its composing weight decreases.The normalized weights satisfy Σ_i w_i = 1.
- Related Work: Prior work includes arithmetic LoRA composition in NLP and Vision & Language, gradient-free weight estimation, and arithmetic visual-concept composition.PEMs, LoRAhub, and SVDiff address different composition settings.
- Reference Tuning-Based Composition: Reference tuning improves composition through gradient fusion and controllable sampling but requires manually designed masks and retraining for new LoRAs or masks.It also retrains large models, increasing computational costs.
- Reference Tuning-Based Composition: Reference tuning uses a fundamentally different position-mask principle, so the paper primarily compares its method with linear arithmetic composition.The authors identify direct comparisons with reference tuning as potentially inappropriate.
3 METHOD
MOLE composes multiple trained LoRAs by learning layer-level weights, motivated by evidence that direct composition can damage generative ability while normalization can weaken LoRA-specific characteristics. It combines learnable gating with balanced training objectives while freezing the pretrained model and LoRA parameters.
- Motivating Observations: Direct composition can impair generative ability, while weight normalization preserves capacity but may sacrifice individual LoRA characteristics.In V&L, direct composition produced meaningless outputs and normalization compromised characteristics; in NLP, composing four or more LoRAs disordered outputs and normalization reduced performance.
- Motivating Observations: Individual LoRA layers encode distinct traits that cumulatively define the LoRA’s overall attributes.Observed layer-specific features include dog coat color and facial features in V&L, while NLP layers varied in performance across QNLI, ANLI-R1, and ANLI-R2.
- Mixture of LoRA Experts: MOLE treats LoRA layers as experts and uses learnable gating functions to determine composition weights from their outputs at each layer.The method implements hierarchical weight control and applies gating within transformer blocks to learn composition weights for multiple trained LoRAs.
- Mixture of LoRA Experts: The gating function concatenates and normalizes LoRA outputs, reduces them to N dimensions, and multiplies each expert output by its corresponding gate value.The gating parameters include a learnable parameter e and a learnable temperature scalar τ.
- Training Objective: MOLE adds a gating balancing loss because training can concentrate probability on an early well-performing LoRA and suppress other characteristics.The balancing loss encourages balanced dispatching and is combined with the domain-specific loss using coefficient α.
- Training Objective: MOLE uses domain-specific objectives and optimizes only gating-function parameters while freezing the pretrained model and trained LoRAs.V&L training uses local and global CLIP guidance, NLP follows the FLAN-T5 loss, and the overall objective is their weighted combination with the balancing loss.
4 EXPERIMENTS
MOLE is evaluated for composing multiple LoRAs in Vision & Language and Natural Language Processing tasks. Across both domains, it is compared with existing LoRA composition methods using task-specific benchmarks and alignment metrics.
- V&L domain: In V&L experiments, MOLE composes three DreamBooth LoRAs for multi-subject text-to-image generation.The experiments use Stable Diffusion V2.1 and evaluate compositions of three visual subjects.
- V&L domain: MOLE improves Text-alignment by 0.031 on average over SVDiff across 15 three-subject compositions.It also improves Image-alignment for concept 1 by 0.037 on average over SVDiff.
- V&L domain: MOLE retains user-specified subjects more accurately, whereas comparative methods may omit subjects or mix their appearance features.Qualitative comparisons report stronger subject fidelity in generated images.
- NLP domain: In NLP experiments, MOLE is evaluated on Translation, NLI, Struct to Text, Closed-Book QA, and BBH tasks using Flan-T5 LoRAs.The compared baselines include LoRAHub and PEMs.
- NLP domain: On BBH, MOLE improves average performance by 3.8 over LoRAHub and 9.0 over PEMs.It also exceeds LoRAHub and PEMs by 1.5 and 2.7 on Translation, and 2.1 and 2.6 on Struct to Text.
5 ANALYSIS
The analysis examines MOLE’s gating balance, scalability, granularity, generalization, flexibility, and hierarchical control. Results show benefits from learned fine-grained weighting, while very large LoRA collections remain challenging.
- Gating balance: Adding Lbalance produces more uniform gating weights and improves performance over MOLE without Lbalance.Increasing temperature alone addresses imbalance but restricts dynamic LoRA exploration and produces inferior outcomes.
- Scalability: MOLE surpasses LoRAHub by 2.5 and 3.0 when composing 48 and 128 LoRAs, respectively.All methods, including MOLE, decline with an extremely large number of LoRAs, while MOLE remains strongest across the tested V&L counts.
- Gating granularity: Intermediate gating granularities, b-MOLE and l-MOLE, achieve the highest performance among the evaluated matrix-, layer-, block-, and network-wise variants.The analysis attributes improved results to finer-grained weight combination methods.
- Generalization: When trained on NLI tasks and evaluated on BBH, MOLE improves average performance by 2.4 over LoRAHub and 3.7 over PEMs.These results are reported as evidence of stronger generalization to new datasets.
- Flexibility: MOLE preserves individual LoRA features and can mask unwanted experts without changing its weights, enabling flexible composition of fewer LoRAs.Reference tuning-based composition cannot provide the same flexibility according to the comparison.
- Hierarchical control: MOLE’s gating functions adaptively assign different weights to LoRA experts across layers, and finer-grained combinations yield superior results.The pattern is observed in both NLP and V&L analyses.
6 CONCLUSION AND LIMITATIONS
The paper presents MOLE as a dynamic, versatile method for composing multiple trained LoRAs. Evaluations in NLP and V&L report that MOLE outperforms existing LoRA composition methods, although very large-scale composition remains challenging.
- Conclusion: MOLE uses learnable gating functions to determine composition weights from multiple LoRA outputs at each layer.This provides a dynamic approach to composing multiple trained LoRAs.
- Conclusion: MOLE outperforms existing LoRA composition methods across evaluations in both NLP and V&L domains.The conclusion presents this as the paper’s comprehensive evaluation finding.
- Limitations: At 128 LoRAs, performance declines for all composition methods, indicating that MOLE still faces challenges with large-scale LoRA composition.The paper identifies better handling of large-scale composition as an area for further research.