PaperlayerView exact revision on arXiv

Source-linked AI summary

Higher Layers Need More LoRA Experts

Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, VS Subrahmanian

arXiv:2402.08562v1cs.CLcs.AI

TL;DR

PEFT lowers fine-tuning costs, but performance remains limited, and prior LoRA-MoE methods generally use a fixed number of experts per layer despite potential redundancy. MoLA allows expert counts to vary across Transformer layers; on six benchmarks, it outperforms other baselines, with higher-layer-heavy allocation performing best among its configurations.

  • Problem

    PEFT reduces fine-tuning costs but its performance remains limited, while LoRA-MoE methods generally use fixed expert counts across layers despite potential redundancy.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    MoLA combines LoRA and MoE with flexible layer-wise expert allocation and evaluates different configurations on Transformer models.

    Source: §1 · p. 2

  • Results

    On six benchmarks, MoLA significantly outperforms other baselines, and MoLA-▽ achieves the best performance among the configurations.

    Source: §7 · p. 8

  • Takeaways & Limitations

    With a fixed total number of experts, allocating more LoRA experts to higher Transformer layers enhances effectiveness, and MoLA is presented as a plug-and-play approach for diverse tasks.

    Source: §1 · p. 2

    The allocation study compares four configurations based on different assumptions, including one with fewer experts in lower layers and more in higher layers.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite promising results, research on improving the efficiency of LoRA with MoE is still in its early stages. Recent studies have shown that experts in the MoE architecture have different strengths and also exhibit some redundancy. Does this statement also apply to parameter-efficient MoE? In this paper, we introduce a novel parameter-efficient MoE method, \textit{\textbf{M}oE-L\textbf{o}RA with \textbf{L}ayer-wise Expert \textbf{A}llocation (MoLA)} for Transformer-based models, where each model layer has the flexibility to employ a varying number of LoRA experts. We investigate several architectures with varying layer-wise expert configurations. Experiments on six well-known NLP and commonsense QA benchmarks demonstrate that MoLA achieves equal or superior performance compared to all baselines. We find that allocating more LoRA experts to higher layers further enhances the effectiveness of models with a certain number of experts in total. With much fewer parameters, this allocation strategy outperforms the setting with the same number of experts in every layer. This work can be widely used as a plug-and-play parameter-efficient tuning approach for various applications. The code is available at https://github.com/GCYZSL/MoLA.

1. Introduction

PEFT reduces fine-tuning costs, but its performance remains limited, and existing LoRA-MoE methods motivate questions about redundant experts and layer-wise allocation. MoLA studies flexible expert allocation and finds that higher-layer-heavy configurations perform best while lower-layer experts are more redundant.

1. Introduction

  • PEFT reduces the computational demands of fine-tuning large models, but its performance remains limited; LoRA-MoE research is still in its early stages.Source: §1 · p. 1
  • The paper asks whether parameter-efficient MoE experts are redundant and how LoRA experts should be distributed across Transformer layers.Source: §1 · pp. 1–2
  • MoLA combines LoRA and MoE with flexible layer-wise expert counts, and the study evaluates configurations on six NLP and commonsense question-answering benchmarks.Source: §1 · p. 2
  • With a fixed total number of experts, allocating more LoRA experts to higher layers outperforms other configurations.Source: §1 · p. 2
  • Across six benchmarks, all MoLA configurations significantly outperform other PEFT baselines.Source: §1 · p. 2
  • Experts in lower layers are more similar to each other and therefore exhibit greater redundancy.Source: §1 · p. 2

2. Preliminaries

MoE routes inputs to selected experts, while LoRA adapts frozen pretrained weights using trainable low-rank matrices. The overview depicts MoLA applying LoRA-MoE with different expert counts across layers.

2. Preliminaries

  • MoE uses a router to select the top K experts for an input, and a load-balancing loss encourages equitable workload across experts.Source: §2 · p. 2
  • LoRA adapts a pretrained linear layer with trainable low-rank matrices A and B while keeping its original weight matrix frozen.Source: §2 · p. 2
  • In the MoLA overview, each Transformer layer has a different number of LoRA experts, and training updates the experts while pretrained weights remain frozen.Source: Figure 1 · p. 3

3. MoE-LoRA with Layer-wise Allocation

MoLA replaces fixed per-layer LoRA expert counts with flexible layer-wise allocation in Transformer models. This addresses concern that expert redundancy may persist when every layer uses the same count.

3. MoE-LoRA with Layer-wise Allocation

  • MoLA lets users assign different numbers of LoRA experts to different Transformer layers, addressing the fixed-count design of prior LoRA-MoE methods.The method is presented as a parameter-efficient tuning approach for Transformer models.Source: §3 · p. 3

3.1. The MoLA Architecture

  • Each layer’s experts are LoRA adapter pairs, and a learned router selects the top K experts for each input token.MoLA preserves the MoE load-balancing loss at each layer.Source: §3.1 · p. 3

3.2. Configurations of Layer-wise Expert Allocation

  • Four allocations encode competing assumptions: triangle emphasizes lower layers, inverted triangle higher layers, hourglass both ends, and rectangle equal counts.The rectangle configuration serves as the baseline.Source: §3.2 · p. 4, Figure 2 · p. 4

4. Experiments

Experiments evaluate MoLA across six NLP and commonsense QA benchmarks using direct and instruction-tuning settings. MoLA variants outperform LoRA, and allocating more experts to higher layers is reported to improve performance.

4.1. Experiment Settings

  • The experiments compare direct downstream fine-tuning with instruction tuning followed by downstream fine-tuning.Allocation variants use equal total expert counts; the LLaMA-2 model has 32 layers, and the reported trainable parameter count is 105,635,840.Source: §4.1 · p. 4

4.2. Task and Data

  • Evaluation covers three NLP datasets and three commonsense QA benchmarks, alongside a 50,000-sample OpenOrca instruction-tuning subset.The NLP datasets are MRPC, RTE, and COLA; the QA benchmarks are ScienceQA, CommonsenseQA, and OpenbookQA.Source: §4.2 · p. 5

4.3. Recent Competitive Baselines

  • Baselines include Prompt Tuning, LLaMA-Adapter, and rank-64 LoRA, with full-parameter fine-tuning also evaluated.COLA, RTE, and CommonsenseQA are evaluated on validation sets; the other tasks use test sets.Source: §4.3 · p. 5

4.4. Implementation

  • Experiments use LLaMA2-7B, rank-8 LoRA experts, a top-2 router, and AdamW with a 3e-4 learning rate.The implementation uses eight A100-40G GPUs.Source: §4.4 · p. 5

4.5. Results

  • Across six benchmarks, MoLA variants outperform LoRA; MoLA-▽ beats LoRA on all six and exceeds MoLA-□(8888) on three with nearly 40% fewer parameters.Table 1 covers direct downstream fine-tuning.Source: Table 1 · p. 6, §4.5 · p. 6
  • With instruction tuning, MoLA variants significantly outperform LoRA on all datasets, and MoLA-▽ is equal or better than MoLA-□(8888) throughout despite fewer parameters.This setting evaluates transfer learning capability.Source: Table 2 · p. 6, §4.5 · p. 6
  • MoLA-▽ generally performs best, beating other variants on five benchmarks; its superiority over baselines was statistically significant (FDR-corrected Wilcoxon p-values below 0.05).The significance test compares MoLA-▽ with Prompt Tuning, LLaMA-Adapter, and LoRA across six benchmarks.Source: §4.5 · p. 6
  • Higher-layer-heavy allocation outperforms other strategies; the authors argue that pruning should reduce experts in lower layers.The experiments compare layer-wise allocations, and the authors identify top-layer expert counts as important.Source: §4.5 · p. 6

5. Model Analysis and Ablation Studies

MoLA variants outperform LoRA in continuous learning, while layer-wise analysis finds greater expert redundancy in lower Transformer layers. Both findings support allocating more experts to higher layers when the total expert count is fixed.

5.1. Analysis of Layer-wise Experts Redundancy

  • Average Frobenius distances between expert weight matrices rise from lower to higher layers, indicating greater similarity and redundancy among lower-layer experts.The pattern also appears in other MoLA configurations, and the authors conclude that allocating more experts to higher layers is more effective with a fixed total.Source: §5.1 · p. 7, Figure 3 · p. 7
  • Experts in lower layers remain more redundant even though experts across layers receive similar router weights and selection workloads.The authors interpret the similar utilization alongside greater lower-layer similarity as further support for the redundancy finding.Source: §5.1 · p. 7

5.2. Continuous Learning

  • The continuous-learning study sequentially fine-tunes MoLAs on five ScienceQA domains and examines performance on previously trained domains.The selected topics are biology, physics, chemistry, economics, and earth science; the study measures overall performance and performance drop.Source: §5.2 · p. 7
  • MoLAs achieve better overall performance than LoRA in continuous learning; MoLA-▽ records a -0.47 performance drop score.The authors associate this result with avoiding domain knowledge forgetting.Source: §5.2 · p. 8, Table 3 · p. 8

6. Related Work

Related work covers parameter-efficient tuning and methods combining it with MoE. The paper distinguishes MoLA through layer-wise expert allocation across Transformer dense weight matrices.

6.1. Parameter-Efficient Tuning

  • LoRA uses low-rank decomposition matrices to adapt Transformer dense weights without additional inference costs, while prompting methods add embedding tokens to sequences.The paper notes that extra tokens may compromise inference efficiency, especially for long contexts.Source: §6.1 · p. 8

6.2. Parameter-Efficient MoE

  • Earlier MoE–PEFT methods do not consider layer-wise expert allocation, which MoLA introduces as a varying expert count across Transformer layers.The paper also distinguishes MoLA from approaches limited to feed-forward layers or requiring task-type prior knowledge for routing.Source: §6.2 · p. 8
  • Prior work combines MoE with LoRA for medical-domain language-model fine-tuning; Dou et al. (2023) introduced LoRAMoE in Transformer feed-forward layers, while Zadouri et al. (2023) proposed MoLORA and MoV.Source: §6.2 · p. 8

7. Conclusion

The conclusion presents MoLA as a parameter-efficient combination of LoRA and MoE with layer-wise expert allocation. It reports the strongest performance for the configuration assigning more experts to higher layers and identifies dynamic allocation and broader task coverage as future work.

7. Conclusion

  • MoLA-▽ achieves the best performance among the four configurations, supporting allocation of more experts to higher layers when the total is fixed.The conclusion reports that experiments across six NLP and commonsense question-answering benchmarks show MoLA significantly outperforming other baselines.Source: §7 · p. 8
  • The analysis finds greater expert redundancy in lower Transformer layers, while the paper presents MoLA as a plug-and-play PEFT approach for diverse tasks.The paper also describes its findings as a research direction for enhancing MoE and PEFT.Source: §7 · p. 8
  • Future work will explore dynamic layer-wise expert allocation and apply MoLA to more diverse tasks.Source: §7 · p. 8

Broader Impact

The authors describe MoLA as a plug-and-play PEFT method intended for diverse tasks and claim that it reduces fine-tuning costs. They also flag possible biases in the pretrained LLMs on which the research relies.

Broader Impact

  • The authors report that their approach provides LLM fine-tuning performance while dramatically reducing training costs.Source: Broader Impact · p. 9
  • Because the research relies on pretrained LLMs, the authors caution that those models may exhibit preferential biases.They advise users to consider the possible implications of those biases.Source: Broader Impact · p. 9
  • The authors suggest that efficient training and performance could support energy saving, AI decarbonization, and wider practical use.Source: Broader Impact · p. 9

A. Continuous Learning

The section evaluates LoRA and MoLAs in a domain-continuous-learning setting, fine-tuning across five domains in sequence and reporting results in Table 4.

A. Continuous Learning

  • LoRA and MoLAs are evaluated by sequential fine-tuning across biology, physics, chemistry, economics, and earth science.Training proceeds through the listed domains in that order.Source: A. Continuous Learning · p. 11
  • Table 4 reports results for the continuous-learning setting.Source: Table 4 · p. 11
  • Bio-Phy denotes testing on physics after training on biology.Source: Table 4 · p. 11

B. Frobenius Norms of Different Experts Allocation

Across the examined MoLA configurations, differences between experts’ weight matrices become larger at higher layers.

B. Frobenius Norms of Different Experts Allocation

C. Analysis of Average Fusion Weights and Selected Times of Experts

The analysis reports average router fusion weights and expert-selection counts, finding similar fusion weights for selected experts and frequent use of most experts.

C. Analysis of Average Fusion Weights and Selected Times of Experts

Related papers

Loading 2402.08562v1…