PaperlayerView exact revision on arXiv

Source-linked AI summary

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, Kang Liu

arXiv:2402.12851v1cs.CL

TL;DR

Fine-tuning LLMs is computationally costly, and PEFT methods with limited global parameters can struggle to combine modules flexibly for downstream tasks. MoELoRA treats LoRA modules as gated experts and uses contrastive learning to encourage distinct expert features. It reports higher average performance than LoRA across math and common-sense reasoning tasks, with competitive results against GPT-3.5 on some benchmarks.

  • Problem

    Fine-tuning LLMs with billions of parameters is costly, while PEFT methods using limited global parameters face challenges flexibly combining computational modules for downstream tasks.

    Source: §1 · p. 1

  • Method

    MoELoRA treats LoRA modules as mixture-of-experts components selected by a gating network and uses contrastive learning to encourage experts to learn distinct features.

    Source: §1 · pp. 1–2

  • Results

    With the same number of parameters, MoELoRA averaged 4.2% higher performance than LoRA in math reasoning and 1.0% higher in common-sense reasoning, outperforming LoRA across all tasks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Across 11 math- and common-sense-reasoning datasets, MoELoRA outperformed LoRA, and it showed competitive performance with GPT-3.5.

    Source: §6 · p. 8

    Common-sense-task gains are limited because performance depends substantially on knowledge accumulated during pretraining, which PEFT cannot supply when relevant knowledge is absent.

    Source: §5.1 · p. 7, §6 · p. 8

Abstract

from arXiv · show

Fine-tuning is often necessary to enhance the adaptability of Large Language Models (LLM) to downstream tasks. Nonetheless, the process of updating billions of parameters demands significant computational resources and training time, which poses a substantial obstacle to the widespread application of large-scale models in various scenarios. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prominent paradigm in recent research. However, current PEFT approaches that employ a limited set of global parameters (such as LoRA, which adds low-rank approximation matrices to all weights) face challenges in flexibly combining different computational modules in downstream tasks. In this work, we introduce a novel PEFT method: MoELoRA. We consider LoRA as Mixture of Experts (MoE), and to mitigate the random routing phenomenon observed in MoE, we propose the utilization of contrastive learning to encourage experts to learn distinct features. We conducted experiments on 11 tasks in math reasoning and common-sense reasoning benchmarks. With the same number of parameters, our approach outperforms LoRA significantly. In math reasoning, MoELoRA achieved an average performance that was 4.2% higher than LoRA, and demonstrated competitive performance compared to the 175B GPT-3.5 on several benchmarks.

1. Introduction

Fine-tuning large language models is computationally costly, while PEFT methods using global parameters can struggle to combine modules flexibly for downstream tasks. MoELoRA treats LoRA modules as gated experts and uses contrastive learning to encourage distinct features, reporting gains over LoRA across reasoning tasks.

1. Introduction

  • Fine-tuning LLMs with billions of parameters demands substantial computational resources, motivating PEFT methods that freeze weights or add smaller trainable modules.The introduction also identifies limited global parameters as a challenge for flexibly combining downstream-task modules.Source: §1 · p. 1

6. Conclusions and Future Work

  • 4.2% higher average performance in math reasoning and 1.0% higher in common-sense reasoning than LoRA; MoELoRA outperformed LoRA across all tasks and was competitive with GPT-3.5.The reported comparisons use the same number of parameters; experiments covered 11 math- and common-sense-reasoning datasets.Source: §1 · p. 2

1. Introduction

  • Contrastive learning treats outputs from the same expert as positive samples and outputs from different experts as negative samples, encouraging experts to learn distinct features.The approach addresses random routing in MoE applied to LoRA.Source: §1 · p. 2
  • MoELoRA treats LoRA as a mixture of experts, dynamically combining modules so the method can better cater to downstream-task requirements.A gating network activates only selected LoRA experts during training and inference; Figure 1 contrasts fine-tuning, LoRA, and MoELoRA architectures.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Experiments cover 11 math- and common-sense-reasoning datasets, with ablations reporting downstream-task improvement from contrastive learning and routing analysis examining model decision-making.The introduction reports that MoELoRA outperforms LoRA in all tasks.Source: §1 · p. 2

2. Related Work

Related work covers parameter-efficient adaptation, mixture-of-experts routing, and contrastive learning. MoELoRA brings MoE into PEFT and applies contrastive learning to expert outputs.

2.1. Parameter-Efficient Fine-Tuning

  • LoRA uses two low-rank matrices, while single LoRA cannot flexibly combine downstream-task modules; MoELoRA sets up multiple LoRAs as experts and combines them dynamically.PEFT adjusts a limited number of parameters or adds trainable parameters instead of fine-tuning the entire backbone.Source: §2.1 · p. 2

2.2. Mixture-of-Experts

2.3. Contrastive Learning

  • Contrastive learning aligns related representations in vision and NLP; MoELoRA applies it to MoE outputs to maximize discrepancy among experts’ output distributions and capture diverse downstream features.The passage frames this as a way to mitigate the random-routing phenomenon described by Zuo et al. (2021).Source: §2.3 · p. 3

3. The Proposed Method

MoELoRA treats LoRA modules as sparse, gated experts and combines their outputs. Its method addresses load imbalance and random routing with load-balancing and experts-contrastive losses.

3.1. Framework of MoELoRA

  • MoELoRA treats different LoRA modules as experts, gates inputs across branches, and combines their weighted outputs with top-k routing to keep computation sparse.Each LoRA branch projects inputs down to a low-rank dimension and back up before the branches are combined.Source: §3.1 · pp. 3–4

3.2.1. Load Imbalance

  • Top-k routing can send many tokens to a few experts and few or none to others, which can lead to poor performance.Prior work used load-balancing loss to encourage more balanced routing.Source: §3.2.1 · p. 4

3.2.2. Random Routing

  • When gating appears random, experts receive randomly routed tokens and learn content that does not differ significantly, contrary to MoE's goal of dividing problems among experts.Source: §3.2.2 · p. 4

3.3.1. Load Balancing Loss

  • Load-balancing loss takes the dot product of nondifferentiable token-assignment proportions f and differentiable average gate probabilities P, balancing assignments and gating outputs.The loss reaches its minimum when the gating network's average probability distribution is uniform across experts.Source: §3.3.1 · p. 4

3.3.2. Experts Contrastive Loss

  • Experts Contrastive Loss treats outputs from the same expert as positives and outputs from different experts as negatives, encouraging experts to learn distinct features.Each expert updates a queue with tokens selected for it, and the contrastive loss uses samples from those queues; temperature τ controls the distribution shape of q · k.Source: §3.3.2 · pp. 4–5, Figure 2 · p. 5

4. Experiments

Experiments compare MoELoRA with LoRA and adapter baselines across six math and five common-sense benchmarks, then ablate its contrastive loss and top-k expert selection. With equal trainable parameters, MoELoRA achieves higher average performance than LoRA in both task groups.

4.1.1. Dataset

  • The evaluation covers six math-reasoning benchmarks and five common-sense benchmarks, using datasets and rationales drawn from prior work.The math rationales were generated with zero-shot chain-of-thought on GPT-3.5 without error filtering.Source: §4.1.1 · p. 5

4.1.2. Implementation Details

  • With LLaMA-7B, the experiments matched trainable parameter counts using LoRA rank 36 and eight rank-4 MoELoRA experts, with both methods added to q_proj and p_proj.MoELoRA’s total LoRA rank was 32.Source: §4.1.2 · pp. 5–6

4.2. Main Results

  • With the same number of trainable parameters, MoELoRA improves average accuracy over LoRA by 4.2 points across six math-reasoning benchmarks, outperforming LoRA on each task.Source: Table 1 · p. 6, §4.2 · p. 6

4.3.1. Ablations on Auxiliary Loss

  • Removing Experts Contrastive Loss lowers average performance by 3.0 in math and 0.9 in common-sense tasks, with R=32, eight experts, and top-2 routing fixed.The authors interpret these ablations as evidence of improvement from the loss.Source: Table 3 · p. 6, Table 4 · p. 6, §4.3.1 · pp. 6–7

4.3.2. Ablations on Selecting Top-k per Token

  • With R=32 and eight experts fixed, top-2 expert selection significantly improves performance over top-1 and top-4 selection in the reported math and common-sense experiments.Tables 5 and 6 present the respective task results.Source: Table 5 · p. 7, Table 6 · p. 7, §4.3.2 · p. 7

5. Analysis

The analysis attributes common-sense performance limits partly to the model’s pretrained knowledge reservoir. Routing analysis finds numerical-token specialization in early layers, later becoming more uniform, while token-frequency variation leaves expert loads unbalanced.

5.1. Why The Improvement In Common-sense Tasks Is So Inconspicuous?

  • Common-sense PEFT cannot address missing relevant knowledge in the model, and performance depends more on its pretraining knowledge reservoir.The paper discusses this boundary for common-sense tasks including ARC, BoolQ, OBQA, and PIQA.Source: §5.1 · p. 7

5.2. Tracing tokens through Experts

  • Specific experts often handle numerical tokens; tokens such as ’2’ and ’4’ route to specific experts early, then more uniformly as attention accumulates information.Figures 3–5 depict routing for numeric tokens and the examples ’2’ and ’4’.Source: §5.2 · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 8
  • Expert load is not balanced, which the authors relate to token-frequency variation; they still consider load-balancing loss necessary to prevent experts from receiving no tokens.The discussion points to Table 7 for token occurrence frequencies.Source: §5.2 · p. 8

6. Conclusions and Future Work

MoELoRA combines LoRA experts with contrastive learning and outperforms LoRA across the reported reasoning tasks, though improvements on common-sense tasks are limited. Future work proposes reframing those tasks as knowledge editing or training only a gate over frozen task-specific LoRA modules.

6. Conclusions and Future Work

  • MoELoRA uses contrastive learning to mitigate random routing in its LoRA-based mixture-of-experts approach.The experiments cover 11 math reasoning and common-sense reasoning datasets.Source: §6 · p. 8
  • 4.2% higher performance in math reasoning and 1.0% higher in common-sense reasoning than LoRA; MoELoRA also shows competitive performance against GPT-3.5.The authors report consistent outperformance of LoRA across tasks.Source: §6 · p. 8
  • Common-sense task improvements are limited; proposed directions include reframing these tasks as knowledge editing and training only the gate over frozen task-specific LoRA modules.The proposed experts would use LoRA modules trained on different tasks.Source: §6 · p. 8

A. Case Study on Common-sense Tasks

The case study investigates why improvements on common-sense tasks are relatively small. It covers four benchmark formats, each of which essentially requires relevant knowledge from the LLM.

A. Case Study on Common-sense Tasks

  • The case study investigates why improvements on common-sense tasks are relatively small.It examines several benchmarks for this task family.Source: A. Case Study on Common-sense Tasks · p. 11
  • Table 8 presents four different formats of benchmarks for common-sense tasks.Source: Table 8 · p. 12
  • Each of the four benchmark formats essentially requires the LLM to possess relevant knowledge.Source: Table 8 · p. 12
Loading 2402.12851v1…