PaperlayerView exact revision on arXiv

Source-linked AI summary

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

Ahin Lee, Sehyun Yun, Joonha Park, Taesik Gong

arXiv:2609.06072v1cs.LGcs.AIcs.CL

TL;DR

MoE expert-wise LoRA fragments adaptation, while full fine-tuning is costly. ACE consolidates functionally similar experts into shared higher-rank adapters with grouped execution, improving adaptation quality and efficiency across evaluations while accelerating training.

  • Problem

    Expert-wise LoRA fragments capacity, gradient supervision, and execution in MoE models, while full fine-tuning remains substantially more expensive than in comparable dense models.

    Source: §1 · pp. 1–2

  • Method

    ACE groups experts by adapter similarity on task-relevant inputs and replaces their adapters with shared higher-rank LoRA modules within the same trainable-parameter budget, using grouped execution.

    Source: §1 · pp. 1–2

  • Results

    Across 12 datasets and four MoE backbones, ACE improves adaptation quality and training efficiency; it delivers 1.31×–1.48× wall-clock training speedups without increasing peak memory.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ACE achieves a stronger quality-efficiency trade-off than parameter-matched PEFT baselines across multiple MoE backbones and benchmarks.

    Source: §6 · p. 9

    Grouped adapter execution applies to up and gate projections, not down projections, and excessive consolidation can erase useful distinctions when tasks demand strong expert specialization.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation in three ways: capacity is split across narrow low-rank updates, gradient supervision becomes sparse and imbalanced under sparse routing, and execution is decomposed into many small GEMMs. We find that such expert-wise separation is often unnecessary, as subsets of LoRA adapters become functionally similar during fine-tuning, revealing redundancy among expert-specific adapters. Based on this redundancy, we propose ACE (Adapter Consolidation across Experts), which groups redundant experts and replaces their expert-specific adapters with group-shared higher-rank LoRA modules under the same PEFT budget. ACE further introduces grouped adapter execution, which consolidates fragmented expert-wise adapter computations into fewer, larger group-level GEMMs. Across evaluations covering 12 datasets and four MoE backbones, ACE achieves the highest observed mean accuracy among the parameter-matched PEFT methods on the three backbones with complete baseline coverage, while providing $1.31\times$ to $1.48\times$ wall-clock training speedup over expert-wise LoRA without increasing peak memory. Our code is available at https://github.com/UbiquitousAILab/ACE.

1 Introduction

ACE addresses fragmentation in expert-wise LoRA for MoE models by consolidating functionally similar expert adapters into group-shared, higher-rank modules. Across 12 datasets and four MoE backbones, it improves adaptation quality and training efficiency under matched parameter budgets.

1 Introduction

  • Expert-wise LoRA fragments a fixed parameter budget across low-rank adapters, while sparse routing makes their gradients sparse and imbalanced and their computations many small GEMMs.These constraints affect adaptation capacity, supervision, and execution, respectively.Source: §1 · p. 1
  • ACE groups experts with functionally similar adapters and replaces their separate modules with group-shared higher-rank adapters under the same trainable parameter budget.The overview describes similarity-based grouping and rank consolidation as the core reallocation.Source: §1 · p. 1, Figure 1 · p. 2
  • Each shared adapter can receive gradients whenever any expert in its group is active, while grouped execution consolidates fragmented computations into fewer, larger operations.This combines denser group-level updates with reduced small-GEMM overhead.Source: §1 · p. 2
  • On OLMoE, average accuracy across eight commonsense benchmarks rises from 74.11% to 75.61% (+1.50 points) over expert-wise LoRA, with 1.31×–1.48× wall-clock training speedups and no peak-memory increase.The evaluations cover 12 datasets and four MoE backbones.Source: §1 · p. 2

2 Related Work

Prior PEFT work develops efficient updates and budget-allocation strategies, largely for dense architectures. MoE fine-tuning work instead explores selective or routed adaptation, while ACE consolidates similar adapters into shared group-level modules.

2 Related Work

  • PEFT adapts pretrained models by training a small set of additional parameters while keeping the backbone frozen.LoRA is described as a standard PEFT approach.Source: §2 · p. 2
  • LoRA parameterizes weight updates efficiently without changing the backbone, and later methods improve fixed-budget use through rank allocation or alternative low-rank parameterizations.These approaches are designed for dense architectures, where each adapted module receives gradients from all inputs.Source: §2 · p. 2
  • In MoE fine-tuning, ESFT tunes task-relevant experts, PERFT-R routes LoRA adapters, and EPnG reallocates LoRA capacity by routing importance.ACE instead keeps pretrained experts and the router intact while consolidating functionally similar adapters into shared group-level modules.Source: §2 · p. 2

3 Methodology

Expert-wise LoRA fragments capacity, supervision, and execution under sparse MoE routing. ACE groups functionally similar expert adapters and consolidates them into shared higher-rank modules, with shared updates and grouped execution.

3.1 Problem: Fragmentation in MoE LoRA

  • Expert-wise LoRA gives each MoE expert an independent low-rank update, fragmenting adaptation under sparse routing.Source: §3.1 · p. 3

3.1.1 Capacity Fragmentation

  • With a fixed layer-level budget, increasing the number of experts linearly reduces the rank available to each adapter.The budget is divided among E narrow updates rather than stronger task-specific corrections.Source: §3.1.1 · p. 3

3.1.2 Gradient Fragmentation

  • Sparse routing limits each expert adapter’s updates to routed tokens, and imbalanced routing can leave some adapters with even fewer updates.With balanced routing, each adapter receives updates from only a k/E fraction of the batch on average.Source: §3.1.2 · p. 3

3.1.3 Execution Fragmentation

  • Expert-wise execution invokes many small adapter-side GEMMs across routed experts, incurring repeated launch overhead and poor GPU utilization.The resulting runtime overhead can be non-negligible even with a small LoRA budget replicated across many experts.Source: §3.1.3 · p. 3

3.2 Motivation: Functional Grouping of Expert LoRA Adapters

  • Adapters within a MoE layer often form functionally similar groups, while other adapters remain distinct.Figure 2’s high-similarity blocks motivate consolidating similar adapters to pool capacity, densify supervision, and reduce execution fragmentation.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3 Rank Consolidation

  • After warm-up, ACE compares adapters on shared probe inputs with CKA, forms Louvain groups, and replaces expert-specific modules with shared higher-rank adapters at unchanged budget.Source: §3.3 · p. 4

3.4 Shared Adapter Updates

  • In ACE, a group-shared adapter receives an update whenever any expert in its group is routed, providing denser and more balanced supervision.Source: §3.4 · p. 4

3.5 Grouped Adapter Execution

  • For shared-input projections, ACE computes each active group’s adapter output once and reuses it for routed experts, replacing repeated calls with fewer, larger GEMMs.Source: §3.5 · p. 5

4 Experiments

Across evaluated MoE backbones, ACE achieves the best average performance under matched trainable-parameter budgets, and it also leads on most specialized-domain benchmarks. The experiments compare PEFT methods across reasoning and generation tasks, with computational efficiency evaluated against expert-wise LoRA.

4 Experiments

  • The experiments cover commonsense and domain-specific tasks across MoE models, comparing PEFT methods with comparable trainable-parameter budgets.The commonsense setting uses three backbones and eight benchmarks; methods adapt the same expert projections.Source: §4 · p. 5, §4.1 · p. 5

4.1 Commonsense Reasoning

  • ACE achieves the best average performance across evaluated MoE backbones under matched trainable-parameter budgets.Source: §4.1 · p. 5

4.2 Mathematical Reasoning, Code Generation, and Legal Understanding

  • ACE achieves the best average on both evaluated backbones and leads on most individual math, code-generation, and legal-understanding benchmarks.Source: §4.2 · p. 5

4.3 Computational and Memory Efficiency

  • The computational-efficiency evaluation compares ACE with expert-wise LoRA.Source: §4.3 · p. 5

OLMoE

ACE reduces training time relative to expert-wise LoRA without increasing peak memory. Grouped adapter execution consolidates adapter computations into larger group-level operations.

OLMoE

  • Table 1 reports per-benchmark accuracy and three-run mean accuracy with standard deviation across MoE backbones.Its # Params column gives the percentage of trainable adapter parameters.Source: Table 1 · p. 6
  • 1.31× on OLMoE is the lowest reported ACE wall-clock training speedup over expert-wise LoRA, with peak memory unchanged.Across the evaluated backbones, speedups range from 1.31× to 1.48×.Source: OLMoE · p. 6, Table 2 · p. 6
  • ACE evaluates shared adapters at the group level, reducing small-GEMM overhead while preserving the trainable parameter budget.Instead of computing separate adapters for each routed expert, ACE consolidates operations into fewer, larger group-level GEMMs.Source: OLMoE · p. 6

5 Further Analysis

ACE's analysis finds that consolidation recovers effective adapter capacity, densifies gradient supervision, and reduces execution fragmentation. CKA-based grouping reflects functional interchangeability, while ablations distinguish the roles of grouping, rank consolidation, and grouped execution.

5.1 Does ACE Mitigate MoE LoRA Fragmentations?

  • At the LoRA-r=2 budget, ACE's effective-rank distribution approaches LoRA-r=64, whose mean effective rank is 46.8 but uses roughly 32× more trainable parameters.LoRA-r=2 has a mean effective rank of 1.77.Source: §5.1 · p. 6, Figure 3 · p. 7
  • ACE achieves the best average on both specialized-domain backbones and leads on most individual benchmarks.Table 3 covers mathematical reasoning, code generation, and legal understanding.Source: Table 3 · p. 7
  • On OLMoE up_proj, ACE lowers gradient-norm CV from 1.449 to 0.531 and the zero-gradient ratio from 3.18% to 1.17%.Both gradient-fragmentation metrics fall across Qwen, OLMoE, and Moonlight.Source: §5.1 · p. 7, Figure 4 · p. 7
  • ACE reduces profiler-level matrix-multiplication calls, CUDA kernel events, and CUDA execution time, with a smaller reduction in wall-clock prefill time.The profiled region also includes non-adapter computation, CPU-side overhead, and framework overhead.Source: §5.1 · p. 7, Table 4 · p. 7

5.2 CKA Similarity Reflects Functional Redundancy

  • High-CKA adapter swaps cause substantially less NLL degradation than low-CKA or random swaps, supporting CKA as a measure of functional interchangeability.Positive CKA gaps also occur across four backbones on commonsense reasoning (0.298–0.438) and across several domains on OLMoE and Qwen.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 Ablation Study

  • Full Merge performs worse than LoRA, while disabling grouped execution leaves accuracy unchanged but increases runtime from 1.89 to 2.83 hours.The ablation attributes adapter selection to similarity grouping, effective-capacity gains to rank consolidation, and runtime gains to grouped execution.Source: §5.3 · p. 8, Table 5 · p. 8

5.4 Expert Adaptation Is a Primary Target in MoE PEFT

  • MoE-only ACE outperforms attention-only adaptation while using fewer trainable parameters on both OLMoE and Qwen.ACE also improves over the LoRA counterpart when attention and MoE adapters are trained together.Source: §5.4 · pp. 8–9

6 Conclusion

ACE consolidates functionally aligned expert adapters into group-shared higher-rank modules with grouped execution. The paper reports a stronger quality-efficiency trade-off than parameter-matched PEFT baselines across multiple MoE backbones and benchmarks.

6 Conclusion

  • ACE addresses capacity, gradient-supervision, and execution fragmentation in expert-wise LoRA for MoE models.It consolidates functionally aligned adapters into group-shared higher-rank modules with grouped execution.Source: §6 · p. 9
  • ACE achieves a stronger quality-efficiency trade-off than parameter-matched PEFT baselines across multiple MoE backbones and benchmarks.The conclusion reports this comparison for ACE's consolidated adapters and grouped execution.Source: §6 · p. 9
  • The results suggest that sharing adaptation across experts can be beneficial compared with fully separate expert-wise adapters in MoE fine-tuning.Source: §6 · p. 9

Limitations

ACE's scope is constrained by grouped execution's input-sharing requirement and by the risk that excessive consolidation removes useful expert specialization. The appendices document the experiment configurations, comparisons, data, and licensing context.

Limitations

  • Grouped adapter execution applies to up and gate projections, but not the down projection, whose expert-specific input prevents the required shared-input execution.When tasks demand strong expert specialization, excessive consolidation can erase useful distinctions; grouping granularity may need tuning.Source: Limitations · p. 9

A Model Configurations / B Computing Infrastructure

  • Experiments use OLMoE, Qwen, Moonlight, and Kimi-Linear backbones; compute measurements use an Ubuntu 22.04.5 workstation with CUDA 12.8 and, unless specified, NVIDIA H200 GPUs.Peak GPU memory is measured with torch.cuda.max_memory_allocated(), and wall-clock time is end-to-end elapsed training time.Source: A Model Configurations · p. 12, B Computing Infrastructure · p. 12

B.1.1 Hyperparameters

  • Unless specified otherwise, the shared setup uses AdamW, weight decay 0.1, a constant learning-rate schedule, sequence length 256, batch size 64, rank 2, and three epochs.ACE uses Tinit=150 and Louvain resolution γ=0.8, and applies consolidation to up and gate projections.Source: §B.1.1 · pp. 12–13

B.1.2 Baselines / B.1.3 Notes on MoE-Specific Baselines

  • The baseline set includes LoRA variants and MoE-aware methods, while PERFT-R results use a unified setup that may differ from its original paper.ESFT is discussed as related work rather than included in the main quantitative comparison because its trainable parameter count is difficult to match to adapter-based methods.Source: §B.1.2 · p. 13, §B.1.3 · p. 13

B.1.4 Seed-wise Consistency

  • ACE has higher average accuracy than the strongest fully evaluated non-ACE baseline in all nine matched backbone-seed comparisons.The comparison covers three random seeds for each of three backbones.Source: §B.1.4 · pp. 13–14, Table 8 · p. 14

B.2 Configuration for Table 3 / B.3 Dataset Statistics

  • The mathematical-reasoning, code-generation, and legal evaluations use rank 8; the evaluation spans 30,403 examples across 12 benchmarks.The listed training datasets comprise 764,455 source examples.Source: §B.2 · p. 14, §B.3 · p. 14

B.4 Configuration for Table 5

  • The Table 5 ablation matches trainable-parameter budgets for LoRA, Random, Full Merge, w/o Grp. Exec., and ACE, but reports w/o Rank Cons. separately.Source: §B.4 · p. 14

B.5 Licenses and Terms of Use

  • The paper uses publicly released backbones and datasets under their respective licenses and terms, while noting that some datasets lack unambiguous standalone license information.Source: §B.5 · p. 14

C.1 Stability Analysis for Fixed Grouping and Warm-up Duration

Expert groupings stabilize after early training, supporting a fixed grouping after warm-up. ACE’s performance remains robust across warm-up duration, grouping resolution, CKA centering, probe-set size, and adapter rank.

C.1 Stability Analysis for Fixed Grouping and Warm-up Duration

  • Consecutive groupings change rapidly early in training and then stabilize, supporting fixed grouping after warm-up and the Tinit = 150 default.The figure measures Jaccard similarity between groupings recomputed every 15 steps.Source: §C.1 · pp. 14–15, Figure 6 · p. 15
  • ACE and LoRA differ by at most 0.0008 in load entropy and 0.0046 in auxiliary loss across training.These measurements indicate little effect from fixed group assignments on aggregate routing statistics in this setting.Source: §C.1 · p. 15, Table 10 · p. 15
  • CKA Pearson/Spearman correlations with the final structure rise from 0.577/0.587 at step 150 to 0.901/0.896 at step 1000.At step 500, the correlations are 0.771/0.772; the means cover 32 layer–projection combinations.Source: §C.1 · p. 15
  • ACE outperforms LoRA at every warm-up duration, with a maximum performance gap of 0.47 points and the best average score at Tinit = 150.The warm-up choice balances grouping reliability against the remaining time for shared-adapter optimization.Source: Table 9 · p. 15, §C.1 · p. 16

C.2 Sensitivity to the Louvain Resolution Parameter

  • At γ = 0.8, ACE gains +1.50 over LoRA while reducing evaluation time by 17.3%; across 46–242 groups, ACE beats LoRA with only 1.23 points of average-accuracy variation.Fully merging all experts drops below LoRA (73.57 vs. 74.11), while evaluation time decreases as γ shrinks.Source: §C.2 · p. 16, Table 11 · p. 16

C.3 Necessity of Centering in CKA Calculation

  • Centered and uncentered CKA yield broadly comparable grouping quality and downstream performance, so the centering choice does not critically affect ACE.Both variants use the same ACE pipeline and differ only in whether representations are mean-centered.Source: §C.3 · pp. 16–17, Table 12 · p. 16

C.4 Sensitivity to Probe-Set Size

  • With 512 probe tokens, the grouping has ARI = 0.922 against the 10k-token reference, while 5k tokens yield nearly identical assignments.This supports robustness to substantial probe-set reductions.Source: §C.4 · p. 17, Table 13 · p. 17

D Rank Variation and Optimization Dynamics

E Analysis of Group-Anchor Strategies

CKA-center is the strongest and most efficient tested group-anchor strategy. ACE also performs strongly in dataset-specific training and leads the three ROUGE metrics on XSum.

E Analysis of Group-Anchor Strategies

  • CKA-center provides the strongest overall performance and is most efficient among the tested group-anchor strategies.It reuses similarity statistics from grouping, whereas Avg. ∆W requires an averaged full update and SVD refitting, and Max ∆W adds update-magnitude scoring.Source: E Analysis of Group-Anchor Strategies · pp. 17–18, Table 14 · p. 17

F Details for Figure 2

  • The Figure 2 matrix compares pairwise expert CKA similarities for gate-projection outputs in the 0th, 4th, and 14th layers.Scores use HellaSwag; values closer to 1 indicate higher similarity and values closer to 0 indicate lower similarity.Source: F Details for Figure 2 · p. 18

G Results with Dataset-Specific Training

  • ACE achieves stronger performance than competing methods on most dataset-specific benchmarks and consistently beats baselines across several datasets.The comparison uses nearly identical trainable-parameter budgets.Source: G Results with Dataset-Specific Training · p. 18

H Free-Form Generation on XSum

  • ACE achieves the highest score on all three ROUGE metrics on XSum, extending its gains to free-form text generation.Methods are compared under matched trainable-parameter budgets.Source: H Free-Form Generation on XSum · p. 18

I.1 Expert Swap on Up Projection

Adapter swaps with high CKA similarity cause smaller NLL increases than low-similarity or random swaps. The additional analyses examine redundancy across settings, grouping construction, execution scope, and runtime effects.

I.1 Expert Swap on Up Projection

  • At ρ = 0.05, high-similarity swaps raise NLL by approximately 0.0034, versus 0.0187 for low-similarity swaps and about 0.0051 for random swaps.At ρ = 0.20, the increases are 0.0109, 0.0396, and 0.0235, respectively.Source: §I.1 · p. 18, Figure 8 · p. 18

I.2 Functional Redundancy across Models and Tasks

  • On an independent OLMoE split, discovered groups retain a CKA gap of 0.293, while random groupings yield −0.0006 ± 0.0034.Across the four backbones, the reported commonsense CKA gaps are 0.299, 0.298, 0.438, and 0.344 for OLMoE, Qwen, Moonlight, and Kimi.Source: §I.2 · p. 18, OLMoE · p. 19

J Extended Backbone Comparison Including Kimi

K.1 Grouping Implementation Details

  • ACE builds layer-specific graphs from CKA similarities and applies Louvain community detection independently to each MoE layer.The probe uses 10,000 training-split tokens; graph weights retain similarities above the layer-specific median baseline, and Louvain greedily groups experts to increase modularity.Source: §K.1 · p. 19, §K.2 · pp. 19–20

K.3 Comparison with Fixed-Threshold Grouping

  • Fixed-threshold grouping scores 75.50 versus 75.61 for Louvain on OLMoE commonsense reasoning, and avoids dependence on a specific grouping algorithm.The variants use the same ACE pipeline and differ only in how CKA-based expert groups are formed.Source: §K.3 · pp. 20–21, Table 18 · p. 20

L.1 Implementation Details

  • Grouped adapter execution applies to up and gate projections, but not down projections, whose inputs are expert-specific.The shared up/gate correction can be computed once per active group and scattered to routed experts; down-projection acceleration would require extra merging or weight materialization.Source: §L.1 · p. 21

L.2 Fine-Grained Runtime Profiling

  • Grouped execution reduces post-consolidation step time by 43.92%, with adapter-forward execution reduced by 89.16%.The adapter-forward reduction accounts for approximately 91% of measured forward-time reduction; these synchronized component measurements are not directly comparable to end-to-end wall-clock results.Source: §L.2 · p. 21, Table 19 · p. 21

M Gradient Supervision Analysis Details

For gate-projection adapters, ACE reduces gradient imbalance and zero-gradient ratios across MoE backbones. The section also describes gradient-measurement and training-efficiency evaluation protocols.

M Gradient Supervision Analysis Details

N Additional Training Efficiency Results

O Overhead Analysis of the Consolidation Pipeline

ACE’s consolidation pipeline adds at most 1.83% to total training time, while grouped execution yields at least a 1.31× end-to-end speedup over LoRA on every backbone. After merging, ACE also shows steeper loss decreases and lower final training loss across evaluated dataset–backbone pairs.

O Overhead Analysis of the Consolidation Pipeline

  • CKA computation accounts for the overwhelming majority of one-time consolidation cost, while Louvain detection and adapter consolidation contribute negligible fractions even at the largest scale.The pipeline comprises pairwise CKA, Louvain community detection, and adapter consolidation; warm-up adds no extra cost beyond LoRA training.Source: O Overhead Analysis of the… · pp. 22–23
  • At most 1.83% of total ACE training time is spent on consolidation, and grouped execution yields at least a 1.31× end-to-end speedup over LoRA on every backbone.Pairwise CKA dominates the one-time overhead; its runtime can be reduced by using a smaller probe set.Source: O Overhead Analysis of the… · pp. 22–23, Table 21 · p. 23

P Optimization Dynamics

  • After the merge step, ACE decreases training loss more steeply than LoRA and reaches lower final loss across evaluated dataset–backbone pairs.The improvement disappears immediately after merging, then appears after stabilization and persists through training.Source: Figure 13 · p. 23, P Optimization Dynamics · p. 23, Figure 10 · p. 23

Q LLM Usage

  • An LLM was used to refine writing, improve grammatical accuracy, and assist with code organization and cleanup.Source: Q LLM Usage · p. 23
Loading 2609.06072v1…