PaperlayerView exact revision on arXiv

Source-linked AI summary

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

Zhentao Tan, Chang Liu, Yao Liu, Yue Wu, Jieping Ye

arXiv:2609.25655v1cs.LGcs.AI

TL;DR

As LLMs scale, dense adaptation faces computational and memory bottlenecks, while activated experts' sparse internal responses make whole-expert tuning too coarse. NSFT selects task-relevant internal channel groups within activated experts and uses learning-rate and gradient scaling to support sparse partial updates. On MMedC, NSFT achieves a better accuracy–parameter trade-off than ESFT and LoRA variants, saving up to 66.9% trainable parameters versus LoRA-QKVO+MLP at comparable or stronger performance.

  • Problem

    As LLMs scale, dense adaptation faces computational and memory bottlenecks, while activated experts' sparse internal responses make whole-expert tuning too coarse.

    Source: §1 · p. 1

  • Method

    NSFT selects task-relevant internal channel groups within activated experts and uses learning-rate and gradient scaling to support sparse partial updates.

    Source: §1 · p. 2

  • Results

    On MMedC, NSFT achieves a better accuracy–parameter trade-off than ESFT and LoRA variants, saving up to 66.9% trainable parameters versus LoRA-QKVO+MLP at comparable or stronger performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest expert-level sparsity remains too coarse and fine-grained sub-expert structure is a promising direction for efficient MoE adaptation.

    Source: §5 · p. 10

    Evaluation covers two MoE models; larger and more heterogeneous models remain future work for characterizing scaling behavior.

    Source: L LIMITATIONS · p. 21

Abstract

from arXiv · show

As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectures such as Mixture-of-Experts (MoE) models. This shift raises a key question for parameter-efficient fine-tuning (PEFT): at what granularity should parameters be selected and updated? Existing PEFT methods such as LoRA operate on predefined weight matrices, while expert-level sparse tuning methods update entire selected experts. However, we observe that activated experts are internally sparse, with only a small fraction of intermediate channels strongly responding to downstream tasks, indicating that expert-level adaptation is still too coarse. We propose NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts. NSFT decomposes each expert along the intermediate dimension into structured channel groups and selects task-relevant sub-experts by combining routing importance with intra-expert activation saliency. To optimize sparse partial updates, NSFT further introduces learning-rate scaling and dynamic gradient scaling to compensate for the reduced effective update magnitude. Experiments on OLMoE and Ling-mini-2.0 across challenging domain-specific tasks and general benchmarks show that NSFT consistently outperforms representative PEFT and expert-level sparse tuning baselines, while using substantially fewer trainable parameters and preserving competitive general capability. These results suggest that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.

1 INTRODUCTION

The introduction asks how finely MoE parameters should be selected for adaptation, arguing that whole-expert updates overlook internal channel sparsity. It presents NSFT, which selects structured sub-experts and adjusts sparse updates, and reports improved efficiency and adaptation performance across two MoE backbones.

1 INTRODUCTION

  • Dense full-parameter adaptation faces growing computational and memory bottlenecks as LLMs scale, making the granularity of parameter-efficient updates a central question.The introduction links this choice to parameter efficiency, training cost, modularity, knowledge sparsity, and parameter redundancy.Source: §1 · p. 1
  • Activated MoE experts contain highly concentrated channel activations, so updating each whole expert introduces redundancy and limits parameter efficiency.The introduction contrasts expert-level selection with the small fraction of internal channels strongly involved in a target task.Source: §1 · p. 2
  • NSFT decomposes experts along the intermediate dimension into structured sub-experts and selects task-relevant channel groups for fine-tuning.It treats sub-experts as compact functional units and focuses updates on responsive internal groups.Source: §1 · p. 2
  • Sparse partial updates can shrink effective update magnitude, so NSFT adds learning-rate scaling and gradient scaling to strengthen selected sub-experts' updates.The two components compensate for reduced global step size and redistribute update strength according to relative importance.Source: §1 · p. 2
  • Evaluation spans OLMoE and Ling-mini across domain-specific tasks in medicine, science, RAG, mathematics, code, and table question answering.The tasks cover factual recall, logical reasoning, instruction following, and structured output synthesis.Source: §1 · p. 2
  • NSFT saves up to 66.9% trainable parameters versus LoRA-QKVO+MLP at comparable or stronger performance, while improving domain adaptation over ESFT and LoRA across two backbones.The conclusion also reports competitive general capability.Source: §1 · pp. 2–3

2 RELATED WORKS

Related work traces adaptation granularity from dense-model operators to whole MoE experts, then motivates structured sub-expert units as a middle ground between expert- and neuron-level updates.

Adaptation Granularity in Dense LLMs

  • Dense-model PEFT uses auxiliary modules or prompts, low-rank updates to linear layers, and selective parameter updates, but remains closely tied to linear transformations.Examples include Adapters (Wang et al., 2022), Soft Prompting, and LoRA.Source: §2 · p. 3

Expert-Level Adaptation in Sparse Architectures

  • MoE routing makes expert-level adaptation a natural sparse update unit, but ESFT, DES-MoE, and DR-LoRA still treat each selected expert as indivisible.ESFT updates experts responsive to target data; DES-MoE uses multi-stage selection, while DR-LoRA assigns different LoRA ranks within experts.Source: §2 · pp. 3–4

From Full Experts to Sub-Expert Units

  • MoNE (Cheng et al., 2025) supports neuron-level selection, but isolated-neuron tuning can be unstable and weak in update capacity; NSFT groups neurons into structured sub-experts.The grouped units aim to retain neuron-level flexibility while avoiding isolated-neuron updates.Source: §2 · p. 4

3 METHOD

NSFT treats MoE experts as collections of task-relevant channel groups rather than indivisible units. It selects groups using expert activity and channel saliency, then adapts sparse updates with learning-rate and gradient scaling.

3.1 PRELIMINARIES

  • An MoE expert can be expressed as a sum of neuron-wise transformations, motivating selection and updating of task-relevant channel groups instead of whole experts.The standard MoE layer routes inputs to selected experts; each expert's intermediate channels contribute individual transformations.Source: §3.1 · p. 4

3.2 FINE-GRAINED SUB-EXPERT SELECTION

  • NSFT groups consecutive intermediate channels into sub-experts, targeting responsive channels that expert-level tuning would update redundantly.Only a small portion of channels inside activated experts is consistently responsive to downstream tasks.Source: §3.2 · p. 4
  • For each expert, NSFT combines routed-token channel responses with expert routing importance to score channel groups by task relevance.It aggregates channel scores within groups of G consecutive channels; G = 1 yields individual-channel selection, while larger groups are coarser.Source: §3.2 · p. 5
  • NSFT selects layer-wise the smallest set of groups whose cumulative importance exceeds threshold τ, retaining high-scoring groups for tuning.Selection ranks all groups in a layer by their group-level scores.Source: §3.2 · p. 5

3.3 ADAPTIVE TRAINING FOR SUB-EXPERT FINE-TUNING

  • Learning-rate scaling compensates for sparse updates' reduced step size, while group-level gradient scaling allocates update strength by channel relevance.The learning-rate scale is based on the inverse active ratio and is clipped for stability; gradient scaling masks frozen channels and scales selected groups.Source: §3.3 · pp. 5–6
  • Because Adam-style optimization can weaken fixed gradient scaling, NSFT refreshes scaling coefficients from online activation statistics with EMA smoothing.Entropy-adaptive modulation controls contrast among groups; when group energy is concentrated, γ moves closer to 0.5 to reduce extreme disparities and improve stability.Source: §3.3 · p. 6

4 EXPERIMENTS

Across OLMoE and Ling-mini, NSFT is evaluated against parameter-efficient and expert-level baselines on domain-specific and general benchmarks, with ablations testing selection granularity and adaptive scaling. The reported results favor sub-expert tuning for adaptation efficiency, while group size and scaling-update frequency affect the balance across tasks.

4.1 IMPLEMENTATION DETAILS

  • Experiments compare OLMoE-7B and Ling-mini-2.0-16B across domain-specific adaptation tasks and general benchmarks, against base, full fine-tuning, LoRA, and expert-level ESFT baselines.The evaluation covers mathematics, medicine, scientific reasoning, retrieval-augmented QA, code, TableQA, and multilingual tasks, alongside four general-purpose benchmarks.Source: §4.1 · p. 7

4.3 ABLATION ANALYSIS

  • Across thresholds on MMedC and SciRIFF-4k, NSFT consistently outperforms ESFT; both use τ = 0.2 in the main experiments for fair comparison and training-cost balance.Figure 3's left panel presents the threshold comparison; its NSFT results use NSFT-SG.Source: Figure 3 · p. 8, §4.3 · p. 9
  • At G = 16, zero-shot general performance peaks while in-domain MMedC remains among the highest; increasing group size from 1 to 16 improves the general average while MMedC stays stable.The authors describe the trade-off as flexibility versus expressiveness: very small groups fragment update capacity, while very large groups add redundant channels.Source: Figure 3 · p. 8, §4.3 · p. 9
  • +SLR improves most MMedC languages and both weighted and unweighted averages, whereas +SG improves several languages and the weighted average less consistently.Figure 4 compares gains over the NSFT baseline with sub-expert selection alone and no adaptive training strategy.Source: Figure 4 · p. 8, §4.3 · p. 9

4.2 MAIN RESULTS

  • 46.62 in-domain average for NSFT-OLMoE exceeds ESFT-OLMoE and both LoRA variants, with 5.67% average trainable parameters and a 45.96 general average.Full FT reaches a 49.91 in-domain average but its general-benchmark average is 35.98; ESFT-OLMoE's general average is 46.24.Source: §4.2 · p. 8
  • 34.44 PoT average for NSFT-Ling exceeds ESFT-Ling's 17.68 on TableQA, using 2.86% trainable parameters and achieving a 66.18 general average.NSFT-Ling also surpasses the strongest LoRA variant, LoRA-r64, on TableQA.Source: §4.2 · p. 8
  • NSFT's responsive sub-expert updates are presented as more precise than whole-expert ESFT or generic LoRA updates, with a better efficiency–generalization trade-off.The comparison attributes ESFT's coarseness to updating entire experts and contrasts LoRA's generic low-rank updates with NSFT's targeting of task-relevant components.Source: §4.2 · p. 8

4.3 ABLATION ANALYSIS

  • Dynamic gradient scaling generally improves in-domain adaptation over static scaling, but the best interval varies: iter1 on MMedC and iter5 on SciRIFF and RAG-Instruct general retention.Table 3 compares fixed scaling with dynamic masks refreshed every one or five training steps.Source: Table 3 · p. 9, §4.3 · p. 9

5 CONCLUSION

NSFT refines MoE adaptation from whole experts to task-relevant channel groups and adds learning-rate and dynamic gradient scaling for partial updates. The conclusion reports that it outperforms representative baselines with fewer trainable parameters while retaining competitive general capability; the accompanying setup describes its models, tasks, and training protocol.

5 CONCLUSION

  • NSFT selects task-relevant internal channel groups within activated experts and adds learning-rate scaling and dynamic gradient scaling to compensate for reduced update magnitude.The conclusion frames this as refining sparse adaptation from whole experts to sub-experts.Source: §5 · p. 9
  • NSFT consistently outperforms representative PEFT and expert-level sparse tuning baselines while using substantially fewer trainable parameters and preserving competitive general capability.The conclusion identifies sub-expert structure as a promising direction for efficient MoE adaptation.Source: §5 · p. 10

A.1 TRAINING AND EVALUATION SETUP

  • The experiments use OLMoE and Ling-mini-2.0 across seven downstream tasks and train for two epochs with AdamW, a peak learning rate of 3e-5, and sequences up to 8,192 tokens.The setup specifies a global batch size of 8 and experiments on eight A100 80G GPUs.Source: §A.1 · p. 13

A.2.1 DATASET DETAILS

  • The dataset suite spans table understanding, scientific instruction following, math, code, and biomedical retrieval QA, with math, code, and RAG scores averaged across paired benchmarks or subsets.The listed pairs are GSM8k and MATH500, HumanEval and MBPP, and PubMedQA and Health Claims.Source: §A.2.1 · p. 13

A.2.2 ON THE CHOICE OF EVALUATION DATASETS AND BENCHMARKS

The evaluation suite is designed to span challenging tasks across reasoning, domain knowledge, retrieval comprehension, and cross-lingual generalization. Table 4 separately details Ling-mini-2.0 comparisons on TableQA and general benchmarks.

A.2.2 ON THE CHOICE OF EVALUATION DATASETS AND BENCHMARKS

  • The benchmark suite spans structured reasoning, scientific and medical knowledge, long-context retrieval, math and code reasoning, and cross-lingual generalization to assess performance on demanding tasks.The text presents these tasks as a rigorous assessment of whether PEFT methods maintain competitive performance where they may fall short of full fine-tuning.Source: §A.2.2 · pp. 13–14
  • Table 4 compares Full FT, LoRA, ESFT, and NSFT settings on four Ling-mini-2.0 TableQA subtasks, the PoT average, and general-benchmark average.The caption says bold values identify the best result within each column.Source: Table 4 · p. 14

B ADDITIONAL RESULTS ON TABLEQA

On TableQA, NSFT delivers competitive or stronger performance than LoRA and ESFT with substantially fewer trainable parameters, while retaining general capability. The results also suggest that finer-grained updates better fit Ling-mini-2.0’s expert organization.

B ADDITIONAL RESULTS ON TABLEQA

  • 34.44 PoT average with 2.86% trainable parameters for NSFT-G4-th0.2, versus 35.35 with 29.69% for LoRA-128 MLP+QKVO.NSFT-G4-th0.2 outperforms all ESFT variants and approaches the strongest LoRA variant on TableQA.Source: B ADDITIONAL RESULTS ON TABLEQA · p. 14
  • 21.65 PoT average for the best ESFT variant, while five NSFT variants at th0.2 outperform ESFT-th0.3 by a large margin.The authors attribute the gap to whole-expert updates including task-irrelevant channels that sub-expert selection can avoid.Source: B ADDITIONAL RESULTS ON TABLEQA · p. 15
  • 66.45 to 62.59 general average: Full FT reduces general performance, while several NSFT variants retain averages around 66.0–66.5 as TableQA performance improves.Large LoRA variants also tend to underperform the base model on general benchmarks.Source: B ADDITIONAL RESULTS ON TABLEQA · p. 15
  • Ling-mini-2.0’s 256 experts versus OLMoE’s 64 motivates finer adaptation: whole-expert updates may remain too coarse for Ling-mini’s architecture.The discussion suggests sub-expert updates better match Ling-mini’s internal organization.Source: B ADDITIONAL RESULTS ON TABLEQA · p. 15

C ROBUSTNESS ACROSS SELECTION THRESHOLDS

Across selection thresholds, NSFT outperforms ESFT on MMedC and SciRiFF, while group-size results vary across backbones and tasks. The appendix also reports concentrated channel activations across domains, supporting sub-expert selection.

C ROBUSTNESS ACROSS SELECTION THRESHOLDS

  • NSFT consistently outperforms ESFT on MMedC and SciRiFF across selection thresholds, and performance generally improves as τ increases; optimal granularity depends on the target domain.The NSFT comparisons use different sparsity budgets, and the threshold trend reflects retaining more task-relevant units.Source: C ROBUSTNESS ACROSS SELECTION THRESHOLDS · pp. 15–16

D GROUP SIZE SENSITIVITY ACROSS MOE BACKBONES

  • 34.44 PoT performance is achieved at G = 4 on Ling-mini-2.0, unlike OLMoE, where larger groups such as G = 16 are often more favorable.The authors recommend treating group size as a backbone- and task-dependent hyperparameter, with a small validation sweep informed by expert coverage and intra-expert activation ratio.Source: D GROUP SIZE SENSITIVITY ACROSS MOE… · p. 16, Table 7 · p. 16
  • As G falls from 32 to 1, involved experts rise from 230 to 1024, broadening coverage; the authors suggest this fits TableQA’s heterogeneous capability demands.The proposed explanation is that smaller groups let NSFT select sub-experts from a broader set of experts.Source: D GROUP SIZE SENSITIVITY ACROSS MOE… · p. 16
  • 17.68 TableQA PoT at G = 1024, where sub-expert selection becomes expert-level selection, is far below the best G = 4 setting.The authors interpret this as further evidence that whole-expert updates are too coarse for this setting.Source: D GROUP SIZE SENSITIVITY ACROSS MOE… · p. 16, Table 7 · p. 16

E.1 INTRA-EXPERT ACTIVATION SPARSITY ACROSS DOMAINS

  • Across medical, scientific, retrieval, mathematical, code, and table-reasoning domains, most channels have near-zero gate scores and a small subset carries relatively large activation.The top-50% cumulative activation threshold lies in a low-value region, indicating that selected experts contain many weakly activated channels.Source: §E.1 · pp. 16–17, Figure 5 · p. 17

E.2 CONSISTENT SPARSITY ACROSS EXPERTS

Randomly sampled experts from shallow, middle, and deep layers show similar intra-expert sparsity. Additional comparisons find that dynamic gradient scaling generally improves in-domain adaptation, with the best refresh schedule varying by task.

E.2 CONSISTENT SPARSITY ACROSS EXPERTS

  • Across shallow, middle, and deep layers, sampled experts show near-zero gate scores for most channels, with only a small fraction forming a higher-activation tail.The top-50% cumulative activation threshold lies in a very small-value region across the sampled cases, supporting the authors’ conclusion that sparsity is not layer- or expert-specific.Source: §E.2 · pp. 17–18

F ADDITIONAL RESULTS ON STATIC AND DYNAMIC GRADIENT SCALING

  • Dynamic gradient scaling generally improves in-domain adaptation over static scaling, but the best update frequency differs between MMedC and SciRiFF.On MMedC, per-step updates lead on weighted and general-benchmark averages, while five-step updates lead on the unweighted average; on SciRiFF, five-step updates achieve the highest weighted and unweighted validation averages under both settings.Source: F ADDITIONAL RESULTS ON STATIC AND… · p. 18, Table 8 · p. 18, Table 9 · p. 18

G LOW-BUDGET AND BUDGET-MATCHED COMPARISONS

NSFT retains a stronger accuracy–parameter trade-off than ESFT in low-budget and closely matched-budget comparisons. The additional tests also examine grouping choices, optimization components, and entropy-adaptive gradient scaling.

G LOW-BUDGET AND BUDGET-MATCHED COMPARISONS

  • Across all three lower thresholds, NSFT achieves higher MMedC performance while updating fewer parameters than ESFT.At τ = 0.05, ESFT’s coarse selection retains nearly one complete expert per layer, while NSFT can reduce its budget further by selecting within experts.Source: G LOW-BUDGET AND BUDGET-MATCHED… · p. 18, Table 10 · p. 19
  • Under closely matched trainable-parameter budgets, NSFT remains stronger than ESFT on both SciRIFF and MMedC.The NSFT threshold is adjusted to τ = 0.22 to approach the ESFT budget at τ = 0.20; the authors conclude that a larger trainable budget does not explain NSFT’s improvement.Source: G LOW-BUDGET AND BUDGET-MATCHED… · p. 19, Table 11 · p. 19
  • 44.01 MMedC average for ESFT with weight decay 0, versus 44.31 with its original weight decay of 0.1.The slightly lower controlled result indicates that NSFT’s advantage is not caused by the baselines’ different weight-decay settings.Source: G LOW-BUDGET AND BUDGET-MATCHED… · p. 19

H ALTERNATIVE SUB-EXPERT GROUPING STRATEGIES

  • Consecutive-channel grouping gives the best overall balance among the tested grouping strategies on MMedC.Random and activation-clustering groups mitigate the general-capability degradation of individual-channel selection but remain weaker than consecutive grouping.Source: H ALTERNATIVE SUB-EXPERT GROUPING… · p. 19, Table 12 · p. 20

I CONTRIBUTIONS OF SELECTION AND ADAPTIVE OPTIMIZATION

J EFFECT OF THE ENTROPY-ADAPTIVE EXPONENT

K TRAINING EFFICIENCY AND DYNAMIC-SCALING OVERHEAD

NSFT’s selection-only variant adds little training-time overhead, while dynamic scaling adds cost but can be adjusted or omitted according to compute budget.

K TRAINING EFFICIENCY AND DYNAMIC-SCALING OVERHEAD

  • NSFT-selection takes close to ESFT’s training time while achieving higher accuracy, and all NSFT variants use memory close to ESFT and below Full FT.Table 15 reports training time and peak per-GPU memory for MMedC; measurements use eight and four A100 GPUs, respectively.Source: K TRAINING EFFICIENCY AND… · p. 20, Table 15 · p. 21
  • Removing dynamic updates cuts training time from 35m 29s to 20m 38s without increasing memory usage.The added cost of full NSFT comes from collecting updated activation statistics and refreshing the scaling mask.Source: K TRAINING EFFICIENCY AND… · p. 20
  • Users can choose selection-only, static-scaling, or dynamic-scaling NSFT; more frequent updates track changing sub-expert importance, while less frequent updates may be more stable with noisy or heterogeneous statistics.Source: K TRAINING EFFICIENCY AND… · p. 20

L LIMITATIONS

The paper identifies boundaries in model coverage and hyperparameter selection, and notes both resource-related benefits and potential safety risks from efficient domain adaptation.

L LIMITATIONS

  • Evaluation covers OLMoE-7B and Ling-mini-2.0-16B; larger and more heterogeneous MoE LLMs remain to be examined to characterize scaling behavior.Source: L LIMITATIONS · p. 21
  • For new domains or substantially different MoE architectures, a lightweight validation sweep may help select the efficiency–performance trade-off controlled by group size G and threshold τ.Source: L LIMITATIONS · p. 21

M BROADER IMPACTS

  • More parameter-efficient adaptation can reduce computational cost, memory use, and energy consumption compared with full-parameter fine-tuning, potentially widening access for users with limited hardware.Source: M BROADER IMPACTS · p. 21
  • Efficient adaptation may lower barriers to harmful or misleading customization, while domain fine-tuning can amplify training-data biases, errors, or unsafe behaviors; the paper recommends curation, safety evaluation, and monitoring.Source: M BROADER IMPACTS · p. 21

Related papers

Loading 2609.25655v1…