PaperlayerView exact revision on arXiv

Source-linked AI summary

Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning

Mehreen Hossain Chowdhury, Nowshin Mahjabin, Ahmed Shafin Ruhan, Md Azam Hossain, Abu Raihan Mostofa Kamal, Md Tahmid Rahman Laskar

arXiv:2609.03150v1cs.LGcs.CL

TL;DR

The paper asks whether token-level routing alone prevents negative transfer inside shared LoRA adapters. It diagnoses adapter-gradient contention and introduces SpawnLoRA, which adds gated sub-adapters within experts. Across the tested models, SpawnLoRA reduces negative transfer relative to fixed- and rank-adaptive LoRA.

  • Problem

    It remains unclear whether routing-based specialization alone prevents adapter-level negative transfer, since domain-specific updates may still interfere within shared low-rank adapters.

    Source: §1 · p. 1

  • Method

    The paper measures expert sharing and adapter-gradient alignment, then uses SpawnLoRA to add gated LoRA sub-adapters inside experts when contention is detected.

    Source: §1 · p. 2

  • Results

    Across both models, SpawnLoRA reduces negative transfer versus fixed- and rank-adaptive LoRA; on OLMoE code+GSM8K, it removes 83% and 44% of Run B/C degradation versus 6% and 2% for matched-rank standard LoRA.

    Source: §6 · p. 5, §5.2 · p. 5

  • Takeaways & Limitations

    The reported results favor structural separation inside experts over shared-capacity expansion for reducing negative transfer in the tested settings.

    Source: §6 · p. 5

    The evaluation covers only two MoE families, leaving generalization to architectures with different routing granularity, including shared-expert designs, open.

    Source: Limitations · p. 6

Abstract

from arXiv · show

Multi-domain fine-tuning often combines MoE routing with LoRA, assuming that token-level routing separates domain-specific updates. We test this assumption in MoE+LoRA using Python code paired with biomedical text and mathematical reasoning. Although these domains show near-disjoint expert routing, adding biomedical data substantially increases code perplexity, indicating that routing separation alone may not prevent negative transfer. To localize the failure, we introduce Jaccard routing overlap and adapter-gradient cosine similarity, which measure expert sharing and update compatibility, respectively. These diagnostics indicate that interference arises mostly from nearly orthogonal domain gradients competing within the same low-rank adapter subspace. We address this issue with SpawnLoRA, which dynamically adds gated sub-adapters inside MoE experts when adapter-level contention is detected, while keeping the router fixed. We evaluate SpawnLoRA on Phi-tiny-MoE-instruct and OLMoE-1B-7B across multiple mixture settings and find that it effectively reduces negative transfer compared with standard and rank-adaptive LoRA. These results demonstrate that structural separation inside experts provides benefits beyond routing or rank expansion alone.

1 Introduction

The introduction asks whether token-level MoE routing can prevent negative transfer inside shared LoRA adapters, even when routing separates domains. It diagnoses adapter-level contention and proposes gated sub-adapters within experts as a structural response.

1 Introduction

  • Prior routing-based specialization does not settle whether routing alone prevents adapter-level negative transfer, because domain updates may still interfere within shared low-rank adapters.Prior work used routing over lightweight or LoRA-based experts to encourage specialization (Zadouri et al., 2024; Li et al., 2024; Dou et al., 2024).Source: §1 · p. 1
  • The central question is whether expert-level routing separation suffices to prevent negative transfer in the LoRA adapters attached to those experts.The study treats Python code as the protected domain and pairs it with biomedical text or mathematical reasoning.Source: §1 · p. 1
  • The proposed diagnostics measure expert sharing and LoRA-update alignment, locating interference mainly in nearly orthogonal gradients competing within the same low-rank adapter.The paper calls this failure mode intra-adapter subspace contention and reports that increasing shared rank is insufficient.Source: §1 · p. 2
  • SpawnLoRA adds gated LoRA sub-adapters inside an expert when adapter-level contention is detected, targeting negative transfer through structural separation.The paper evaluates the method across two MoE backbones and multiple conflict levels.Source: §1 · p. 2

2 Related Work

Related work addresses multi-domain interference through capacity adjustment, update modification, routing, and structural separation. The paper distinguishes SpawnLoRA by selectively creating separate pathways inside experts where gradient-space contention is detected.

2 Related Work

  • Rank-adaptive methods adjust capacity during training, but DR-LoRA expands rank within existing adapters, leaving domain updates able to interact in the shared low-rank subspace.AdaLoRA reallocates rank based on parameter importance, while DR-LoRA periodically expands expert ranks according to expert saliency (Deng et al., 2026).Source: §2 · p. 2
  • Gradient-surgery methods modify conflicting updates, while MoE-style PEFT uses routing to encourage specialization; near-orthogonal gradients can still compete inside one adapter despite routing separation.Gradient surgery methods include Yu et al. (2020) and Wang et al. (2021); routing approaches include Zadouri et al. (2024) and Li et al. (2024).Source: §2 · p. 2
  • Prior structural approaches include per-expert, hierarchical, or dynamically expanded adapters, whereas domain-specific adapters rely on predefined specialization and SpawnLoRA selectively adds ReLU-gated sub-adapters inside contended experts.The passage names Zadouri et al. (2024), Li et al. (2024), Peng et al. (2026), and Huynh et al. (2025) among structural-separation work; MixLoRA and LoRAMoE combine LoRA-style adaptation with expert or mixture structures (Li et al., 2024; Dou et al., 2024).Source: §2 · p. 2

3 Experimental Setup

The experiments use two MoE backbones and Python code as the protected evaluation domain, paired with biomedical text or mathematical reasoning. The setup uses controlled mixtures and matched code-only controls, while treating the small-scale results as within-method changes rather than converged-model perplexities.

3 Experimental Setup

  • The study tests Phi-tiny-MoE-instruct and OLMoE-1B-7B to assess whether contention and SpawnLoRA's mitigation extend across architectures.Phi-tiny-MoE-instruct has 1.1B active/3.8B total parameters, 16 experts, and top-2 routing; OLMoE-1B-7B has 1B active/7B total, 64 experts, and top-8 routing.Source: §3 · p. 2
  • Held-out HumanEval code perplexity measures negative transfer relative to a code-only run, using pooled next-token loss and token count.The reported calculation is PPL = exp(total loss/total tokens).Source: §3 · p. 2
  • Code is paired with PubMedQA biomedical text for severe mismatch or GSM8K reasoning for a more moderate shift, across 80/20 and 50/50 mixtures and a code-only reference.Each pair uses a 2,000-example training budget and 3,000 steps; matched code-only controls account for code exposure in the mixture runs.Source: §3 · pp. 2–3
  • The deliberately small-scale setup uses 1–8B active parameters and 3,000 steps to isolate contention, so absolute perplexity is not comparable to converged models.Negative transfer is reported as a within-method change relative to Run A.Source: §3 · p. 3

4 Diagnostic Study

The diagnostic study asks whether negative transfer reflects routing overlap or conflict inside LoRA adapters. For Phi-tiny-MoE-instruct on code plus biomedical data, routing overlap is low while adapter gradients are nearly orthogonal, pointing to contention within the shared subspace.

4 Diagnostic Study

  • Across layers and training steps, Phi-tiny-MoE-instruct has average code–biomedical Jaccard routing overlap of approximately 0.056, indicating largely separate expert paths.The study treats 0 as disjoint routing and 1 as identical expert usage.Source: §4 · p. 3
  • Across 20 trials, code and biomedical adapter gradients have cosine similarity −0.002 ± 0.003, with 75% below zero, indicating unrelated rather than aligned updates.The paper defines +1 as aligned, −1 as directly opposed, and 0 as orthogonal updates.Source: §4 · p. 3
  • The domains interfere because their adapter gradients are nearly orthogonal within the shared LoRA subspace, a pattern routing separation cannot prevent.The diagnostic study examines Phi-tiny-MoE-instruct on code plus biomedical data; corresponding OLMoE gradient analysis is in Appendix D.1.Source: §4 · p. 3

5 SpawnLoRA

SpawnLoRA targets contention within shared expert adapters by adding gated sub-adapters while leaving token routing unchanged. Across the evaluated mixtures, it reduces negative transfer compared with standard and rank-adaptive LoRA.

5.1 Method

  • Each expert keeps frozen weights and an always-active base LoRA while gated sub-adapters provide input-dependent pathways without changing which tokens reach each expert.The j-th pathway’s learned ReLU gate controls its contribution for input x.Source: §5.1 · pp. 3–4
  • A sub-adapter is spawned only when stalled adapter importance and a persistent smoothed-loss gap between domains both hold over a rolling window.After spawning, the window resets and each expert is capped at a maximum number of sub-adapters.Source: §5.1 · p. 3
  • At 20 spawn events, code/interfering-domain gradient cosine ranged from −0.013 to +0.140, corresponding to near-orthogonal gradients.The reported angles were 82°–91°.Source: §5.1 · p. 4

5.2 Results and Discussion

  • SpawnLoRA also yields the lowest negative transfer in both Phi biomedical mixture settings, while DR-LoRA performs worse than fixed-rank LoRA.The reported comparison is for Phi-tiny-MoE-instruct and the code–biomedical pair.Source: §5.2 · p. 4, Table 1 · p. 4
  • On OLMoE, SpawnLoRA eliminates negative transfer in biomedical Run B and reduces code perplexity 26.9% in Run C; matched-rank LoRA cuts degradation only 6% and 2%.The matched-rank comparison uses r = 17 against base rank r = 16.Source: Table 2 · p. 4, §5.2 · p. 5
  • For OLMoE with GSM8K, SpawnLoRA reduces code perplexity over LoRA by 7.8% in Run B and 11.1% in Run C.The reported negative transfer in this setting includes reduced code exposure, not only interference.Source: §5.2 · p. 5, Table 3 · p. 5
  • Under severe OLMoE conflict, SpawnLoRA reaches 33.18% MBPP token accuracy versus 19.71% for LoRA, and a 30.0% syntactic pass rate versus 22.0% for LoRA and 2.7% for DR-LoRA.All adapted models degrade relative to the base model on syntactic pass rate.Source: §5.2 · p. 5, Table 5 · p. 5
  • On Phi Run B, adding high-overlap layer L1 or L3 reduces code perplexity from 684.52 to 361.24 or 352.45, respectively.The authors identify targeted layer-wise expansion as practical when additional compute is available.Source: §5.2 · p. 5, Table 4 · p. 5

6 Conclusion

Successful MoE routing does not prevent multi-domain interference: near-orthogonal domain gradients can compete within shared low-rank adapters. SpawnLoRA reduces negative transfer through gated sub-adapters, with multi-layer spawning further eliminating it under moderate conflict on Phi-tiny-MoE-instruct.

6 Conclusion

  • SpawnLoRA reduces negative transfer relative to fixed- and rank-adaptive LoRA across both evaluated models.Multi-layer spawning further eliminates negative transfer under moderate conflict on Phi-tiny-MoE-instruct.Source: §6 · p. 5
  • Near-orthogonal domain gradients can compete within a shared low-rank adapter despite successful MoE routing.Source: §6 · p. 5
  • The results favor structural separation inside experts over shared-capacity expansion alone.Source: §6 · p. 5

Limitations

The study is a controlled diagnostic evaluation, and its conclusions are bounded by the domain pairs, model scale, training budget, and architectures tested. Broader domain, scale, functional, and architectural evaluations remain open.

Limitations

  • The diagnosis uses Python paired with biomedical text or mathematical reasoning; less-dissimilar domain pairs and scaling beyond two domains remain untested.The authors identify broader domain pairs and possible nonlinear growth of contention as future work.Source: Limitations · p. 6
  • Experiments use 1–8B-parameter models, 3,000 training steps, and fixed 2,000-example mixtures; stability at ≥70B parameters or longer runs remains open.The authors also note the spawn trigger may require recalibration in those regimes.Source: Limitations · p. 6
  • Headline evaluation relies on perplexity because Pass@1 and PubMedQA accuracy were uninformative at this training scale; broader downstream evaluation remains future work.Syntactic pass and MBPP serve as additional functional checks for code.Source: Limitations · p. 6
  • Evaluation covers two MoE families; generalization to architectures with different routing granularity, including shared-expert designs, remains open.Shared experts may receive tokens from all domains by construction, potentially producing different contention patterns.Source: Limitations · p. 6

Ethical Considerations

The appendix records the study’s research-only safeguards and compute setup, then describes how importance scores and calibrated triggers guide dynamic capacity allocation in DR-LoRA and SpawnLoRA.

Ethical Considerations

  • The experiments use public datasets and checkpoints for research, make no new human-subject or personally identifiable data collection, and do not establish clinical validity or suitability.The authors also warn that adapted models may inherit errors, biases, or unsafe code patterns; those failure modes were not evaluated, and high-stakes deployment requires independent evaluation.Source: Ethical Considerations · p. 6
  • Experiments ran on single GPUs: an NVIDIA L4 for Phi-tiny-MoE-instruct and an NVIDIA A100 for OLMoE-1B-7B, with no distributed training.Observed peak memory use was 13.9 GB and 27.8 GB, respectively.Source: Ethical Considerations · pp. 6–7

A.1 Hyperparameter Values

  • The SpawnLoRA hyperparameter tables distinguish model-specific loss-gap thresholds from shared base rank and other allocation settings, while training settings are generally shared across methods.The window and importance threshold were insensitive over the tested ranges; after a spawn, resetting the rolling window prevents another allocation within fewer than W steps.Source: §A.1 · p. 8, Table 6 · p. 8, Table 7 · p. 8

A.2 Importance Scores for Dynamic Capacity Allocation

  • DR-LoRA uses gradient–parameter sensitivity to score rank directions, averages active directions per expert, smooths over time, and combines the result with routing frequency and a rank penalty.The penalty discourages repeatedly expanding experts that already have many active ranks; the top-k experts by saliency receive new rank slots.Source: §A.2 · p. 9

SpawnLoRA.

  • SpawnLoRA uses parameter-norm importance rather than gradients and spawns a sub-adapter only when the base adapter plateaus while the smoothed domain-loss gap remains above threshold.Its importance score measures each rank’s output-scale contribution to the current low-rank update.Source: SpawnLoRA · p. 9

A.3 Parameter Count Analysis

This appendix quantifies SpawnLoRA’s added capacity, describes its algorithm and experimental resources, and reports trigger sensitivity and per-seed evaluation details.

A.3 Parameter Count Analysis

  • Each SpawnLoRA sub-adapter adds 2·rs·dmodel+dmodel trainable parameters for its low-rank matrices and gating vector; both backbones reached the cap of 10 spawns in Runs B and C.At peak allocation, Phi-tiny-MoE-instruct used 2,095,104 parameters and OLMoE-1B-7B used 4,542,464.Source: §A.3 · p. 9, Table 8 · p. 9, Table 9 · p. 10
  • A fixed-rank budget match requires approximately rank 21 for Phi-tiny-MoE-instruct and rank 17 for OLMoE-1B-7B, but a direct rank-21 comparison for Phi remains future work.The authors note that SpawnLoRA concentrates extra parameters in the conflicted expert; OLMoE’s gains with only Δr = 1 overhead suggest they are not parameter-driven.Source: §A.3 · p. 10

A.4 Dataset Provenance

  • The experiments use publicly available datasets without modification and run on single-GPU instances, with no DeepSpeed or FSDP.Table 10 identifies the datasets and reports sample counts as examples drawn per run, up to 1,000 per domain.Source: §A.4 · p. 10, Table 10 · p. 10

A.5 Per-Seed Results for Phi-tiny-MoE-instruct

  • The Phi code-plus-biomedical evaluation reports mean ± standard deviation over seeds 42, 123, and 456, defining negative transfer as PPLcode(Run X) − PPLcode(Run A).Under severe conflict, SpawnLoRA has the lowest reported standard deviation: ≤32.43 versus 87.16 for LoRA and 153.82 for DR-LoRA.Source: §A.5 · p. 10, Table 11 · p. 10, Table 12 · p. 10

B.1 SpawnLoRA Algorithm

  • SpawnLoRA initializes frozen MoE experts with a base LoRA, adds ReLU-gated sub-adapters, and triggers spawning after sustained plateau and domain-loss conflict conditions.New sub-adapters are initialized using leading right-singular vectors of the base-adapter gradient, with a Kaiming-uniform fallback if SVD fails.Source: §B.1 · p. 11

C.1 Trigger Sensitivity

  • +574.79 negative transfer under the default setting fell to +540.51 after relaxing the loss-gap threshold and raising the spawn cap, a further reduction of approximately 6%.Both settings reached their cap; the first spawn moved earlier by only one training step, suggesting the cap—not the trigger threshold—was the main limit under severe conflict.Source: §C.1 · p. 12
  • The threshold grid tests nine δ-by-τ combinations over 300 steps at 50% conflict; lower δ consistently yields more spawns, while τplateau has negligible effect in the tested range.The selected δ values mark the boundary for active but not excessive spawning: 0.3 for OLMoE-1B-7B and 1.0 for Phi-tiny-MoE-instruct.Source: Table 13 · p. 12, §C.1 · p. 12

C.2 DR-LoRA Baseline Results

The baseline comparison finds that scheduled rank expansion performs poorly under severe biomedical conflict but comparably to fixed-rank LoRA on the milder GSM8K pair; gradient measurements also contrast shared and separated adapter pathways.

C.2 DR-LoRA Baseline Results

  • +1311.93 negative transfer for DR-LoRA under Run C was 68% worse than LoRA (+778.78) and 128% worse than SpawnLoRA (+574.79).DR-LoRA expands rank on a fixed schedule independent of conflict detection; on the milder GSM8K pair, it performs comparably to fixed-rank LoRA at both conflict levels.Source: §C.2 · p. 12, Table 14 · p. 12, Table 15 · p. 12

D.1 Gradient-Space Conflict Measurement

  • 114.7° for LoRA versus 83.7° for SpawnLoRA: the measured gradient angle shifts from opposing directions to near-orthogonality under the separated sub-adapter pathways.The measurements are from OLMoE-1B-7B, Layer 0, Expert 6, Run C, after 1,500 training steps at 50% conflict.Source: §D.1 · p. 12, Table 16 · p. 13

D.2 Failure Mode Characterisation

The characterized SpawnLoRA failure modes arise under severe domain imbalance and when the sub-adapter cap is exhausted; both can limit separation during training.

D.2 Failure Mode Characterisation

  • With a 50% conflict ratio and a smaller primary-domain dataset, the EMA loss tracker can be dominated by the larger domain, delaying or suppressing spawning.In the Phi experiments, the example imbalance is 164 HumanEval examples versus 775 medical examples; the trigger may then leave SpawnLoRA behaving like standard LoRA.Source: §D.2 · p. 13
  • After Jmax is reached, no further sub-adapters can be added, so new conflicting gradients are absorbed by existing sub-adapters and negative transfer continues to accumulate.The cap therefore sets a ceiling on gradient-space separation, and choosing it for a given conflict ratio remains an open design question.Source: §D.2 · p. 13
  • The authors characterize both failure modes as dataset-ratio and cap-configuration effects and suggest balanced or capped sampling, plus a higher Jmax when compute permits.The recommendation is conditional on substantially different domain sizes and expected conflict ratio.Source: §D.2 · p. 13
Loading 2609.03150v1…