PaperlayerView exact revision on arXiv

Source-linked AI summary

When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging

Shangge Liu, Yuehan Yin, Yinghuan Shi, Lei Wang, Wenbin Li

arXiv:2608.27518v1cs.LG

TL;DR

Continual learning and model merging face distinct-looking failures that the paper identifies as a shared task-interference phenomenon, while optimizer geometry has received limited attention. The paper formalizes interference, derives a spectral-norm bound, and interprets Muon as controlling that factor by construction; experiments report positive gains across both paradigms, including up to +5.02 accuracy points in model merging.

  • Problem

    Continual learning and model merging address multi-task learning but are challenged by catastrophic forgetting and weight-disentanglement error, while optimizer geometry is usually treated as an implementation detail.

  • Method

    The paper unifies both difficulties as task interference, reduces them to ⟨∆Wℓ, Jℓ(x)⟩F, derives a spectral-norm upper bound, and analyzes Muon’s spectral-norm steepest descent.

  • Results

    Muon delivers uniformly positive continual-learning gains across the reported protocols and improves eight-task model-merging accuracy over AdamW by up to +5.02 points across three CLIP backbones.

  • Takeaways & Limitations

    The results position Muon as an optimizer-centric approach that complements existing continual-learning and model-merging solutions by controlling the spectral factor of interference.

Abstract

from arXiv · show

Continual learning (CL) and model merging (MM) both aim to obtain a single model that performs well across multiple tasks, challenged respectively by catastrophic forgetting and weight-disentanglement error. In the literature, these difficulties are merely treated separately and mitigated through a variety of solutions, while the geometry induced by the base optimizer is treated as an implementation detail. In this work, we show that the two difficulties are in fact two instances of the same phenomenon: a parameter update useful for one task shifts the model's outputs on another. We formalize this shared phenomenon as \textit{task interference} and reduce it to a common layer-wise Frobenius inner product $\langle ΔW_\ell, J_\ell(x)\rangle_F$. This quantity, in turn, is utilized to expose the role of the optimizer. We theoretically derive an upper bound that isolates the spectral norm $\|ΔW_\ell\|_2$ as an optimizer-controllable factor of task interference, and a per-mode analysis shows that this bound tracks the dominant part of the empirical interference. Specifically, we then identify the recent Muon optimizer as a mechanism that regulates this factor by construction. Our work reveals that its elegant control on spectral norm tightens the interference bound for both CL and MM, positioning Muon as a principled optimizer-centric approach complementary to existing solutions. Our theoretcal analysis is well validated by experimental results. Replacing the AdamW optimizer with Muon improves accuracy by up to +5.02 points on the eight-task model-merging benchmark across three CLIP backbones. For continual learning, Muon also delivers uniformly positive gains across ten class-incremental protocols, three task-incremental protocols, and the 11-task MTIL benchmark.

1 Introduction

The paper unifies catastrophic forgetting in continual learning and weight-disentanglement error in model merging as task interference, then analyzes how optimizer geometry controls it. Muon’s spectral-norm update geometry is linked to tighter interference bounds and positive empirical gains across both paradigms.

  • Motivation: Existing continual-learning and model-merging methods address interference through data access, objectives, architectures, parameter spaces, or merging rules, while optimizer geometry is usually treated as an implementation detail.
  • Unified problem: Catastrophic forgetting and weight-disentanglement error are instances of task interference caused by updates for one task changing outputs on another.The shared layer-wise quantity is the Frobenius inner product ⟨∆Wℓ, Jℓ(x)⟩F.
  • Spectral analysis: The interference decomposition isolates the update spectral norm ∥∆Wℓ∥2 as the factor controlled by training and the optimizer.A per-mode analysis investigates how strongly this factor governs empirical interference.
  • Muon: Muon regulates the optimizer-controllable factor through spectral-norm steepest descent without changing the loss, parameterization, or data pipeline.This provides a single optimizer-level mechanism for both continual learning and model merging.
  • Empirical validation: +5.02, +3.42, and +3.82 points are the absolute accuracy gains from replacing AdamW with Muon on the eight-task model-merging benchmark across ViT-B/32, ViT-B/16, and ViT-L/14.Muon also yields uniformly positive gains across ten class-incremental and three task-incremental protocols, improving all three metrics on the 11-task MTIL benchmark.
  • Empirical validation: The study validates its unified analysis across eight-task model merging and CIL, TIL, and MTIL continual-learning benchmarks.

2 Related Work

Related work treats continual learning and model merging through specialized mitigation strategies, whereas this paper emphasizes the base optimizer’s update geometry as a shared perspective. It positions Muon as distinct from approaches that add task-specific projection or merging mechanisms.

  • Continual Learning: Continual-learning methods include regularization, replay, gradient projection, representation-based approaches, and task-specific parameter allocation.
  • Model Merging: Model-merging methods either design combination algorithms for trained checkpoints or modify fine-tuning to produce more mergeable models.
  • Muon Optimizer: Muon orthogonalizes hidden-layer gradients during each update, encouraging balanced gradient flow across each weight matrix’s singular-value spectrum.Prior work mainly studies it for single-task pretraining and optimization efficiency.
  • Muon Optimizer: Muon-OGD adds orthogonal gradient projection for continual learning, while this work studies how Muon’s base optimizer geometry bounds interference in both continual learning and model merging.

3 The Proposed Unified Framework

The framework unifies catastrophic forgetting and weight-disentanglement error as layer-wise task interference, then identifies spectral-norm control as the optimizer-dependent lever for reducing it. Muon applies this control through spectral-norm steepest descent, while empirical mode analyses show that dominant interference is concentrated in the modes it suppresses.

  • Common setup: CL and MM approach a shared multi-task objective through sequential updates and task-vector composition, respectively.
  • Task interference: Both catastrophic forgetting and disentanglement error reduce to the layer-wise task-interference term ⟨∆Wℓ, Jℓ(x)⟩F.A first-order expansion expresses both output shifts relative to a fixed reference point through this common quantity.
  • Spectral-norm bound: Theorem 1 bounds task interference by ∥Jℓ(x)∥∗·∥∆Wℓ∥2, separating backbone-and-data factors from the optimizer-controlled update spectral norm.The backbone and source-task distribution determine the nuclear-norm factor, whereas training and optimizer choice shape ∥∆Wℓ∥2.
  • Empirical validation: Under AdamW, the top 10% of singular modes account for 63.5% of total interference on ViT-B/32, whereas Muon reduces this fraction to 29.0% with a nearly flat curve.The per-mode analysis supports the bound’s empirical relevance by showing that dominant modes carry much of the interference under AdamW.
  • Muon optimizer: Muon realizes spectral-norm steepest descent and produces updates with uniformly equal singular values, tightening the interference bound across CL and MM.Its update geometry directly matches the spectral norm isolated by Theorem 1.

4 Experiments on Model Merging

Model-merging experiments show that Muon improves merged-model performance and reduces interference without improving single-task experts. Its effects align with higher stable rank, flatter mode contributions, and tighter spectral-norm control.

  • Per-task performance: Figure 6 compares zero-shot, independently fine-tuned, and merged-model accuracies, showing nearly identical experts but substantially expanded merged accuracies under Muon.
  • Muon improves Abs. Acc by +5.02, +3.42, and +3.82 points over AdamW on ViT-B/32, ViT-B/16, and ViT-L/14, respectively.
  • OrthoReg + Muon achieves 78.05%, 82.21%, and 89.83% Abs.Acc. on the three backbones, while TTA + Muon gains +2.73 points on ViT-B/32.
  • Task interference: Muon reduces relative interference on every source task, with EuroSAT decreasing from 1.01 to 0.67.
  • Mode-wise contribution: Under AdamW, the top-10% singular modes contribute 63.5% of interference, compared with 29.0% under Muon.
  • Stable rank: Muon raises mean stable rank from 15.1 to 171.5, about 11×, uniformly across all blocks and datasets.

5 Experiments on Continual Learning

Continual-learning experiments find uniformly positive gains from replacing AdamW with Muon across LoRA, MoE-Adapters4CL, CIL, TIL, and MTIL settings. The results are accompanied by higher stable rank and flatter mode contributions.

  • LoRA baseline: Muon improves Avg. by +4.15 on ImageNet-R 20-task CIL and +4.26 on CIFAR-100 20-task TIL with plain LoRA.
  • LoRA baseline: On CIFAR-100, the LoRA improvement grows with sequence length from +1.69 to +3.39 to +4.26 for 5, 10, and 20 tasks.
  • MTIL: Muon improves Transfer, Average, and Last on the 11-task MTIL benchmark by +0.7, +1.4, and +1.6 points, respectively.
  • Stable rank: On MTIL, Muon raises stable rank from the AdamW range [2.3, 4.4] to [27.0, 36.8], roughly 10× higher.
  • MoE-Adapters4CL: Muon uniformly improves both Avg. and Last across all seven MoE-Adapters4CL CIL protocols.
  • Mode-wise contribution: Under AdamW, mode contributions concentrate on leading modes, whereas under Muon they remain flat; the same Frobenius energy is spread across more effective modes.

6 Conclusion

The conclusion unifies forgetting and disentanglement error as task interference and identifies Muon’s spectral-norm control as a shared mechanism for reducing it. Experiments report uniformly positive gains across continual learning and model merging.

  • The paper unifies catastrophic forgetting and weight-disentanglement error as task interference and bounds it using an optimizer-controllable spectral norm.
  • Muon tightens the interference bound for both paradigms by keeping the spectral-norm factor uniformly small by construction.
  • Experiments show uniformly positive gains from replacing AdamW with Muon across continual learning and model merging.
  • Future work will extend the interference bound beyond the NTK regime and test the mechanism on billion-parameter foundation models and longer task sequences.

A Detailed Proofs

The proofs establish optimizer-independent control of the Jacobian nuclear norm, derive the spectral-norm steepest-descent solution, and explain Muon’s approximate implementation through Newton–Schulz iterations.

  • Optimizer-independent Jacobian bound: The Jacobian nuclear norm is bounded using local Lipschitz continuity and a bounded fine-tuning radius around the pretrained parameters.
  • Optimizer-independent Jacobian bound: The bound’s right-hand quantities depend on the backbone, architecture, activations, and matched fine-tuning protocol rather than the optimizer.
  • Spectral-norm steepest descent: Spectral–nuclear duality identifies the polar factor U V^⊤ as the maximizer of the Frobenius pairing under spectral-norm constraint.
  • Spectral-norm steepest descent: For updates written as ∆Wℓ = −c · T, optimization selects T⋆ = UℓV⊤ and c⋆ = ∥Gℓ∥∗/λ.
  • Muon update: Muon uses Nesterov-smoothed gradients followed by Newton–Schulz orthogonalization, approximating spectral-norm steepest descent for hidden-layer matrices.
  • Newton–Schulz analysis: Newton–Schulz iterations preserve singular vectors and drive singular values toward unity, with practical K = 5 to 10 steps reaching within 10−3.
  • Relation to continual learning: The paper frames Muon’s orthogonal update structure as a theoretical mechanism for controlling catastrophic forgetting, extending prior single-task optimization analyses.

C More Details of Model Merging Experiments

The model-merging experiments evaluate optimizer effects across diverse tasks, backbones, and pre-merging strategies. Muon is paired with existing methods to isolate optimizer-level spectral control from parameterization- and loss-side mechanisms.

  • Benchmark: The eight-task benchmark spans scene, fine-grained, remote-sensing, satellite, digit, traffic-sign, and texture classification across largely disjoint visual domains.Standard train, validation, and test splits are used.
  • Setup: Experiments use CLIP-pretrained ViT-B/32, ViT-B/16, and ViT-L/14 backbones with frozen zero-shot classification heads and fine-tuned image encoders.Task vectors are directly comparable across tasks and optimizers under this protocol.
  • Compared methods: Muon is compared with AdamW across non-linear fine-tuning, Tangent Task Arithmetic, and OrthoReg.These methods represent optimizer-, parameterization-, and loss-side approaches to improving mergeability.
  • Experimental design: Pairing each pre-merging strategy with Muon isolates optimizer-level spectral control and tests whether its predicted improvement is orthogonal to existing mechanisms.The comparison keeps the pre-merging strategy fixed while changing the optimizer.

Merging Coefficient and Metrics

The experiments evaluate merged models with shared scaling coefficients and two complementary accuracy metrics. Muon reaches a higher peak and is less sensitive to coefficient misspecification than AdamW on ViT-L/14.

  • Merging coefficient: The shared merging coefficient is selected from {0, 0.05, . . . , 1.0}, with α = 0.25 for ViT-B/32 and ViT-B/16 and α = 0.35 for ViT-L/14.The same selected α is used across optimizers.
  • Metrics: Absolute Accuracy is the mean top-1 accuracy across eight tasks, while Normalized Accuracy averages each merged-model accuracy relative to its single-task expert.Normalized Accuracy factors out the raw difficulty of individual datasets and reflects preserved single-task capability.
  • Sensitivity: Muon reaches a higher peak than AdamW and degrades more gracefully as α moves away from the optimum on ViT-L/14.The sensitivity curve is visibly flatter for Muon.

D More Details of Continual Learning Experiments

Continual-learning experiments compare Muon with plain LoRA and MoE-Adapters4CL while holding the architecture and training conditions fixed. The baselines represent minimal and architectural approaches to interference mitigation.

  • Choice of base methods: Muon is evaluated on plain LoRA and MoE-Adapters4CL to isolate optimizer effects from architectural mechanisms.The two baselines lie at opposite ends of the interference-mitigation spectrum.
  • LoRA baseline: Plain LoRA uses low-rank adapters in every attention and MLP projection while freezing the backbone, with no rehearsal, routing, regularization, or gradient projection.It is evaluated in both class-incremental and task-incremental protocols.
  • MoE baseline: MoE-Adapters4CL suppresses interference through DDAS distribution selection, top-2 expert routing, and incremental activate–freeze training.It is described as a strong parameter-efficient baseline for class-incremental learning and MTIL.
  • Controlled comparison: For both baselines, “+ Muon” changes only the optimizer for matrix-valued adapter parameters while preserving architecture, data order, training budget, and relevant DDAS checkpoints.This design attributes differences to optimizer choice rather than other experimental changes.

Benchmarks and Protocols

The continual-learning evaluation covers class-incremental, task-incremental, and multi-domain protocols spanning heterogeneous visual domains. Experiments use frozen CLIP backbones and compare controlled optimizer assignments for trainable matrix and non-matrix parameters.

  • Multi-domain task-incremental learning: The 11-task MTIL sequence includes Aircraft, Caltech101, CIFAR-100, DTD, EuroSAT, Flowers, Food-101, MNIST, Oxford-IIIT Pet, Stanford Cars, and SUN397.The sequence spans fine-grained, object, texture, remote-sensing, digit, food, and scene recognition.
  • Metrics: Transfer measures zero-shot generalization, Average is the mean of all matrix entries, and Last emphasizes retention after the complete task sequence.These metrics characterize performance before observation, overall performance, and final-sequence retention.
  • Backbone: All continual-learning experiments use a CLIP-pretrained ViT-B/16 image encoder with frozen backbone parameters and frozen text-derived zero-shot classification heads.This follows the standard MoE-Adapters4CL protocol.
  • Optimizer assignment: Muon optimizes eligible matrix-valued parameters, while auxiliary AdamW handles vector and scalar parameters; the AdamW baseline uses one AdamW instance for the same parameter set.The Muon branch uses five Newton–Schulz iterations, cosine learning-rate decay, and zero weight decay in the MoE configuration.
  • DDAS control: AdamW and Muon share the same Adam-trained DDAS checkpoints, so optimizer comparisons exclude differences from task-identification checkpoints.DDAS uses TinyImageNet as its reference dataset with threshold 0.0655.

Results

Muon produces consistent continual-learning gains across LoRA and MoE-Adapters4CL protocols, including longer task sequences and the 11-task MTIL benchmark. The improvements persist when Muon is added to a framework that already suppresses interference.

  • LoRA4TIL: Muon improves LoRA4TIL Avg./Last by +1.69/+1.64, +3.39/+1.79, and +4.26/+3.20 at 5, 10, and 20 tasks on CIFAR-100.The widening gains with sequence length are reported as evidence that spectral step normalization benefits longer task sequences.
  • LoRA4CIL: Muon improves LoRA4CIL Avg. by +2.12 and +4.15 and Last by +3.90 and +3.51 at 10 and 20 tasks on ImageNet-R.CIFAR-100 and TinyImageNet also show consistently positive improvements across tested splits.
  • MoE-Adapters4CL: Muon improves MoE-Adapters4CL Avg./Last across CIFAR-100 and TinyImageNet task-length protocols, including +0.77/+0.53, +0.88/+0.96, and +0.69/+0.18 on TinyImageNet.These gains occur on top of top-2 expert routing and incremental activate–freeze, which already mitigate interference.
  • Multi-Domain Task-Incremental Learning: Muon improves all three mean MTIL metrics over AdamW: Transfer by +0.6, Average by +1.4, and Last by +1.6 points.The largest reported single-dataset improvement is +11.8 points on all three Aircraft metrics.
  • Overall: Across both baselines and all protocols, Muon provides a systematic continual-learning benefit attributable solely to changing the optimizer.Architecture, data order, training budget, and DDAS checkpoints are held constant.
Loading 2608.27518v1…