PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S Dhillon, Devvrit Khatri

arXiv:2610.22293v1cs.LGcs.AI

TL;DR

Fine-tuning can erase capabilities acquired during pretraining, and prior work associates forgetting with the directions in which weights are updated. SST projects the final optimizer update into the non-principal subspace; it preserves prior capabilities while maintaining target-task accuracy, with retention strengthening at larger model sizes. The authors identify fully tuned learning–retention frontiers and scaling to much larger models and longer task sequences as future work.

  • Problem

    Fine-tuning can severely degrade capabilities acquired during pretraining, while the original pretraining data is typically unavailable and post-hoc repair may recover only part of what was lost.

    Source: §1 · p. 2

  • Method

    SST projects the final optimizer update into a weight matrix’s non-principal subspace, removing update components along principal directions.

    Source: §1 · p. 2

  • Results

    Despite protecting the top 99% of singular directions, SST matches unconstrained fine-tuning’s target-task accuracy while retaining prior capabilities; it also induces 2–4× less KL divergence than unconstrained recipes.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Across sequential fine-tuning chains, SST lets trained skills hold their level as later stages are learned, whereas SFT degrades dormant skills.

    Source: §1 · p. 3

    Fully tuned learning–retention frontiers remain future work, as do scaling SST to models with hundreds of billions of parameters and to longer task sequences.

    Source: §5 · p. 11

Abstract

from arXiv · show

Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99\% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior strengthens with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing $2$--$4\times$ less KL drift than the baseline recipes. Overall, SST advances the learning--forgetting Pareto frontier, making it a practical default for fine-tuning pre-trained language models.

1 Introduction

Fine-tuning can erase capabilities built during pretraining, and prior work links forgetting to where weight updates land. This section introduces Subspace Tuning (SST), which protects principal singular directions while enabling new-task learning and stronger retention across settings.

1 Introduction

  • Prior work links principal singular directions to model functionality and reports that on-policy methods update outside these directions and forget less than SFT.These observations motivate preserving principal singular subspaces while leaving capacity elsewhere for new-task learning.Source: §1 · p. 2
  • SST restricts weight updates to a non-principal subspace and projects the final optimizer update, making the applied step stay outside principal directions.Projecting the applied update makes SST optimizer-agnostic and compatible with fine-tuning recipes without other training changes.Source: §1 · p. 2
  • Despite protecting the top 99% of singular directions, SST matches unconstrained fine-tuning’s target-task accuracy while retaining prior capabilities across domains, recipes, and optimizers.The reported settings include Science, Medicine, Pathology, Finance, and Tool Use; SFT and SDFT; and Adam and Muon.Source: §1 · p. 2
  • With larger models, SST forgetting stays within ∼1% of the base model from 8B onward, while SFT forgets progressively more with training data.SST also reaches higher target-task accuracy at every model size shown in Figure 2.Source: §1 · p. 2, Figure 2 · p. 3
  • SST induces 2–4× less KL divergence from the pretrained model than unconstrained recipes, while preserving behaviors that accuracy alone does not capture.Examples include natural reasoning length under SST where SFT collapses it, and fewer fabricated reference-solution attributions when SST is added to SDFT.Source: §1 · p. 3
  • Across two- and three-stage fine-tuning chains, SST lets each trained skill hold its level as later stages are learned, unlike SFT, which degrades dormant skills.The paper concludes that forgetting depends on where the update lands in weight space, rather than being an inevitable consequence of learning.Source: §1 · p. 3

2 Subspace Tuning

Because leading singular directions are associated with important functionality, SST protects them while adapting in the remaining subspace. It formalizes two-sided projection, applies it after optimizer updates, and refreshes the protected basis to trade retention for adaptability.

2.1 Projection Formulation

  • With a matched 45k-example budget, SST stays flat while SFT drifts under both fresh-data and repeated-data exposure, and SST learns the task better.The repeated-data arm cycles 9k examples for five epochs.Source: Figure 3 · p. 4
  • Two-sided projection keeps only the free-to-free update block, discarding blocks that touch protected directions and preserving protected singular triplets exactly.The protected subspace comprises the top-k singular directions; choosing a rank fraction near 1 leaves only the low-energy tail free.Source: §2.1 · pp. 3–4

2.2 Projecting the Optimizer Update

  • 0.13 held-out NLL increase accompanied exact leakage elimination under applied-update projection, versus 69% leakage for gradient projection.In the Qwen2.5-0.5B Adam study, re-projecting the cumulative change eliminated leakage but increased held-out NLL by 3.28; unconstrained fine-tuning increased it by 13.0.Source: Figure 4 · p. 5, §2.2 · p. 5

2.3 Refreshing the Protected Basis

  • Refreshing the basis every K optimizer steps opens new adaptation directions and can improve task learning, but mildly weakens protection relative to the original model.Refresh re-anchors to current weights and recomputes the protected singular vectors; K controls the adaptation–retention trade-off.Source: §2.3 · p. 5

3 Experiments

Across the tested recipes and domains, SST improves retention while preserving target-task learning, and its retention benefits persist as model and data scale increase and across successive fine-tuning stages.

3 Experiments

  • The experiments ask whether restricting updates to the non-principal subspace enables new-task learning while retaining prior knowledge, measuring in-distribution accuracy and average performance on held-out general-purpose benchmarks.The evaluation includes Qwen3-8B tasks in Science, Finance, Pathology, and Medicine, with SFT and SDFT baselines and SST variants.Source: §3 · p. 6

3.1 SST learns without forgetting

  • SFT and SDFT broadly reduce prior capability, while post-hoc KL repair recovers only part of it and reduces in-domain gains.The per-benchmark figure compares prior-capability changes against the base model; SST variants stay closer across benchmarks and domains.Source: §3.1 · p. 6, Figure 6 · p. 7, §3.3 · p. 7
  • SST stays close to the base model while matching the in-domain performance of unconstrained recipes across domains, and R10 preserves target-task learning with a small retention cost.The reported results also find that SST composes with SDFT and Muon; more frequent refresh favors adaptation at a small retention cost.Source: §3.1 · pp. 6–7, Figure 5 · p. 6

3.2 SST scales well

  • As model size grows, SST preserves the base model better while remaining competitive with or improving on SFT for the new task; with more training data, it maintains prior capability while target-task performance improves.The data-scaling finding holds for both a novel stream and a 9k-example subset repeated for five epochs under a 45,000-example budget.Source: §3.2 · p. 7

3.3 SST stays close to the base model

  • Across training-data and model-scale studies, SST remains substantially closer to the base distribution than unconstrained models, with 2–4× less KL drift across model sizes and recipes.In the answer-only medical setting, SST keeps reasoning length close to the base model, and adding SST to SDFT substantially reduces fabricated reference-solution attributions.Source: §3.3 · pp. 7–8, Figure 7 · p. 8, Figure 8 · p. 8

3.4 SST preserves gains across successive fine-tuning stages

  • Across tested two-task chains, SST+R10 outperforms SFT in both retention of the first task and learning of the second.The study also compares independently trained experts: SST experts can be merged with far less interference than SFT experts.Source: §3.4 · pp. 8–9, Figure 9 · p. 9
  • In the FinQA → Pathology → Science chain, each SST-trained domain remains near its trained level through later stages, while plain SFT loses 28 and 18 points on untrained Pathology and Science during FinQA training and gives back 9 FinQA points later.More frequent refresh improves current-task accuracy at a minor cost to retention on previously learned tasks.Source: §3.4 · p. 9, Figure 10 · p. 9

4 Related Work

Prior approaches reduce forgetting through data or by protecting parts of the weight spectrum, but differ from SST in what they constrain and how they adapt. SST constrains the applied weight update and adds subspace refresh; comparisons report stronger retention than Sculpting and OPLoRA at comparable in-domain accuracy.

4 Related Work

  • Classical continual-learning constraints rely on stored data or earlier-task gradients and task boundaries, while SST needs neither; self-distillation and on-policy RL instead induce favorable update geometry through on-policy data.SST constrains weight updates and can compose with on-policy methods.Source: §4 · pp. 9–10

4.1 Comparison with subspace-protecting methods

  • Sculpting projects gradients while freezing top singular vectors, whereas OPLoRA trains a low-rank adapter between fixed projectors; SST trains weights directly and constrains the applied update.Sculpting omits two cross-block components, and even removing all three blocks from its gradient leaves 69% of the applied update in protected directions; OPLoRA adapts at low rank while SST protects 99% of the spectrum and adapts densely in the tail.Source: §4.1 · p. 10
  • SST adds refresh by re-anchoring the protected basis every K steps, an explicit retention–adaptation control swept from no refresh to R10.In the four-domain comparison, Sculpting and OPLoRA lose 18–26 average prior-capability points versus about one for no-refresh SST at comparable in-domain accuracy; OPLoRA still trails SST on three of four domains when given SST’s protection level.Source: §4.1 · p. 10

5 Conclusion

SST confines optimizer steps to the non-principal weight subspace, preserving prior capabilities while learning new ones. The appendix extends this picture to related methods, expert merging, and two- and three-stage SDFT chains.

5 Conclusion

  • Across recipes, model sizes, and data scales, SST confines applied updates to the non-principal subspace, retains prior capability, and makes sequential fine-tuning cumulative.The conclusion reports application across SFT, SDFT, and Muon, from 1.7B to 32B parameters; it also notes 2–4× closer KL to the base distribution.Source: §5 · p. 10

A Extended Related Work

  • Prior continual-learning methods estimate protected subspaces from data, while SST uses the weight spectrum and requires neither stored examples nor task boundaries.Unlike parameter-efficient spectral methods that encode their choice in a fixed low-rank factorization, SST projects the dense optimizer update after stateful preconditioning and can refresh its protected basis.Source: A Extended Related Work · p. 17

B Additional Experiments & Details

  • The appendix adds model-merging, Qwen2.5, scaling, Qwen3, policy-drift, toy-study, and comparison results, alongside algorithm and evaluation details.It also lists reverse-KL and protected-rank controls.Source: B Additional Experiments & Details · p. 17

B.1 Model Merging

  • SST experts interfere less when merged: pairwise merges stay close to their single-domain experts, and three-expert degradation grows more slowly than for SFT.The experiment compares merged models with their own single-domain experts; no merge coefficients are tuned.Source: §B.1 · p. 17, Figure 11 · p. 18

B.2 Two-stage continual learning with SDFT

  • Across two-stage chains, SDFT improves target learning over SFT but has chain-dependent retention, while SST+R10 gives the most consistent retention–learning trade-off.Plain SDFT loses 20.0 points on FinQA → Science; Figure 12 describes the source-task loss and target-task gain shown for the chains.Source: Figure 12 · p. 18, §B.2 · p. 18

B.3 Three-stage continual learning with SDFT

  • In the Medicine → Pathology → Science chain, plain SDFT degrades Medicine and Science performance, while SST stabilizes the trajectories and R10 accumulates learning across all three domains.Figure 13 tracks all three domains across the stages, with dotted lines marking base-model accuracy.Source: §B.3 · p. 18, Figure 13 · p. 19

B.4 Instruct tuning without forgetting

On Qwen2.5-7B-Instruct, SFT improves target-task performance while reducing prior capability, and SDFT is not consistently protective across domains. Adding SST recovers much of the lost capability on Tool Use and Medical while remaining competitive on in-domain accuracy.

B.4 Instruct tuning without forgetting

  • The ToolAlpaca, HuatuoGPT-o1, and SciKnowEval setup uses r=0.99 except for Science at r=0.90, because the free tail at r=0.99 was too small to fit that task.This was the only paper setting requiring a looser protected rank.Source: §B.4 · p. 19
  • SFT improves target-task performance but reduces prior capability, with the clearest losses in instruction following and knowledge; SDFT is not consistently protective across domains.The setup uses a non-thinking instruction model with shorter sequences, where forgetting is less severe.Source: §B.4 · p. 19
  • Adding SST recovers much of the lost prior capability on Tool Use and Medical while remaining competitive on in-domain accuracy.Science is a narrower trade-off: SST variants can improve task learning, but no single variant dominates both learning and retention.Source: §B.4 · p. 19

B.5 Scaling Experiments

Across model and data scaling comparisons, SST preserves more prior capability than unconstrained tuning while remaining competitive on the target task. Its retention advantage strengthens with model size, while SFT drifts steadily under repeated and novel exposure.

B.5 Scaling Experiments

  • SST preserves substantially more prior capability than unconstrained tuning while remaining competitive on the target task, and its retention advantage strengthens with model size.The matched-budget data comparison shows SFT drifting steadily while SST remains stable and continues learning under repeated and novel exposure.Source: §B.5 · p. 19

B.6 Qwen3-8B thinking: full result tables

The full Qwen3-8B thinking-model sweep reports that adding SST keeps the model closer to baseline while retaining strong target-task performance. Across model sizes, SST also remains closer to the pretrained base distribution than SFT.

B.6 Qwen3-8B thinking: full result tables

  • Across Science, Finance, Pathology, and Medicine, adding SST keeps the model closer to baseline while retaining strong target-task performance.Table 4 reports the per-recipe and per-benchmark values.Source: §B.6 · p. 19
  • Table 1 reports Qwen2.5-7B-Instruct results with ID accuracy, a five-benchmark LM average, and deltas relative to the pretrained baseline of 68.5.Table 2 gives five-epoch Science fine-tuning results by model size, with LM averages and deltas relative to each model’s own base.Source: Table 1 · p. 20, Table 2 · p. 21

B.7 Policy drift across model sizes

  • Across model sizes, SST remains substantially closer to each pretrained base distribution than SFT on final-checkpoint WikiText-2 KL divergence, including at the largest scale.Figure 14 reports the exact drift values for each model.Source: §B.7 · p. 21

B.8 Toy study: full specification

In a controlled toy study, all enforcement schemes learn the task, but projecting the optimizer’s applied update is the only scheme reported to eliminate protected-subspace leakage without substantially changing held-out capability.

B.8 Toy study: full specification

  • In the controlled setup, all schemes learn the toy task, but only projecting the optimizer’s applied update eliminates protected-subspace leakage without substantially changing held-out capability.Source: §B.8 · p. 21
  • The toy task uses next-token cross-entropy on completion tokens from eight fixed prompt–completion pairs, with prompt tokens masked.The task is deliberately easy, so the discriminator is forgetting on out-of-domain tasks.Source: §B.8 · p. 21
  • SST leaves the optimizer untouched and projects its proposed applied step into the allowed free subspace; parameters outside constrained matrices remain optimizer-controlled.Projecting only the gradient can permit leakage because Adam’s coordinate-wise rescaling does not commute with the projection.Source: §B.8 · p. 24

B.9 Comparison with subspace-constrained baselines

SST preserves more prior capability than factorised subspace baselines across the reported domains, while refresh trades some retention for stronger task learning. Reverse-KL regularisation does not provide a reliable learning–retention trade-off.

B.9 Comparison with subspace-constrained baselines

  • Sculpting and OPLoRA reduce forgetting relative to SFT but remain below no-refresh SST in prior capability across four domains; refresh shifts SST toward stronger task learning at some retention cost.The comparison uses method-specific official settings with shared datasets and evaluation; Table 5 reports exact domain-level endpoints.Source: §B.9 · p. 26, Table 5 · p. 28
  • On Medicine and Pathology, both SST variants preserve substantially more prior capability than Sculpting and OPLoRA; on FinQA, no-refresh SST is closest to base while R10 reaches higher target accuracy in both exposure regimes.Medicine and Pathology use 9,000 examples repeated for five epochs; FinQA compares matched 6,000-example exposure regimes.Source: §B.9 · p. 26, Figure 15 · p. 27

B.10 KL regularisation does not prevent forgetting

  • Reverse-KL regularisation gives no reliable learning–retention trade-off: weak regularisation leaves forgetting essentially unchanged, while stronger regularisation can reduce target-task learning without restoring prior capability.The effect varies across tasks, unlike the consistent retention from constraining the update subspace.Source: §B.10 · p. 28, Table 6 · p. 28

B.11 Rank ablation on FinQA

The rank sweep links greater subspace protection to stronger retention, with target accuracy declining at the highest rank. SST is more robust than SFT across tested optimizer choices, and its added cost depends on refresh frequency.

B.11 Rank ablation on FinQA

  • Prior capability improves monotonically as the protected rank fraction increases, with held-out benchmarks moving together rather than one task driving the average.The FinQA sweep tests r values of 0.1, 0.5, and 0.9 against the same base control and plain SFT run.Source: §B.11 · pp. 28–29
  • In-domain accuracy improves across the rank sweep until r = 0.99, where target-task accuracy declines; lower-r projections can also disrupt general behavior when the projection matrix is dated.The reported example of disruption is response-length collapse.Source: §B.11 · p. 29

B.12 Effect of Optimizer Hyperparameters on SST

  • Across six learning-rate and schedule configurations, SST is more robust than SFT and exceeds SFT in-domain accuracy in four, but SFT has the highest individual scores at the lowest learning rate.R10 shifts the balance toward higher average in-domain accuracy at some cost to retention robustness.Source: §B.12 · pp. 29–30

B.13 Computational Cost

  • Projection adds 26.2 ms per step (1.5%) in the measured Qwen3-8B setup; refresh overhead is 5.1% for R100 and 37.6% for R10.Splitting SVD computation across eight GPUs reduces those total overheads to 0.6% and 4.7%, respectively.Source: §B.13 · p. 31

B.14 Training and evaluation details

The Qwen3 and Olmo studies share training settings across paired recipe/SST comparisons unless an official baseline recipe specifies otherwise. The appendix describes the datasets and evaluation procedures used for the Qwen2.5 and medical studies.

B.14 Training and evaluation details

  • Unless an official baseline recipe applies, Qwen3 and Olmo studies use a 5×10^-5 learning rate and effective batch size 16, shared across paired recipe/SST comparisons.Training is full-parameter apart from the factored Sculpting and OPLoRA baselines.Source: §B.14 · p. 31
  • The Qwen2.5 instruction study uses ToolAlpaca, HuatuoGPT-o1, and SciKnowEval; Qwen3 Medicine and Pathology use MedMCQA-derived traces.The archived medical sets contain 8,996 Medicine traces and 8,999 Pathology traces.Source: §B.14 · p. 31
  • Prior capability is the average of GPQA-Diamond, MMLU-Pro, GSM8K, HumanEval, and IFEval, while in-domain accuracy uses each task’s native answer extractor.Olmo-3 medical open-ended accuracy is judged by a Bedrock model, and its fabrication metric flags explicit absent-source claims.Source: §B.14 · p. 31

Related papers

Loading 2610.22293v1…