PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Support Learning

Assaf Ben-Kish, Akarsh Kumar, James Glass, Raja Giryes

arXiv:2610.02126v1cs.LGcs.AI

TL;DR

Finetuning can overwrite previously learned capabilities, and scalable retention without prior data remains a challenge. LSL pairs a weight adapter with a support-estimating gate to localize updates to current-phase activations. On LLMs up to 7 billion parameters, it retains pretrained and finetuned capabilities across phases while learning at full capacity.

  • Problem

    Finetuning can overwrite capabilities acquired during pretraining and earlier phases, while scalable retention without prior data remains an open problem.

    Source: §1 · pp. 1–2

  • Method

    LSL pairs a standard adapter with a GMM-based gate that applies updates only to activations within the current phase’s data support.

    Source: §1 · p. 2

  • Results

    LSL learns at full capacity on LLMs up to 7 billion parameters while retaining pretrained and previously finetuned capabilities across phases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Localizing adapter updates offers a way to retain prior capabilities without access to prior data.

    Source: §1 · p. 2

    Inference cost grows linearly with the number of phases because adapters cannot be merged into the base weights, though computation is parallelizable.

    Source: §6 · p. 10

Abstract

from arXiv · show

We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.

1 INTRODUCTION

Finetuning can overwrite capabilities acquired during pretraining and earlier phases, while scalable retention remains an open problem. Local Support Learning (LSL) confines adapter updates to activations from the current phase’s data and retains capabilities across phases in models up to 7 billion parameters.

1 INTRODUCTION

  • Gradient-based updates can alter prior inputs that are not orthogonal to the update, motivating a retention objective that confines changes to the current phase’s input support.The paper identifies the exact definition of change and how to prevent it as open problems.Source: §1 · p. 2
  • LSL pairs a standard loss-minimizing weight adapter with a gate that applies it only to activations within the current phase’s data support.Its GMM-based gate uses rapidly decaying likelihood to tend to stay closed on inputs it has not seen.Source: §1 · p. 2
  • LSL retains pretrained and previously finetuned capabilities while learning at full capacity across multiple phases, including on LLMs up to 7 billion parameters.Retention is largely unaffected by hyperparameter choice, and the method uses no more memory than a LoRA adapter.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses forgetting through replay, regularization, optimization constraints, or limits on model change, but large-scale pretraining creates practical constraints for these approaches. LSL instead localizes updates to the activations from their training distribution.

Catastrophic Forgetting

  • Replay buffers, data-dependent regularization, and optimization constraints are earlier mitigation strategies, but auxiliary mechanisms scale poorly for retaining large-scale pretraining capabilities.The paper frames the precise definition of model change and how to prevent it as open questions.Source: §2 · p. 3

Catastrophic Forgetting in Large Pretrained Models

  • LoRA can forget less than full fine-tuning but reduces learning capacity, while weight-subspace constraints and orthogonal adapters have heuristic or phase-specific limits.LoRA: Hu et al. (2021); weight-subspace constraints: Xiong & Xie (2025) and Nayak et al. (2025); orthogonal adapters: Wang et al. (2023) and Liang & Li (2024).Source: §2 · p. 3
  • LSL differs from methods that apply gradient updates to all inputs by gating each update to activations from its training distribution.Learning without Forgetting instead penalizes deviations from base-model outputs on finetuning data via knowledge distillation (Hinton et al., 2015).Source: §2 · p. 3

3 FORMULATION AND PROBLEM INVESTIGATION

In a two-phase toy classification problem, ordinary finetuning learns new classes while deforming earlier decision boundaries; local-support updates preserve the earlier behavior. The analysis motivates estimating the current phase’s support, while a parametric gate avoids the token-count scaling of a local estimator.

3 FORMULATION AND PROBLEM INVESTIGATION

  • Continual learning proceeds phase by phase with bounded retained memory, avoiding the impractical replay-buffer solution of storing and retraining on all earlier data.Evaluation averages performance across tasks from all phases.Source: §3 · p. 3
  • Ordinary finetuning correctly classifies the two new classes but heavily deforms earlier decision boundaries, causing severe test loss on pretraining data.The toy setup uses six classes in two phases: four for pretraining and two for finetuning.Source: §3 · p. 4
  • A gradient update changes prior-sample logits unless that input is orthogonal to the training sample, because the update operates on all inputs.This applies to gradient-based optimizers including Adam and Muon.Source: §3 · p. 4
  • LSL estimates the region occupied by current-phase data and gates the adapter there, unlike prior-data orthogonality methods that become infeasible at pretraining scale.An always-open gate corresponds to standard finetuning and makes the update global.Source: §3 · p. 4
  • A gate’s error separates into deficit and excess: training optimizes deficit but cannot observe excess, which measures when the gate stays open outside the training distribution.The paper finds that density estimators can recover optimal gates under explicit conditions.Source: §3 · p. 4
  • The local-support estimator adapts fully to finetuning data with zero interference on pretraining data, preserving a perfect classifier; its parametric successor scales with phases rather than tokens.The initial estimator’s complexity scales with the number of tokens encountered, whereas the proposed efficient parametric gate scales with the number of phases.Source: §3 · p. 5

4 METHOD

Local Support Learning gates each weight-matrix adapter with GMMs that distinguish current-phase inputs from a generic pretraining reference distribution. Exponential moving-average smoothing adds temporal continuity to token-level gate decisions, while training fits phase-specific adapters and gates for inference-time routing.

Gate Design

  • Each weight-matrix gate compares likelihoods from a current-phase GMM and a generic-pretraining GMM, applying the adapter when the former is higher.The negative GMM avoids requiring a manually tuned density threshold.Source: §4 · p. 5

Intuition for Φneg

  • The generic reference GMM approximates distribution width rather than full pretraining support; the gate opens on 16% of out-of-distribution tokens and retains 96.6% of pretrained performance.With temporal smoothing, these figures are 5% and 98.8%, respectively.Source: §4 · p. 5

Temporal Smoothing

  • An exponential moving average smooths independent token-level GMM decisions; the GMM raises retention from 76.6% to 96.6%, and smoothing raises it to 98.8%.The authors describe smoothing as a lightweight complement to the GMM.Source: §4 · p. 6

Local Support Learning

  • Training fits Φneg once, then fits each phase’s adapter and Φpos; inference applies adapters using smoothed gate decisions and is parallelizable during prefill.Figure 3 presents inference per weight-matrix module and training for the full model.Source: Figure 3 · p. 5, §4 · p. 6

5 EXPERIMENTAL RESULTS

Across post-training settings, LSL learns new tasks while retaining pretrained capabilities, and it preserves gains across sequential finetuning phases. Ablations support local support as the key retention property, with per-matrix gates and GMM efficiency providing additional benefits.

5.1 POST-TRAINING WITH LOCAL SUPPORT LEARNING

  • LSL maintains close-to-optimal retention while learning new tasks, outperforming LoRA, OP-LoRA, and LwF across the evaluated benchmarks.The comparison evaluates Qwen2.5-7B-Instruct on cybersecurity, Igbo translation, and chemistry adaptation, alongside three pretrained-capability benchmarks.Source: §5.1 · p. 6, Figure 4 · p. 7
  • Across sequential phases, LSL preserves pretrained capabilities and prior finetuning gains, while LoRA loses task performance after phases end.LoRA translation drops 8% after two phases, chemistry drops 5% after one, and zero-shot cybersecurity drops 11% after translation.Source: Figure 5 · p. 7, §5.1 · p. 7
  • From 1.5B to 7B, LSL retention improves with model size and remains above LoRA, whose improvement is slower.Retention is measured as pretraining-task performance before versus after finetuning.Source: §5.1 · p. 8, Figure 7 · p. 8
  • On Qwen2.5-7B-Instruct with an Nvidia B200, LSL adds 13.8MB for GMMs, 86ms inference latency per pass, and 1.64× total training time.The GMM memory overhead is 6.9MB per phase; training time excludes one-time Φneg fitting, and neither inference nor training was fully optimized.Source: §5.1 · p. 8, Figure 8 · p. 8

5.2 ABLATIONS

  • GMM and Union of Spheres gates achieve near-optimal retention at full learning capacity, whereas the non-local MLP gate lags in retention.The MLP has a slight in-distribution accuracy edge but degrades on all three OOD pretraining tasks; UoS requires about 5.67GB, versus 13.8MB for the GMM.Source: §5.2 · p. 9, Figure 9 · p. 9
  • A single token-level router recovers only 50–77% of LSL’s new-task improvement and also degrades cybersecurity retention.Per-matrix gates use contextual features at each depth, and a routing error affects only its own matrix.Source: §5.2 · p. 9, Figure 10 · p. 9
  • With two GMM components, LSL achieves near-optimal retention; temporal smoothing raises retention from 69.8 to 71.1, while the GMM raises it from 55.3 to 69.8.The respective base-model-relative scores are 96.6% to 98.8% for smoothing and 76.6% to 96.6% for the GMM.Source: §5.2 · p. 10
  • On in-distribution data, LSL and LoRA change model behavior similarly, but LSL changes it much less on OOD pretraining benchmarks.The comparison groups per-token output distributions by the fraction of LSL gates each token opens.Source: §5.2 · p. 10

6 CONCLUSION

LSL localizes updates to activations from each phase’s training distribution, retaining pretrained and previously finetuned capabilities in models up to 7 billion parameters. The conclusion also highlights efficiency and robustness, while noting practical and theoretical limitations.

6 CONCLUSION

  • Across sequential phases, LSL learns at full capacity while retaining pretrained capabilities and previously finetuned skills in LLMs up to 7 billion parameters.The conclusion also reports robustness to hyperparameter choice and a small memory footprint independent of training-token count.Source: §6 · p. 10
  • Adapters cannot be merged into base weights, so inference cost grows linearly with phases; the theory is single-matrix, and GMM gates rely on the quality of base-model representations.Many tasks can be learned jointly within one phase, and the phase computations are highly parallelizable.Source: §6 · p. 10

AI USE STATEMENT

  • Generative AI tools assisted with implementation, related-work discovery, manuscript fluency, and mathematical claims; authors reviewed the work and verified code correctness.The authors state that AI tools did not generate data or run experiments.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The authors state that continual-learning methods, including LSL, inherit limitations of their pretrained model and training data, and call for careful evaluation before deployment beyond research settings.The ethics statement anticipates positive impact through greater model efficiency and reduced energy consumption.Source: ETHICS STATEMENT · pp. 10–11

REPRODUCIBILITY STATEMENT

  • The paper provides source code, experiment configurations, datasets, models, hardware, and the metrics used in each experiment..Source: REPRODUCIBILITY STATEMENT · p. 11

A EXTENDED RELATED WORK

  • Replay methods protect capabilities represented in replayed samples, while LSL trains the whole model on full finetuning datasets; related editing methods target individual facts at one layer.The cited RL-based approaches are confined to RL settings and require in-context learning and a full rollout at each step; they target training objectives and data, making them complementary to LSL’s architectural approach.Source: A EXTENDED RELATED WORK · p. 16

B COMPLEXITY AND ACCELERATION

  • LSL adds one GMM per weight matrix per phase; projection to dimension E ≪ D and diagonal-covariance GMMs reduce memory to O(PLKE + LDE) and time to O(PBL(DE + KE)).Memory per phase is independent of token count, but inference remains linear in phases and fitting GMMs on large datasets may need acceleration.Source: B COMPLEXITY AND ACCELERATION · pp. 16–17

C EXPLAINING OP-LORA’S PERFORMANCE

The section examines why OP-LoRA performs similarly to LoRA by measuring how much LoRA update energy lies in a pretrained-weight subspace. It finds that the measured overlap is small, suggesting forgetting is only weakly related to preserving that subspace.

C EXPLAINING OP-LORA’S PERFORMANCE

D IMPLEMENTATION DETAILS

The implementation section specifies experiment sweeps, model and software choices, and the configurations used for single-task and multiphase evaluations. It also describes baseline-specific training procedures.

D IMPLEMENTATION DETAILS

  • Hyperparameters were selected by sweeping learning rates {3e−6, 1e−5, 3e−5, 1e−4, 3e−4}, batch sizes {8, 16, 32, 64}, and adapter ranks {8, 32, 128, 512}; GMM runs swept K ∈ {16, 32} and α ∈ {0.1, 0.3, 1.0}.Each experiment is averaged over three seeds; GMM runs use the optimal LoRA parameters because the LoRA-learning phase is identical.Source: D IMPLEMENTATION DETAILS · p. 17
  • The code uses the official Hugging Face implementation, Pomegranate for GMMs, and PyTorch for the remaining code.The tested checkpoints include Qwen2.5-7B-Instruct, Qwen2.5-3B-Instruct, and Qwen2.5-1.5B-Instruct.Source: D IMPLEMENTATION DETAILS · pp. 17–18
  • OP-LoRA sweeps top-k values {16, 128, 512}, while LwF uses KL-divergence regularization with coefficients {0.1, 0.3, 1.0, 3.0, 10.0}.LwF adds a forward pass to obtain base-model logits for comparison with adapter-model logits.Source: D IMPLEMENTATION DETAILS · p. 17
  • The MLP gate has hidden layers of dimensions 256 and 128 and uses binary cross-entropy to classify current-distribution versus generic-pretraining embeddings.Unlike the GMM, the MLP trains on examples from both labels; its gate loss is added to the main objective with a scaling factor.Source: D IMPLEMENTATION DETAILS · pp. 17–18
  • Tables 2 and 3 list the shared and per-task hyperparameters, respectively..Source: Table 2 · p. 18, Table 3 · p. 18
  • Multiphase runs use learning rate 1e−4, batch size 32, rank 128, and three seeds; evaluation is capped at 300 samples per benchmark.Igbo translation and chemistry run for three epochs each, cybersecurity for one; LSL uses 16 Φpos components, 32 Φneg components, and smoothing coefficient 0.1.Source: D IMPLEMENTATION DETAILS · p. 18

E THEORETICAL MOTIVATION - FULL DERIVATION

The formulation models sequential tasks using a current input distribution P and earlier, unknown distributions Q. It defines the target gate by the support of P and frames gate error against unknown prior distributions as a minimax objective.

E.1 FORMULATION

  • The formulation assumes a bounded feature domain, with P occupying a proper subset and having density bounded below there, while all densities are bounded above.Earlier task distributions Q are unknown during training, and the gate may train on P and a small buffer Q0.Source: §E.1 · p. 19
  • The target gate applies the update on supp P, the minimum-volume set carrying all of P’s mass; where P and Q overlap, the newer task’s update takes precedence.The minimum-volume characterization is the α = 1 case of Scott & Nowak (2006).Source: §E.1 · p. 19
  • Because Q is unknown, gate error is evaluated against the worst admissible test distribution, making minimization of disagreement volume with the target set the minimax objective.The gate’s error is the probability under Q of the symmetric difference between its accept set and the target.Source: §E.1 · p. 19

E.2 THE GATE’S TRAINING OBJECTIVE IS BLIND TO THE EXCESS

The gate’s error separates into a deficit, which rejects part of the target support, and an excess, which accepts inputs outside it. Training on the current distribution addresses the deficit but cannot directly control the excess.

E.2 THE GATE’S TRAINING OBJECTIVE IS BLIND TO THE EXCESS

  • The deficit is the part of the target support that the gate rejects, and the fitting procedure handles it explicitly.The section therefore focuses on excess rather than deficit.Source: §E.2 · p. 19
  • The excess is the region the gate accepts outside the target support, where the current distribution has no mass.A training loss averaged over the current distribution is blind to behavior in that region.Source: §E.2 · p. 19
  • A gate can accept an arbitrary off-support region without changing its training objective, even though a prior-task distribution concentrated there would be accepted.The objective cannot distinguish such gates, so data alone does not determine which one training returns.Source: §E.2 · p. 19

E.3 CONTROLLING WHAT THE OBJECTIVE CANNOT CONTROL

The objective on current-phase data cannot control excess by itself, so control must come from the estimator, hypothesis class, or fitting procedure. The section compares these sources of control and their costs, then describes GMM fitting, adapter training, gate placement, and function-space behavior.

E.3 CONTROLLING WHAT THE OBJECTIVE CANNOT CONTROL

  • The objective is blind to excess, so its control must come from elsewhere; each source of control pays through assumptions, memory, or reference samples.The conclusion contrasts parametric densities, local estimators, and uniform buffers by their respective costs.Source: §E.3 · pp. 19, 21
  • 16% of out-of-distribution pretraining tokens open gates, while LSL retains 96.6% of pretrained performance; temporal smoothing changes these figures to 5% and 98.8%.The GMM threshold is input-dependent, with Φpos as the density estimate and Φneg as the threshold; its positive minimum supports the stated bound.Source: §E.3 · p. 20
  • Classification against an arbitrary reference distribution does not guarantee support recovery, and a zero-training-error classifier can still accept an off-support region of volume 10.The outcome depends on the reference distribution; likelihood can also leave off-data mass unidentified.Source: §E.3 · pp. 20–21
  • A parametric density estimator offers constant memory but can retain excess if misspecified; local estimators avoid shape assumptions but use growing memory and may need many samples in high dimensions.Uniform negatives also require very large samples when the target support is small in a high-dimensional feature domain.Source: §E.3 · pp. 19–21

F FITTING THE GMM AND THE WEIGHT ADAPTER

  • Fitting more than 100 high-dimensional GMMs is demanding, so the method fits them module by module, retaining inputs for only one module at a time.Because each GMM depends only on its module’s inputs, this procedure is exact.Source: F FITTING THE GMM AND THE WEIGHT ADAPTER · p. 21
  • The adapter is trained as usual with the gate open throughout; this avoids bootstrapping, and the observed train-inference mismatch does not affect performance.The gate is kept open because training data all comes from the new task’s distribution.Source: F FITTING THE GMM AND THE WEIGHT ADAPTER · p. 22

G WHY PER-MATRIX GATES? - EXTENDED DISCUSSION

  • Per-matrix gates make contextual decisions in one pass; a single intermediate-layer gate would require an extra pass, while per-token gating handles mixed-domain streams.A single gate after the embedding layer falls short of per-matrix gates in learning and retention.Source: G WHY PER-MATRIX GATES? - EXTENDED… · p. 22

H FUNCTION BEHAVIOR CHANGE

I UNION OF SPHERES - RADIUS CALIBRATION

A single radius does not work well across UoS gates because their dimensions and input distributions differ. The section calibrates a radius per gate using ROC curves and compaction constraints.

I UNION OF SPHERES - RADIUS CALIBRATION

Loading 2610.02126v1…