PaperlayerView exact revision on arXiv

Source-linked AI summary

Stability-Plasticity Balance via Singular-Vector Selection in LLM Continual Learning

Lingxiang Wang, Hainan Zhang, Liang Pang, Hongwei Zheng, Zhiming Zheng

arXiv:2610.11076v1cs.LGcs.AI

TL;DR

Continual domain adaptation must acquire new capabilities while preserving those learned during pretraining, yet existing PEFT methods lack a principled unit for allocating plasticity. SVC estimates channel-level adaptation benefit and forgetting cost to select updates, and experiments report effective sequential adaptation with consistent preservation of pretrained general capabilities.

  • Problem

    Continual LLM adaptation must balance acquiring domain knowledge with preserving pretrained capabilities, while existing approaches lack a principled unit for deciding which directions to update.

    Source: §1 · pp. 1–2

  • Method

    SVC estimates singular-vector channels’ adaptation benefits and forgetting costs from current-task gradients and general-domain reference activations, then adaptively selects channels for updates.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Across four LLM families and eight downstream tasks, SVC effectively adapts to sequential tasks while consistently preserving pretrained general capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments support singular-vector channels as units for balancing adaptation benefits against forgetting costs in continual LLM learning.

    Source: §1 · p. 2, §5 · p. 9

    Longer task sequences and a broader range of general-capability benchmarks remain to be evaluated.

    Source: §5 · p. 10

Abstract

from arXiv · show

Domain-specific continual adaptation of LLMs risks catastrophic forgetting, creating a fundamental tension between acquiring new capabilities and preserving those learned during pretraining. PEFT mitigates this problem by restricting the number of trainable parameters, but existing methods lack a principled unit for deciding where plasticity should be allocated and stability should be preserved. We identify the singular-vector channel as a natural unit for managing this trade-off. Each channel represents an input-output transformation, which can be updated to acquire new knowledge or fixed to preserve pretrained capabilities. Based on this perspective, we introduce SVC, a parameter-efficient continual-learning method that selectively updates Singular-Vector Channels. Before fine-tuning, SVC uses domain-specific data to estimate each channel's adaptation benefit and a fixed public general-domain corpus only as a history activation proxy for estimating forgetting cost. It then adaptively selects trainable channels based on these scores via knee-based cost screening, Pareto-front filtering, and Otsu thresholding. Experimental results across four LLM families and eight downstream tasks show that SVC better preserves pretrained capabilities while achieving strong downstream performance relative to existing PEFT baselines. Further analysis of channel scoring and selection demonstrates that selective plasticity at the singular-vector-channel level enables effective continual LLM adaptation.

1 INTRODUCTION

Continual domain adaptation must balance learning new knowledge against preserving pretrained capabilities, but existing PEFT constraints do not consistently control forgetting. SVC selects singular-vector channels for task-specific updates, and experiments report effective adaptation alongside consistent preservation of general capabilities.

1 INTRODUCTION

  • The central challenge is balancing plasticity to acquire domain knowledge with stability to preserve pretrained capabilities when adapting LLMs.Pretraining data and signals are typically unavailable, while overly strong constraints may impede adaptation and weak constraints may fail to prevent forgetting.Source: §1 · pp. 1–2
  • Singular-vector channels—the paired left and right directions associated with a singular value—are proposed as the unit for weighing adaptation benefit against forgetting cost.SVC identifies these channels as a principled unit for controlling the stability-plasticity trade-off.Source: §1 · p. 2
  • SVC estimates each channel’s adaptation benefit and forgetting cost, then selects task-specific channels using knee-based screening, Pareto-front filtering, and Otsu thresholding.It updates all singular values and only the selected channels’ singular vectors.Source: §1 · p. 2, Figure 2 · p. 3
  • Across four LLM families and eight downstream tasks, SVC effectively adapts to sequential tasks while consistently preserving pretrained general capabilities.Further analyses support the channel-level benefit-cost estimates and multistage selection mechanism.Source: §1 · p. 2
  • SVC is presented as a parameter-efficient continual-learning method that adaptively balances stability and plasticity across tasks without prior-task samples or task-specific modules.The method overview describes reference activation statistics, current-task gradients, channel selection, and updates to selected directions.Source: §1 · p. 2, Figure 2 · p. 3

2 RELATED WORK

Prior PEFT and spectral methods constrain updates in different ways, but generally do not select directions using task-specific estimates of adaptation benefit and forgetting cost. SVC treats singular-vector channels as units for making that selection.

2 RELATED WORK

  • O-LoRA uses orthogonal low-rank subspaces from previous tasks, but does not directly estimate whether changing a direction benefits the current domain or compromises pretrained capabilities.Its orthogonality constraint is defined relative to previously learned low-rank subspaces.Source: §2 · p. 3
  • SVF, SVDiff, and SAM-PARSER keep singular-vector bases fixed and optimize diagonal SVD coefficients, while SVFT learns sparse coefficients that can combine different basis vectors.These methods specify trainable spectral coefficients before fine-tuning.Source: §2 · p. 3
  • Unlike these approaches, SVC adaptively selects singular-vector channels using task-specific estimates of adaptation benefit and general-capability forgetting cost.PEFT methods restrict trainable parameters, and O-LoRA reduces interference through orthogonal subspaces.Source: §2 · p. 3

3 METHOD

SVC treats each singular-vector channel as an adaptation unit, scoring its benefit and estimated forgetting cost before selecting which channels to update. Its procedure keeps model size fixed across tasks and adds no persistent task-specific modules.

3.1 PRELIMINARIES

  • Each singular-vector channel pairs the input and output directions associated with a singular value as one adaptation unit.SVD represents each linear-layer weight matrix as rank-one components.Source: §3 · p. 3, §3.1 · p. 4

3.2 BENEFIT–COST ESTIMATION FOR SINGULAR-VECTOR CHANNELS

  • Channel benefit is estimated from current-task loss reduction, while forgetting cost uses linear-layer output drift on historical activations as a proxy.Because historical activations are unavailable, their statistics are approximated using an unlabeled general-domain corpus.Source: §3.2 · pp. 4–5

3.3 PARETO-FRONTIER-BASED ADAPTIVE CHANNEL SELECTION

  • SVC screens high-cost channels at a knee, removes dominated candidates with Pareto-front filtering, then adaptively thresholds benefit–cost efficiency.This three-stage procedure avoids requiring a trade-off coefficient that may vary across layers and tasks.Source: §3.3 · p. 5

3 METHOD

  • SVC applies SVD to linear layers and scores channels using activation second moments from general-domain data and average gradients from current-task data.The method follows PEFT and operates on linear layers.Source: §3 · p. 3, §3.4 · p. 6

3.4 SVC

  • SVC selects channels before fine-tuning, updates all singular values and only selected singular-vector directions, then carries the merged weights forward without task-specific modules or task-wise model growth.Selection remains fixed during training on the task.Source: §3.4 · p. 6

4 EXPERIMENTS

SVC is evaluated across multiple model families, task orders, and benchmarks, with results indicating strong preservation of pretrained capabilities and sequential learning. Channel-level analyses support the benefit and cost scores used to guide selection.

4.1 EXPERIMENTAL SETUP

  • Experiments span four model families, three task orders, and TRACE tasks across science, finance, summarization, multilingual, code, and mathematics.A fixed 30,000-example unlabeled C4 and Stack-v2-Python corpus, sampled at a 2:1 ratio, supplies activation second moments.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • Across 12 settings, SVC ties or leads in 10 HumanEval and 11 GSM8K settings, with mean gains of 1.68 and 5.18 percentage points over strongest baselines, respectively.SVC also ranks first or second in 46 of 48 model, task-order, and metric combinations.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 ANALYSIS AND DISCUSSION

  • Pretrained-capability retention improves as selected channels move from high to lower cost, while high-cost selection also affects sequential learning.The comparison matches SVC’s channel count and holds other settings fixed.Source: §4.3 · p. 8, Table 2 · p. 9
  • At matched costs, high-benefit channel groups achieve higher average performance than low-benefit groups in both cost regions.This supports B as a measure of channels’ contribution to current-task adaptation.Source: §4.3 · p. 9, Table 2 · p. 9

2. How important is each selection stage?

Ablations show that the three channel-selection stages contribute to SVC’s stability-plasticity balance, while joint selection of singular values and vectors performs worse than the default design.

How important is each selection stage?

  • Removing knee-based cost screening has a smaller but still negative effect on mathematical-reasoning retention and sequential performance, while increasing forgetting.The authors conclude that reliable selection depends on the complete three-stage procedure.Source: §2 · p. 9
  • Removing Pareto-front filtering causes the largest degradation, nearly eliminating code-generation and mathematical-reasoning retention and sharply reducing sequential performance.Figure 3 reports BWT in absolute value.Source: Figure 3 · p. 9, §2 · p. 9
  • Jointly selecting singular values and vectors lowers HumanEval, GSM8K, OP, and BWT by 1.22, 2.80, 0.65, and 0.56 points, respectively, versus default SVC.The authors retain selective vector updates while optimizing all singular values.Source: §2 · p. 9

5 CONCLUSION

The conclusion presents SVC as a channel-level approach to balancing adaptation benefits and forgetting costs in continual LLM learning. It reports results across four model families and three task orders, and identifies broader evaluation and dynamic channel selection as future work.

5 CONCLUSION

  • SVC treats singular-vector channels as the basic units for balancing adaptation benefits against forgetting costs in continual LLM learning.It estimates both quantities using current-task gradients and activation statistics from an unlabeled general-domain reference corpus, then adaptively selects trainable channels.Source: §5 · p. 9
  • Across four LLM families and three task orders, SVC improves continual-learning performance while preserving pretrained general capabilities.The conclusion characterizes this as a task-adaptive route to stable and scalable continual domain adaptation.Source: §5 · p. 9
  • Future evaluation will examine longer task sequences, broader general-capability benchmarks, and the limits of robustness and computational efficiency.The authors also propose broader reference-corpus coverage and robustness to distribution shifts, plus channel-selection strategies that account for interactions and change dynamically during training.Source: §5 · pp. 9–10

A DERIVATION OF ADAPTATION BENEFIT AND FORGETTING COST

The section defines adaptation benefit and forgetting cost for individual singular-vector channels. It derives default scores from singular-vector perturbations and treats singular-value perturbations separately in an ablation.

A DERIVATION OF ADAPTATION BENEFIT AND FORGETTING COST

  • Each singular-vector channel is the paired left and right singular-vector directions associated with a singular value, and benefit–cost analysis is performed per channel.Together with its singular value, the pair defines a rank-one component.Source: A DERIVATION OF ADAPTATION BENEFIT AND… · p. 13
  • Adaptation benefit is the reduction in new-task loss, while forgetting cost measures linear-layer output drift on old-task input activations.The cost uses the second-moment matrix of old-task activations.Source: A DERIVATION OF ADAPTATION BENEFIT AND… · p. 13
  • In default SVC, selection controls whether the two singular-vector directions are trainable, while all singular values are optimized regardless of selection.The alternative ablation includes singular-value perturbations in the channel score.Source: A DERIVATION OF ADAPTATION BENEFIT AND… · pp. 13, 15
  • The default scores isolate orthogonal singular-vector perturbations, using first-order loss analysis and projected effective gradients.The singular-value perturbation is considered separately in the ablation.Source: A DERIVATION OF ADAPTATION BENEFIT AND… · p. 14
  • The singular-value-inclusive ablation scores the joint perturbation of a singular value and its associated singular-vector pair.The default scores instead quantify the marginal effect of enabling the singular-vector directions.Source: A DERIVATION OF ADAPTATION BENEFIT AND… · p. 15

B.1 CONTINUAL-LEARNING TASKS

The continual-learning sequences use eight heterogeneous TRACE language tasks, with 1,000 training examples per task and corresponding test sets for evaluation. The tasks span domain-specific, multilingual, code-completion, and mathematical-reasoning settings.

B.1 CONTINUAL-LEARNING TASKS

  • The evaluation uses eight TRACE tasks, with 1,000 training examples per task and corresponding test sets.The tasks were selected for their challenging and heterogeneous character and room for adaptation.Source: §B.1 · p. 16
  • Domain-specific tasks cover text-only ScienceQA, FOMC stance classification, and MeetingBank summarization of city-council proceedings.The FOMC data combine meeting minutes, press-conference transcripts, and speeches.Source: §B.1 · p. 16
  • The multilingual tasks are Chinese target-based stance detection on C-STANCE and German text simplification on 20Minuten.C-STANCE test-set targets are not observed during training.Source: §B.1 · p. 16
  • Py150 tests code completion by asking the model to predict subsequent Python code from a code context.The task uses source files collected from public GitHub repositories and structured inputs and outputs.Source: §B.1 · p. 16
  • NumGLUE-cm and NumGLUE-ds provide complementary mathematical-reasoning tasks requiring numerical computation and multistep reasoning.Both derive from NumGLUE and use natural-language problem statements.Source: §B.1 · p. 16

B.2 CAPABILITY-PRESERVATION BENCHMARKS

The evaluation measures retention on held-out code-generation and mathematical-reasoning benchmarks, uses a fixed unlabeled corpus to estimate forgetting costs, and compares SVC with continual-learning methods spanning several baseline families.

B.2 CAPABILITY-PRESERVATION BENCHMARKS

  • HumanEval and GSM8K measure pretrained code-generation and mathematical-reasoning retention, respectively, and are excluded from continual-training sequences.GSM8K uses its complete official test split of 1,319 problems.Source: §B.2 · p. 17

B.3 GENERAL-DOMAIN REFERENCE CORPUS

  • SVC estimates forgetting costs from layer activations on a fixed corpus of 30,000 examples sampled from C4 and Python-subset The Stack v2 at a 2:1 ratio.The corpus is shared across models, tasks, and task orders; its input texts provide no training supervision or parameter optimization.Source: §B.3 · p. 17

C.1 BASELINES

  • SVC is compared with regularization and gradient-constraint methods, replay methods, PEFT methods, and full or constrained full-parameter fine-tuning.Unless specified otherwise, methods share task data, task orders, optimization schedule, and evaluation protocols.Source: §C.1 · p. 17

C.1.1 REGULARIZATION AND GRADIENT-CONSTRAINT METHODS

  • EWC penalizes changes to parameters important for previous tasks, GEM constrains gradients against episodic-memory references, and LwF distills predictions from a frozen preceding model.The methods represent parameter-importance regularization, gradient constraints, and knowledge distillation.Source: §C.1.1 · pp. 17–18

C.1.2 REPLAY-BASED METHODS

  • Replay rehearses stored earlier-task samples after current-task training, whereas Replay-online mixes historical samples into training mini-batches.Both use a unified memory buffer of real examples from prior tasks.Source: §C.1.2 · p. 18

C.1.3 PARAMETER-EFFICIENT FINE-TUNING METHODS

  • The PEFT baselines include sequential and orthogonal LoRA, PiSSA initialized from principal singular components, and MiLoRA initialized from minor components.The LoRA-based methods use rank 8 adapters on q proj and v proj modules, with the stated dropout configurations.Source: §C.1.3 · pp. 18–19

C.1.4 FULL AND CONSTRAINED FINE-TUNING METHODS

  • SeqFT updates all parameters without explicit regularization, replay, or subspace constraints, while OSFT projects updates away from protected singular subspaces.OSFT allocates protected subspaces using layer importance estimated from activations; SVC selects channels adaptively across all linear modules without a manually specified global ratio.Source: §C.1.4 · p. 19

C.2 TASK ORDERS

The experiments repeat all eight TRACE tasks under three shared task orders. Table 4 examines target-module coverage for Llama3-8B under Order 1.

C.2 TASK ORDERS

  • Three task orders each contain all eight TRACE tasks, and the same orders are used for every model and baseline.The orders account for sensitivity of continual-learning performance to task ordering.Source: §C.2 · p. 19
  • Table 4 compares target-module coverage for Llama3-8B under Order 1, with higher values indicating better performance across metrics.The table includes default PiSSA and MiLoRA results and the SVC result reproduced from Table 1.Source: Table 4 · p. 20

C.3 MOTIVATION EXPERIMENT DETAILS

This section describes the independent LoRA/SVF TRACE comparison and tests whether expanding PEFT target modules alone explains SVC’s advantage.

C.3 MOTIVATION EXPERIMENT DETAILS

  • Figure 1 compares independent LoRA and SVF adaptation across eight TRACE tasks from the same Llama3-8B checkpoint, using five epochs and a learning rate of 10−4.Source: §C.3 · p. 20
  • Figure 1(a) reports GSM8K Pass@1 before and after adaptation, while Figure 1(b) reports performance on each task’s test set.Source: §C.3 · p. 20

D EFFECT OF EXPANDING PEFT TARGET MODULES

Loading 2610.11076v1…