PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning via Self-Probe Gradients

Dongkyu Cho, Rumi Chunara, Sungmin Cha

arXiv:2609.32771v1cs.LGcs.AI

TL;DR

When only a few past samples remain, their sparse and narrow evidence makes it harder to preserve prior behavior during adaptation. CPLUS combines gradients from retained samples and frozen-model self-probes to scale down conflicting updates. Across the reported evaluations, it reduces forgetting while learning new data, with stronger gains when anchors are scarce and an increasing share of SFT forgetting recovered as Qwen3 models grow.

  • Problem

    Users adapting released pretrained models often lack the original training corpus or train-time statistics, leaving few anchors as sparse and narrow evidence of prior behavior to preserve.

    Source: §1 · p. 1

  • Method

    CPLUS combines gradients from anchors and frozen-model self-probes to identify conflicting new-data updates and scales them down with a smoothed parameter-wise mask.

    Source: §1 · p. 2, §3.2 · p. 4, §3.2.2 · p. 5

  • Results

    CPLUS achieves higher Retention than all main baselines in 118 of 120 settings, with its advantage clearest when anchors are scarce.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Self-probes can provide actionable evidence for preserving prior behavior as gradients that guide updates, rather than as replay training examples.

    Source: §1 · p. 2, §5 · p. 9

    Self-probes reflect the frozen model’s behavior rather than verified truth, and the reported scale and sequential results do not establish preservation of every capability or robustness to arbitrary task sequences.

    Source: F LIMITATIONS · p. 26

Abstract

from arXiv · show

Adapting pretrained models to new data can cause catastrophic forgetting of previously learned behavior. When only a few past samples remain, they give continual learning methods sparse and narrow evidence about what to preserve. We show that language models can expand this evidence through self-probing, in which the frozen model generates new inputs from the retained samples and records its own predictions on them. Unlike prior work that replays such data as training examples, our method, CPLUS uses self-probe and past-sample gradients to scale down parameter updates that conflict with prior behavior. Experiments with five language models on four benchmarks show three results. First, the same probes preserve more prior behavior as gradient signals than as replay data. Second, CPLUS learns the new data while consistently reducing forgetting more than existing baselines, especially when past data are scarce, and this protection extends to benchmarks not used for training. Third, we observe that CPLUS also becomes more effective as models grow: within the Qwen3 model family, it recovers an increasing share of the forgetting caused by standard fine-tuning.

1 INTRODUCTION

With few retained examples, continual-learning methods have sparse, narrow evidence of prior behavior, making forgetting harder to prevent. CPLUS uses gradients from anchors and model-generated self-probes to constrain updates, reducing forgetting while learning new data.

1 INTRODUCTION

  • Retained anchors provide sparse and narrow evidence because users of publicly released pretrained models typically lack the original training corpus or train-time statistics.Preserving the anchored task does not guarantee preservation on other benchmarks.Source: §1 · p. 1
  • Self-probes can broaden preservation evidence, whereas replay treats the model’s potentially incorrect predictions as training labels and does not show which new-data updates disrupt prior behavior.CPLUS uses self-probe gradients to guide updates rather than replaying probes as examples.Source: §1 · pp. 1–2
  • CPLUS combines anchor and self-probe gradients to identify parameter updates likely to disrupt prior behavior, then scales those updates down with a smoothed mask.Self-probes guide how much of a proposed update is applied; they are not training targets.Source: §1 · p. 2
  • CPLUS achieves higher Retention than all main baselines in 118 of 120 settings across five models, four benchmarks, and anchor budgets from 4 to 128.With four or eight anchors, it leads in all 24 Qwen3 settings by an average of 7.73 points.Source: §1 · p. 2
  • The authors report that matched comparisons preserve more prior behavior when self-probes guide updates than when the same probes are replayed as training samples.CPLUS also protects performance across five benchmarks not used for training.Source: §1 · p. 2

2 RELATED WORK

Continual-learning methods differ in how they preserve prior behavior, but commonly depend on past data or information derived from it. CPLUS uses generated-data gradients to guide updates rather than training directly on generated samples.

Continual Learning

  • Continual-learning approaches include regularization, rehearsal, optimization, and expansion methods, which balance preserving prior behavior with learning new data.These approaches commonly rely on past data or information derived from it; CPL uses parameter-wise gradients to identify updates predicted to avoid interference when past data are available.Source: §2 · pp. 2–3
  • Open-weight models are typically released without their original training data or associated statistics, limiting access to evidence used by prior-data-dependent methods.CPLUS follows optimization-based work but rescales each coordinate of the proposed update rather than projecting the whole update as A-GEM and OGD do.Source: §2 · p. 3

Self-Generated Data

  • Unlike generative replay and self-distillation methods that train on generated data, CPLUS uses self-probe gradients to identify parts of a new-data update that may interfere with prior behavior.New data determine what the model learns, while self-probes guide how it learns.Source: §2 · p. 3

3 CPLUS: MOTIVATION AND METHOD

When only a few past samples remain, their preservation evidence can be sparse and narrow. CPLUS generates self-probes from the frozen model and combines their gradients with anchor gradients to assess and smooth parameter-wise update decisions.

3 CPLUS: MOTIVATION AND METHOD

  • CPLUS addresses limited past-data access by seeking a broader preservation signal than the few retained anchors provide.The method is motivated by evidence that preservation weakens as anchors become scarce and that preserving the anchored task may not preserve other capabilities.Source: §3.1 · p. 3

3.1 CONTINUAL LEARNING UNDER LIMITED PAST-DATA ACCESS

  • Self-probes record the frozen model’s behavior and their gradients assess proposed new-data updates, rather than treating potentially incorrect predictions as replay labels.Replay can reinforce errors through confirmation bias, and noisy replay labels can exacerbate forgetting.Source: §3.1 · p. 3

3.2 PROPOSED METHOD

  • CPLUS uses anchor and self-probe gradients to identify whether each parameter’s proposed update is collaborative or conflicting with prior behavior.A parameter is collaborative when the gradient-update product is nonpositive; these labels depend on the current update and preservation signal.Source: §3.2 · p. 4

3.2.1 SELF-PROBES

  • The frozen base model uses anchor demonstrations to generate a fixed self-probe bank containing new inputs and the model’s own predictions.Neither anchor nor self-probe loss is added to the objective producing the proposed new-data update; their gradients determine how much of that update is applied.Source: §3.2.1 · p. 4

3.2.2 GRADIENT BLENDING

  • CPLUS blends mean anchor and self-probe gradients into a preservation gradient, using equal coefficients without normalizing or rescaling either source.The blended signal is not added to the new-data training objective; it is used to mask proposed updates.Source: §3.2.2 · pp. 4–5

3.2.3 SMOOTH UPDATE MASKING

  • At each step, CPLUS masks each proposed update according to its alignment with the blended preservation gradient, then smooths the binary decisions with a bias-corrected exponential moving average.The smoothed mask can limit an update but cannot amplify or reverse it; smoothing trades the instantaneous sign constraint for less sensitivity to isolated flips.Source: §3.2.3 · p. 5

4 EXPERIMENTS

Across five models and four benchmarks, the experiments examine CPLUS under limited anchors, beyond-task retention, alternative probe uses, sequential learning, compute tradeoffs, and learning dynamics.

4.1 EXPERIMENTAL SETTING

  • Across Qwen3-0.6B, 1.7B, and 4B, Gemma-2-2B-it, and Phi-3.5 Mini, the evaluation uses four benchmarks and anchor budgets from 4 to 128.Training uses initially incorrect examples; Retention is measured on initially correct examples, separating learning from forgetting.Source: §4.1 · p. 5

4.2 PRESERVATION WITH LIMITED ANCHORS

  • CPLUS beats the strongest baseline in 70 of 72 Qwen3 settings, including all 24 with four or eight anchors, with a 7.73-point mean gain at those budgets.It also leads in all 48 Gemma and Phi-3.5 Mini settings; all methods attain 100% Acquisition.Source: Figure 3 · p. 6, §4.2 · p. 7

4.3 PRESERVATION BEYOND THE REPRESENTED TARGET

  • After CommonsenseQA adaptation, CPLUS stays near or above base-model performance on five benchmarks; external-benchmark averages rise by 1.15, 0.88, and 0.14 points after ARC-C, MMLU, and MedQA adaptation.CPLUS has the highest external average after ARC-C and MedQA, and is within 0.02 points of SFT after MMLU adaptation.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 HOW SHOULD SELF-PROBES BE USED?

  • Across four anchor budgets, CPLUS retains more previously correct answers than either probe-replay variant, although all methods attain near-perfect Acquisition.The matched Figure 5 comparison uses the same probes and update budget.Source: Figure 5 · p. 7, §4.4 · p. 8
  • Combining anchor and probe gradients performs best in the component ablation; smoothing adds a smaller gain, norm alignment changes little, and matched-norm or randomized-mask controls do not reproduce CPLUS’s Retention at full Acquisition.The ablation uses Qwen3-0.6B on CommonsenseQA with eight anchors and one seed; the controls suggest coordinate selection matters beyond aggregate attenuation in that diagnostic.Source: Table 1 · p. 7, §4.4 · p. 8

4.5 SEQUENTIAL MULTI-TASK ADAPTATION

  • After sequential adaptation to four tasks with eight anchors per preceding task, CPLUS reaches 78.87% Average Task Accuracy, versus 69.73% for replay and 69.65% for CPL.CPLUS also has the highest accuracy on every task in the sequence.Source: §4.5 · p. 8, Figure 6 · p. 8

4.6 EFFICIENCY AND OPTIMIZATION ROBUSTNESS

  • Across the displayed model sizes, CPLUS achieves a larger Retention gain than the other evaluated methods against training FLOPs relative to SFT.Figure 7 uses eight anchors for preservation methods and excludes one-time preparation costs.Source: §4.6 · p. 8, Figure 7 · p. 9
  • In a separate eight-anchor diagnostic, CPLUS reaches 100% Acquisition after five epochs, versus ten for CPL, then maintains the highest Retention as early forgetting partly recovers.Replay initially forgets less than SFT but trails CPLUS after both reach full Acquisition by epoch five.Source: §4.6 · p. 8, Figure 8 · p. 9

5 CONCLUSION

CPLUS uses gradients from anchors and frozen-model self-probes to softly constrain new-data updates. Across the tested models, benchmarks, and anchor budgets, it improves Retention while maintaining Acquisition, including beyond the anchored task and in a four-task sequence.

5 CONCLUSION

  • CPLUS uses self-probe and anchor gradients as preservation signals to softly constrain updates on new data when only a limited anchor set is available.Self-probes are not replayed as training examples.Source: §5 · p. 9
  • Across model sizes, benchmarks, and anchor budgets, CPLUS improves Retention while maintaining Acquisition, including beyond the anchored task and in a tested four-task sequence.The conclusion frames self-probes as actionable evidence for preserving prior behavior during continual adaptation with limited past-data access.Source: §5 · p. 9
  • The results show that self-probes can guide preservation as gradient signals rather than being replayed as training examples.This conclusion is scoped to continual adaptation under limited access to past data.Source: §5 · p. 9

A EXPERIMENT INDEX

The appendix identifies an experiment index and adds details on estimating preservation signals and evaluating masked updates.

A EXPERIMENT INDEX

  • Table 2 is identified as the experiment index.The supplied passage gives its title but no further index contents.Source: Table 2 · p. 14

B ADDITIONAL PROBLEM-FORMULATION DETAILS

  • The formulation estimates the unavailable full past-data gradient from available evidence and evaluates the proposed optimizer update against that estimate.The mask acts on the proposed update, so relevant estimation errors depend on that update.Source: B ADDITIONAL PROBLEM-FORMULATION DETAILS · p. 14
  • Suppressing an update to zero avoids forgetting but also prevents acquisition; diagnostics distinguish selective blocking of harmful updates from suppressing most update energy.The formulation classifies retained coordinates with positive past-gradient/update products as harmful mass and suppressed negative-product coordinates as useful mass.Source: B ADDITIONAL PROBLEM-FORMULATION DETAILS · p. 14

C DETAILED CPLUS METHOD

CPLUS uses frozen-model self-probes and retained anchors as preservation evidence, not as training targets, to selectively attenuate proposed updates on new data. Its procedure builds and caches probes, blends their gradients with anchor gradients, and smooths the resulting update mask over time.

C.1 CONSTRUCTING THE SELF-PROBE BANK

  • CPLUS generates a fixed bank of self-probe input–prediction pairs from contexts containing retained anchors; the frozen model’s predictions are behavioral targets, not verified labels.Invalid-interface outputs and duplicate inputs are removed; the reported configuration uses K = 2, T = 0.8, and N = 512.Source: §C.1 · p. 15
  • In a one-seed diagnostic, adding a topic cue did not consistently improve Retention.The paper notes that self-probes broaden behavioral coverage rather than reconstructing the unavailable past-data distribution, and their usefulness depends on what the frozen model can infer from context.Source: §C.1 · p. 15

C.3 WHY USE SELF-PROBES AS PRESERVATION EVIDENCE?

  • Direct replay can make probe gradients an optimization direction and may reinforce inherited errors; CPLUS instead uses probes only to control the supervised update proposal.A self-probe can only attenuate a proposed update component, keeping acquisition supervision separate from preservation evidence.Source: §C.3 · p. 15

C.4 ESTIMATING AND BLENDING PRESERVATION GRADIENTS

  • At each epoch’s refresh, CPLUS blends mean anchor and self-probe gradients; the anchor signal is verified but sparse, while the probe signal is broader but inherits base-model behavior.The default is a raw blend without norm alignment or source-specific rescaling, so equal coefficients need not yield equal coordinate-wise influence.Source: §C.4 · p. 16

C.5 MASKING THE OPTIMIZER UPDATE

  • CPLUS masks the optimizer’s complete proposed update, retaining coordinates whose contribution to the refreshed blended-gradient loss linearization is nonpositive.Preservation gradients refresh once per epoch, while the mask is recomputed at each adaptation step.Source: §C.5 · p. 16

C.6 TEMPORAL SMOOTHING

  • An exponential moving average turns instantaneous binary decisions into soft coefficients in [0, 1], which can attenuate but not amplify or reverse updates.This smoothing is not a strict per-step safety guarantee: an EMA-smoothed update need not satisfy the instantaneous mask’s nonpositive inner-product condition.Source: §C.6 · pp. 16–17

C.7 COMPLETE PROCEDURE AND COST

  • CPLUS generates probes once, refreshes and blends preservation gradients at each epoch’s start, then optimizes only on new data while anchors and probes affect updates through the mask.Each refresh adds forward and backward passes over A + N reference examples, and the method stores one blended gradient and one EMA state.Source: §C.7 · p. 17

D CPLUS PSEUDOCODE

  • The pseudocode cycles through frozen probe construction, epoch-level anchor and probe gradient blending, and per-minibatch proposal masking, smoothing, and parameter updates.The parameter update applies the soft mask elementwise to the optimizer proposal.Source: D CPLUS PSEUDOCODE · p. 17

E.1 MODEL-SCALE ANALYSES

The analyses examine CPLUS across model scale, development-only hyperparameter choices, and epoch-level learning dynamics. Retention rises with model size for both compared methods, while CPLUS reaches full Acquisition earlier than CPL in the reported training trajectory.

E.1 MODEL-SCALE ANALYSES

  • Across model scale, absolute Retention increases for both access-matched CPL and CPLUS; the figures compare their four-anchor results and show CPLUS’s recovered share of SFT forgetting across benchmarks.The headroom-normalized view measures the share of each model’s available forgetting recovered, accounting for differences in native forgetting.Source: §E.1 · p. 18, Figure 9 · p. 18, Figure 10 · p. 18

E.2 HYPERPARAMETER SELECTION

  • At A = 4, two-example generation with T = 0.8 had the highest observed Retention, and equal-weight blending outperformed self-probe-only and 75:25 blending at every tested anchor budget.The hyperparameters were selected using only Qwen3-0.6B on CommonsenseQA development data; mean reduction with β = 0.9 was favored in the separate reference-reduction and smoothing screen.Source: §E.2 · pp. 18–19, Table 3 · p. 19, Figure 11 · p. 19

E.3 EPOCH-LEVEL ACQUISITION–RETENTION DYNAMICS

  • CPLUS reached complete Acquisition by epoch 5 and maintained a higher Retention plateau, whereas CPL first reached complete Acquisition at epoch 10.Retention first dropped sharply as Acquisition rose, then partially recovered; early evaluation can miss recovery, while final Retention can hide temporary forgetting.Source: §E.3 · p. 19

E.4 ANCHOR COMPOSITION ACROSS TASKS

These analyses test how anchor composition, aggressive optimization, and update controls affect preservation. Across the reported stress test and matched controls, CPLUS preserves more prior behavior than the comparison methods under the stated settings.

E.4 ANCHOR COMPOSITION ACROSS TASKS

  • Base-correct anchors more strongly preserved established predictions, while base-wrong anchors improved correction and overall accuracy in the MedQA adaptation test.All rows achieved 100% MedQA Acquisition and 100% accuracy on the eight supplied anchors; evaluation excluded those anchors.Source: §E.4 · pp. 19–20, Table 4 · p. 20

E.5 FIXED AGGRESSIVE-UPDATE STRESS TEST

  • 17.22 percentage points more of the base model’s correct predictions were retained by CPLUS than SFT, while both reached 98.97% Acquisition under the fixed aggressive schedule.Replay attained complete Acquisition but retained less, and CPL reached 42.63% Acquisition.Source: §E.5 · p. 20, Table 5 · p. 21

E.6 MATCHED UPDATE CONTROLS

  • Neither norm-matched SFT nor a randomized mask reproduced CPLUS’s Retention at the same Acquisition in the matched update-control diagnostic.The controls matched CPLUS’s update norm or within-tensor mask-coefficient multiset, respectively.Source: §E.6 · p. 21, Table 6 · p. 21

E.7 WHERE THE PRESERVATION MASK ACTS DURING TRAINING

The mask audits examine which parts of proposed updates CPLUS and CPL admit, and whether similarity to a full-data mask explains CPLUS’s advantage. The reported diagnostics point to coordinate placement rather than aggregate admitted update magnitude alone, while remaining limited to specific settings.

E.7 WHERE THE PRESERVATION MASK ACTS DURING TRAINING

  • 71.05% of initially correct behavior was retained by CPLUS versus 50.48% by CPL, although both admitted approximately half their coordinates and proposal energy from nine epochs onward.CPLUS admitted fewer coordinates and less candidate-update energy in early states; both methods achieved 100% Acquisition.Source: §E.7 · p. 21, Figure 13 · p. 22
  • Permuting CPLUS’s mask coefficients within each parameter tensor lowered Retention to 47.13% at the same 100% Acquisition, supporting the importance of which coordinates are attenuated.The permutation preserved the coefficient multiset, distinguishing coordinate placement from aggregate mask density or admitted update energy.Source: §E.7 · p. 21, Figure 13 · p. 22

E.8 MASK BEHAVIOR

  • At A = 128, update-weighted agreement was 0.95 for CPL and 0.89 for CPLUS, yet CPLUS had higher Retention under limited access.The diagnostic indicates that CPLUS’s advantage does not come from greater agreement with its own full-data reference; it covers one model–benchmark pair and is not presented as a universal mechanistic explanation.Source: §E.8 · p. 22, Figure 14 · p. 22, Figure 15 · p. 23

E.9 SEQUENTIAL MULTI-TASK LEARNING

In sequential learning, CPLUS maintains separate preservation evidence for each preceding task; it also remains closest to the base capability profile on five benchmarks not used for adaptation. Its layer-wise mask audit finds that method-matched agreement varies with anchor budget and depth, while compute comparisons use reduced probe counts for CPLUS.

E.9 SEQUENTIAL MULTI-TASK LEARNING

  • In sequential learning, CPLUS averages preservation gradients from separate past-example sets and self-probe banks for preceding tasks before masking the current-task update.All methods start from the same post-CommonsenseQA checkpoint and continue from their own preceding checkpoint.Source: §E.9 · p. 23
  • After adaptation to MedQA, CPLUS achieves the highest accuracy on every task encountered in the sequence.Table 8 reports final-stage accuracy; Table 7 reports average task accuracy across stages.Source: Table 8 · p. 23, Table 7 · p. 23

E.10 BASE-NORMALIZED CAPABILITY SCORES

  • CPLUS remains closest to the base capability profile across the five benchmarks not used for adaptation.Figure 16 normalizes post-adaptation scores against each benchmark’s base-model score; values above 100% indicate improvement.Source: §E.10 · pp. 23–24, Figure 16 · p. 24

E.11 LAYER-WISE MASK AUDIT

  • CPL remains closer to its full-data reference than CPLUS across budgets and depth, with both methods showing a drop at A = 8.Figure 17 presents method-matched comparisons across transformer layers; the audit does not claim monotonic convergence at the smallest budgets.Source: §E.11 · p. 24, Figure 17 · p. 25
  • CPLUS and the all-real CPL reference suppress different updates, so mask similarity is not treated as evidence of broader behavioral preservation.The cross-method comparison in Figure 18 is diagnostic rather than an oracle target; broader function is evaluated behaviorally.Source: §E.11 · pp. 24–25, Figure 18 · p. 25

E.12 EVALUATION AND COMPUTE DETAILS

  • For the compute comparison, CPLUS uses N = 32 probes rather than the default 512, and reported FLOPs exclude evaluation and one-time preparation.FLOPs estimate arithmetic cost, not elapsed time.Source: §E.12 · pp. 25–26

F LIMITATIONS

The paper limits its claims to the tested models and task order, and notes that self-probes reflect the frozen model rather than independently verified truth. Its related-work discussion distinguishes forgetting from acquisition and positions CPLUS among methods that use prior information to guide updates.

F LIMITATIONS

  • Self-probes record frozen-model behavior rather than verified truth, so bias or poor probe coverage can limit useful adaptation updates.The scale findings characterize tested models, and sequential results concern the tested task order; neither establishes preservation of every capability or arbitrary sequences.Source: F LIMITATIONS · p. 26

G EXTENDED RELATED WORK

  • Continual-learning methods use parameter penalties, episodic-memory gradients, orthogonal projections, or coordinate-wise suppression to mitigate interference.CPLUS complements limited past samples with frozen-model self-probes and blends the two signals before masking adaptation updates.Source: G EXTENDED RELATED WORK · p. 26
  • CPLUS uses self-probes to construct a preservation mask, not as replay samples or targets for new behavior.The related-work discussion contrasts this role with model-generated replay and on-policy predictions used as supervision.Source: G EXTENDED RELATED WORK · p. 26
  • Forgetting and loss of plasticity are distinct, so the discussion motivates measuring retention and acquisition separately.It also distinguishes native robustness, absolute CPLUS performance, and incremental gains over matched baselines.Source: G EXTENDED RELATED WORK · p. 26
  • The evaluation uses hyperparameters selected on one development setting, applying self-probe and masking settings unchanged across reported conditions.The discussion also motivates comparisons with replay and access-matched CPL, alongside a simple global gradient blend.Source: G EXTENDED RELATED WORK · pp. 26–27

Related papers

Loading 2609.32771v1…