PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning

Shengtao Wen, Yunying Yang, Xiang Chen, Lingbing Guo, Yu Tian, Sheng-Jun Huang

arXiv:2609.29711v1cs.LGcs.AI

TL;DR

PPCL must reduce sensitive-content reproduction while retaining useful knowledge across sequential tasks, and operational output control is distinct from formal privacy guarantees. SPARK first builds a stable post-task checkpoint through self-distillation replay, then corrects annotated-PII likelihood around that reference. Across task orders and model families, the paper reports stable utility and selective suppression.

  • Problem

    PPCL must mitigate sensitive-content reproduction over sequential tasks while retaining useful knowledge, and operationally lowering sensitive-output likelihood is distinct from formal privacy guarantees.

    Source: §1 · p. 1

  • Method

    SPARK first builds a stable checkpoint with Self-Distillation Replay, then uses Post-Task Privacy Correction to lower annotated-PII likelihood while anchoring current- and old-task non-PII behavior.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Across task orders and model families, SPARK shows stable utility and consistent likelihood reduction; Δsel = 0.746 with a 95% source-cluster bootstrap interval of [0.385, 1.169].

    Source: §6 · p. 8, §5.2 · p. 8

  • Takeaways & Limitations

    Fixed-reference post-task correction is reported as an effective way to balance knowledge retention and sensitive-output control, complementary to formal privacy mechanisms.

    Source: §6 · p. 8

    The operational measurements do not provide differential privacy, certify parameter-level erasure, or establish non-extractability against arbitrary adaptive black-box attacks.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Privacy-preserving continual learning (PPCL) must reduce the reproduction of sensitive content while retaining useful knowledge across sequential tasks. Formal privacy guarantees characterize randomized mechanisms, whereas operational output control concerns whether a trained model selectively reduces the likelihood of sensitive content in its outputs. In this work, we investigate the latter together with continual-learning utility under realistic task evolution. Retention and privacy correction operate at different granularities: task acquisition requires broad preservation of current- and old-task behavior, whereas privacy correction targets sparse annotated positions. Joint optimization leaves the current-task preservation target continually changing. We propose SPARK, a retention-correction decomposition that first freezes the learned post-task distribution and then applies selective correction around this stable reference. Self-Distillation Replay learns the current task while distilling behavior from previous tasks, and Post-Task Privacy Correction reduces annotated-PII likelihood while anchoring current- and old-task non-PII behavior to the resulting checkpoint. Extensive evaluations demonstrate that SPARK achieves effective selective PII suppression while preserving strong continual-learning utility and knowledge retention across diverse settings. Code and data will be released upon publication.

1 Introduction

PPCL seeks to reduce sensitive-content reproduction while retaining useful knowledge across sequential tasks, but broad retention and sparse privacy correction require different controls. SPARK separates these objectives by first constructing a stable post-task checkpoint and then correcting annotated PII around it.

1 Introduction

  • SPARK targets operational output control and complements formal privacy mechanisms rather than providing a differential-privacy guarantee.The introduction distinguishes selective reduction of annotated-PII likelihood from formal privacy guarantees.Source: §1 · pp. 1–2
  • PPCL combines broad preservation of current- and previous-task behavior with sparse changes to annotated sensitive-token probabilities, yet joint optimization leaves the current-task preservation target moving.Privacy correction therefore needs a reference checkpoint that already reflects current-task learning and relevant old-task behavior.Source: §1 · p. 2
  • In the evaluated PeCL configuration, embedding perturbation changes PII and low-sensitivity probabilities similarly at the training noise scale, while the coupled task/unlearning update exhibits task-sensitive acquisition behavior.The authors present this as a supporting case study, not as a premise required by SPARK's fixed-reference design.Source: §1 · p. 2
  • SPARK uses Self-Distillation Replay to learn the current task while distilling previous-task behavior, then applies Post-Task Privacy Correction to reduce annotated-PII likelihood and anchor non-PII behavior.The correction anchors both current- and old-task non-PII distributions to the frozen checkpoint.Source: §1 · p. 2
  • Difficulty-matched controls and source-cluster bootstrap establish selectivity on the primary benchmark, while cross-order and cross-backbone results show stable utility and absolute annotated-PII likelihood reduction.The conclusion also reports near-zero forgetting across task orders and model families.Source: §1 · p. 2

2 Related Work

Prior work addresses continual learning through regularization, replay, distillation, or architectural separation, while privacy methods include formal protections and post-training unlearning. The paper situates SPARK at the intersection of distillation-based retention and targeted post-task output correction.

Continual Learning for LLMs

  • Continual-learning approaches include regularization such as EWC [16], replay such as ER [27] and GEM [19], distribution distillation such as DER++ [5], and architecture methods such as O-LoRA [38].The paper relates Self-Distillation Replay to continual-learning distillation [25] [33] and distinguishes its previous-task checkpoint teacher from SDFT's in-context signals [29].Source: §2 · p. 2
  • Self-Distillation Replay uses the previous-task checkpoint as teacher and integrates distillation with sensitivity analysis to identify PII tokens for later correction, unlike SDFT's in-context generated signals.The paper describes Post-Task Privacy Correction as inspired by unlikelihood training [39].Source: §2 · p. 2

Privacy-Preserving Methods for LLMs

  • Differentially private training includes DPSGD [1] [34], token-level approaches DP-MLM [21] and PMixED [11], while machine-unlearning methods include SISA [37] and gradient ascent [14] [20].The paper notes that these methods may require expensive retraining or degrade overall model capabilities [18] [26].Source: §2 · p. 2

What Output Selectivity Needs

  • The motivation distinguishes broad retention from sparse PII correction: embedding perturbation alone does not establish output selectivity, whereas direct correction uses PII targets and a fixed preservation reference.The proposed ordering is to retain knowledge before sparse post-task probability correction.Source: Figure 1 · p. 3

Privacy and Memorization Evaluation

  • PII likelihood alone is confounded by token rarity or difficulty, so the evaluation uses same-example, pretrained-difficulty-matched controls with source-cluster bootstrap as primary selectivity evidence.Vocabulary ranking supports the evaluation, while canary-ranking and membership diagnostics are treated as stress tests rather than extensions of the main selectivity claim.Source: §2 · p. 3

3 Preliminaries

The formulation considers sequential task learning with privacy-aware token sensitivity and operational evaluation of annotated-PII output likelihood. It motivates direct output correction after embedding perturbation is shown not to selectively control PII likelihood in the evaluated setting.

3.1 Problem Formulation

  • The task stream comprises sequential input-output tasks, with token sensitivity scores reflecting predicted private or task-specific information; the objective preserves prior-task performance, reduces sensitive-token memorization, and retains non-sensitive utility.The benchmark table reports final post-stream accuracy for task columns across six tasks.Source: §3.1 · p. 3, Table 1 · p. 4
  • The threat scope is operational: known-prefix likelihood, fixed-candidate canary ranking, and score-based membership diagnostics do not provide differential privacy, parameter-level erasure, or arbitrary adaptive-attack non-extractability.“Privacy correction” refers to selective mitigation of annotated-PII output likelihood within this scope.Source: §3.1 · p. 3

3.2 Why Is Direct Output Control Needed?

  • At TDP's training noise scale of 0.01, PII and low-sensitivity token probabilities both decrease by approximately 4%, with negligible selectivity of 0.29 pp.The passage says increasing noise to 0.05–0.50 strongly suppresses both groups; embedding perturbation alone is not direct selective output control, and this does not negate the formal LDP guarantee.Source: §3.2 · pp. 3–4
  • SPARK's sequence learns the current task with response-only supervision while reusing a frozen previous-task checkpoint as replay teacher, then anchors current- and old-task non-PII behavior to that checkpoint.The resulting post-task correction provides direct control after current-task learning.Source: §3.2 · p. 4

4 Methodology

SPARK separates broad knowledge retention from sparse privacy correction by first learning a post-task reference and then correcting around that fixed checkpoint.

4 Methodology

  • SPARK first learns a post-task checkpoint that preserves current- and old-task behavior, then applies privacy correction around that fixed reference.This separates retention from correction because the current-task preservation target changes during task acquisition.Source: §4 · pp. 4–5, §4.2 · p. 6

4.1 Self-Distillation Replay: Knowledge Retention

  • Phase 1 combines response-only current-task supervision with replay: low-sensitivity positions use cross-entropy, while high-sensitivity valid positions distill the previous checkpoint through teacher top-K KL.Replay avoids direct reinforcement at sensitive positions.Source: §4.1 · p. 5

4.2 PTPC: Post-Task Privacy Correction

  • Phase 2 initializes from the learned checkpoint, penalizes observed PII tokens with unlikelihood, and anchors current- and old-task non-PII behavior to the same frozen post-task teacher.The resulting privacy-corrected checkpoint initializes the next task’s learning phase.Source: §4.2 · pp. 5–6
  • PTPC targets output-level conditional likelihood, not parameter erasure, and the reported tests do not establish general extraction or membership improvements or arbitrary-attack resistance.The PeCL case study is supporting evidence only under its evaluated configuration, not a claim that every coupled objective fails.Source: §4 · p. 5, §4.2 · p. 6

5 Experiments

SPARK combines sequential-task learning and selective PII correction, then evaluates utility, selectivity, robustness, and operational costs. It achieves strong continual-learning results and matched-control evidence of PII suppression, with results also reported across task orders and configurations.

5.1 Experimental Setup

  • The six-domain benchmark uses sequential prompted classification; utility is Last, Avg, and BWT, while selectivity compares annotated PII with difficulty-matched same-example non-PII spans.The main model is LLaMA-2-7B with LoRA; the benchmark also evaluates Qwen2.5-1.5B.Source: §5.1 · p. 6

5.2 Main Results

  • SPARK led evaluated PPCL and privacy-augmented methods at Last 0.799 and Avg 0.812, and on Qwen2.5-1.5B stayed within 0.9 points of ER while improving over PeCL†.BWT was −0.009, compared with +0.001 for SD-Replay without PTPC.Source: §5.2 · pp. 6–8, Figure 4 · p. 7

5.3.2 Task-Order Robustness.

  • Across three orders, Last/Avg averaged 0.813/0.818 with BWT −0.001, matching ER utility; Figure 5 reports consistent absolute suppression, not matched selectivity.SPARK’s Last ranged from 0.799 to 0.820 and BWT from −0.010 to +0.017.Source: Figure 5 · p. 7, §5.3.2 · p. 8

5.3.3 Hyperparameter Sensitivity Analysis.

  • Across evaluated configurations, Last stays 0.78–0.81; increasing αUL raises PII NLL from 2.65 to 4.85, with default 2 capturing most of the gain.PTPC takes 7.98 minutes (4.0% of runtime); total SPARK runtime is 20.3% above ER.Source: Figure 6 · p. 7, §5.3.3 · p. 8, §5.3.4 · p. 8

5.2 Main Results

  • SPARK achieves Δsel = 0.746 (95% interval [0.385, 1.169]) and lowers PII Top-1 from 47.13% for the pretrained model to 23.75%; the conclusion applies to the pooled benchmark.Canary ranking and membership inference showed no improvements, and the matched set was dominated by FOMC and MentILL.Source: §5.2 · p. 8
  • Ablations assign distinct roles: SD-Replay constructs the preservation reference, unlikelihood drives PII demotion, dKL limits overcorrection, and anchors constrain current- and old-task drift.Without SD-Replay, Last falls to 0.236 and BWT to −0.684; without PTPC, Last is 0.804 but most PII suppression disappears.Source: §5.2 · p. 8

5.3.1 Case Analysis.

  • In a FOMC example, SPARK retains the dovish prediction while raising injected-PII NLL from 4.30 to 5.32.The example illustrates targeted conditional-likelihood reduction without changing the downstream task decision.Source: §5.3.1 · p. 8

6 Conclusion and Future Outlook

SPARK separates task retention from privacy correction by freezing a post-task checkpoint, then selectively reducing annotated-PII likelihood while anchoring non-PII behavior. The reported evaluations support selective suppression alongside continual-learning utility, with operational output control distinct from formal privacy guarantees.

6 Conclusion and Future Outlook

  • Across task orders and model families, SPARK shows stable utility, near-zero forgetting, and reduced annotated-PII likelihood while selectively suppressing PII beyond comparable non-PII drift.The conclusion frames fixed-reference post-task correction as complementary to formal privacy mechanisms.Source: §6 · p. 8

A PeCL Reproduction and Operational Audit

  • Yelp accuracy under reproduced joint PeCL falls to near-random immediately after Task 2, while SPARK does not show comparable instability.The reproduced configuration serves as an operational audit and supplies checkpoints for the difficulty-matched evaluation.Source: A PeCL Reproduction and Operational… · p. 10, Table 5 · p. 11

B Detailed Objective Definitions

  • PTPC uses full-sequence PII positions and anchors current-task and replayed non-PII distributions to the same frozen post-task teacher.The demotion teacher removes the observed PII token and renormalizes; losses for empty selected sets are defined as zero.Source: B Detailed Objective Definitions · p. 11

C Algorithm

  • Algorithm 1 summarizes SPARK’s complete training pipeline.The supplied passage identifies the algorithm as the summary of the full pipeline.Source: C Algorithm · p. 11

D Sequence Construction and Loss Masks

  • Phase 2 uses the full input sequence and attention mask, so annotated PII in prompts enters PTPC even though Phase 1 task supervision is response-only.This distinction matters because the synthetic password and SSN canaries occur in task text while the supervised response is the class label.Source: D Sequence Construction and Loss Masks · p. 11

E Embedding-Perturbation Selectivity Diagnostic

  • At noise scale 0.01, mean probability changes are −4.01% for PII and −3.72% for low-sensitivity tokens, indicating little output-level selectivity in the evaluated configuration.Larger scales strongly suppress both groups; the diagnostic is not a theorem about RMSNorm or a refutation of DP post-processing.Source: E Embedding-Perturbation Selectivity… · p. 12

F.1 Hyperparameter Sensitivity

  • Last accuracy remains 0.78–0.81 across evaluated configurations; αUL primarily governs PII conditional-likelihood suppression, while λpii saturates beyond 8.0.The observed best balance uses λold = 1.0, and the privacy-step sweep exhibits a threshold near 200 steps.Source: §F.1 · p. 12

G Detailed Results for Task Order Analysis

Across three task orders, SPARK maintains continual-learning performance and lower conditional likelihood for annotated PII. A Qwen2.5-1.5B evaluation reports a similar pattern, while its likelihood comparisons remain unmatched diagnostics.

G Detailed Results for Task Order Analysis

  • Across three task orders, SPARK maintains high Last/Avg accuracy and near-zero BWT while obtaining the highest NLLpii and Canary NLL among evaluated methods, but these unmatched values do not establish selectivity or extraction resistance.The difficulty-matched selectivity claim is evaluated separately on the primary task order.Source: G Detailed Results for Task Order… · p. 12

H Cross-Backbone Evaluation on Qwen2.5

  • On Qwen2.5-1.5B, SPARK remains close to unconstrained ER in utility, improves retention over PeCL, and assigns lower conditional likelihood to annotated PII than evaluated baselines.The likelihood columns are unmatched diagnostics and indicate cross-backbone consistency, not an additional difficulty-matched selectivity test.Source: H Cross-Backbone Evaluation on Qwen2.5 · p. 12

I Internal PTPC Ablations

PTPC ablations distinguish the roles of demotion and the two non-PII anchors: stronger PII-likelihood suppression can accompany worse utility, while the old-task anchor most strongly limits retention loss.

I Internal PTPC Ablations

  • Removing dKL yields the strongest PII-likelihood suppression, but Last, Avg, BWT, and low-sensitivity NLL all deteriorate.The pattern supports interpreting dKL as constraining probability redistribution and overcorrection rather than directly demoting PII.Source: I Internal PTPC Ablations · p. 13
  • Removing the current-task anchor raises low-sensitivity NLL from 2.9031 to 3.4055 while reducing Last by only 0.44 percentage points, indicating that it primarily constrains current-task non-PII drift.Source: I Internal PTPC Ablations · p. 13
  • Removing the old-task anchor lowers BWT from −0.93% to −3.07% and final FOMC accuracy from 79.67% to 70.00%, the largest retention degradation among the reported ablations.The old-task anchor is therefore the strongest constraint against accumulated drift on previously learned tasks.Source: I Internal PTPC Ablations · p. 13

J Difficulty-Matched Selectivity Evaluation

The evaluation compares annotated PII with same-example non-PII spans matched on pretrained-model difficulty, then assesses selectivity and observed-token vocabulary rank. The reported matched-control evidence is positive for SPARK full and the variant without dKL.

J Difficulty-Matched Selectivity Evaluation

  • The matched-control set pairs all 70 annotated PII spans from 51 source examples with non-PII spans from the same examples, matching pretrained NLL within 0.5.The set contains 261 PII token pieces; confidence intervals use 2,000 source-cluster bootstrap replicates.Source: J Difficulty-Matched Selectivity… · p. 13
  • Δsel is NLLPII − NLLMatched; a confidence interval above zero indicates selectively lower conditional likelihood for annotated PII than comparable non-PII content.Within-example pairing and pretrained-difficulty matching reduce confounding from context and token frequency.Source: J Difficulty-Matched Selectivity… · p. 13
  • Only SPARK full and the variant without dKL have pooled matched-control Δsel intervals entirely above zero.Source: Table 10 · p. 14
  • For observed PII-token vocabulary ranking, SPARK full shows its clearest separation at Top-1; Top-5 and Top-10 intervals overlap more.The ranking evaluates how often an observed PII token appears among the model’s highest-probability next-token predictions.Source: Table 11 · p. 14

K PTPC Convergence Analysis

PTPC applies losses to distinct token sets and anchors non-PII behavior; experiments report empirical optimization stability, while membership-inference results do not support broad sample-level leakage reduction.

K PTPC Convergence Analysis

  • Privacy losses target PII positions, while current- and old-task anchor losses target separate non-PII sets; normalization prevents sparse PII loss from dilution.Source: K PTPC Convergence Analysis · p. 14
  • Across six tasks under three task orders, total loss decreased monotonically with a 1 × 10−5 learning rate and 200 optimization steps, without observed instability.Source: K PTPC Convergence Analysis · p. 14
  • Across runs, final loss fell below 95% of its initial value, NLLpii rose, NLLlow stabilized after approximately 50 steps, and |ΔLast| stayed below 0.01.Source: K PTPC Convergence Analysis · p. 15

L Membership Inference Attack Evaluation

M Canary Conditional-Likelihood and Candidate-Ranking Evaluation

The canary evaluation distinguishes lower absolute likelihood from candidate-ranking resistance. Its results support selective suppression on the pooled annotated benchmark, but not broader privacy guarantees or attack resistance.

M Canary Conditional-Likelihood and Candidate-Ranking Evaluation

  • The canary set has 300 annotations for 50 planted canaries, each compared with 511 negative candidates; ranking uses −avg_nll and 2,000 canary-clustered bootstrap replicates.Source: M Canary Conditional-Likelihood and… · p. 15
  • SPARK full had normalized full rank 227.90 versus pretrained 239.00 and Full Top-10 3.67% versus 3.33%, with overlapping intervals; lower canary NLL did not imply harder candidate distinction.Source: M Canary Conditional-Likelihood and… · p. 15

N Scope of the Privacy Evidence

  • Evidence supports selective conditional-likelihood suppression on the pooled annotated benchmark, but not general candidate-ranking resistance, reduced membership leakage, formal DP, parameter erasure, or arbitrary black-box non-extractability.Source: N Scope of the Privacy Evidence · p. 15
  • Table 12 treats AUC near 0.5 and lower TPR as preferable, while Table 14's broad interval overlap does not show general candidate-ranking resistance.Source: Table 12 · p. 16, Table 14 · p. 16
Loading 2609.29711v1…