PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries

Yanhang Li, Zhichao Fan, Zexin Zhuang

arXiv:2605.18891v1cs.LGcs.AI

TL;DR

Reasoning-model unlearning audits can mistake parser-split trace leakage for evidence of retained weight-level memorization. The paper tests this interpretation with fixed-weight decode-time and teacher-forced probes on LoRA-memorized fictional authors, finding that prefill and format changes alter the gap and motivating a paired sanity check.

  • Problem

    It is unclear whether a positive parser-split bypass gap identifies weight-level memorization or reflects the measurement process itself.

  • Method

    The study audits NPO unlearning on 60 LoRA-memorized fictional authors using six-token head-conditioned parser-split, fixed-prefill, and teacher-forced probes.

  • Results

    On Qwen-7B, out acc falls from 1.00 to 0.60 while thk leak stays at 0.83 and ∆=+0.23, but a prefill swap changes out acc from 0.60 to 0.37 on the same weights.

  • Takeaways & Limitations

    A positive parser-split gap does not by itself distinguish residual weight-level memorization from decode-time prefix sensitivity, so audits should pair template swaps with teacher-forced scoring.

  • Takeaways & Limitations

    The evidence is limited to synthetic, small-scale LoRA memorization and two DeepSeek-R1 distillates, with seed-sensitive prefill effects and no claim about practical privacy or IP risk.

Abstract

from arXiv · show

Evaluations of unlearning on reasoning models sometimes show a bypass pattern. The answer side looks unlearned, but the model's own thinking trace keeps emitting the forgotten content, and the gap is taken as evidence that the weights still remember. We audit this reading on DeepSeek-R1-Distill-Qwen-7B with LoRA-memorized fictional authors and NPO unlearning, conditioned on a six-token canary head. On one seed, swapping the thinking trace for a short non-canary prefill on the same weights drops the answer rate by as much as the bypass gap itself, whether the prefill mimics the training template or not. On a second seed the bypass gap shrinks rather than vanishing, and the prefill swap reverses direction and brings the answer rate to ceiling. A positive parser-split bypass gap thus does not by itself identify hidden weight-level memorization, and does not rule it out either. On a different distillate the same metric flips sign because the parser cannot find the closing tag. We recommend a decode-time template swap as a cheap sanity check alongside the canonical audit.

1. Introduction

The paper audits whether parser-split bypass gaps in reasoning-model unlearning reveal residual weight-level memorization or instead reflect decode-time context and parsing artifacts. Fixed-weight prefill and teacher-forced probes show that the gap is not self-identifying evidence, motivating a paired audit paradigm.

  • Motivation: Reasoning-model audits split generations into a thinking trace and answer, but answer-masked unlearning leaves the trace outside the forgetting loss.The audited format is ⟨think⟩τ⟨/think⟩a, with unlearning applied to a and parsing performed after the closing tag.
  • Motivation: The parser-split bypass reading treats a suppressed answer and persistent trace emission as evidence that weights retain forgotten content.The paper tests this as a measurement claim rather than assuming it is established.
  • Setup: The audit uses 60 fictional authors memorized into DeepSeek-R1-Distill-Qwen-7B with LoRA and then unlearned using NPO.Each probe is conditioned on the first six canary tokens, making answer and trace rates head-conditioned continuation scores.
  • Results: After NPO, out acc falls from 1.00 to 0.60 while thk leak remains 0.83, yielding ∆=+0.23.In 86% of bypass cases, the answer is only the question prefix while the trace contains the trained bio.
  • Results: A short non-canary prefill on the same weights lowers out acc from 0.60 to 0.37, showing that the gap is sensitive to decode-time context.BIO and META prefills produce the same rate, while EMPTY lowers it to 0.20; the intervention also perturbs ordinary QA probes.
  • Implications: Teacher-forced continuation scoring remains 0.90–1.00 across prefills, while Llama-8B produces −0.92 because format drift leaves the parser with an empty trace.The paper recommends a paired fixed-prefill decode-time swap plus teacher-forced scoring as one audit paradigm.
  • Scope: The conclusions are limited to synthetic, small-scale LoRA memorization and two DeepSeek-R1 distillates, and the prefill effect is seed-sensitive.The paper does not claim to establish or bound practical privacy or IP extraction risk.

2. Related Work and Positioning

The paper positions its contribution as a construct-validity audit of reasoning-model unlearning metrics. It connects prior answer-only unlearning, trace leakage, and evaluation-robustness work while arguing that no single parser-split metric should be trusted alone.

  • LLM unlearning: Prior language-model unlearning methods include Gradient Ascent, WMDP representation misdirection, anchored rewriting, and NPO, with TOFU supplying synthetic-bio forgetting evaluations.The paper studies measurement confounding rather than proposing or improving an unlearning method.
  • Reasoning-model unlearning: Concurrent 2025 studies show that answer-level unlearning can leave residual content in reasoning traces and develop trace-aware objectives or suppression attacks.This work asks whether the parser-split gap itself identifies weight-level memorization.
  • Evaluation robustness: Robustness research argues that no single unlearning metric suffices, while extraction attacks show that deleted information can remain recoverable.The paper adds a reasoning-specific confound from the trained trace template.
  • Positioning: The construct-validity argument is that the surface form of the thinking trace is not a transparent readout of model weights, consistent with prompt-format sensitivity and faithfulness concerns.A cheap decode-time intervention is presented as a diagnostic for this confound.

3. Setup and the Bypass Metric

The audit measures head-conditioned canary continuation separately in the thinking trace and answer, then tests whether their gap remains invariant when the fixed-weight decode prefix changes. Its interpretation is limited because the trace is trained to restate the bio, the forgetting loss targets only the answer, and substring matching can misread prefixes or paraphrases.

  • Setup: The audit memorizes 60 fictional authors with LoRA, trains bio-restating thinking traces, and applies NPO with forgetting loss masked to the answer span.Each author has one canary probe in the headline bypass audit.
  • Bypass Metric: The parser splits each generation at the closing thinking tag and tests exact containment of each full canary in the answer and thinking trace.The six-token canary head is supplied in the prompt, so the scores measure conditioned continuation rather than free recall.
  • Interpretation: A positive bypass gap is naturally read as residual weight-level memorization, but it is also consistent with template echo because the bio template receives no forgetting gradient.The two interpretations predict different behavior when the weights are fixed and the thinking prefix changes.
  • Prefill Intervention: The prefill intervention replaces autoregressive thinking with BIO, META, or EMPTY prefixes and compares answer accuracy while holding the weights fixed.In prefill arms, thinking leakage is zero by construction, so the comparison concerns answer accuracy and decode-time invariance.
  • Metric Caveats: The metric averages binary answer and thinking indicators over n=60 authors, while exact containment may overcount prefix echoes or undercount paraphrases.The table distinguishes the fixed-weight contrast ∆AB from the parser-split bypass gap ∆.

4. Results

The parser-split bypass replicates on Qwen-7B, but fixed-weight prefill and teacher-forced probes show that its magnitude and interpretation are not stable. On Llama-8B, format drift makes the same metric a parser failure rather than evidence about weights.

  • 4.1. The bypass replicates under the bio template: NPO reduces Qwen-7B out acc from 1.00 to 0.60 while thk leak remains 0.83, producing a bypass gap of +0.23 at K=1600.The 95% CI is [0.13, 0.35] over 60 authors; the gap grows from −0.02 at K=100 to +0.23 at K=1600.
  • 4.2. Inference-time prefill on identical weights: On Qwen-7B seed 0, replacing autoregressive thinking with a non-canary prefill lowers out acc from 0.60 to 0.37, while EMPTY lowers it to 0.20.BIO and META prefills both yield 0.37, so the intervention estimates decode-time non-invariance rather than isolating a scratchpad mechanism.
  • 4.2. Inference-time prefill on identical weights: Across K on seed 0, ∆AB remains positive and tracks the bypass gap, whereas seed-1 bypass gaps vary in magnitude and include both negative and positive values.Seed-0 ∆AB values are {0.10, 0.27, 0.27, 0.23}; seed-1 gaps are {−0.10, +0.08, +0.03, +0.18}.
  • 4.4. Parser-field failure under format drift: On Llama-8B, missing closing tags move the bio outside the parsed thinking trace, leaving out acc at 1.00 while parser-reported thk leak approaches zero.The tag is missing on at least 60% of probes, making this a parser false negative under format drift.
  • 4.4. Parser-field failure under format drift: The Llama-8B bypass gap changes from −0.92 to +0.92 when the parser fallback convention is reversed on the same outputs.This demonstrates that the reported sign depends on parser convention in this failure case.

5. Discussion

Across seeds and models, the parser-split metric is not invariant to thinking-trace prefills at fixed weights, so a positive bypass gap cannot by itself identify residual weight-level memorization. The paper recommends paired decode-time and teacher-forced checks, while limiting its claims to a synthetic, small-scale audit setting.

  • 5. Discussion: On Qwen seed 0, the +0.23 bypass gap matches the out-accuracy change after a prefill swap; on seed 1, the swap reverses direction and lifts out acc to ceiling.The stable conclusion is fixed-weight non-invariance, not a universal effect size or direction.
  • 5. Discussion: The authors recommend paired prefill and teacher-forced arms, plus trace-aware unlearning methods, as future evaluation directions for larger real-world forget corpora.The prefill check adds one decode pass per probe, while trace-aware methods would apply forgetting pressure to the reasoning trace.

A. Training-time template comparison

The training-time comparison varies the thinking-trace template across separately memorized Qwen-7B adapters before identical answer-span NPO unlearning. The bypass gap appears only with BIO, while residual emissions, prompt shifts, and incomplete META sweeping limit interpretation.

  • A. Training-time template comparison: Three adapters memorize the same forget set under BIO, META, or NONE thinking-trace templates before NPO unlearning.The comparison is weaker than the fixed-weight inference-time intervention because the adapters have different LoRA deltas.
  • A. Training-time template comparison: Under the hidden-channel hypothesis, bypass should survive META or NONE, whereas template echo predicts confinement to BIO.
  • A. Training-time template comparison: The bypass gap is confined to BIO; META output is pinned to 1.00, while NONE reverses the gap sign.META carries no bio content, so its uniform template leaves no output suppression for a bypass to overcome.
  • A. Training-time template comparison: The NONE adapter still emits non-empty traces on ∼45% of canary probes and uses prompts ∼60 tokens shorter than the bio-template prompt.Thus the training-time comparison mixes residual emission with prompt-distribution effects.
  • A. Training-time template comparison: The fixed-weight prefill intervention holds weights and prompts constant but jointly varies trace presence, prefix length/style, and canary content.
  • A. Training-time template comparison: META was run only at K=100 because output accuracy was 1.00 and thinking leak was 0, leaving no room for a gap to move.No META sweep was run at K ∈{400, 800, 1600}.
  • B. Full NPO K-sweep under the bio template: Under the bio template, the autoregressive gap is near zero at K=100 and grows monotonically through higher K values.The reported prefill-sweep comparison labels AUTO − BIO-prefill as ∆AB rather than a causal scratchpad-content effect.

D. Mechanistic anatomy of the bypass cases

The bypass cases are usually truncated answer generations paired with intact bio traces, rather than clear evidence that the model knows the answer but declines to provide it. Their pattern is consistent with template echo but does not prove that mechanism.

  • D. Mechanistic anatomy of the bypass cases: On 12 of 14 bypass probes (86%), the output is the question prefix verbatim with no continuation.For Zephyr Okonkwo, the answer stops after “The Crimson Tide of Calabar was,” while the thinking trace contains the full bio template.
  • D. Mechanistic anatomy of the bypass cases: Both channels carry the canary on probes with mean output length 94 chars, versus 36 chars for bypass cases.
  • D. Mechanistic anatomy of the bypass cases: The bypass pattern is consistent with answer-side truncation plus an untouched thinking-side template echo, not proof that the model knows the answer but chooses not to say it.The inference-time prefill probe remains sensitive to several possible patterns.
  • D. Mechanistic anatomy of the bypass cases: Gradient Ascent at K ≥400 drives both output accuracy and thinking leak to exactly 0.00 on canary and QA probes.The trained-empty arm shows the same 0.00/0.00 collapse from K=400 onward.

F. Seed replication

The second seed reproduces a positive, significant K=1600 gap but changes intermediate estimates and reverses the prefill response. This shows that the bypass metric and prefill effect are seed-sensitive.

  • F. Seed replication: At K={100, 400, 800, 1600}, seed-1 gaps are −0.10 [−0.18, −0.03], +0.08 [−0.05, 0.22], +0.03 [−0.12, 0.18], and +0.18 [0.05, 0.32].
  • F. Seed replication: At K=1600, both seeds have positive gaps with confidence intervals excluding zero, but point estimates differ: +0.23 versus +0.18.Intermediate seed-1 gaps are positive but have intervals that straddle zero.
  • F. Seed replication: On seed 1, prefill output accuracy is 1.00 for BIO and META and 0.98 for EMPTY.This contrasts with the seed-0 autoregressive drop from 1.00 to 0.60.
  • F. Seed replication: The seed-1 results show full canary retention under every thinking-template arm, so the seed-0 prefill drop does not replicate as a weight-level memorization loss.The finding is consistent with the matched teacher-forced result on the same adapter.

G. Retain-set utility on Qwen-7B

The retain-set check evaluates generic knowledge separately from forget-side behavior. On Qwen-7B, NPO causes a substantial K=1600 retain-accuracy loss and can produce fictional-author substitutions.

  • G. Retain-set utility on Qwen-7B: The utility evaluation uses 60 generic-knowledge probes unrelated to the fictional-author forget set.Probes cover capitals, chemical symbols, famous literature, and basic biology.
  • G. Retain-set utility on Qwen-7B: By K=1600, retain accuracy drops by 25 percentage points on Qwen-7B.
  • G. Retain-set utility on Qwen-7B: Some retain-set failures answer generic questions with names from the fictional-author forget set.For example, “the largest mammal” receives a fictional author’s name.

H. Inference-time prefill implementation

The implementation replaces autoregressive thinking generation with fixed prefills, then resumes generation on the same adapter weights across multiple template modes.

  • Prefill intervention: The intervention replaces autoregressive ⟨think⟩ generation with a fixed prefill before allowing the model to continue.This implementation is used to estimate decode-time non-invariance while keeping weights fixed.
  • Distribution control: The prefill strings reuse training-data opening sentences so each mode remains in-distribution for its corresponding adapter.This is intended to avoid adding a new distribution shift beyond the one being measured.
  • Prefill modes: Bio, meta, and empty modes provide distinct prefill choices, including an empty string for the no-prefill condition.The bio and meta strings use author- and task-specific prompts, while none returns an empty prefill.
  • Evaluation setup: The driver evaluates bio-trained adapters under all three prefilled thinking templates at fixed weights.The command-line configuration includes bio, meta, and none modes for the prefill comparison.

I. Teacher-forced canary log-probability probe

The teacher-forced probe scores continuation behavior for the same canary under matched prefills, separating canary-conditioned token preference from greedy decoding effects.

  • Probe design: Teacher forcing scores the canary continuation after supplying its first six tokens as a head-conditioned context.The score is not a free-recall estimate because the canary head is provided to the model.
  • Uncertainty: Confidence intervals are computed with an author-clustered bootstrap over n=60.
  • Controls: The probe removes autoregressive drift, substring-match thresholds, and prefix-length interactions with next-token entropy in greedy decoding.It does not remove differences in prefill length or style within the context itself.
  • Results: On Qwen-7B seed-0 NPO-K=1600, greedy substring recall falls 1.00 →0.60 while teacher-forced top-1 match remains 0.96 for bio and meta prefills or 0.90 for empty.On seed-1, teacher-forced top-1 is ≥0.997 across prefills; on Llama-8B it is ≥0.998 regardless of prefill.
Loading 2605.18891v1…