PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring the Depth of LLM Unlearning via Activation Patching

Jaeung Lee, Dohyun Kim, Jaemin Jo

arXiv:2605.24614v1cs.CLcs.AIcs.LG

TL;DR

Auditing whether unlearned knowledge is genuinely removed is difficult because output-level metrics can miss recoverable internal representations, while existing white-box methods lack a generalizable score. The paper introduces UDS, an activation-patching metric, and finds it achieves the highest faithfulness and robustness across 20 metrics evaluated on 150 models spanning 8 methods.

  • Problem

    Existing output-level metrics can miss recoverable internal knowledge, while white-box approaches often require auxiliary training or dataset-specific adaptations, leaving no generalizable score for comparison.

  • Method

    UDS uses two-stage activation patching with a retain-model baseline to identify knowledge-encoding layers and quantify erased knowledge on a 0–1 scale.

  • Results

    AUC-ROC 0.971 faithfulness and HM 0.932 robustness were highest among 20 metrics evaluated on 150 unlearned models spanning 8 methods.

  • Takeaways & Limitations

    UDS provides a causal metric for detecting residual knowledge and supports integration into existing benchmarking frameworks and streamlined evaluation pipelines.

  • Takeaways & Limitations

    Rigorous UDS quantification requires a retain model, which may be unavailable in some deployment contexts.

Abstract

from arXiv · show

Large language model (LLM) unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether target knowledge is truly erased remains challenging. Existing output-level metrics fail to detect when this knowledge remains recoverable from internal representations. Recent white-box studies reveal such residual knowledge but often rely on auxiliary training or dataset-specific adaptations, leaving no generalizable metric. To address these limitations, we propose the Unlearning Depth Score (UDS), a metric that quantifies the mechanistic depth of unlearning via activation patching. UDS first identifies layers that encode the target knowledge using a retain model baseline, then measures how much of it is erased in the unlearned model on a 0-1 scale. In a meta-evaluation across 20 metrics on 150 unlearned models spanning 8 methods, UDS achieves the highest faithfulness and robustness, confirming our causal approach as the most reliable for unlearning evaluation. Case studies further reveal that white-box metrics can disagree at the layer level and that erasure depth varies across examples. We provide guidelines for integrating UDS into existing benchmarking frameworks and streamlining the evaluation pipeline. Code and data are available at https://github.com/gnueaj/unlearning-depth-score

1 Introduction

LLM unlearning aims to remove sensitive or hazardous target knowledge while preserving general capabilities, but output-level evaluation can miss knowledge that remains recoverable internally. The paper introduces UDS, a training-free causal metric based on activation patching, and reports the strongest faithfulness and robustness in a meta-evaluation of 20 metrics across 150 models and 8 methods.

  • Motivation: LLMs memorize training data, creating privacy and AI-safety risks, while unlearning seeks to remove target knowledge without sacrificing general capabilities.The intended goal is a model indistinguishable from one trained without the target data.
  • Evaluation challenge: Output-based metrics can miss residual knowledge because lightweight fine-tuning or activation manipulation may restore ostensibly erased information.This motivates white-box analyses that inspect internal representations rather than relying only on output logits.
  • Proposed metric: UDS is a training-free, causal, dataset-invariant metric that measures mechanistic unlearning depth through two-stage activation patching and produces per-example scores from 0 to 1.It first identifies knowledge-encoding layers using retain-model hidden states, then tests whether knowledge remains recoverable across layers.
  • Results: 20 metrics on 150 unlearned models spanning 8 methods yielded UDS’s highest faithfulness, AUC-ROC 0.971, and robustness, HM 0.932.Case studies found residual knowledge obscured by representational shifts and showed that erasure depth varies across examples within one method.

2 Background and Related Work

Machine unlearning seeks to remove forget-set knowledge while preserving general capabilities, using optimization-, response-, representation-, and distribution-based methods. Evaluation combines memorization, privacy, and utility benchmarks with white-box analyses showing that residual knowledge can persist internally.

  • Objectives: Machine unlearning aims to produce a model indistinguishable from one trained only on the retain set while preserving general capabilities.
  • Methods: Gradient ascent maximizes forget-set loss but can cause catastrophic collapse, while GradDiff jointly retains Dr performance yet remains fragile because opposing gradients must be balanced.
  • Methods: NPO, IdkNLL, IdkDPO, AltPO, RMU, and UNDIAL respectively use preference reformulation, alternative responses, positive feedback, representation intervention, or adjusted-logit self-distillation.
  • Evaluation Frameworks: Unlearning evaluation typically measures memorization, privacy, and utility, with TOFU, MUSE, WMDP, and OpenUnlearning providing benchmark frameworks.
  • White-box Analysis: White-box methods including CKA, Logit Lens, Fisher Information, activation patching, probes, causal tracing, value-vector projections, and parameter restoration reveal internally retained forget-set knowledge.These techniques compare representations, decode hidden states, measure parameter sensitivity, test causal effects, or inspect knowledge traces across model components.

3 The UNLEARNING DEPTH SCORE

UDS measures unlearning depth by using activation patching to identify where forget-set knowledge is encoded and quantify how much remains recoverable after unlearning. It aggregates layer-wise erasure relative to a retain-model baseline into a 0–1 score, with efficient reuse across models and consistent ordering across scales.

  • Core procedure: UDS uses two-stage activation patching: Mret into Mfull identifies knowledge-encoding layers, then Munl quantifies residual recoverability.Mfull is the target because it learned forget-set knowledge and can decode it from patched hidden states.
  • Score aggregation: The Layer Erasure Ratio measures each layer’s erasure as a fraction of its baseline, clipped to [0, 1], and UDS weights layers by baseline knowledge encoding.A score of 1 matches Mret-level erasure, whereas 0 indicates fully intact knowledge; examples without KE layers are excluded.
  • Efficient implementation: Stage 1 references and KE layer sets can be cached, so evaluating another unlearned model requires only hidden-state extraction and Stage 2 patched forward passes.Teacher forcing computes all entity-token predictions in one forward pass per layer, avoiding autoregressive generation latency.
  • Scale validation: Across Llama 1B, 3B, and 8B, UDS preserves the ordering full < retain99 < retain95 < retain90 at every scale.UDS values track the fraction of the forget set not seen during training and decrease slightly with scale.

4 Meta-Evaluation

UDS is meta-evaluated against 19 alternatives for faithfulness and robustness on unlearning models, using pool separation and intervention stability tests. It ranks first overall, achieving the highest faithfulness and aggregate robustness.

  • Evaluation Protocol: The evaluation measures faithfulness with AUC-ROC for separating P-pool and N-pool models, and robustness under 4-bit quantization and one-epoch relearning.Robustness is restricted to models retaining at least 80% of Mfull utility and classified as unlearned by each metric’s faithfulness threshold.
  • Faithfulness: UDS achieves the highest faithfulness (AUC 0.971), exceeding Truth Ratio (0.947), the second-best and top output-level metric.CKA (0.648) and Fisher (0.712) show poor separation among white-box baselines.
  • Robustness: UDS ranks first in aggregate robustness (HM = 0.932), with balanced quantization (Q = 0.968) and relearning (R = 0.900) stability.It evaluates intermediate representations through Mfull’s unperturbed computational pathways and bypasses the unlearned model’s output head.
  • Robustness: CKA collapses under relearning (R = 0.013), Fisher is most vulnerable to quantization (Q = 0.583), and Truth Ratio degrades under relearning (R = 0.234).ROUGE variants also show relearning stability of R = 0.06–0.20.
  • Overall: UDS achieves the highest overall score (0.951) across all 20 metrics, supporting causal evaluation as the most reliable approach to knowledge erasure.The comparison includes output-level, retain-referenced, and internal metrics.

5 Case Studies

Case studies show that UDS detects residual knowledge hidden by observational metrics and exposes substantial variation in erasure depth across prompt types and examples. In particular, representational shifts can make Logit Lens falsely indicate erasure, whereas UDS identifies recoverable knowledge and deeper failures in specific question types.

  • 5.1 Observational vs. causal evaluation: UDS’s causal evaluation avoids Logit Lens failures caused by representational shifts that make a frozen decoder miss retained forget-set knowledge.UDS measures whether knowledge remains recoverable despite rotations or distortions of the internal vector space.
  • 5.2 Prompt-type variation: 0.624: Yes/No questions show much higher UDS erasure than other prompt types, which score 0.025–0.049 in one IdkNLL model.The “I don’t know” response negates factual answers for Yes/No questions and modifies deeper knowledge-encoding layers, unlike other prompt types.
  • 5.1 Observational vs. causal evaluation: 0.209: UDS finds the same IdkDPO model’s knowledge highly recoverable across mid-layers, with LER ≤0.230, while both metrics converge to 0.254 at the final layer.The final-layer convergence occurs where representations directly determine output logits, but the mid-layer divergence reveals retained knowledge.
  • 5.2 Prompt-type variation: 0.039–0.253: UDS differentiates IdkNLL configurations that all score approximately 0.0 on normalized MIA variants.This demonstrates that output-level behavior can appear uniformly successful while internal erasure depth differs across configurations.

6 Practical Implications

UDS broadens privacy evaluation beyond output-level membership-inference metrics by exposing internal erasure differences and changing method and configuration choices. Its stability under quantization and relearning supports using pre-perturbation UDS to reduce evaluation overhead.

  • Integrating UDS into the privacy evaluation axis: Adding UDS to the privacy axis swaps NPO and SimNPO’s ranks, revealing internal erasure differences that MIAagg alone misses.The integrated formula uses Privacy = HM(MIAagg, UDS), whereas the original uses Privacy = MIAagg.
  • Integrating UDS into the privacy evaluation axis: NPO combines MIAagg = 0.875 with UDS = 0.619, while SimNPO has higher UDS = 0.739 despite lower MIAagg.SimNPO’s length-normalized, reference-free objective drives unlearning pressure deeper into internal representations.
  • Streamlining hyperparameter selection: With the UDS formula, AltPO and NPO shift toward higher learning rates or longer training, steering hyperparameter selection toward deeper internal erasure.The retention coefficient is α, and both methods improve UDS after these configuration shifts.
  • Streamlining the evaluation pipeline: UDS remains highly stable under quantization and relearning, yielding the highest aggregate robustness and enabling pre-perturbation evaluation.Current frameworks otherwise require perturbing each model and rerunning the evaluation suite, creating substantial computational overhead across large model pools.

7 Conclusion

The paper presents UDS, a two-stage activation-patching metric that quantifies the mechanistic depth of unlearning through knowledge recoverability. UDS yields 0–1 erasure scores and achieved the highest faithfulness and robustness among 20 evaluated metrics.

  • UDS quantifies unlearning depth by measuring knowledge recoverability through two-stage activation patching.
  • 0–1 erasure scores let UDS complement output-only metrics with causal intervention.
  • 20 metrics were compared in a meta-evaluation, where UDS achieved the highest faithfulness and robustness.

Limitations

UDS has practical and methodological limitations: rigorous quantification requires a retain model, clipping can hide over-unlearning, evaluation is currently limited to TOFU, and its scope beyond localized entity spans remains uncertain.

  • Requirement of a Retain Model: Rigorous UDS quantification requires a retain model, which may be unavailable in some deployment contexts.Without one, practitioners can use quantification alone by patching hidden states from the unlearned model into the original pre-unlearning model, though this omits retain-model information.
  • Clipping and Over-Unlearning: UDS clipping caps scores at 1.0, making over-unlearning mathematically indistinguishable from perfect unlearning.Practitioners should jointly monitor UDS and utility to diagnose representations that deviate beyond the retain model.
  • Dataset: The current evaluation focuses on the TOFU benchmark, so validation on other benchmarks is needed to strengthen UDS’s generality.The scope follows the meta-evaluation framework of Dorna et al. (2025).
  • Entity Span: UDS currently targets localized entity spans under teacher forcing, leaving its extension to long-form or multi-step reasoning targets open.The automatic extraction pipeline handles structured QA pairs but may not generalize to all data formats.

Broader Impact … B.4 Score Computation for Method Ranking

The paper positions UDS as a causal audit for incomplete unlearning that should complement comprehensive safety pipelines, and situates evaluation within a multi-method, retain-referenced benchmarking framework. The appendix defines comparison metrics, sensitivity analysis, and harmonic-mean score computation for method ranking.

  • Broader Impact: UDS can detect incomplete unlearning for responsible deployment and regulatory compliance, but should not be treated as an absolute guarantee of data removal.The authors recommend integrating UDS into a comprehensive safety pipeline.
  • Broader Impact: The activation-patching framework could audit layered architectures beyond autoregressive language models, although adapting its concrete metric formulation remains future work.Potential targets include diffusion models and vision transformers.
  • B Metric Definitions: The metric definitions span output-level extraction, memorization, probability, truth-ratio, ROUGE, paraphrase, jailbreak, and membership-inference measures.These include ES, EM, Prob, ParaProb, Truth Ratio, ROUGE variants, and four MIA scoring functions aggregated by AUC.
  • B.1 Output-Level Metrics: Retain-referenced and white-box metrics include normalized MIA, CKA, Logit Lens, and Fisher Masked, measuring deviation, geometry, decodable knowledge, or parameter importance.Normalized MIA higher scores indicate less deviation from retain and therefore more erased knowledge.
  • B.3 Mask Fraction Sensitivity: Fisher Masked robustness and faithfulness are evaluated across three mask fractions, with p = 0.1% selected as representative.The reported robustness dimensions are quantization and relearning.
  • B.4 Score Computation for Method Ranking: Method-ranking scores use HM(Memorization, Privacy, Utilityrel), with Privacy computed as MIAagg alone or HM(MIAagg, UDS).Utilityrel normalizes utility against the full model, while Model Utility is a harmonic mean over nine QA metrics and Fluency measures nongibberish generations.

C Dataset Details … E.1 Robustness Attack Settings

The paper specifies systematic entity-span extraction, evaluates activation-patching choices and threshold sensitivity, tests entity-length bias, and defines quantization and relearning robustness attacks. These settings support consistent measurement of knowledge signals and unlearning depth.

  • C Dataset Details: Each TOFU forget10 benchmark contains 400 forget-set QA pairs, with answers partitioned into contextual prefixes and target entity spans.GPT-5.2 supplies the exact character index separating each prefix from its factual entity without modifying the original text.
  • C Dataset Details: Representative prefix–entity partitions are provided for the prompt types used in the dataset construction.Table 10 documents examples of these partitions.
  • D.1 Component Patching: Full layer output h_l produces the largest mean S1 delta across all tested layers, so UDS patches the residual stream by default.Four patching locations were compared to identify which component carries the knowledge signal.
  • D.2 KE Threshold Sensitivity: The KE layer set is determined from S1 deltas between the retain and full models, with its size evaluated across six threshold values.Table 11 reports how many knowledge-erasing layers are selected and notes examples skipped when no layer has ΔS1.
  • D.2 KE Threshold Sensitivity: τ=0.05 is the default because skipped examples remain stable at 6 (1.5%) from τ=0.03 to 0.05, then increase to 14 (3.5%) at τ=0.10.The threshold stabilizes the UDS denominator and filters noise from low-delta layers.
  • D.3 Entity Length Bias: UDS shows no consistent directional bias from entity token length, with mixed-sign correlations satisfying |ρ| < 0.24 across methods.The analysis directly tests whether entity length biases the score.
  • E.1 Robustness Attack Settings: Quantization uses BitsAndBytes 4-bit NF4 with bfloat16 computation and applies no post-quantization calibration or fine-tuning.Relearning uses one epoch on D_f with learning rate = 2 × 10^-5, batch size = 8, gradient accumulation = 4, effective batch size = 32, and AdamW.

E.2 Full Per-Metric Plots

Per-metric plots evaluate robustness under NF4 quantization and one-epoch relearning, with filtered and unfiltered utility analyses. White-box metrics are plotted as 1 − score so higher values indicate more knowledge, while reference lines and recovery counts expose metric stability.

  • Robustness analyses: Figures 6 and 8 plot quantization and relearning robustness after applying both utility (≥0.8) and faithfulness filters.Figure 6 compares metric values before versus after NF4 4-bit quantization; Figure 8 compares values before versus after one epoch of relearning.
  • Robustness analyses: Figure 7 applies utility filtering only and shows score drops across many models for Extraction Strength and ROUGE.The plots report each metric before and after NF4 4-bit quantization, alongside counts of recovery and destruction.
  • Metric visualization: CKA, Fisher, Logit Lens, and UDS are plotted as 1 − score, aligning higher values with more knowledge.This transformation makes white-box metrics consistent with the output-based metrics in the robustness plots.
  • Plot interpretation: Quantization plots include recovery or destruction counts, sample sizes n, and a y = x stability reference with background shading for deviation.Figure 6 defines n as models passing both filters, whereas Figure 7 defines n using the utility filter only.
  • Plot interpretation: Relearning plots report over-recovery and under-recovery relative to y = x + ∆ret, with y = x shown as a dotted reference.The shifted dashed line represents expected behavior given the retain model’s shift.
Loading 2605.24614v1…