PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Recoverability of Private Information Unlearning in Large Language Models

Shicheng Hu, Runzhi Tian, Ziqiao Wang, Yongyi Mao

arXiv:2608.29943v1cs.LGcs.CL

TL;DR

LLMs can retain sensitive information after unlearning, making it difficult to determine whether privacy-relevant data were erased or merely hidden. The paper constructs the FPI dataset and a white-box auditing framework, then shows that restricted inverse-greedy decoding can recover supposedly forgotten information, motivating stronger auditing and unlearning methods.

  • Problem

    The paper addresses whether private information in LLMs is genuinely removed by unlearning or merely suppressed, a question central to data privacy.

  • Method

    The paper builds the synthetic FPI dataset and evaluates unlearned models with white-box auditing and restricted inverse greedy decoding.

  • Results

    Similar recovery occurs across existing unlearning methods, with up to 97% of supposedly forgotten postcodes recovered after gradient-ascent unlearning with KL regularization.

  • Takeaways & Limitations

    Current unlearning techniques often fail to fully eliminate sensitive information, so privacy auditing must test internal recoverability rather than direct outputs alone.

  • Takeaways & Limitations

    The recovery strategies use only model logits and do not fully exploit other white-box signals such as retain data, unlearning configurations, or latent representations.

Abstract

from arXiv · show

Large language models (LLMs) can memorize sensitive information, raising serious privacy concerns. Machine unlearning offers a potential solution to remove such information, but it remains unclear whether existing methods truly erase it or merely hide it within the model. A key challenge is quantifying the persistence of sensitive data under a unified evaluation framework. To address this, we construct a synthetic dataset containing fake private information and propose a white-box auditing framework to systematically assess whether claimed-forgotten information is genuinely removed. Using this framework, we evaluate five existing unlearning methods and find that a simple "inverse greedy" decoding -- selecting the least likely token at each step -- can recover supposedly forgotten private information. Our results reveal that current unlearning approaches often fail to fully eliminate sensitive information, highlighting the need for more reliable methods to ensure privacy in deployed LLMs.

1 Introduction

PI unlearning seeks to remove structured, sensitive information while verifying that it is genuinely erased rather than merely hidden. The paper introduces a synthetic benchmark and white-box auditing approach, showing that inverse-greedy recovery can expose supposedly forgotten information.

  • Motivation: PI unlearning targets sensitive information about individuals, motivated by privacy requirements and the need to verify genuine removal.Unlike general knowledge unlearning, private information is often structured and unique, with identifiable ground-truth values.
  • Motivation: Existing knowledge-unlearning datasets and findings do not necessarily transfer to PI unlearning because private information has distinct structural and semantic properties.Private information is generally structured, unique, and less entangled than factual knowledge.
  • Auditing framework: White-box auditing gives the auditor the unlearning algorithm and model, enabling direct testing of whether target information was erased or concealed.The framework is intended to assess recovery using internal model details rather than black-box behavior alone.
  • Dataset: The FPI dataset uses synthetic question–answer pairs covering year of birth, blood type, social insurance number, and postcode.Its controllable fields support attribute-specific ground-truth tracking while avoiding privacy risks from real data.
  • Auditing framework: Restricted inverse greedy decoding searches low-likelihood tokens to recover information displaced there by loss-increasing unlearning methods.The method selects tokens from a restricted candidate set using the unlearned model’s logits.
  • Findings: Up to 97% of supposedly forgotten postcodes remained recoverable after gradient-ascent unlearning with KL regularization.Direct queries produced nonsensical outputs, but restricted probing of postcode-relevant letters and digits recovered the information.
  • Limitations and future work: The auditing algorithms are simple and far from optimal, so the reported recovery may underestimate the security weaknesses of current unlearning methods.The authors call for stronger auditors and iterative evaluation against increasingly capable recovery strategies.

2 Preliminaries

The paper distinguishes prompt-, finetuning-, and decoding-based unlearning by which LLM component they modify. It focuses on finetuning-based methods, which alter model parameters while using retained data to preserve general utility, yet may conceal rather than remove target information.

  • Unlearning paradigms: Machine unlearning seeks a model behaving as if trained without the forget set, without the impracticality of full retraining.Methods modify prompts, transformer parameters, or decoding strategies.
  • Unlearning paradigms: Prompt-based and decoding-based unlearning leave the original model parameters unchanged, so target information is not truly removed.These approaches modify inputs or generation using auxiliary logits while preserving the original transformer.
  • Finetuning-based unlearning: Finetuning-based unlearning modifies the original transformer using a forget set and a retain set intended to preserve general utility.The retain set provides a lightweight reference for normal data during training.
  • Finetuning-based unlearning: Gradient-ascent objectives increase loss on forgotten examples while regularization on retained data helps preserve performance on normal data.The regularizer can use negative cross-entropy or negative KL divergence terms.
  • Evaluated methods: The evaluated finetuning-based methods include Gradient Ascent, Gradient Difference, Gradient Ascent with KL, Preference Optimization, and Negative Preference Optimization.These algorithms correspond to different instantiations of the general finetuning objective.
  • Unlearning limitation: Even after parameter modification, target information may remain concealed in the unlearned model and become recoverable through a carefully designed decoder.The vulnerability arises when the original decoding strategy is replaced.

3 White-box Auditing

White-box auditing formalizes recovery as an auditor using the unlearned model, unlearning procedure, retained data, and forget prompts to reconstruct forgotten information. The framework defines effectiveness and robustness through forget-quality bounds and focuses experimentally on finetuning-based methods.

  • Auditing setup: The framework evaluates forget quality using a loss over candidate and reference sequences, with the forget set measuring how well information is forgotten.The loss is bounded between 0 and 1.
  • Auditing setup: A white-box auditor receives the unlearned model, unlearning algorithm, retain set, and forget prompts, then outputs a recovered model.The forget prompts exclude the original responses.
  • Auditing setup: An unlearned model is α-effective β-robust when its forget quality is at least α and an auditor can recover it to quality at most β.For fixed α, smaller β indicates greater information-leakage risk.
  • Auditing setup: The robustness definition permits auditors to exploit available prior knowledge and a broad range of auditing algorithms to recover unlearned information.This makes the framework a lower-bound characterization of vulnerability under white-box auditing.
  • Scope: Auditing prompt-based and decoding-based methods is straightforward because their original model parameters remain unchanged.The paper therefore focuses subsequent white-box auditing experiments on finetuning-based unlearning.

4 The FPI dataset, evaluation metrics, and auditing

The paper constructs the synthetic FPI dataset and attribute-specific evaluation procedures, then audits unlearned models with restricted decoding strategies designed to recover masked private attributes.

  • The FPI dataset: The FPI dataset contains fictitious profiles with four private-information attributes: year of birth, blood type, postcode, and social insurance number.These attributes span numerical, categorical, and sequential formats.
  • The FPI dataset: The dataset yields 6,400 question–answer pairs by creating four paraphrased questions for each attribute of each individual while keeping answers identical.The profiles are used to construct multiple QA pairs for memorization and unlearning evaluation.
  • Evaluation metrics: Private-information memorization is measured by extracting candidate attribute values from model outputs and comparing them with ground-truth values using attribute-specific metrics.The extraction operators identify canonical patterns such as digits, Canadian postcodes, and valid blood types.
  • Auditing unlearned models: The auditing setup gives the auditor masked answers and attempts recovery by combining altered model inputs with restricted decoding strategies.For each forget-set question, the question and masked answer are concatenated before decoding.
  • Auditing unlearned models: Restricted inverse greedy decoding selects the lowest-likelihood token from an attribute-specific candidate set rather than the highest-likelihood token.Candidate sets encode prior structure, including digits for SINs and alternating letters and digits for Canadian postcodes.
  • Auditing unlearned models: For blood types, the auditor evaluates every possible label and predicts the candidate with the lowest sequence likelihood.This extends inverse greedy decoding from token-level choices to complete blood-type strings.
  • Auditing unlearned models: The auditing rationale is that fine-tuning-based unlearning may displace targeted information into high-loss regions instead of fully erasing it.Inverse decoding explicitly searches those regions for supposedly forgotten content.
  • Auditing unlearned models: Restricted candidate sets model attackers who use prior knowledge of the target attribute to strengthen recovery attempts.Restricted Greedy decoding is used for Preference Optimization because that method modifies question–rejection response pairs rather than directly increasing loss on the original forget set.

5 Experiments

The experiments evaluate five finetuning-based unlearning methods on FPI forget sets and test recovery using restricted decoding, retraining, quantization, and related audits. Apparent forgetting often does not survive white-box recovery, while stronger recovery resistance can trade off against utility.

  • Experimental setup: Five finetuning-based methods—GA, GD, GA+KL, PO, and NPO—are evaluated after creating original and forget sets from the FPI dataset.The original model is built by finetuning DeepSeek-7B on FPI, then selecting target attributes and individuals for forgetting.
  • Recovery results: GA, GD, and GA+KL often exceed 80% apparent forget quality, but RIG substantially lowers forget quality by recovering the targeted information.For GA+KL postcode unlearning, direct outputs are nonsensical and the original forget quality reaches 100%, yet RIG recovers nearly 97% of the supposedly forgotten postcodes.
  • Recovery results: PO appears effective under ordinary decoding, but RG reduces year-of-birth forget quality from 82% to 3%, showing that private information remains extractable.The result indicates that PO biases outputs toward rejection responses instead of removing the sensitive knowledge itself.
  • Baselines and additional audits: The retrained gold-standard model remains comparatively stable under RIG, whereas quantization alone provides limited recovery and decoding strategies have a larger recovery effect.RIG leaves postcode and SIN forget quality unchanged for the retrained model, with only slight changes for blood type and year of birth.
  • Additional observations: Training configurations reveal a trade-off among forget quality, white-box recovery robustness, and utility preservation.Configurations that resist recovery often substantially degrade utility, while utility-preserving configurations retain recoverable traces.
  • Limitations and future work: The recovery methods use logits but do not exploit other white-box signals such as the retain set, unlearning configurations, or latent representations.These additional signals are identified as directions for stronger auditing methods.

B Finetuning-based unlearning methods

The paper instantiates finetuning-based unlearning through gradient-based objectives, preference optimization, and negative preference optimization. These methods differ in how they penalize forget data and preserve behavior on retained data.

  • Gradient-based methods: GA and GD use token-level averaged cross-entropy loss, with GA omitting regularization and GD setting the regularizer to negative cross-entropy.GA uses λ = 0, whereas GD takes R(z, π) = −l(z, π).
  • Gradient-based methods: GA+KL uses the GD loss and a regularizer defined as the negative KL divergence between the original and unlearned models.The objective encourages the unlearned model to match the original output distribution on retained data while degrading performance on forget data.
  • Preference optimization: PO replaces original forget answers with designated rejection responses and minimizes loss on the modified forget set and retain set.Examples of rejection responses include “I don’t know” and “I cannot help you with that”.
  • Preference optimization: NPO extends DPO by treating forget data as negative samples, with γ > 0 and a regularizer instantiated as negative cross-entropy or negative KL divergence.Π(z) and Πo(z) denote sequence probabilities under the current and original models, respectively.

C Additional Information for FPI Dataset Creation

The FPI dataset is built from fake private profiles and templated question–answer pairs. Questions vary through paraphrased templates, while answers follow a fixed attribute-value format.

  • Dataset examples: Figure 3 illustrates fake private profiles together with their associated question–answer pairs.The figure provides examples of the synthetic records used in the dataset.
  • Question templates: Questions are generated by inserting each attribute into predefined templates, including paraphrases for birth year, postcode, SIN, and blood type.The templates vary wording while targeting the same private attribute.
  • Answer template: Answers use the template “{name}’s {attribute} is {value}.”This format keeps the answer structure consistent across attributes and profiles.

D Experimental Settings

The experiments use fixed forget–retain configurations and select unlearning hyperparameters by balancing forget quality with utility. The selected settings are summarized in Table 3.

  • The retain set is fixed at ten times the forget set, with m = 20 for year of birth and blood type, and m = 5 for SIN and postcode.
  • Unlearning configurations vary learning rate, weight decay, LoRA rank, and training epochs.The selected configuration maximizes the average of forget quality and utility.
  • The utility term is defined as 1 − 1/2 F(f) + 1/2 U(f).
  • The final hyperparameters, including the regularization weight λ, are reported in Table 3.

E Omitted Experimental Results

The omitted results include a SIN-task reproduction and sample outputs comparing the original model, unlearned model, and RG/RIG recovery methods.

  • Figure 4 reproduces the Figure 2 experiment on the SIN unlearning task.
  • A sample blood-type question has the original model output the ground-truth answer before recovery decoding is applied.
  • Table 4 compares outputs from the original and unlearned models with outputs produced by RG and RIG.RG and RIG use modified questions as model inputs, and long outputs are truncated.

F Experimental Results Reproduced on Qwen3-8B

Additional experiments reproduce the main experiments on Qwen3-8B and report quantization-based recovery results for year-of-birth and SIN tasks.

  • Figure 5 reproduces the Figure 1 experiments on Qwen3-8B.
  • Table 5 reproduces the Table 2 experiments on Qwen3-8B.
  • Table 6 reports forget quality for year-of-birth and SIN tasks under different quantization settings.Each table entry contains year-of-birth and SIN results in that order.

G Additional Discussion on White-box Auditing

The discussion explains why white-box auditing can bypass prompt-based and decoding-based unlearning, while finetuning-based unlearning requires testing whether information remains encoded in changed parameters.

  • Auditing prompt- and decoding-based unlearning: Prompt-based unlearning changes the input function, whereas decoding-based unlearning changes only the decoding function while leaving the base transformer unchanged.
  • Auditing prompt- and decoding-based unlearning: A white-box auditor may modify any component of the unlearned model.
  • Auditing prompt- and decoding-based unlearning: Removing the injected prompt or replacing altered decoding with greedy decoding can reveal the supposedly removed information.
  • Auditing prompt- and decoding-based unlearning: Because the base transformer is unchanged, prompt- and decoding-based approaches mask information rather than remove it.
  • Auditing finetuning-based unlearning: Finetuning-based unlearning changes model parameters, making residual implicit encoding harder to determine.
  • Quantization-based auditing: With NPO, combining RG with int8 quantization gives the best recovery, while decoding strategies matter more than quantization overall.

I Additional Observations

Across unlearning configurations, RIG often recovers information that standard decoding suggests was forgotten. Robustness to recovery can instead coincide with severe utility degradation, exposing a tension among forget quality, recoverability, and utility.

  • GD trajectory: GD’s standard forget quality rises across iterations while RIG-based forget quality falls, indicating increasingly recoverable supposedly forgotten information.The divergent curves suggest GD redirects information into inverted representations rather than erasing it.
  • Learning-rate effects: Figure 7 reports original forget quality, RIG-adjusted forget quality, and utility for each unlearning method and learning-rate setting.The evaluated methods are GA, GD, GA+KL, and NPO, each trained for 50 iterations under multiple learning rates.
  • Learning-rate effects: RIG reduces forget quality under most tested learning-rate settings, with green markers below red markers across configurations.The comparison uses original forget quality versus forget quality after RIG.
  • Learning-rate effects: Some configurations resist RIG only alongside near-zero or substantially degraded utility.GA+KL at learning rate 0.001 is one example, while GD, GA+KL, and NPO at 0.002 show the same pattern on postcode and SIN tasks.
  • Trade-offs: The results indicate a tension in which strong forget quality with preserved utility can leave latent representations recoverable under white-box auditing.Configurations that appear robust to recovery tend to degrade overall model capability.
Loading 2608.29943v1…